Change the order in the manifest

[SchSeba]

I take a look on the std-ci logs and found that it takes time for the service account to exist.

daemonset.extensions/bridge-marker created
clusterrole.rbac.authorization.k8s.io/bridge-marker-cr created
clusterrolebinding.rbac.authorization.k8s.io/bridge-marker-crb created
serviceaccount/bridge-marker created
+ timeout=300
+ sample=30
+ current_time=0
+ true
++ kubectl describe daemonset bridge-marker -n kube-system
++ cluster/kubectl.sh describe daemonset bridge-marker -n kube-system
+ describe_result='Name:           bridge-marker
Selector:       app=bridge-marker,tier=node
Node-Selector:  beta.kubernetes.io/arch=amd64
Labels:         app=bridge-marker
                tier=node
Annotations:    <none>
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=bridge-marker
                    tier=node
  Service Account:  bridge-marker
  Containers:
   bridge-marker:
    Image:      quay.io/kubevirt/bridge-marker:latest
    Port:       <none>
    Host Port:  <none>
    Args:
      -node-name
      $(NODE_NAME)
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type     Reason        Age   From                  Message
  ----     ------        ----  ----                  -------
  Warning  FailedCreate  1s    daemonset-controller  Error creating: pods "bridge-marker-" is forbidden: error looking up service account kube-system/bridge-marker: serviceaccount "bridge-marker" not found'
++ echo 'Name:           bridge-marker
Selector:       app=bridge-marker,tier=node
Node-Selector:  beta.kubernetes.io/arch=amd64
Labels:         app=bridge-marker
                tier=node
Annotations:    <none>
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=bridge-marker
                    tier=node
  Service Account:  bridge-marker
  Containers:
   bridge-marker:
    Image:      quay.io/kubevirt/bridge-marker:latest
    Port:       <none>
    Host Port:  <none>
    Args:
      -node-name
      $(NODE_NAME)
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type     Reason        Age   From                  Message
  ----     ------        ----  ----                  -------
  Warning  FailedCreate  1s    daemonset-controller  Error creating: pods "bridge-marker-" is forbidden: error looking up service account kube-system/bridge-marker: serviceaccount "bridge-marker" not found'
++ grep 'Desired Number of Nodes Scheduled'
++ awk -F : '{print $2}'
+ desired_nodes=' 0'
++ echo 'Name:           bridge-marker
Selector:       app=bridge-marker,tier=node
Node-Selector:  beta.kubernetes.io/arch=amd64
Labels:         app=bridge-marker
                tier=node
Annotations:    <none>
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=bridge-marker
                    tier=node
  Service Account:  bridge-marker
  Containers:
   bridge-marker:
    Image:      quay.io/kubevirt/bridge-marker:latest
    Port:       <none>
    Host Port:  <none>
    Args:
      -node-name
      $(NODE_NAME)
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type     Reason        Age   From                  Message
  ----     ------        ----  ----                  -------
  Warning  FailedCreate  1s    daemonset-controller  Error creating: pods "bridge-marker-" is forbidden: error looking up service account kube-system/bridge-marker: serviceaccount "bridge-marker" not found'
++ grep 'Number of Nodes Scheduled with Up-to-date Pods'
++ awk -F : '{print $2}'
+ current_nodes=' 0'
+ '[' 0 -eq 0 ']'
+ break

I think if we move the daemonset to the end of the manifest file it will help.

[SchSeba]

@mmirecki Can you take a look on this please?

[mmirecki]

I'm unable to reproduce the issue.
Tried the following commands multiple times on both k8s and os
./cluster/kubectl.sh delete -f examples/bridge-marker.yml
./cluster/kubectl.sh apply -f examples/bridge-marker.yml
./cluster/kubectl.sh describe ds -n kube-system bridge-marker
but have not see a failed event once.

With Sebastian's ack, I'm closing the issue.

[RamLavi]

We also see this issue on cnao, sometimes (~1 in 3 times) our ci tests fail because of this issue (there is an open issue for flakyness). We currently use version 0.3.0 of bridge marker.
@mmirecki can you please reopen the issue?

prow logs:
[1] example PR, where it currently fails.
[2] upgrade_test, when operator in version 0.27.0 is installed and supported spec configured and it is upgraded to the latest release it should report expected deployed container images and leave no leftovers from the previous version
[3] upgrade test, when operator in version 0.27.0 is installed and supported spec configured should transition reported versions while being upgraded to version 99.0.0

@qinqon FYI

[phoracek]

It is an annoying issue but not a cause for our failure IIUIC. Even if we fail on the order, on the re-reconcile, we should apply all the manifests again and thus eventually get them all applied.

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[kubevirt-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

[kubevirt-bot]

@kubevirt-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.