[FLAKE] tests failing on " waited 4m0s but couldn't find running pod matching selector"

[kadel]

There has been a lot of tests failing on waited 4m0s but couldn't find running pod matching selector recently.

Creating a new project: pumdnyfvww
Running odo with args [odo project create pumdnyfvww -w -v4]
[odo]  •  Waiting for project to come up  ...
[odo]  ✓  Waiting for project to come up [368ms]
[odo]  ✓  Project 'pumdnyfvww' is ready for use
[odo]  ✓  New project created and now using project : pumdnyfvww
Created dir: /tmp/493410141
Running git with args [git clone https://github.com/openshift/nodejs-ex /tmp/493410141/nodejs-ex]
[git] Cloning into '/tmp/493410141/nodejs-ex'...
Running odo with args [odo component create nodejs nodejs --project pumdnyfvww --context /tmp/493410141/nodejs-ex --app app]
[odo]  •  Validating component  ...
[odo]  ✓  Validating component [44ms]
[odo] Please use `odo push` command to create the component with source deployed
Running odo with args [odo push --context /tmp/493410141/nodejs-ex]
[odo] Validation
[odo]  •  Checking component  ...
[odo]  ✓  Checking component [60ms]
[odo] 
[odo] Configuration changes
[odo]  ✓  Initializing component
[odo]  •  Creating component  ...
[odo]  ✓  Creating component [196ms]
[odo] 
[odo] Pushing to component nodejs of type local
[odo]  •  Checking files for pushing  ...
[odo]  ✓  Checking files for pushing [1ms]
[odo]  •  Waiting for component to start  ...
[odo]  ✗  Waiting for component to start [4m]
[odo]  ✗  waited 4m0s but couldn't find running pod matching selector: 'deploymentconfig=nodejs-app'
Deleting project: pumdnyfvww
Running odo with args [odo project delete pumdnyfvww -f]
[odo] This project contains the following applications, which will be deleted
[odo] Application app
[odo] This application has following components that will be deleted
[odo] component named nodejs
[odo] No services / could not get services
[odo]  •  Deleting project pumdnyfvww  ...
[odo]  ✓  Deleting project pumdnyfvww [6s]
[odo]  ✓  Deleted project : pumdnyfvww
Deleting dir: /tmp/493410141
• Failure [249.085 seconds]
odo push command tests
/go/src/github.com/openshift/odo/tests/integration/cmd_push_test.go:15
  when push command is executed
  /go/src/github.com/openshift/odo/tests/integration/cmd_push_test.go:40
    should not build when changes are detected in a ignored file [It]
    /go/src/github.com/openshift/odo/tests/integration/cmd_push_test.go:168
    No future change is possible.  Bailing out early after 240.561s.
      
    Running odo with args [odo push --context /tmp/493410141/nodejs-ex]
    Expected
        <int>: 1
    to match exit code:
        <int>: 0
    /go/src/github.com/openshift/odo/tests/helper/helper_run.go:33
------------------------------
odo push command tests when push command is executed 
  should be able to create a file, push, delete, then push again propagating the deletions
  /go/src/github.com/openshift/odo/tests/integration/cmd_push_test.go:57

from #2242 (comment)

[amitkrout]

@kadel It's a flake, i have seen the same issue in other prs too. It would be great if somebody look into the odo code implementation side. cc_ @mik-dass

[amitkrout]

Can you please change the title to [Flake]Push fails on " waited 4m0s but couldn't find running pod matching selector"

[dharmit]

I'm not sure but I think this happens when something's off with the project to which we're trying to do odo push. I saw this in my environment once. I did an odo delete --all, odo create and odo push which fixed this for me.

[cdrage]

For the life of me, I can't seem to replicate this.. going to see if this pops back up on the CI side again.

[kadel]

We can try to reproduce it by increasing number of parallel test

[kadel]

Hi, @cdrage do you have any leads to what might be causing this?
There is also #2392 which might be related to this.

[cdrage]

@kadel No idea, but I'll investigate and see what may be causing this.. Really difficult to re-produce, but we can at least add some checks make sure the DeploymentConfig is completely deployed.

[cdrage]

Been trying to replicate it multiple times but nothing... Here's my script:

#!/bin/bash    
    
set -e    
    
# Delete project and folders    
rm -rf backend frontend || true    
odo project delete myproject -f || true    
    
git clone https://github.com/openshift-evangelists/Wild-West-Backend backend || true    
git clone https://github.com/openshift-evangelists/Wild-West-Frontend frontend || true    
    
odo project create myproject    
    
# Import image    
oc import-image openjdk18 \    
--from=registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift --confirm    
oc annotate istag/openjdk18:latest tags=builder    
    
# Deploy backend    
cd backend    
mvn package    
odo create openjdk18 backend --binary target/wildwest-1.0.jar    
odo push -v 4    
cd ..    
    
# Deploy frontend    
cd frontend    
odo create nodejs frontend    
odo push -v 4    
    
# Link both    
odo link backend --port 8080    
    
cd ..  

[cdrage]

Hmm. I'm getting to something. I got a failure to start since PVC was taking too long. We actually do not check for PVC nor "output" anything regarding it...

I'm going to create some output for this at least.

So going to at least

Name:               nodejs-nodejs-ex-uhda-app-1-7vqs6
Namespace:          myproject
Priority:           0
PriorityClassName:  <none>
Node:               ip-172-16-15-175.ec2.internal/172.16.15.175
Start Time:         Wed, 13 Nov 2019 12:01:26 -0500
Labels:             deployment=nodejs-nodejs-ex-uhda-app-1
                    deploymentconfig=nodejs-nodejs-ex-uhda-app
Annotations:        alpha.image.policy.openshift.io/resolve-names=*
                    kubernetes.io/limit-ranger=LimitRanger plugin set: cpu, memory request for container nodejs-nodejs-ex-uhda-app; cpu, memory limit for container nodejs-nodejs-ex-uhda-app; cpu, memory request for init ...
                    openshift.io/deployment-config.latest-version=1
                    openshift.io/deployment-config.name=nodejs-nodejs-ex-uhda-app
                    openshift.io/deployment.name=nodejs-nodejs-ex-uhda-app-1
                    openshift.io/scc=restricted
Status:             Pending
IP:                 
Controlled By:      ReplicationController/nodejs-nodejs-ex-uhda-app-1
Init Containers:
  copy-files-to-volume:
    Container ID:  
    Image:         172.30.16.196:5000/openshift/nodejs@sha256:71ff339d7221be9095961a95c5acea25f87f2b9c4c92dbb9b139c9b2940bb9ce
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      
SRC=/opt/app-root
DEST=/mnt/app-root

if [ -f $DEST/.delete-volume ]; then
    rm -rf $DEST
fi
 if [ -d $DEST ]; then
    if [ -f $DEST/.sync-volume ]; then
        if ! [[ "$JUPYTER_SYNC_VOLUME" =~ ^(false|no|n|0)$ ]]; then
            JUPYTER_SYNC_VOLUME=yes
        fi
    fi
     if [[ "$JUPYTER_SYNC_VOLUME" =~ ^(true|yes|y|1)$ ]]; then
        rsync -ar --ignore-existing $SRC/. $DEST
    fi
     exit
fi
 if [ -d $DEST.setup-volume ]; then
    rm -rf $DEST.setup-volume
fi

mkdir -p $DEST.setup-volume
tar -C $SRC -cf - . | tar -C $DEST.setup-volume -xvf -
mv $DEST.setup-volume $DEST
                        
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  512Mi
    Requests:
      cpu:        100m
      memory:     256Mi
    Environment:  <none>
    Mounts:
      /mnt from nodejs-nodejs-ex-uhda-app-s2idata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99rdz (ro)
  copy-supervisord:
    Container ID:  
    Image:         registry.access.redhat.com/openshiftdo/odo-init-image-rhel7:1.0.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/cp
    Args:
      -r
      /opt/odo-init/.
      /opt/odo/
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  512Mi
    Requests:
      cpu:        100m
      memory:     256Mi
    Environment:  <none>
    Mounts:
      /opt/odo/ from odo-supervisord-shared-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99rdz (ro)
Containers:
  nodejs-nodejs-ex-uhda-app:
    Container ID:  
    Image:         172.30.16.196:5000/openshift/nodejs@sha256:71ff339d7221be9095961a95c5acea25f87f2b9c4c92dbb9b139c9b2940bb9ce
    Image ID:      
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /opt/odo/bin/go-init
    Args:
      -main
      /opt/odo/bin/supervisord -c /opt/odo/conf/supervisor.conf
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  512Mi
    Requests:
      cpu:     100m
      memory:  256Mi
    Environment:
      DEBUG_PORT:                5858
      ODO_S2I_SCRIPTS_URL:       /usr/libexec/s2i
      ODO_S2I_SCRIPTS_PROTOCOL:  image://
      ODO_S2I_SRC_BIN_PATH:      /tmp
      ODO_S2I_DEPLOYMENT_DIR:    /opt/app-root/src
      ODO_S2I_WORKING_DIR:       /opt/app-root/src
      ODO_S2I_BUILDER_IMG:       rhoar-nodejs/nodejs-10
      ODO_SRC_BACKUP_DIR:        /opt/app-root/src-backup
    Mounts:
      /opt/app-root from nodejs-nodejs-ex-uhda-app-s2idata (rw)
      /opt/odo/ from odo-supervisord-shared-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99rdz (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  odo-supervisord-shared-data:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  nodejs-nodejs-ex-uhda-app-s2idata:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  nodejs-nodejs-ex-uhda-app-s2idata
    ReadOnly:   false
  default-token-99rdz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-99rdz
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  type=compute
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
Events:
  Type     Reason              Age              From                                    Message
  ----     ------              ----             ----                                    -------
  Normal   Scheduled           3m               default-scheduler                       Successfully assigned myproject/nodejs-nodejs-ex-uhda-app-1-7vqs6 to ip-172-16-15-175.ec2.internal
  Warning  FailedAttachVolume  3m (x2 over 3m)  attachdetach-controller                 AttachVolume.Attach failed for volume "pvc-38ce5c02-0637-11ea-b4b9-12d3ba54a54b" : "Error attaching EBS volume \"vol-0cde4c5fcc80f93ff\"" to instance "i-0eb54d0efaf9e2db7" since volume is in "creating" state
  Warning  FailedMount         1m               kubelet, ip-172-16-15-175.ec2.internal  Unable to mount volumes for pod "nodejs-nodejs-ex-uhda-app-1-7vqs6_myproject(3ae2079f-0637-11ea-b4b9-12d3ba54a54b)": timeout expired waiting for volumes to attach or mount for pod "myproject"/"nodejs-nodejs-ex-uhda-app-1-7vqs6". list of unmounted volumes=[nodejs-nodejs-ex-uhda-app-s2idata]. list of unattached volumes=[odo-supervisord-shared-data nodejs-nodejs-ex-uhda-app-s2idata default-token-99rdz]

[cdrage]

Another failure I was able to encounter (for my notes..):

To replicate: quickly do a odo delete -f and then odo push and you'll encounter an error with DeploymentConfig complaining regarding the deployer pod going missing...

mkdir -p $DEST.setup-volume
tar -C $SRC -cf - . | tar -C $DEST.setup-volume -xvf -
mv $DEST.setup-volume $DEST

    Environment:        <none>
    Mounts:
      /mnt from nodejs-nodejs-ex-qpfd-app-s2idata (rw)
   copy-supervisord:
    Image:      registry.access.redhat.com/openshiftdo/odo-init-image-rhel7:1.0.0
    Port:       <none>
    Host Port:  <none>
    Command:
      /usr/bin/cp
    Args:
      -r
      /opt/odo-init/.
      /opt/odo/
    Environment:        <none>
    Mounts:
      /opt/odo/ from odo-supervisord-shared-data (rw)
  Containers:
   nodejs-nodejs-ex-qpfd-app:
    Image:      172.30.16.196:5000/openshift/nodejs@sha256:71ff339d7221be9095961a95c5acea25f87f2b9c4c92dbb9b139c9b2940bb9ce
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      /opt/odo/bin/go-init
    Args:
      -main
      /opt/odo/bin/supervisord -c /opt/odo/conf/supervisor.conf
    Environment:
      DEBUG_PORT:               5858
      ODO_S2I_SCRIPTS_URL:      /usr/libexec/s2i
      ODO_S2I_SCRIPTS_PROTOCOL: image://
      ODO_S2I_SRC_BIN_PATH:     /tmp
      ODO_S2I_DEPLOYMENT_DIR:   /opt/app-root/src
      ODO_S2I_WORKING_DIR:      /opt/app-root/src
      ODO_S2I_BUILDER_IMG:      rhoar-nodejs/nodejs-10
      ODO_SRC_BACKUP_DIR:       /opt/app-root/src-backup
    Mounts:
      /opt/app-root from nodejs-nodejs-ex-qpfd-app-s2idata (rw)
      /opt/odo/ from odo-supervisord-shared-data (rw)
  Volumes:
   odo-supervisord-shared-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
   nodejs-nodejs-ex-qpfd-app-s2idata:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  nodejs-nodejs-ex-qpfd-app-s2idata
    ReadOnly:   false

Deployment #1 (latest):
        Name:           nodejs-nodejs-ex-qpfd-app-1
        Created:        about a minute ago
        Status:         Failed
        Replicas:       0 current / 0 desired
        Selector:       deployment=nodejs-nodejs-ex-qpfd-app-1,deploymentconfig=nodejs-nodejs-ex-qpfd-app
        Labels:         app.kubernetes.io/instance=nodejs-nodejs-ex-qpfd,app.kubernetes.io/managed-by-version=v1.0.1,app.kubernetes.io/managed-by=odo,app.kubernetes.io/name=nodejs,app.kubernetes.io/part-of=app,app.openshift.io/runtime-version=latest,app=app,openshift.io/deployment-config.name=nodejs-nodejs-ex-qpfd-app                                             
        Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed

Events:
  Type          Reason                  Age     From                            Message
  ----          ------                  ----    ----                            -------
  Normal        DeploymentCreated       1m      deploymentconfig-controller     Created new replication controller "nodejs-nodejs-ex-qpfd-app-1" for version 1
  Warning       Failed                  32s     deployer-controller             Deployer pod "nodejs-nodejs-ex-qpfd-app-1-deploy" has gone missing

[priyankakhetawat]

getting while odo push
X Waiting for component to start [4m]
X waited 4m0s but couldn't find running pod matching selector: 'deploymentconfig=frontend-app'