Failing tests on PSI - Failed Error: ImagePullBackOff

[kadel]

I'm noticing a lot of failing runs due to Error: ImagePullBackOff on PSI.

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_odo/4450/pull-ci-openshift-odo-master-v4.6-e2e-4x-psi/1362767243833774080

✓  Validating the devfile [78628ns]
[ssh:Fedora 32] [odo] 
[ssh:Fedora 32] [odo] Creating Kubernetes resources for component zerjru
[ssh:Fedora 32] [odo] I0219 15:23:21.060188 2797004 utils.go:228] Updating container devruntime entrypoint with supervisord
[ssh:Fedora 32] [odo] I0219 15:23:21.060223 2797004 utils.go:123] Updating container devruntime with supervisord volume mounts
[ssh:Fedora 32] [odo] I0219 15:23:21.060236 2797004 utils.go:133] Updating container devruntime env with run command
[ssh:Fedora 32] [odo] I0219 15:23:21.060248 2797004 utils.go:150] Updating container devruntime env with run command's workdir
[ssh:Fedora 32] [odo] I0219 15:23:21.060258 2797004 utils.go:186] Updating container devruntime env with debug command
[ssh:Fedora 32] [odo] I0219 15:23:21.060265 2797004 utils.go:203] Updating container devruntime env with debug command's workdir
[ssh:Fedora 32] [odo] I0219 15:23:21.060271 2797004 utils.go:212] Updating container devruntime env with debug command's debugPort
[ssh:Fedora 32] [odo] I0219 15:23:21.060317 2797004 preference.go:222] The path for preference file is /tmp/424561106/preference.yaml
[ssh:Fedora 32] [odo] I0219 15:23:21.065947 2797004 adapter.go:410] Creating deployment zerjru
[ssh:Fedora 32] [odo] I0219 15:23:21.065975 2797004 adapter.go:411] The component name is zerjru
[ssh:Fedora 32] [odo] I0219 15:23:21.085794 2797004 adapter.go:454] Successfully created component zerjru
[ssh:Fedora 32] [odo] I0219 15:23:21.122934 2797004 adapter.go:462] Successfully created Service for component zerjru
[ssh:Fedora 32] [odo] I0219 15:23:21.122970 2797004 deployments.go:101] Waiting for zerjru deployment rollout
[ssh:Fedora 32] [odo]  •  Waiting for component to start  ...
[ssh:Fedora 32] [odo] I0219 15:23:21.125148 2797004 deployments.go:134] Deployment Condition: {"type":"Progressing","status":"True","lastUpdateTime":"2021-02-19T15:22:40Z","lastTransitionTime":"2021-02-19T15:22:40Z","reason":"NewReplicaSetCreated","message":"Created new replica set \"zerjru-85c48c455\""}
[ssh:Fedora 32] [odo] I0219 15:23:21.125167 2797004 deployments.go:152] Waiting for deployment spec update to be observed...
[ssh:Fedora 32] [odo] I0219 15:23:21.145726 2797004 deployments.go:134] Deployment Condition: {"type":"Progressing","status":"True","lastUpdateTime":"2021-02-19T15:22:40Z","lastTransitionTime":"2021-02-19T15:22:40Z","reason":"NewReplicaSetCreated","message":"Created new replica set \"zerjru-85c48c455\""}
[ssh:Fedora 32] [odo] I0219 15:23:21.145760 2797004 deployments.go:134] Deployment Condition: {"type":"Available","status":"False","lastUpdateTime":"2021-02-19T15:22:40Z","lastTransitionTime":"2021-02-19T15:22:40Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}
[ssh:Fedora 32] [odo] I0219 15:23:21.145769 2797004 deployments.go:141] Waiting for deployment "zerjru" rollout to finish: 0 out of 1 new replicas have been updated...
[ssh:Fedora 32] [odo] I0219 15:23:21.145775 2797004 deployments.go:152] Waiting for deployment spec update to be observed...
[ssh:Fedora 32] [odo] I0219 15:23:21.214254 2797004 deployments.go:134] Deployment Condition: {"type":"Available","status":"False","lastUpdateTime":"2021-02-19T15:22:40Z","lastTransitionTime":"2021-02-19T15:22:40Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}
[ssh:Fedora 32] [odo] I0219 15:23:21.214289 2797004 deployments.go:134] Deployment Condition: {"type":"Progressing","status":"True","lastUpdateTime":"2021-02-19T15:22:40Z","lastTransitionTime":"2021-02-19T15:22:40Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"zerjru-85c48c455\" is progressing."}
[ssh:Fedora 32] [odo] I0219 15:23:21.214296 2797004 deployments.go:145] Waiting for deployment "zerjru" rollout to finish: 0 of 1 updated replicas are available...
[ssh:Fedora 32] [odo] I0219 15:23:21.214302 2797004 deployments.go:152] Waiting for deployment spec update to be observed...
[ssh:Fedora 32] [odo] 
I0219 15:27:46.354206 2797004 events.go:52] Warning Event: Count: 5, Reason: Failed, Message: Error: ImagePullBackOff
[ssh:Fedora 32] [odo]  ✗  Waiting for component to start [5m] [WARNING x5: Failed]
[ssh:Fedora 32] [odo]  ✗  Failed to start component with name zerjru. Error: Failed to create the component: error while waiting for deployment rollout: timeout while waiting for zerjru deployment roll out\nFor more information to help determine the cause of the error, re-run with '-v'.
[ssh:Fedora 32] [odo] See below for a list of failed events that occured more than 5 times during deployment:
[ssh:Fedora 32] [odo] 
[ssh:Fedora 32] [odo]  NAME                                     COUNT  REASON  MESSAGE                 
[ssh:Fedora 32] [odo] 
[ssh:Fedora 32] [odo]  zerjru-85c48c455-cnjfc.16652f7dad588715  5      Failed  Error: ImagePullBackOff 
[ssh:Fedora 32] [odo] 
[ssh:Fedora 32] [odo] 

[dharmit]

@mohammedzee1000 can you help here?

[kadel]

I think that QEs should start getting more involved in investigating errors like this.

[rnapoles-rh]

I ran the failing test locally against the PSI cluster and it passes:

ginkgo -randomizeAllSpecs -slowSpecThreshold=120 -timeout 7200s -nodes=2 -focus="odo debug support for devfile components" tests/e2escenarios
Running Suite: odo e2e scenarios
================================
Random Seed: 1614777265 - Will randomize all specs
Will run 30 specs

Running in parallel across 2 nodes

SSSSSSSSSSSSSSSSSS•SSSS•S•SSS•
Ran 4 of 30 Specs in 98.932 seconds
SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 26 Skipped

Looking at Stabilize tests on PSI #4459 to try to identify the root cause.

[rnapoles-rh]

I ran the e2escenarios against PSI and all passed as expected

ginkgo  -randomizeAllSpecs -slowSpecThreshold=120 -timeout 7200s -nodes=2 tests/e2escenarios/
Running Suite: odo e2e scenarios
================================
Random Seed: 1616089979 - Will randomize all specs
Will run 31 specs

Running in parallel across 2 nodes
.
.
.
Ran 26 of 31 Specs in 694.885 seconds
SUCCESS! -- 26 Passed | 0 Failed | 0 Pending | 5 Skipped

Ginkgo ran 1 suite in 11m38.145983455s
Test Suite Passed

Closing the issue based on this.