Recurring CI failures while trying to verify support for nodejs images #4347

dharmit · 2021-01-06T08:13:07Z

CI's failing quite frequently while trying to verify support for nodejs-10 & nodejs-12 images.

nodejs-10 image

 • Failure [73.051 seconds]
odo supported images e2e tests
/go/src/github.com/openshift/odo/tests/e2escenarios/e2e_images_test.go:16
  odo supported images deployment on amd64
  /go/src/github.com/openshift/odo/tests/e2escenarios/e2e_images_test.go:84
    Should be able to verify the nodejs-10-centos7 image [It]
    /go/src/github.com/openshift/odo/tests/e2escenarios/e2e_images_test.go:96
    Expected
      
        <string>: total 0
        147020556 drwxr-xr-x. 2 1006720000 root  6 Jan  5 17:53 .
        250054292 drwxrwxrwt. 1 root       root 31 Jan  5 17:53 ..
        
    to contain substring
        <string>: a.txt
    /go/src/github.com/openshift/odo/tests/helper/helper_generic.go:70

nodejs-12 image

 • Failure [130.010 seconds]
odo supported images e2e tests
/go/src/github.com/openshift/odo/tests/e2escenarios/e2e_images_test.go:16
  odo supported images deployment on amd64
  /go/src/github.com/openshift/odo/tests/e2escenarios/e2e_images_test.go:84
    Should be able to verify the nodejs-12-centos7 image [It]
    /go/src/github.com/openshift/odo/tests/e2escenarios/e2e_images_test.go:101
    Expected
      
        <string>: total 0
        343997299 drwxr-xr-x. 2 1006490000 root  6 Jan  5 17:45 .
        354471742 drwxrwxrwt. 1 root       root 31 Jan  5 17:45 ..
        
    to contain substring
        <string>: a.txt
    /go/src/github.com/openshift/odo/tests/helper/helper_generic.go:70

Logs can be found on test failure dashboard: https://gist.github.com/jgwest/5b28687514970ae6baa5942ee82d5755.

/kind failing-test
/priority high

The text was updated successfully, but these errors were encountered:

prietyc123 · 2021-01-06T14:25:40Z

@dharmit I tried looking into the issue and it seems particular to the nodejs centos image because it works fine for other nodejs images. Steps I run locally:

$ odo project create vqwcxappzr
 ✓  Project 'vqwcxappzr' is ready for use
 ✓  New project created and now using project: vqwcxappzr

$ oc --request-timeout 5m import-image nodejs:latest --namespace=vqwcxappzr --from=docker.io/centos/nodejs-10-centos7:latest --confirm
imagestream.image.openshift.io/nodejs imported
[...]

$ oc annotate istag/nodejs:latest --namespace=vqwcxappzr tags=builder --overwrite
imagestreamtag.image.openshift.io/nodejs:latest annotated

$ odo create --s2i nodejs:latest nodejs-app --project vqwcxappzr --context . --app app
Validation
[...]
Please use `odo push` command to create the component with source deployed

$ odo push --context .
Validation
 ✓  Checking component [892ms]
[...]
 ✓  Changes successfully pushed to component

$ odo url create --port 8080 --context .
 ✓  URL nodejs-app-8080 created for component: nodejs-app
[...]

$ odo push --context .
Validation
 ✓  Checking component [679ms]
[...]

$ odo watch -v 4 --context .
I0106 15:08:35.417942   66280 util.go:418] path devfile.yaml doesn't exist, skipping it
[...]
Waiting for something to change in /Users/pkumari/go/src/github.com/openshift/odo/testim
I0106 15:10:57.447199   66280 watch.go:197] filesystem watch event: "/Users/pkumari/go/src/github.com/openshift/odo/testim/a.txt": CREATE
[...]
Pushing files...
I0106 15:10:59.136709   66280 watch.go:339] Copying files [/Users/pkumari/go/src/github.com/openshift/odo/testim/a.txt] to pod
[...]
I0106 15:11:00.058693   66280 pods.go:75] Container Status: {"name":"nodejs-app-app","state":{"running":{"startedAt":"2021-01-06T09:24:29Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/centos/nodejs-10-centos7@sha256:fab0a2ffe461277f0f5e38bb664df3fa581322821bdabe8a2724ce528d207e14","imageID":"docker.io/centos/nodejs-10-centos7@sha256:fab0a2ffe461277f0f5e38bb664df3fa581322821bdabe8a2724ce528d207e14","containerID":"cri-o://69e60c7902c1fbd9ddc8dace19f485ff95046b07ab1816ef14af17df00c96695","started":true}
I0106 15:11:00.058729   66280 pods.go:79] Pod nodejs-app-app-1-cz4qn is Running
 ✓  Waiting for component to start [420ms]
 •  Syncing files to the component  ...
I0106 15:11:00.058927   66280 component.go:736] Copying files [...]
/Users/pkumari/go/src/github.com/openshift/odo, srcFile: testim/a.txt, destBase: /tmp/src, destFile: a.txt
I0106 15:11:00.059144   66280 sync.go:127] Corrected destinations: base: /tmp/src file: a.txt
I0106 15:11:00.386339   66280 events.go:36] Quitting collect events
 ✓  Syncing files to the component [3s]
 •  Building component  ...
I0106 15:11:03.111467   66280 exec.go:27] Executing command [/opt/odo/bin/assemble-and-restart] for pod: nodejs-app-app-1-cz4qn in container: 
+ set -eo pipefail
+ export ODO_UTILS_DIR=/opt/odo
+ ODO_UTILS_DIR=/opt/odo
+ export ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
+ ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
++ /opt/odo/bin/getlanguage
+ IMAGE_LANG=
+ '[' '!' -z /opt/app-root/src ']'
+ '[' /tmp '!=' /opt/app-root/src ']'
+ '[' -n /opt/app-root/src-backup ']'
+ '[' '!' -d /opt/app-root/src-backup ']'
---> Installing application source ...
+ rsync -rlO /tmp/src/. /opt/app-root/src-backup/src/
+ b_IFS=' 
'
+ b_OFS=
---> Installing all dependencies
+ OIFS=' 
'
+ IFS='
'
++ ls -A /tmp/src/
+ for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
+ rm -fr /opt/app-root/src/a.txt
+ OIFS=
+ IFS=' 
'
+ '[' -n centos7/nodejs-10-centos7 ']'
+ '[' centos7/nodejs-10-centos7 == redhat-[openjdk-18](https://issues.redhat.com/browse/openjdk-18)/openjdk18-openshift ']'
+ '[' -f /tmp/src/.s2i/bin/assemble ']'
+ '[' -n '' ']'
+ '[' -n /usr/libexec/s2i ']'
+ rm -rf /opt/app-root/src/.git
+ /usr/libexec/s2i/assemble

> nodemon@2.0.6 postinstall /opt/app-root/src/node_modules/nodemon
> node bin/postinstall || exit 0
npm WARN node-echo@0.0.1 No repository field.
npm WARN node-echo@0.0.1 No license field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.1 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.1: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
added 119 packages from 53 contributors and audited 120 packages in 2.968s

11 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

---> Building in production mode

---> Pruning the development dependencies
npm WARN node-echo@0.0.1 No repository field.
npm WARN node-echo@0.0.1 No license field.
removed 119 packages in 0.908s
found 0 vulnerabilities

/opt/app-root/src/.npm is not a mountpoint
---> Cleaning the npm cache /opt/app-root/src/.npm
+ '[' '!' -z /opt/app-root/src ']'
+ '[' -n /opt/app-root/src-backup ']'
+ '[' /tmp '!=' /opt/app-root/src ']'
+ '[' /opt/app-root/src '!=' /opt/app-root/src ']'
/tmp is not a mountpoint
+ '[' -z '' ']'
+ PV_MNT_PT=/opt/app-root
+ sed -i s/autostart=false/autostart=true/g /opt/app-root/conf/supervisor.conf
+ /opt/odo/bin/supervisord ctl stop run
---> Cleaning the /tmp/npm-*
run: stopped
+ /opt/odo/bin/supervisord ctl start run
run: started
 ✓  Building component [11s]

Creating/deleting and modifying the file within the context directory while odo watch

$ touch a.txt

Content pushed to Environment variable dir

$ oc get pods --namespace vqwcxappzr --show-labels
NAME                      READY   STATUS      RESTARTS   AGE   LABELS
nodejs-app-app-1-cz4qn    1/1     Running     0          21m   deployment=nodejs-app-app-1,deploymentconfig=nodejs-app-app
nodejs-app-app-1-deploy   0/1     Completed   0          21m   openshift.io/deployer-pod-for.name=nodejs-app-app-1

$ oc get dc nodejs-app-app --namespace vqwcxappzr -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}:{.value}{"\n"}{end}'
DEBUG_PORT:5858
ODO_S2I_SCRIPTS_URL:/usr/libexec/s2i
ODO_S2I_SCRIPTS_PROTOCOL:image://
ODO_S2I_SRC_BIN_PATH:/tmp
ODO_S2I_DEPLOYMENT_DIR:/opt/app-root/src
ODO_S2I_WORKING_DIR:/opt/app-root/src
ODO_S2I_BUILDER_IMG:centos7/nodejs-10-centos7
ODO_SRC_BACKUP_DIR:/opt/app-root/src-backup

$ oc exec nodejs-app-app-1-cz4qn --namespace vqwcxappzr -- ls -lai /tmp/src
total 0
341848214 drwxr-xr-x. 2 1000730000 root  6 Jan  6 09:41 .
 10686604 drwxrwxrwt. 1 root       root 31 Jan  6 09:24 ..

$ oc exec nodejs-app-app-1-cz4qn --namespace vqwcxappzr -- ls -lai /opt/app-root/src
total 88
  13 drwxrwsr-x. 9 1000730000 root        4096 Jan  6 09:41 .
  12 drwxr-sr-x. 6 1000730000 1000730000  4096 Jan  6 09:24 ..
  [...]
  47 -rw-rw-r--. 1 1000730000 root           0 Jan  6 09:40 a.txt
1537 -rw-rw-r--. 1 1000730000 root         326 Jan  6 09:24 package.json
  45 -rw-rw-r--. 1 1000730000 root         562 Jan  6 09:41 server.js

$ oc exec nodejs-app-app-1-cz4qn --namespace vqwcxappzr -- ls -lai /opt/app-root/src-backup/src
total 32
[...]
40 -rw-r--r--. 1 1000730000 1000730000    0 Jan  6 09:51 a.txt
28 -rw-r--r--. 1 1000730000 1000730000  329 Jan  6 09:24 package.json
42 -rw-r--r--. 1 1000730000 1000730000  562 Jan  6 09:51 server.js

I ran the entire steps for registry.access.redhat.com/rhoar-nodejs/nodejs-10:latest image as well and its quite weird that ODO_S2I_SRC_BIN_PATH/src has all the expected content for this image.

$ oc exec nodejs-app-app-1-kf8lr --namespace blah -- ls -lai /tmp/src
total 12
[...]
257956134 drwxr-xr-x. 2 1000720000 root  25 Jan  6 11:26 .odo
255859333 -rw-r--r--. 1 1000720000 root   0 Jan  6 11:27 a.txt 
[...]

Not sure why the contents are not getting pushed to the ODO_S2I_SRC_BIN_PATH environment for nodejs centos image while it works fine with other nodejs images. One more interesting point is env like ODO_S2I_DEPLOYMENT_DIR and ODO_SRC_BACKUP_DIR has all the expected data in it. Due to this contradicting behaviour, I suspect this to be a failure from image side. I need to look into the image details to get more clarity on this.

prietyc123 · 2021-01-07T06:32:06Z

According to my recent observation I suspect when we start assemble script it cleans up the sources from destination dir while moving it to working directory in centos/nodejs-10-centos7 case. And as non of the checks satisfy the condition https://github.com/openshift/odo-init-image/blob/master/assemble-and-restart#L70-L72 , it is not able to copy the source from backup. Hence we don't see the contents inside the destination dir and it even complains about /tmp is not a mount point . Why I am saying this because before starting of the assemble script the sources were observed there. Exact logs which shows the behaviour:

$ odo watch -v 4 --context .
[...]
I0106 15:11:03.111467   66280 exec.go:27] Executing command [/opt/odo/bin/assemble-and-restart] for pod: nodejs-app-app-1-cz4qn in container: 
+ set -eo pipefail
+ export ODO_UTILS_DIR=/opt/odo
+ ODO_UTILS_DIR=/opt/odo
+ export ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
+ ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
++ /opt/odo/bin/getlanguage
+ IMAGE_LANG=
+ '[' '!' -z /opt/app-root/src ']'
+ '[' /tmp '!=' /opt/app-root/src ']'
+ '[' -n /opt/app-root/src-backup ']'
+ '[' '!' -d /opt/app-root/src-backup ']'
---> Installing application source ...
+ rsync -rlO /tmp/src/. /opt/app-root/src-backup/src/
+ b_IFS=' 
'
+ b_OFS=
---> Installing all dependencies
+ OIFS=' 
'
+ IFS='
'
++ ls -A /tmp/src/
+ for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
+ rm -fr /opt/app-root/src/a.txt
+ OIFS=
+ IFS=' 
'
+ '[' -n centos7/nodejs-10-centos7 ']'
+ '[' centos7/nodejs-10-centos7 == redhat-[openjdk-18](https://issues.redhat.com/browse/openjdk-18)/openjdk18-openshift ']'
+ '[' -f /tmp/src/.s2i/bin/assemble ']'
+ '[' -n '' ']'
+ '[' -n /usr/libexec/s2i ']'
+ rm -rf /opt/app-root/src/.git
+ /usr/libexec/s2i/assemble

> nodemon@2.0.6 postinstall /opt/app-root/src/node_modules/nodemon
> node bin/postinstall || exit 0
npm WARN node-echo@0.0.1 No repository field.
npm WARN node-echo@0.0.1 No license field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.1 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.1: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
added 119 packages from 53 contributors and audited 120 packages in 2.968s

11 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

---> Building in production mode

---> Pruning the development dependencies
npm WARN node-echo@0.0.1 No repository field.
npm WARN node-echo@0.0.1 No license field.
removed 119 packages in 0.908s
found 0 vulnerabilities

/opt/app-root/src/.npm is not a mountpoint
---> Cleaning the npm cache /opt/app-root/src/.npm
+ '[' '!' -z /opt/app-root/src ']'
+ '[' -n /opt/app-root/src-backup ']'
+ '[' /tmp '!=' /opt/app-root/src ']'
+ '[' /opt/app-root/src '!=' /opt/app-root/src ']'
/tmp is not a mountpoint
+ '[' -z '' ']'
+ PV_MNT_PT=/opt/app-root
+ sed -i s/autostart=false/autostart=true/g /opt/app-root/conf/supervisor.conf
+ /opt/odo/bin/supervisord ctl stop run
---> Cleaning the /tmp/npm-*
run: stopped
+ /opt/odo/bin/supervisord ctl start run
run: started
 ✓  Building component [11s]

sarveshtamba · 2021-01-07T07:45:30Z

I am also seeing similar error on Power:-

[odo] I0107 01:13:47.523986 1984452 exec.go:27] Executing command [/opt/odo/bin/assemble-and-restart] for pod: nodejs-app-1-dqxmm in container: 
[odo] 
 ✓  Syncing files to the component [281ms]
[odo]  •  Building component  ...
[odo] + set -eo pipefail
[odo] + export ODO_UTILS_DIR=/opt/odo
[odo] + ODO_UTILS_DIR=/opt/odo
[odo] + export ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
[odo] + ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
[odo] ++ /opt/odo/bin/getlanguage
[odo] + IMAGE_LANG=nodejs
[odo] + '[' '!' -z /opt/app-root/src ']'
[odo] + '[' /tmp '!=' /opt/app-root/src ']'
[odo] + '[' -n /opt/app-root/src-backup ']'
[odo] + '[' '!' -d /opt/app-root/src-backup ']'
[odo] + mkdir -p /opt/app-root/src-backup/src
[odo] + rsync -rlO /tmp/src/. /opt/app-root/src-backup/src/
[odo] + b_IFS=' 	
[odo] '
[odo] + b_OFS=
[odo] + OIFS=' 	
[odo] '
[odo] + IFS='
[odo] '
[odo] ++ ls -A /tmp/src/
[odo] + for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
[odo] + rm -fr /opt/app-root/src/.dummy
[odo] + for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
[odo] + rm -fr /opt/app-root/src/.gitignore
[odo] + for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
[odo] + rm -fr /opt/app-root/src/.odo
[odo] + for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
[odo] + rm -fr /opt/app-root/src/README.md
[odo] + for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
[odo] + rm -fr /opt/app-root/src/helm
[odo] + for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
[odo] + rm -fr /opt/app-root/src/openshift
[odo] + for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
[odo] + rm -fr /opt/app-root/src/package.json
[odo] + for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
[odo] + rm -fr /opt/app-root/src/server.js
[odo] + for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
[odo] + rm -fr /opt/app-root/src/tests
[odo] + for file in '`ls -A ${ODO_S2I_SRC_BIN_PATH}/src/`'
[odo] + rm -fr /opt/app-root/src/views
[odo] + OIFS=
[odo] + IFS=' 	
[odo] '
[odo] + '[' -n rhscl/nodejs-12-rhel7 ']'
[odo] + '[' rhscl/nodejs-12-rhel7 == redhat-[openjdk-18](https://issues.redhat.com/browse/openjdk-18)/openjdk18-openshift ']'
[odo] + '[' -f /tmp/src/.s2i/bin/assemble ']'
[odo] + '[' -n nodejs ']'
[odo] + /opt/odo/language-scripts/nodejs/dev-assemble
[odo] + DEV_MODE=true
[odo] + REFRESH_WORKING_DIR=/tmp/refresh/
[odo] + PACKAGE_SHA_FILE=/tmp/refresh//package.json.sha1
[odo] + mkdir -p /tmp/refresh/
[odo] + '[' -z '' ']'
[odo] + '[' true == true ']'
[odo] + export NODE_ENV=development
[odo] + NODE_ENV=development
[odo] + '[' development == production ']'
[odo] + export DEV_MODE=true
[odo] + DEV_MODE=true
[odo] + '[' true == true ']'
[odo] ---> Installing application source
[odo] + set -x
[odo] + echo '---> Installing application source'
[odo] + cp -Rf /tmp/src/. ./
[odo] ---> package.json modified
[odo] + '[' true == true ']'
[odo] + '[' -f /tmp/refresh//package.json.sha1 ']'
[odo] + echo '---> package.json modified'
[odo] + sha1sum package.json
[odo] + '[' '!' -z ']'
[odo] + '[' '!' -z ']'
[odo] + '[' '!' -z ']'
[odo] + '[' '!' -z ']'
[odo] + '[' '!' -z ']'
[odo] + '[' '!' -z ']'
[odo] + '[' '!' -z '' ']'
[odo] + echo '---> Building your Node application from source'
[odo] + echo -e 'Current git config'
[odo] + git config --list
[odo] ---> Building your Node application from source
[odo] Current git config
[odo] ---> Installing dependencies
[odo] ---> Using 'npm install'
[odo] + '[' '!' -z '' ']'
[odo] + echo '---> Installing dependencies'
[odo] + '[' true == true ']'
[odo] + echo '---> Using '\''npm install'\'''
[odo] + npm install
Setting current dir to: /usr/local/go/src/github.com/openshift/odo/tests/integration
Deleting dir: /tmp/886181273

Attaching the detailed log -
scripts_openshiftci-presubmit-all-tests_4347.log

sarveshtamba · 2021-01-19T11:15:20Z

I am still seeing the error as below:-

https://gist.github.com/sarveshtamba/575daec2a6e821884faafbb7971c58c5

openshift-ci-robot added kind/failing-test priority/High Important issue; should be worked on before any other issues (except priority/Critical issue(s)). labels Jan 6, 2021

This was referenced Jan 6, 2021

sanitize dir name before using it on odo push #4333

Merged

Separates test into its own spec #4345

Merged

Refactor catalog list services #4338

Merged

This was referenced Jan 7, 2021

Revert centos nodejs tags to older version #4348

Merged

Add centos7 nodejs image as supported redhat-developer/odo-init-image#82

Merged

Add centos7 nodejs latest as supported #4349

Merged

mohammedzee1000 mentioned this issue Jan 13, 2021

Setting up baseline POC of internal CI #4063

Merged

openshift-merge-robot closed this as completed in #4349 Jan 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recurring CI failures while trying to verify support for nodejs images #4347

Recurring CI failures while trying to verify support for nodejs images #4347

dharmit commented Jan 6, 2021

prietyc123 commented Jan 6, 2021 •

edited by openshift-ci-robot

Loading

prietyc123 commented Jan 7, 2021

sarveshtamba commented Jan 7, 2021 •

edited by openshift-ci-robot

Loading

sarveshtamba commented Jan 19, 2021

Recurring CI failures while trying to verify support for nodejs images #4347

Recurring CI failures while trying to verify support for nodejs images #4347

Comments

dharmit commented Jan 6, 2021

prietyc123 commented Jan 6, 2021 • edited by openshift-ci-robot Loading

prietyc123 commented Jan 7, 2021

sarveshtamba commented Jan 7, 2021 • edited by openshift-ci-robot Loading

sarveshtamba commented Jan 19, 2021

prietyc123 commented Jan 6, 2021 •

edited by openshift-ci-robot

Loading

sarveshtamba commented Jan 7, 2021 •

edited by openshift-ci-robot

Loading