Create a file, push, delete, then push again does not propagate the deletions on 4.4 cluster

[amitkrout]

/kind bug

What versions of software are you using?

Operating System:
All supported
Output of odo version:
master

How did you run odo exactly?

make test-cmd-push

Scenario which is failing - https://github.com/openshift/odo/blob/master/tests/integration/cmd_push_test.go#L129

Actual behavior

Fails

Expected behavior

should pass

Any logs, error output, etc?

------------------------------
Creating a new project: fztwkumgri
Running odo with args [odo project create fztwkumgri -w -v4]
[odo]  •  Waiting for project to come up  ...
[odo] I0226 10:14:52.166332   18004 occlient.go:538] Project fztwkumgri now exists
[odo]  ✓  Waiting for project to come up [800ms]
[odo]  ✓  Project 'fztwkumgri' is ready for use
[odo]  ✓  New project created and now using project: fztwkumgri
[odo] I0226 10:14:52.180482   18004 odo.go:80] Could not get the latest release information in time. Never mind, exiting gracefully :)
Created dir: /tmp/553271773
Current working dir: /go/src/github.com/openshift/odo/tests/integration
Running git with args [git clone https://github.com/openshift/nodejs-ex /tmp/553271773/nodejs-ex]
[git] Cloning into '/tmp/553271773/nodejs-ex'...
Running odo with args [odo component create nodejs nodejs --project fztwkumgri --context /tmp/553271773/nodejs-ex --app app]
[odo] Warning: nodejs is not fully supported by odo, and it is not guaranteed to work
[odo]  •  Validating component  ...
[odo]  ✓  Validating component [60ms]
[odo] 
[odo] Please use `odo push` command to create the component with source deployed
Running odo with args [odo push --context /tmp/553271773/nodejs-ex]
[odo] Validation
[odo]  •  Checking component  ...
[odo]  ✓  Checking component [94ms]
[odo] 
[odo] Configuration changes
[odo]  ✓  Initializing component
[odo]  •  Creating component  ...
[odo]  ✓  Creating component [311ms]
[odo] 
[odo] Pushing to component nodejs of type local
[odo]  •  Checking files for pushing  ...
[odo]  ✓  Checking files for pushing [1ms]
[odo]  •  Waiting for component to start  ...
[odo]  ✓  Waiting for component to start [31s]
[odo]  •  Syncing files to the component  ...
[odo]  ✓  Syncing files to the component [330ms]
[odo]  •  Building component  ...
[odo]  ✓  Building component [12s]
[odo]  ✓  Changes successfully pushed to component
Running oc with args [oc get pods --namespace fztwkumgri --show-labels]
[oc] NAME                  READY   STATUS      RESTARTS   AGE   LABELS
[oc] nodejs-app-1-deploy   0/1     Completed   0          42s   openshift.io/deployer-pod-for.name=nodejs-app-1
[oc] nodejs-app-1-rkf59    1/1     Running     0          40s   deployment=nodejs-app-1,deploymentconfig=nodejs-app
Running oc with args [oc get dc nodejs-app --namespace fztwkumgri -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}:{.value}{"\n"}{end}']
[oc] 'DEBUG_PORT:5858
[oc] ODO_S2I_SCRIPTS_URL:/usr/libexec/s2i
[oc] ODO_S2I_SCRIPTS_PROTOCOL:image://
[oc] ODO_S2I_SRC_BIN_PATH:/tmp
[oc] ODO_S2I_DEPLOYMENT_DIR:/opt/app-root/src
[oc] ODO_S2I_WORKING_DIR:/opt/app-root/src
[oc] ODO_S2I_BUILDER_IMG:rhscl/nodejs-12-rhel7
[oc] ODO_SRC_BACKUP_DIR:/opt/app-root/src-backup
[oc] 'Running oc with args [oc exec nodejs-app-1-rkf59 --namespace fztwkumgri -- ls -lai /opt/app-root/src]
[oc] total 112
[oc]   17 drwxrwsr-x.  12 1001910000 root        4096 Feb 26 10:15 .
[oc]   12 drwxr-sr-x.   6 1001910000 1001910000  4096 Feb 26 10:15 ..
[oc]   91 drwxrws---.   3 1001910000 root        4096 Feb 26 10:15 .config
[oc]   57 -rw-rw-r--.   1 1001910000 root           0 Feb 26 10:15 .dummy
[oc]   58 -rw-rw-r--.   1 1001910000 root          53 Feb 26 10:14 .gitignore
[oc]   90 drwxrwsr-x.   2 1001910000 root        4096 Feb 26 10:15 .npm-global
[oc]   59 drwxrwsr-x.   2 1001910000 root        4096 Feb 26 10:15 .odo
[oc]   18 drwxrwsr-x.   3 1001910000 root        4096 Jan 29 14:13 .pki
[oc]   62 -rw-rw-r--.   1 1001910000 root       12926 Feb 26 10:14 README.md
[oc]   63 -rw-rw----.   1 1001910000 root          11 Feb 26 10:14 foobar.txt
[oc]   64 drwxrwsr-x.   3 1001910000 root        4096 Feb 26 10:15 helm
[oc] 1591 drwxrwsr-x. 102 1001910000 root        4096 Feb 26 10:15 node_modules
[oc]   75 drwxrwsr-x.   4 1001910000 root        4096 Feb 26 10:15 openshift
[oc]   83 -rw-rw-r--.   1 1001910000 root       31945 Feb 26 10:15 package-lock.json
[oc] 1590 -rw-rw-r--.   1 1001910000 root         818 Feb 26 10:15 package.json
[oc]   84 -rw-rw-r--.   1 1001910000 root        3814 Feb 26 10:14 server.js
[oc]   85 drwxrws---.   2 1001910000 root        4096 Feb 26 10:14 testdir
[oc]   86 drwxrwsr-x.   2 1001910000 root        4096 Feb 26 10:15 tests
[oc]   88 drwxrwsr-x.   2 1001910000 root        4096 Feb 26 10:15 views
Deleting dir: /tmp/553271773/nodejs-ex/foobar.txt
Deleting dir: /tmp/553271773/nodejs-ex/testdir
Running odo with args [odo push --context /tmp/553271773/nodejs-ex -v4]
[odo] I0226 10:15:38.661885   18284 preference.go:147] The path for preference file is /tmp/553271773/config.yaml
[odo] I0226 10:15:38.662190   18284 common_push.go:153] SourceLocation: ./
[odo] I0226 10:15:38.662211   18284 common_push.go:161] Source Path: /tmp/553271773/nodejs-ex
[odo] I0226 10:15:38.807638   18284 preference.go:147] The path for preference file is /tmp/553271773/config.yaml
[odo] Validation
[odo]  •  Checking component  ...
[odo] I0226 10:15:38.807809   18284 occlient.go:2606] Getting DeploymentConfig: nodejs-app
[odo] I0226 10:15:38.911507   18284 component.go:506] Checking source location: ./
[odo] I0226 10:15:38.911573   18284 preference.go:147] The path for preference file is /tmp/553271773/config.yaml
[odo]  ✓  Checking component [103ms]
[odo] 
[odo] Configuration changes
[odo]  •  Retrieving component data  ...
[odo] I0226 10:15:38.983446   18284 occlient.go:2606] Getting DeploymentConfig: nodejs-app
[odo] I0226 10:15:39.014761   18284 component.go:1023] Source for component nodejs is file://./ (local)
[odo] I0226 10:15:39.014800   18284 occlient.go:2606] Getting DeploymentConfig: nodejs-app
[odo] I0226 10:15:39.046159   18284 component.go:1149] Updating component nodejs, from local to ./ (local).
[odo]  ✓  Retrieving component data [134ms]
[odo]  •  Applying configuration  ...
[odo] I0226 10:15:39.106039   18284 occlient.go:748] Found exact image tag match for nodejs:latest
[odo] I0226 10:15:39.180616   18284 url.go:80] Listing routes with label selector: app.kubernetes.io/part-of=app,app.kubernetes.io/instance=nodejs
[odo] I0226 10:15:39.180638   18284 occlient.go:2474] Listing routes with label selector: app.kubernetes.io/part-of=app,app.kubernetes.io/instance=nodejs
[odo]  ✓  Applying configuration [134ms]
[odo] 
[odo] Pushing to component nodejs of type local
[odo]  •  Checking file changes for pushing  ...
[odo] I0226 10:15:39.217617   18284 util.go:569] ignoring path /tmp/553271773/nodejs-ex/.git because of glob rule /tmp/553271773/nodejs-ex/.git
[odo] I0226 10:15:39.217686   18284 file_indexer.go:204] .odo or .git directory detected, skipping it
[odo] I0226 10:15:39.218082   18284 file_indexer.go:244] Deleting file: foobar.txt
[odo] I0226 10:15:39.218096   18284 file_indexer.go:244] Deleting file: testdir
[odo]  ✓  Checking file changes for pushing [1ms]
[odo] I0226 10:15:39.218494   18284 common_push.go:225] List of files to be deleted: +[foobar.txt testdir]
[odo] I0226 10:15:39.218512   18284 common_push.go:248] Copying directory /tmp/553271773/nodejs-ex to pod
[odo] I0226 10:15:39.218527   18284 component.go:635] PushLocal: componentName: nodejs, applicationName: app, path: /tmp/553271773/nodejs-ex, files: [], delFiles: [foobar.txt testdir], isForcePush: false
[odo] I0226 10:15:39.247688   18284 preference.go:147] The path for preference file is /tmp/553271773/config.yaml
[odo] I0226 10:15:39.247738   18284 occlient.go:1803] Waiting for deploymentconfig=nodejs-app pod
[odo]  •  Waiting for component to start  ...
[odo]  ✓  Waiting for component to start [31ms]
[odo] I0226 10:15:39.278935   18284 occlient.go:1827] Status of nodejs-app-1-rkf59 pod is Running
[odo] I0226 10:15:39.278983   18284 occlient.go:1831] Pod nodejs-app-1-rkf59 is Running
[odo]  •  Syncing files to the component  ...
[odo] I0226 10:15:39.279122   18284 component.go:674] propogating deletion of files foobar.txt testdir to pod
[odo] I0226 10:15:39.279155   18284 occlient.go:3173] s2ipaths marked for deletion are [/opt/app-root/src/foobar.txt /opt/app-root/src/testdir /tmp/src/foobar.txt /tmp/src/testdir /opt/app-root/src-backup/src/foobar.txt /opt/app-root/src-backup/src/testdir]
[odo]  ✓  Syncing files to the component [319ms]
[odo]  •  Building component  ...
[odo] + set -eo pipefail
[odo] + export ODO_UTILS_DIR=/opt/odo
[odo] + ODO_UTILS_DIR=/opt/odo
[odo] + export ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
[odo] + ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
[odo] ++ /opt/odo/bin/getlanguage
[odo] + IMAGE_LANG=
[odo] + '[' '!' -z /opt/app-root/src ']'
[odo] + '[' /tmp '!=' /opt/app-root/src ']'
[odo] + '[' -n /opt/app-root/src-backup ']'
[odo] + '[' '!' -d /opt/app-root/src-backup ']'
[odo] + rsync -rlO /tmp/src/. /opt/app-root/src-backup/src/
[odo] + b_IFS=' 	
[odo] '
[odo] + b_OFS=
[odo] + OIFS=' 	
[odo] '
[odo] + IFS='
[odo] '
[odo] ++ ls -A /tmp/src/
[odo] + OIFS=
[odo] + IFS=' 	
[odo] '
[odo] + '[' -n rhscl/nodejs-12-rhel7 ']'
[odo] + '[' rhscl/nodejs-12-rhel7 == redhat-openjdk-18/openjdk18-openshift ']'
[odo] + '[' -f /tmp/src/.s2i/bin/assemble ']'
[odo] ---> Installing application source ...
[odo] + '[' -n '' ']'
[odo] + '[' -n /usr/libexec/s2i ']'
[odo] + rm -rf /opt/app-root/src/.git
[odo] + /usr/libexec/s2i/assemble
[odo] mv: cannot stat '/tmp/src/*': No such file or directory
[odo]  ✗  Unable to build files
[odo] + set -eo pipefail
[odo] + export ODO_UTILS_DIR=/opt/odo
[odo] + ODO_UTILS_DIR=/opt/odo
[odo] + export ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
[odo] + ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
[odo] ++ /opt/odo/bin/getlanguage
[odo] + IMAGE_LANG=
[odo] + '[' '!' -z /opt/app-root/src ']'
[odo] + '[' /tmp '!=' /opt/app-root/src ']'
[odo] + '[' -n /opt/app-root/src-backup ']'
[odo] + '[' '!' -d /opt/app-root/src-backup ']'
[odo] + rsync -rlO /tmp/src/. /opt/app-root/src-backup/src/
[odo] + b_IFS=' 	
[odo] '
[odo] + b_OFS=
[odo] + OIFS=' 	
[odo] '
[odo] + IFS='
[odo] '
[odo] ++ ls -A /tmp/src/
[odo] + OIFS=
[odo] + IFS=' 	
[odo] '
[odo] + '[' -n rhscl/nodejs-12-rhel7 ']'
[odo] + '[' rhscl/nodejs-12-rhel7 == redhat-openjdk-18/openjdk18-openshift ']'
[odo] + '[' -f /tmp/src/.s2i/bin/assemble ']'
[odo] ---> Installing application source ...
[odo] + '[' -n '' ']'
[odo] + '[' -n /usr/libexec/s2i ']'
[odo] + rm -rf /opt/app-root/src/.git
[odo] + /usr/libexec/s2i/assemble
[odo] mv: cannot stat '/tmp/src/*': No such file or directory
[odo] 
[odo]  ✗  Building component [273ms]
[odo] I0226 10:15:39.872734   18284 cmdutils.go:48] Error:
[odo] Failed to push component: nodejs: unable to execute assemble script: error while streaming command: command terminated with exit code 1
[odo]  ✗  command terminated with exit code 1
Deleting project: fztwkumgri
Running odo with args [odo project delete fztwkumgri -f]
[odo] This project contains the following applications, which will be deleted
[odo] Application app
[odo] This application has following components that will be deleted
[odo] component named nodejs
[odo] No services / could not get services
[odo]  •  Deleting project fztwkumgri  ...
[odo]  ✓  Deleting project fztwkumgri [6s]
[odo]  ✓  Deleted project : fztwkumgri
Deleting dir: /tmp/553271773
Setting current dir to: /go/src/github.com/openshift/odo/tests/integration
• Failure [55.475 seconds]
odo push command tests
/go/src/github.com/openshift/odo/tests/integration/cmd_push_test.go:16
  when push command is executed
  /go/src/github.com/openshift/odo/tests/integration/cmd_push_test.go:112
    should be able to create a file, push, delete, then push again propagating the deletions [It]
    /go/src/github.com/openshift/odo/tests/integration/cmd_push_test.go:129
    No future change is possible.  Bailing out early after 1.315s.
      
    Running odo with args [odo push --context /tmp/553271773/nodejs-ex -v4]
    Expected
        <int>: 1
    to match exit code:
        <int>: 0
    /go/src/github.com/openshift/odo/tests/helper/helper_run.go:34
------------------------------

[kadel]

/area component
/priority high

[mik-dass]

The problem is that there is no code in the /tmp/src folder due to changes in the recent builder images. Looking at the logs

[odo] + /usr/libexec/s2i/assemble
[odo] mv: cannot stat '/tmp/src/*': No such file or directory

something in the assemble script of the builder, tries to move files from the /tmp/src folder and since it's empty, it errors out.

[mik-dass]

Looking into the assemble and run scripts for nodejs 10 and nodejs 12, they seem to be completely identical sclorg/s2i-nodejs-container@f4fc901. The files were only renamed. Even the assemble-and-restart scripts in the container seemed completely identical.

[mik-dass]

Looking into the /usr/libexec/s2i/assemble scripts of a nodejs component for 3.11 and 4.4

3.11

#!/bin/bash

set -e

source /usr/libexec/s2i/env

if [ "$DEV_MODE" == true ] ; then
        set -x
fi

echo "---> Installing application source"
cp -Rfp /tmp/src/. ./

if [ ! -z $HTTP_PROXY ]; then
        echo "---> Setting npm http proxy to $HTTP_PROXY"
        npm config set proxy $HTTP_PROXY
fi

if [ ! -z $http_proxy ]; then
        echo "---> Setting npm http proxy to $http_proxy"
        npm config set proxy $http_proxy
fi

if [ ! -z $HTTPS_PROXY ]; then
        echo "---> Setting npm https proxy to $HTTPS_PROXY"
        npm config set https-proxy $HTTPS_PROXY
fi

if [ ! -z $https_proxy ]; then
        echo "---> Setting npm https proxy to $https_proxy"
        npm config set https-proxy $https_proxy
fi

# Change the npm registry mirror if provided
if [ ! -z "$NPM_MIRROR" ]; then
        echo "---> Setting the npm package mirror to $NPM_MIRROR"
        npm config set registry $NPM_MIRROR
fi

echo "---> Building your Node application from source"
echo -e "Current git config"
git config --list

if [ ! -z "$YARN_ENABLED" ]; then
        echo "---> Using 'yarn install' with YARN_ARGS"
        yarn install $YARN_ARGS
else
        echo "---> Installing dependencies"
        if [ "$DEV_MODE" == true ]; then
                echo "---> Using 'npm install'"
                npm install

                #do not fail when there is no build script
                echo "---> Building in development mode"
                npm run build --if-present
        else
                HAS_BUILD=$(node -e "console.log(require('./package.json').scripts.build ? true : false)")

                # check to see if there is a build script by inspecting the package.json
                if [ "$HAS_BUILD" == true ]; then
                        # Do a npm install to get the dev depdencies
                        echo "---> Installing dev dependencies"
                        NODE_ENV=development npm install
                        #do not fail when there is no build script
                        echo "---> Building in production mode"
                        npm run build --if-present
                else
                        echo "---> Using 'npm install -s --only=production'"
                        npm install -s --only=production
                fi

                echo "---> Pruning the development dependencies"
                npm prune
        fi
fi

echo "---> Cleaning up npm cache"
rm -rf .npm

echo "---> Fix permissions on app-root"
fix-permissions /opt/app-root

4.4

#!/bin/bash

# Prevent running assemble in builders different than official STI image.
# The official nodejs:12-onbuild already run npm install and use different
# application folder.
[ -d "/usr/src/app" ] && exit 0

set -e

# FIXME: Linking of global modules is disabled for now as it causes npm failures
#        under RHEL7
# Global modules good to have
# npmgl=$(grep "^\s*[^#\s]" ../etc/npm_global_module_list | sort -u)
# Available global modules; only match top-level npm packages
#global_modules=$(npm ls -g 2> /dev/null | perl -ne 'print "$1\n" if /^\S+\s(\S+)\@[\d\.-]+/' | sort -u)
# List all modules in common
#module_list=$(/usr/bin/comm -12 <(echo "${global_modules}") | tr '\n' ' ')
# Link the modules
#npm link $module_list

safeLogging () {
    if [[ $1 =~ http[s]?://.*@.*$ ]]; then
        echo $1 | sed 's/^.*@/redacted@/'
    else
        echo $1
    fi
}

shopt -s dotglob
if [ -d /tmp/artifacts ] && [ "$(ls /tmp/artifacts/ 2>/dev/null)" ]; then
    echo "---> Restoring previous build artifacts ..."
    mv -T --verbose /tmp/artifacts/node_modules "${HOME}/node_modules"
fi

echo "---> Installing application source ..."
mv /tmp/src/* ./

# Fix source directory permissions
fix-permissions ./

if [ ! -z $HTTP_PROXY ]; then
    echo "---> Setting npm http proxy to" $(safeLogging $HTTP_PROXY)
        npm config set proxy $HTTP_PROXY
fi

if [ ! -z $http_proxy ]; then
    echo "---> Setting npm http proxy to" $(safeLogging $http_proxy)
        npm config set proxy $http_proxy
fi

if [ ! -z $HTTPS_PROXY ]; then
    echo "---> Setting npm https proxy to" $(safeLogging $HTTPS_PROXY)
        npm config set https-proxy $HTTPS_PROXY
fi

if [ ! -z $https_proxy ]; then
    echo "---> Setting npm https proxy to" $(safeLogging $https_proxy)
        npm config set https-proxy $https_proxy
fi

# Change the npm registry mirror if provided
if [ -n "$NPM_MIRROR" ]; then
        npm config set registry $NPM_MIRROR
fi

# Set the DEV_MODE to false by default.
if [ -z "$DEV_MODE" ]; then
  export DEV_MODE=false
fi

# If NODE_ENV is not set by the user, then NODE_ENV is determined by whether
# the container is run in development mode.
if [ -z "$NODE_ENV" ]; then
  if [ "$DEV_MODE" == true ]; then
    export NODE_ENV=development
  else
    export NODE_ENV=production
  fi
fi

if [ "$NODE_ENV" != "production" ]; then

        echo "---> Building your Node application from source"
        npm install

else

        echo "---> Installing all dependencies"
        NODE_ENV=development npm install

        #do not fail when there is no build script
        echo "---> Building in production mode"
        npm run build --if-present

        echo "---> Pruning the development dependencies"
        npm prune

        # Clear the npm's cache and tmp directories only if they are not a docker volumes
        NPM_CACHE=$(npm config get cache)
        if ! mountpoint $NPM_CACHE; then
                echo "---> Cleaning the npm cache $NPM_CACHE"
                #As of npm@5 even the 'npm cache clean --force' does not fully remove the cache directory
                # instead of $NPM_CACHE* use $NPM_CACHE/*.
                # We do not want to delete .npmrc file.
                rm -rf "${NPM_CACHE:?}/"
        fi
        NPM_TMP=$(npm config get tmp)
        if ! mountpoint $NPM_TMP; then
                echo "---> Cleaning the $NPM_TMP/npm-*"
                rm -rf $NPM_TMP/npm-*
        fi

fi

# Fix source directory permissions
fix-permissions ./

The difference between the scripts is that in case of 3.11, it's a cp command which just copies the files but in case of 4.4 it's a mv command which moves the files and thus the /tmp folder becomes empty in case of 4.4.

If we remove the files added and files deleted checks for odo push, then build and push again, it throws the same error.

[kadel]

This whole issue might be just due to the fact that nodejs image is doesn't show as supported image #2603

[mik-dass]

This whole issue might be just due to the fact that nodejs image is doesn't show as supported image #2603

Yes I am getting a similar feeling about this.