v1.0.0: olm upgrade test failed

[pohly]

After tagging v1.0.0, the "olm upgrade" test caused following olm tests to fail because it removed the "deploy/olm-bundle/1.0.0" directory that is needed for other tests. From https://cloudnative-k8sci.southcentralus.cloudapp.azure.com/job/pmem-csi/job/v1.0.0/1/artifact/joblog-jenkins-pmem-csi-v1.0.0-1-test-1.19.log:

pmem-csi-operator-55584cb8b9-p4r6w/pmem-csi-operator@pmem..ker1: I0717 08:11:51.070825       1 reflector.go:255] Listing and watching *v1.RoleBinding from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:241
�[1mSTEP�[0m: Operator is ready!
I0717 08:11:51.107474     724 exec.go:34] "Starting command" command="/bin/rm" args=[rm -rf /mnt/workspace/pmem-csi_v1.0.0/deploy/olm-bundle/1.0.0]
I0717 08:11:51.108784     724 exec.go:52] "Command terminated" command="/bin/rm" stdout-len=0 combined-len=0 error=<nil>
Jul 17 08:11:51.108: INFO: upgrade: Operator changed to: 0.9
Jul 17 08:11:51.108: INFO: Ensuring that the CRs got reconciled
...
I0717 08:13:46.273436     724 exec.go:34] "EnsureDeployment: Starting command" command="test/start-operator.sh" args=[test/start-operator.sh -olm]
I0717 08:13:46.285487     724 exec.go:72] "EnsureDeployment/stderr: ++ ls -1t /mnt/workspace/pmem-csi_v1.0.0/deploy/olm-bundle" command="test/start-operator.sh"
I0717 08:13:46.285530     724 exec.go:72] "EnsureDeployment/stderr: ++ head -1" command="test/start-operator.sh"
I0717 08:13:46.287185     724 exec.go:72] "EnsureDeployment/stderr: + BUNDLE_VERSION=" command="test/start-operator.sh"
I0717 08:13:46.287216     724 exec.go:72] "EnsureDeployment/stderr: + echo 'Starting the operator using OLM....'" command="test/start-operator.sh"
I0717 08:13:46.287233     724 exec.go:72] "EnsureDeployment/stdout: Starting the operator using OLM...." command="test/start-operator.sh"
I0717 08:13:46.287241     724 exec.go:72] "EnsureDeployment/stderr: + BUNDLE_DIR=/mnt/workspace/pmem-csi_v1.0.0/deploy/olm-bundle/" command="test/start-operator.sh"
I0717 08:13:46.287258     724 exec.go:72] "EnsureDeployment/stderr: + BINDIR=/mnt/workspace/pmem-csi_v1.0.0/_work/bin" command="test/start-operator.sh"
I0717 08:13:46.287283     724 exec.go:72] "EnsureDeployment/stderr: + '[' '!' -d /mnt/workspace/pmem-csi_v1.0.0/deploy/olm-bundle/ ']'" command="test/start-operator.sh"
I0717 08:13:46.287298     724 exec.go:72] "EnsureDeployment/stderr: + set +e" command="test/start-operator.sh"
I0717 08:13:46.287645     724 exec.go:72] "EnsureDeployment/stderr: ++ /mnt/workspace/pmem-csi_v1.0.0/_work/govm/ssh.0 kubectl get csv -o 'jsonpath={.items[*].metadata.name}'" command="test/start-operator.sh"
I0717 08:13:46.816153     724 exec.go:72] "EnsureDeployment/stderr: + output=" command="test/start-operator.sh"
I0717 08:13:46.816222     724 exec.go:72] "EnsureDeployment/stderr: + '[' 0 -eq 0 ']'" command="test/start-operator.sh"
I0717 08:13:46.816343     724 exec.go:72] "EnsureDeployment/stderr: + set -e" command="test/start-operator.sh"
I0717 08:13:46.816592     724 exec.go:72] "EnsureDeployment/stderr: ++ mktemp -d" command="test/start-operator.sh"
I0717 08:13:46.817673     724 exec.go:72] "EnsureDeployment/stderr: + tmpdir=/tmp/tmp.jiiaqoASWY" command="test/start-operator.sh"
I0717 08:13:46.817693     724 exec.go:72] "EnsureDeployment/stderr: + TMP_BUNDLE_DIR=/tmp/tmp.jiiaqoASWY/" command="test/start-operator.sh"
I0717 08:13:46.817709     724 exec.go:72] "EnsureDeployment/stderr: + trap 'rm -rf $tmpdir' SIGTERM SIGINT EXIT" command="test/start-operator.sh"
I0717 08:13:46.817747     724 exec.go:72] "EnsureDeployment/stderr: + cp -rv /mnt/workspace/pmem-csi_v1.0.0/deploy/olm-bundle/ /tmp/tmp.jiiaqoASWY/" command="test/start-operator.sh"
I0717 08:13:46.820259     724 exec.go:72] "EnsureDeployment/stdout: '/mnt/workspace/pmem-csi_v1.0.0/deploy/olm-bundle/' -> '/tmp/tmp.jiiaqoASWY/olm-bundle'" command="test/start-operator.sh"
I0717 08:13:46.820526     724 exec.go:72] "EnsureDeployment/stderr: + CSV_FILE=/tmp/tmp.jiiaqoASWY//manifests/pmem-csi-operator.clusterserviceversion.yaml" command="test/start-operator.sh"
I0717 08:13:46.820552     724 exec.go:72] "EnsureDeployment/stderr: + '[' 172.17.0.1:5000 '!=' intel ']'" command="test/start-operator.sh"
I0717 08:13:46.820595     724 exec.go:72] "EnsureDeployment/stderr: + sed -i -e 's^intel/pmem-csi-driver:^172.17.0.1:5000/pmem-csi-driver:v1.0.0^g' /tmp/tmp.jiiaqoASWY//manifests/pmem-csi-operator.clusterserviceversion.yaml" command="test/start-operator.sh"
I0717 08:13:46.821929     724 exec.go:72] "EnsureDeployment/stderr: sed: can't read /tmp/tmp.jiiaqoASWY//manifests/pmem-csi-operator.clusterserviceversion.yaml: No such file or directory" command="test/start-operator.sh"
I0717 08:13:46.822187     724 exec.go:72] "EnsureDeployment/stderr: + rm -rf /tmp/tmp.jiiaqoASWY" command="test/start-operator.sh"
I0717 08:13:46.823355     724 exec.go:52] "EnsureDeployment: Command terminated" command="test/start-operator.sh" stdout-len=124 combined-len=1222 error="exit status 2"

The code is:

pmem-csi/test/e2e/versionskew/operator.go

Lines 81 to 89 in e477f9f

	semver := current + ".0"
	defer func() {
	// Remove generated bundle
	_, err := pmemexec.RunCommand(ctx, "rm", "-rf", root+"/deploy/olm-bundle/"+semver)
	framework.ExpectNoError(err, "remove generated bundle: %v", err)
	}()
	// (Re)Generate olm-bundle with future release version number
	// So that OLM sees it as upgrade.
	make := exec.Command("make", "operator-generate-bundle", "VERSION=v"+semver)

[pohly]

I am not sure how this is supposed to work after tagging a release. Do we need to set a version somewhere? Then this should (ideally) be part of set-version.sh or must be mentioned in our release checklist.

As a stop-gap solution I am going to add this test to the skip list and re-tag v1.0.0.

[okartau]

periodic jobs have also failed, and one reason may be BUNDLE_VERSION being empty. Is this related to current issue?

+ BUNDLE_VERSION=
...
sed: can't read /tmp/tmp.WEQGkI87e9//manifests/pmem-csi-operator.clusterserviceversion.yaml: No such file or directory

BUNDLE_VERSION is used on that path before /manifests/
brief check indicates emtpy BUNDLE_VERSION causes failure?

[pohly]

Same issue. olm.*upgrade is still enabled on the master branch.

[pohly]

The short-term fix of bumping the version from 1.0 to 100.0 (#1034) worked.

[pohly]

Fixed by PR #1035