CI Test Failure under updated kernel 5.x

[cevich]

Steps to reproduce (using same VM images as master):

$ hack/get_ci_vm.sh fedora-cloud-base-30-1-2-1556821664
...cut...

Everything works fine, including the package install/update.

[root@cevich-fedora-cloud-base-30-1-2-1556821664 storage]# contrib/cirrus/build_and_test.sh

The tests pass as normal, then I execute reboot. After waiting a minute, reconnect to the VM and execute tests a second time:

[root@cevich-fedora-cloud-base-30-1-2-1556821664 storage]# reboot
Connection to 35.226.8.25 closed by remote host.
Connection to 35.226.8.25 closed.
ERROR: (gcloud.compute.ssh) [/bin/ssh] exited with return code [255].

Offering to Delete cevich-fedora-cloud-base-30-1-2-1556821664 (Might take a minute or two)

Note: It's safe to answer N, then re-run script again later.
+  sudo podman run -it --rm -e AS_ID=4179 -e AS_USER=cevich --security-opt label=disable -v /tmp/get_ci_vm.sh_tmpdir_NsUYzF:/home/cevich -v /home/cevich/.config/gcloud:/home/cevich/.config/gcloud -v /home/cevich/.config/gcloud/ssh:/home/cevich/.ssh -v /home/cevich/devel/storage:/home/cevich/devel/storage quay.io/cevich/gcloud_centos:latest --configuration=storage --project=storage-240716 compute instances delete --zone us-central1-b --delete-disks=all cevich-fedora-cloud-base-30-1-2-1556821664
The following instances will be deleted. Any attached disks configured
 to be auto-deleted will be deleted unless they are attached to any 
other instances or the `--keep-disks` flag is given and specifies them
 for keeping. Deleting a disk is irreversible and any data on the disk
 will be lost.
 - [cevich-fedora-cloud-base-30-1-2-1556821664] in [us-central1-b]

Do you want to continue (Y/n)?  n

ERROR: (gcloud.compute.instances.delete) Deletion aborted by user.
[cevich@cevich storage]$ hack/get_ci_vm.sh fedora-cloud-base-30-1-2-1556821664
...cut...
[root@cevich-fedora-cloud-base-30-1-2-1556821664 storage]# contrib/cirrus/build_and_test.sh

This time the tests fail in the same/similar manner as in #408

Master currently is: 1a0442e

[cevich]

[root@cevich-fedora-cloud-base-30-1-2-1556821664 storage]# grubby --default-kernel
/boot/vmlinuz-5.2.7-200.fc30.x86_64
[root@cevich-fedora-cloud-base-30-1-2-1556821664 storage]# grubby --default-index
1
[root@cevich-fedora-cloud-base-30-1-2-1556821664 storage]# grubby --set-default-index=0
The default is /boot/loader/entries/f241772f3e32496c92975269b5794615-5.0.9-301.fc30.x86_64.conf with index 0 and kernel /boot/vmlinuz-5.0.9-301.fc30.x86_64
[root@cevich-fedora-cloud-base-30-1-2-1556821664 storage]# reboot
...cut...
[root@cevich-fedora-cloud-base-30-1-2-1556821664 storage]# contrib/cirrus/build_and_test.sh
...cut...
Tests pass!

Conclusion: Kernel /boot/vmlinuz-5.2.7-200.fc30.x86_64 is incompatible with c/storage and/or tests.

[cevich]

On F29:

[root@cevich-fedora-cloud-base-29-1-2-1541789245 storage]# uname -r
4.18.17-300.fc29.x86_64
[root@cevich-fedora-cloud-base-29-1-2-1541789245 storage]# contrib/cirrus/build_and_test.sh 
...tests pass...
[root@cevich-fedora-cloud-base-29-1-2-1541789245 storage]# reboot
...reconnect...
[root@cevich-fedora-cloud-base-29-1-2-1541789245 storage]# contrib/cirrus/build_and_test.sh
...fails...
[root@cevich-fedora-cloud-base-29-1-2-1541789245 storage]# uname -r
5.2.7-100.fc29.x86_64

[cevich]

@nalind @rhatdan PTAL, this is also blocking #418 and basically any future testing (if/when kernel is updated).

[rhatdan]

Does this work on later fedora? We can drop f29?

[cevich]

@rhatdan nope it's broken in F30 as well (same kernel).

[cevich]

oh geeze, my paste-bins of the errors are gone 😠

Here's an example under F30: https://cirrus-ci.com/task/5174065982078976
and one under F29: https://cirrus-ci.com/task/6299965888921600

[cevich]

To be clear, I don't know for-sure this is a kernel problem. It could easily be a race problem in the test which is simply triggered by the different kernel. Probably best way forward is for someone who knows these tests to go in with hack/get_ci_vm.sh and reproduce it (was 100% when I opened the issue).

[cevich]

Thinking more over the last few days, the next step here seems to be getting more data what/why this is failing. I don't think the errors from the test output is detailed enough to pinpoint it. Maybe running the test commands manually would help?

[cevich]

@rhatdan @nalind I remember someone said something like "this is failing because some layer isn't unmounting". Any chance this is at all related to containers/podman#3870 (where I'm also dealing with an updated system encountering test-breakage)?

[cevich]

update: this cannot be related to 3870 since we're not using podman-remote 😖