WIP: Cirrus: Update to XFS root in Fedora VM images

[cevich]

COMPLETELY experimental; DO NOT MERGE

Ref: containers/automation_images#69

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cevich
To complete the pull request process, please assign rhatdan after the PR has been reviewed.
You can assign the PR to them by writing /assign @rhatdan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[cevich]

@rhatdan I have some...errr...results 😁 Both F33 and F34 are running with XFS, and in all cases do not appear to run significantly faster compared to EXT4. Though I'll assume that deeper inspection/debugging would find some/many low-level inefficiencies that need fixing. The test failures are quite a bit more worrying, since they might describe real-world problems that need fixing.

The other alternative is test/environment setup bugs (i.e. not "real world" reproducible) but these too may be worth fixing. In all cases it would be good to put some expert eyeballs on these failures. Especially since switching to XFS appears to cause checkpoint/restore to outright hang 😖

• int podman fedora-34 root host - Ginkgo node hung running checkpoint-pause test.
• int podman fedora-33 root host - Ginkgo node hung
  • Node 1 had systemd and pod/net failures
  • Node 2 ran fine.
  • Node 3 hung during checkpoint+tcp test
• int remote fedora-34 root host - Similar systemd and pod/net failures to int podman fedora-33 root host Node 1 failures
• int remote fedora-33 root host - More flakes than F34 rootless, but again, similar failures in systemd and pod/net tests.

Please let me know how I can help.

[rhatdan]

Could you check to see if reflink is enabled?

xfs_info /var | grep ref
         =                       reflink=1    bigtime=0

[cevich]

~/dev/podman (kickstart_fedora|✔) $ hack/get_ci_vm.sh int podman fedora-34 root host

# Initializing get_ci_vm
...cut...
# Accessing instance cevich-fedora-c6202812959817728
+ Entering Cirrus-CI environment
+ Changing into /var/tmp/go/src/github.com/containers/podman
+ Dropping into a bash login shell inside Cirrus-CI 'int podman fedora-34 root host' task environment
[root@cevich-fedora-c6202812959817728 podman]# xfs_info /var | grep ref
xfs_info: cannot open /var: Is a directory
[root@cevich-fedora-c6202812959817728 podman]# xfs_info / | grep ref
         =                       reflink=1    bigtime=0
[root@cevich-fedora-c6202812959817728 podman]# xfs_info /tmp | grep ref
xfs_info: cannot open /tmp: Is a directory
[root@cevich-fedora-c6202812959817728 podman]#

[rhatdan]

If /tmp is on the / OS then reflinks are enabled and look like they are enabled by default, which is goodness.

[cevich]

If /tmp is on the / OS then reflinks are enabled and look like they are enabled by default, which is goodness.

Yep, so either our bits aren't taking advantage as expected, or the affect isn't very substantial, or both. Still, I'm a bit more concerned about the test failures I mentioned above. They all strike me as things which shouldn't fail due to underlying FS changing. Albeit, I also don't know the details, maybe it makes sense to someone?

[rhatdan]

Yes I have no idea what is causing the timeouts.

[cevich]

The network-create failures are even more strange: Error: container create failed (no logs from conmon): EOF

I wonder if maybe these are all race-conditions in the tests which are simply being exposed by timing changes. If so, the failures are pretty consistent. I got basically the same failures running these jobs multiple times and with BTRFS also 😕

[cevich]

rebased on master

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cevich
To complete the pull request process, please assign rhatdan after the PR has been reviewed.
You can assign the PR to them by writing /assign @rhatdan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@cevich: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[cevich]

Closing this for now, since there's a LOT of additional complexity in the image-build workflow for apparently minimal performance gain. Happy to revisit/re-open this at a later date if there is a need.