Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ext.config.root-reprovision.autosave-xfs and ext.config.root-reprovision.luks.autosave-xfs failing on aarch64 and ppc64le #1458

Closed
marmijo opened this issue Apr 7, 2023 · 1 comment · Fixed by coreos/fedora-coreos-config#2359
Closed

ext.config.root-reprovision.autosave-xfs and ext.config.root-reprovision.luks.autosave-xfs failing on aarch64 and ppc64le #1458

marmijo opened this issue Apr 7, 2023 · 1 comment · Fixed by coreos/fedora-coreos-config#2359

Comments

@marmijo
Copy link
Contributor

marmijo commented Apr 7, 2023

coreos/fedora-coreos-config#2320 added new kola-reprovision tests which are failing on multi-arch aarch64 and ppc64le multi-arch builds. These tests are consistently failing because the agcount is below the expected value of 128.

14:57:03  Apr 07 18:57:03 qemu0 kola-runext-test.sh[2618]: + . /var/opt/kola/extdata/commonlib.sh
14:57:03  Apr 07 18:57:03 qemu0 systemd[1]: Started kola-runext.service.
14:57:03  Apr 07 18:57:03 qemu0 kola-runext-test.sh[2618]: ++ cmdline=($(< /proc/cmdline))
14:57:03  Apr 07 18:57:03 qemu0 kola-runext-test.sh[2618]: + '[' -z '' ']'
14:57:03  Apr 07 18:57:03 qemu0 kola-runext-test.sh[2618]: + '[' '!' -f /run/ignition-ostree-autosaved-xfs.stamp ']'
14:57:03  Apr 07 18:57:03 qemu0 kola-runext-test.sh[2618]: + fatal 'expected autosaved XFS'
14:57:03  Apr 07 18:57:03 qemu0 kola-runext-test.sh[2618]: + echo 'expected autosaved XFS'
14:57:03  Apr 07 18:57:03 qemu0 kola-runext-test.sh[2618]: expected autosaved XFS
14:57:03  Apr 07 18:57:03 qemu0 kola-runext-test.sh[2618]: + exit 1
14:57:03  Apr 07 18:57:03 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
14:57:03  Apr 07 18:57:03 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
14:57:06  --- FAIL: ext.config.root-reprovision.luks.autosave-xfs (100.24s)
14:57:06          cluster.go:162: Error: Unit kola-runext.service exited with code 1
14:57:06          cluster.go:162: 2023-04-07T18:57:03Z cli: Unit kola-runext.service exited with code 1
14:57:06          harness.go:1115: kolet failed: : kolet run-test-unit failed: Process exited with status 1
14:58:02  systemctl status kola-runext.service:
14:58:02  �� kola-runext.service
14:58:02       Loaded: loaded (/etc/systemd/system/kola-runext.service; static)
14:58:02      Drop-In: /usr/lib/systemd/system/service.d
14:58:02               ������10-timeout-abort.conf
14:58:02       Active: failed (Result: exit-code) since Fri 2023-04-07 18:57:57 UTC; 191ms ago
14:58:02     Duration: 10ms
14:58:02      Process: 1966 ExecStart=/usr/local/bin/kola-runext-test.sh (code=exited, status=1/FAILURE)
14:58:02     Main PID: 1966 (code=exited, status=1/FAILURE)
14:58:02          CPU: 4ms
14:58:02  
14:58:02  Apr 07 18:57:57 qemu0 systemd[1]: Started kola-runext.service.
14:58:02  Apr 07 18:57:57 qemu0 kola-runext-test.sh[1966]: ++ cmdline=($(< /proc/cmdline))
14:58:02  Apr 07 18:57:57 qemu0 kola-runext-test.sh[1966]: + '[' '!' -f /run/ignition-ostree-autosaved-xfs.stamp ']'
14:58:02  Apr 07 18:57:57 qemu0 kola-runext-test.sh[1966]: + fatal 'expected autosaved XFS'
14:58:02  Apr 07 18:57:57 qemu0 kola-runext-test.sh[1966]: + echo 'expected autosaved XFS'
14:58:02  Apr 07 18:57:57 qemu0 kola-runext-test.sh[1966]: expected autosaved XFS
14:58:02  Apr 07 18:57:57 qemu0 kola-runext-test.sh[1966]: + exit 1
14:58:02  Apr 07 18:57:57 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
14:58:02  Apr 07 18:57:57 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
14:58:02  --- FAIL: ext.config.root-reprovision.autosave-xfs (53.54s)
14:58:02          cluster.go:162: Error: Unit kola-runext.service exited with code 1
14:58:02          cluster.go:162: 2023-04-07T18:57:57Z cli: Unit kola-runext.service exited with code 1
14:58:02          harness.go:1115: kolet failed: : kolet run-test-unit failed: Process exited with status 1

The journal.txt of the failed kola tests shows something like e.g. ignition-ostree-transposefs[843]: Filesystem agcount is 111; skipping. The autosave-xfs logic is then skipped and the /run/ignition-ostree-autosaved-xfs.stamp file is never created as it should be here

Here's an example journal.txt with the failure:
autosave-xfs.journal.txt
marmijo added a commit to marmijo/fedora-coreos-config that referenced this issue Apr 7, 2023
@marmijo
denylist: deny autosave-xfs and luks.autosave-xfs for aarch64 and ppc…
623a57f 
…64le

These tests are failing and blocking fcos pipeline multi-arch builds.
See: coreos/fedora-coreos-tracker#1458
@marmijo marmijo mentioned this issue Apr 7, 2023
Merged
marmijo added a commit to marmijo/fedora-coreos-config that referenced this issue Apr 7, 2023
@marmijo
denylist: deny autosave-xfs and luks.autosave-xfs for aarch64 and ppc…
7c7f8d0 
…64le

These tests are failing and blocking fcos pipeline multi-arch builds.
See: coreos/fedora-coreos-tracker#1458
marmijo added a commit to marmijo/fedora-coreos-config that referenced this issue Apr 7, 2023
@marmijo
denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and p…
90ecb30 
…pc64le

These tests are failing and blocking fcos pipeline multi-arch builds.
See: coreos/fedora-coreos-tracker#1458
marmijo added a commit to coreos/fedora-coreos-config that referenced this issue Apr 7, 2023
@marmijo
denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and p…
95a07de 
…pc64le

These tests are failing and blocking fcos pipeline multi-arch builds.
See: coreos/fedora-coreos-tracker#1458
coreosbot-releng pushed a commit to coreosbot-releng/os that referenced this issue Apr 8, 2023
@coreosbot
Bump fedora-coreos-config
2b85033 
Jonathan Lebon (7):
      40ignition-ostree: run ignition-ostree-growfs before sysroot mount
      40ignition-ostree: factor out zram-related functions
      40ignition-ostree: skip udev hack if Ignition did not reprovision rootfs
      40ignition-ostree: add autosave-xfs transposefs unit
      tests/kola: add non-exclusive check for growfs
      tests/kola: move LUKS checks to shared file
      tests/kola: add autosave-xfs tests

Michael Armijo (1):
      denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and ppc64le These tests are failing and blocking fcos pipeline multi-arch builds. See: coreos/fedora-coreos-tracker#1458
coreosbot-releng pushed a commit to coreosbot-releng/os that referenced this issue Apr 9, 2023
@coreosbot
Bump fedora-coreos-config
97f90de 
Jonathan Lebon (7):
      40ignition-ostree: run ignition-ostree-growfs before sysroot mount
      40ignition-ostree: factor out zram-related functions
      40ignition-ostree: skip udev hack if Ignition did not reprovision rootfs
      40ignition-ostree: add autosave-xfs transposefs unit
      tests/kola: add non-exclusive check for growfs
      tests/kola: move LUKS checks to shared file
      tests/kola: add autosave-xfs tests

Michael Armijo (1):
      denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and ppc64le These tests are failing and blocking fcos pipeline multi-arch builds. See: coreos/fedora-coreos-tracker#1458
@dustymabe dustymabe mentioned this issue Apr 10, 2023
Merged
@bgilbert
Copy link
Contributor

bgilbert commented Apr 10, 2023 •
edited

They're also consistently failing in Afterburn and coreos-installer upstream CI on x86_64 (example).

coreosbot-releng pushed a commit to coreosbot-releng/os that referenced this issue Apr 11, 2023
@coreosbot
Bump fedora-coreos-config
c132ef6 
Dusty Mabe (1):
      tests/kola: upgrade.extended: get info about booted deployment

Jonathan Lebon (7):
      40ignition-ostree: run ignition-ostree-growfs before sysroot mount
      40ignition-ostree: factor out zram-related functions
      40ignition-ostree: skip udev hack if Ignition did not reprovision rootfs
      40ignition-ostree: add autosave-xfs transposefs unit
      tests/kola: add non-exclusive check for growfs
      tests/kola: move LUKS checks to shared file
      tests/kola: add autosave-xfs tests

Michael Armijo (1):
      denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and ppc64le These tests are failing and blocking fcos pipeline multi-arch builds. See: coreos/fedora-coreos-tracker#1458

Renata Ravanelli (3):
      overlay.d: create new 30gcp-udev-rules overlay
      overlay.d: Add/Update udev rules for GCP
      Add 30gcp-udev-rules overlay to the manifest
jlebon added a commit to jlebon/fedora-coreos-config that referenced this issue Apr 11, 2023
@jlebon
tests/kola: bump minDisk in autosave-xfs tests
ef61335 
The actual agcount on a 64G disk depends on the agsize baked in the
image, which in turn depends on how large the rootfs was calculated to
be (see `estimate-commit-disk-size` in cosa).

Since the starting rootfs size will differ on different arches (notably,
be larger than x86_64) and minor variations are greatly amplified, the
agcount may be lower than expected for a 64G disk and in turn the test
can fail. Even on x86_64, the rootfs size can change e.g. in CI builds
where we ship non-stripped binaries.

Bump the requested disk size to 100G in the test to make it more
foolproof.

Closes: coreos/fedora-coreos-tracker#1458
jlebon added a commit to jlebon/fedora-coreos-config that referenced this issue Apr 11, 2023
@jlebon
tests/kola: bump minDisk in autosave-xfs tests
154f752 
The actual agcount on a 64G disk depends on the agsize baked in the
image, which in turn depends on how large the rootfs was calculated to
be (see `estimate-commit-disk-size` in cosa).

Since the starting rootfs size will differ on different arches (notably,
be larger than x86_64) and minor variations are greatly amplified, the
agcount may be lower than expected for a 64G disk and in turn the test
can fail. Even on x86_64, the rootfs size can change e.g. in CI builds
where we bake non-stripped binaries.

Bump the requested disk size to 100G in the test to make it more
foolproof.

Closes: coreos/fedora-coreos-tracker#1458
@jlebon jlebon mentioned this issue Apr 11, 2023
Merged
dustymabe pushed a commit to coreos/fedora-coreos-config that referenced this issue Apr 11, 2023
@jlebon @dustymabe
tests/kola: bump minDisk in autosave-xfs tests
07f87e3 
The actual agcount on a 64G disk depends on the agsize baked in the
image, which in turn depends on how large the rootfs was calculated to
be (see `estimate-commit-disk-size` in cosa).

Since the starting rootfs size will differ on different arches (notably,
be larger than x86_64) and minor variations are greatly amplified, the
agcount may be lower than expected for a 64G disk and in turn the test
can fail. Even on x86_64, the rootfs size can change e.g. in CI builds
where we bake non-stripped binaries.

Bump the requested disk size to 100G in the test to make it more
foolproof.

Closes: coreos/fedora-coreos-tracker#1458
coreosbot-releng pushed a commit to coreosbot-releng/os that referenced this issue Apr 11, 2023
@coreosbot
Bump fedora-coreos-config
5280817 
Dusty Mabe (1):
      tests/kola: upgrade.extended: get info about booted deployment

Jonathan Lebon (8):
      40ignition-ostree: run ignition-ostree-growfs before sysroot mount
      40ignition-ostree: factor out zram-related functions
      40ignition-ostree: skip udev hack if Ignition did not reprovision rootfs
      40ignition-ostree: add autosave-xfs transposefs unit
      tests/kola: add non-exclusive check for growfs
      tests/kola: move LUKS checks to shared file
      tests/kola: add autosave-xfs tests
      tests/kola: bump `minDisk` in autosave-xfs tests

Michael Armijo (1):
      denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and ppc64le These tests are failing and blocking fcos pipeline multi-arch builds. See: coreos/fedora-coreos-tracker#1458

Renata Ravanelli (3):
      overlay.d: create new 30gcp-udev-rules overlay
      overlay.d: Add/Update udev rules for GCP
      Add 30gcp-udev-rules overlay to the manifest
coreosbot-releng pushed a commit to coreosbot-releng/os that referenced this issue Apr 12, 2023
@coreosbot
Bump fedora-coreos-config
cf0f8b0 
Dusty Mabe (1):
      tests/kola: upgrade.extended: get info about booted deployment

Jonathan Lebon (9):
      40ignition-ostree: run ignition-ostree-growfs before sysroot mount
      40ignition-ostree: factor out zram-related functions
      40ignition-ostree: skip udev hack if Ignition did not reprovision rootfs
      40ignition-ostree: add autosave-xfs transposefs unit
      tests/kola: add non-exclusive check for growfs
      tests/kola: move LUKS checks to shared file
      tests/kola: add autosave-xfs tests
      tests/kola: bump `minDisk` in autosave-xfs tests
      40ignition-ostree: give filesystem type when mounting zram-based XFS

Michael Armijo (1):
      denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and ppc64le These tests are failing and blocking fcos pipeline multi-arch builds. See: coreos/fedora-coreos-tracker#1458

Renata Ravanelli (3):
      overlay.d: create new 30gcp-udev-rules overlay
      overlay.d: Add/Update udev rules for GCP
      Add 30gcp-udev-rules overlay to the manifest
coreosbot-releng pushed a commit to coreosbot-releng/os that referenced this issue Apr 13, 2023
@coreosbot
Bump fedora-coreos-config
f331adf 
Dusty Mabe (2):
      tests/kola: upgrade.extended: get info about booted deployment
      manifests/fedora-coreos-base: surgically remove qcom dtb files on aarch64

Jonathan Lebon (9):
      40ignition-ostree: run ignition-ostree-growfs before sysroot mount
      40ignition-ostree: factor out zram-related functions
      40ignition-ostree: skip udev hack if Ignition did not reprovision rootfs
      40ignition-ostree: add autosave-xfs transposefs unit
      tests/kola: add non-exclusive check for growfs
      tests/kola: move LUKS checks to shared file
      tests/kola: add autosave-xfs tests
      tests/kola: bump `minDisk` in autosave-xfs tests
      40ignition-ostree: give filesystem type when mounting zram-based XFS

Michael Armijo (2):
      denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and ppc64le These tests are failing and blocking fcos pipeline multi-arch builds. See: coreos/fedora-coreos-tracker#1458
      denylist: bump snooze for ext.config.kdump.crash on aarch64 This is still causing issues. See: coreos/fedora-coreos-tracker#1430

Renata Ravanelli (4):
      overlay.d: create new 30gcp-udev-rules overlay
      overlay.d: Add/Update udev rules for GCP
      Add 30gcp-udev-rules overlay to the manifest
      overlay.d: Add 30gcp-udev-rules dracut module
coreosbot-releng pushed a commit to coreosbot-releng/os that referenced this issue Apr 14, 2023
@coreosbot
Bump fedora-coreos-config
58169c9 
Dusty Mabe (2):
      tests/kola: upgrade.extended: get info about booted deployment
      manifests/fedora-coreos-base: surgically remove qcom dtb files on aarch64

Jonathan Lebon (9):
      40ignition-ostree: run ignition-ostree-growfs before sysroot mount
      40ignition-ostree: factor out zram-related functions
      40ignition-ostree: skip udev hack if Ignition did not reprovision rootfs
      40ignition-ostree: add autosave-xfs transposefs unit
      tests/kola: add non-exclusive check for growfs
      tests/kola: move LUKS checks to shared file
      tests/kola: add autosave-xfs tests
      tests/kola: bump `minDisk` in autosave-xfs tests
      40ignition-ostree: give filesystem type when mounting zram-based XFS

Michael Armijo (2):
      denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and ppc64le These tests are failing and blocking fcos pipeline multi-arch builds. See: coreos/fedora-coreos-tracker#1458
      denylist: bump snooze for ext.config.kdump.crash on aarch64 This is still causing issues. See: coreos/fedora-coreos-tracker#1430

Renata Ravanelli (4):
      overlay.d: create new 30gcp-udev-rules overlay
      overlay.d: Add/Update udev rules for GCP
      Add 30gcp-udev-rules overlay to the manifest
      overlay.d: Add 30gcp-udev-rules dracut module
c4rt0 pushed a commit to c4rt0/fedora-coreos-config that referenced this issue May 17, 2023
@marmijo @c4rt0
denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and p…
734e197 
…pc64le

These tests are failing and blocking fcos pipeline multi-arch builds.
See: coreos/fedora-coreos-tracker#1458
c4rt0 pushed a commit to c4rt0/fedora-coreos-config that referenced this issue May 17, 2023
@jlebon @c4rt0
tests/kola: bump minDisk in autosave-xfs tests
c9945ba 
The actual agcount on a 64G disk depends on the agsize baked in the
image, which in turn depends on how large the rootfs was calculated to
be (see `estimate-commit-disk-size` in cosa).

Since the starting rootfs size will differ on different arches (notably,
be larger than x86_64) and minor variations are greatly amplified, the
agcount may be lower than expected for a 64G disk and in turn the test
can fail. Even on x86_64, the rootfs size can change e.g. in CI builds
where we bake non-stripped binaries.

Bump the requested disk size to 100G in the test to make it more
foolproof.

Closes: coreos/fedora-coreos-tracker#1458
HuijingHei pushed a commit to HuijingHei/fedora-coreos-config that referenced this issue Oct 10, 2023
@marmijo @HuijingHei
denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and p…
4f6124b 
…pc64le

These tests are failing and blocking fcos pipeline multi-arch builds.
See: coreos/fedora-coreos-tracker#1458
HuijingHei pushed a commit to HuijingHei/fedora-coreos-config that referenced this issue Oct 10, 2023
@jlebon @HuijingHei
tests/kola: bump minDisk in autosave-xfs tests
97c9ee1 
The actual agcount on a 64G disk depends on the agsize baked in the
image, which in turn depends on how large the rootfs was calculated to
be (see `estimate-commit-disk-size` in cosa).

Since the starting rootfs size will differ on different arches (notably,
be larger than x86_64) and minor variations are greatly amplified, the
agcount may be lower than expected for a 64G disk and in turn the test
can fail. Even on x86_64, the rootfs size can change e.g. in CI builds
where we bake non-stripped binaries.

Bump the requested disk size to 100G in the test to make it more
foolproof.

Closes: coreos/fedora-coreos-tracker#1458
HuijingHei pushed a commit to HuijingHei/fedora-coreos-config that referenced this issue Oct 10, 2023
@marmijo @HuijingHei
denylist: snooze autosave-xfs and luks.autosave-xfs for aarch64 and p…
2af9078 
…pc64le

These tests are failing and blocking fcos pipeline multi-arch builds.
See: coreos/fedora-coreos-tracker#1458
HuijingHei pushed a commit to HuijingHei/fedora-coreos-config that referenced this issue Oct 10, 2023
@jlebon @HuijingHei
tests/kola: bump minDisk in autosave-xfs tests
9ba0d83 
The actual agcount on a 64G disk depends on the agsize baked in the
image, which in turn depends on how large the rootfs was calculated to
be (see `estimate-commit-disk-size` in cosa).

Since the starting rootfs size will differ on different arches (notably,
be larger than x86_64) and minor variations are greatly amplified, the
agcount may be lower than expected for a 64G disk and in turn the test
can fail. Even on x86_64, the rootfs size can change e.g. in CI builds
where we bake non-stripped binaries.

Bump the requested disk size to 100G in the test to make it more
foolproof.

Closes: coreos/fedora-coreos-tracker#1458
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

tests/kola: bump minDisk in autosave-xfs tests jlebon/fedora-coreos-config
2 participants
@bgilbert @marmijo