New buildah/timeout/external container flakes

[edsantiago]

New weird flakes seen in buildah treadmill. I'm filing all under one issue, but they might be unrelated:

not ok 119 podman ps --external
...
# podman build -t thiswillneverexist -
STEP 1/3: FROM quay.io/libpod/testimage:20221018
STEP 2/3: RUN touch /intermediate.image.to.be.pruned
--> 0c116d0cdcf9
STEP 3/3: RUN sleep 30
timeout: sending signal TERM to command '/var/tmp/go/src/github.com/containers/podman/bin/podman'
[ rc=124 (expected) ]
#/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
#|     FAIL: podman build did not get killed within 10 seconds
#| expected: -le 10
#|   actual:     32

(ISTR something changing in buildah signal handling recently)

And:

not ok 248 podman diff with buildah container 
...
$ podman images --all --format {{.Repository}}:{{.Tag}} {{.ID}}
quay.io/libpod/testimage:20221018 f5a99120db64
buildahctr
$ podman diff --format json buildahctr
Error: buildahctr not found: layer not known

And:

not ok 137 podman rm container from storage
...
$ podman container exists --external testimage-working-container
[ rc=1 (** EXPECTED 0 **) ]

All of these are buildah-related. I haven't seen them in any other PRs, but this has been the lightest PR week in memory. If we don't see these in regular podman CI, it might be something in the new buildah vendoring.

• debian-12 : sys podman debian-12 root host boltdb
  • 05-19 09:06 in [sys] podman ps --external
• debian-12 : sys podman debian-12 rootless host boltdb
  • 05-19 09:05 in [sys] podman rm container from storage
  • 05-19 09:05 in [sys] podman diff with buildah container
• rawhide : sys podman rawhide rootless host sqlite
  • 05-19 09:09 in [sys] podman ps --external

[edsantiago]

Flakes being seen in the wild, outside of the treadmill. Next guess: something to do with the buildah version in new VM images?

[edsantiago]

Everything points to something broken in containers/storage.

[edsantiago]

All the 13808 failures below are the buildah treadmill PR.

All the 17831 failures below are from my hammer-CI PR, during a period this weekend when I had cherry-picked #18634, a test PR that included a new containers/storage. That is why I believe the problem is in containers/storage. Today I have reverted that PR from my hammer-CI PR, and am no longer seeing those buildah failures.

• debian-12 : sys podman debian-12 root host boltdb
  • PR DO NOT MERGE: buildah vendor treadmill #13808
    • 05-22 14:16 in [sys] podman ps --external
    • 05-19 19:46 in [sys] podman ps --external
• debian-12 : sys podman debian-12 rootless host boltdb
  • PR DO NOT MERGE: buildah vendor treadmill #13808
    • 05-22 14:16 in [sys] podman rm container from storage
    • 05-22 14:16 in [sys] podman diff with buildah container
    • 05-19 19:45 in [sys] podman rm container from storage
    • 05-19 19:45 in [sys] podman diff with buildah container
• debian-12 : sys podman debian-12 rootless host sqlite
  • PR Ed's pet PR with no flake retries #17831
    • 05-22 11:58 in [sys] podman rm container from storage
    • 05-22 11:58 in [sys] podman diff with buildah container
    • 05-22 10:56 in [sys] podman rm container from storage
    • 05-22 10:56 in [sys] podman diff with buildah container
    • 05-22 09:39 in [sys] podman rm container from storage
    • 05-22 09:39 in [sys] podman diff with buildah container
    • 05-22 08:25 in [sys] podman rm container from storage
    • 05-22 08:25 in [sys] podman diff with buildah container
    • 05-21 22:10 in [sys] podman rm container from storage
    • 05-21 22:10 in [sys] podman diff with buildah container
    • 05-21 20:45 in [sys] podman rm container from storage
    • 05-21 20:45 in [sys] podman diff with buildah container
    • 05-21 08:34 in [sys] podman rm container from storage
    • 05-21 08:34 in [sys] podman diff with buildah container
    • 05-20 06:31 in [sys] podman rm container from storage
    • 05-20 06:31 in [sys] podman diff with buildah container
    • 05-19 23:14 in [sys] podman rm container from storage
    • 05-19 23:14 in [sys] podman diff with buildah container
    • 05-19 22:06 in [sys] podman rm container from storage
    • 05-19 22:06 in [sys] podman diff with buildah container
    • 05-19 21:03 in [sys] podman rm container from storage
    • 05-19 21:03 in [sys] podman diff with buildah container
    • 05-19 19:56 in [sys] podman rm container from storage
    • 05-19 19:56 in [sys] podman diff with buildah container
    • 05-19 18:41 in [sys] podman rm container from storage
    • 05-19 18:41 in [sys] podman diff with buildah container
    • 05-19 17:42 in [sys] podman rm container from storage
    • 05-19 17:42 in [sys] podman diff with buildah container
    • 05-19 16:10 in [sys] podman rm container from storage
    • 05-19 16:10 in [sys] podman diff with buildah container
• fedora-38 : sys podman fedora-38 root host boltdb
  • PR DO NOT MERGE: buildah vendor treadmill #13808
    • 05-22 14:23 in [sys] podman ps --external

[mtrmac]

All the 17831 failures below are from my hammer-CI PR, during a period this weekend when I had cherry-picked #18634, a test PR that included a new containers/storage.

Assuming that was github.com/mtrmac/storage v0.0.0-20230519165505-36c85d0b2b46 , that means upstream c/storage between v1.46.1 and ceaf6ce2c54a0a2b184b831a732c722758979e81 .

a short-list of relevant PRs (skipping dependency updates, non-overlay changes, and entirely new unused features):

• store: ensure lockfile is updated before writes storage#1603
• drivers/overlay.get(): only use the relative workdir for mountOverlayFrom() storage#1601
• store: lock additional store before validating image storage#1589
• pkg/rm: attempt os.RemoveAll() first storage#1581
• Use Go 1.18 generics in the store’s locking helpers storage#1577
• store,imagestore: treat graphRoot as rwImageStore to support Delete storage#1578
• store: add support to split filesystem using imagestore storage#1549
• types: try harder to use overlay with rootless storage#1571
• Don't use bytes.NewBuffer to read data storage#1572
• Extend driver.ListLayers() storage#1563
• Split up DeleteContainer storage#1568
• drivers: chown root mount to root in the userns storage#1564

To me, it seems containers/storage#1549 is the highest-risk change in this area, but there are certainly other candidates.

[edsantiago]

Uh-oh. this might be the same bug, *without( the new containers/storage. It's really hard to tell, because nobody is submitting PRs the last few weeks.

# podman [options] rmi -f rmtest:1
Error: committing container for step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[touch rmtest:8] Flags:[] Attrs:map[] Message:RUN touch rmtest:8 Original:RUN touch rmtest:8}: copying layers and metadata for container "802e6990add8b793b1cb8bbef78ae6cff2d027aa57397fa913d3fdca49170be1": committing the finished image: creating image "cde9f59aba1cf8c2625eceef9009afd9225fa12b2cfacf7d8bc6a850d41eafad": layer not known

[mtrmac]

That (podman image rm - concurrent with shared layers) is, I think at a first glance, entirely unrelated.

It seems to me that the test is buggy — it claims to test concurrent removal, but it tests concurrent (build + removal); and if an image is removed while another is being built from the same layers, I can imagine that failing (a build doesn’t hold a race-free “reference” to a layer that would protect it against concurrent removals).

Actually, well, the lack of a race-free “reference” is arguably a bug, but it’s not the bug reportedly being tested by the test; #9266 only intended to make removes not fail against concurrent removals.

[edsantiago]

Thank you @mtrmac. I just panicked without looking closely.

[mtrmac]

@edsantiago I do think that build&rm one is worth tracking and fixing.

[mtrmac]

Filed #18659 for my concurrent rm theory.

[nalind]

Aside from containers/buildah#4813, I think containers/storage#1571 is causing podman (with a containers/storage that incorporates it) and the system default version of buildah (built with a containers/storage that didn't incorporate it) to default to different storage drivers, so they see different sets of layers and images.

There are multiple options for getting them to agree. Set STORAGE_DRIVER in the environment to $($PODMAN $_PODMAN_TEST_OPTS info -f '{{.Store.GraphDriverName}}'). Create ~/.config/containers/storage.conf. Incorporate the default that podman selects into the configuration file for CI.

[flouthoc]

To me, it seems containers/storage#1549 is the highest-risk change in this area, but there are certainly other candidates.

@mtrmac I was testing it with podman with #18224 and I never saw any race or issues with CI until I rebased recently and saw some debian failures ( debian failures looked unrelated to the PR ). ( But let me take a look again and see if I missed anything )

[flouthoc]

Regarding following tests

[sys] podman rm container from storage
[sys] podman diff with buildah container

I think buildah version in VM might be the very much issue.

There are multiple options for getting them to agree. Set STORAGE_DRIVER in the environment to $($PODMAN $_PODMAN_TEST_OPTS info -f '{{.Store.GraphDriverName}}'). Create ~/.config/containers/storage.conf. Incorporate the default that podman selects into the configuration file for CI.

podman diff with buildah container fails for me when storage driver is not set in storage.conf but error is different from what i see in CI but both tests pass for me with latest buildah binary and when storage is configured correctly.

[edsantiago]

I'm having trouble following the thread, and I don't have a way to play with debian. But I just think it's important to point out that system tests do not (and must not) muck around with setting STORAGE_DRIVER or adding --this --that options. They are intended to test our tools as configured on a real system. If there is an incompatibility between podman and buildah, on a system that could conceivable look like a real-world one, my inclination would be to fix our tools so they handle this situation better.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago update on this issue?

[flouthoc]

I think this is similar what we saw in treadmill and was recently fixed by doing a workaround for debian. But I think @edsantiago can confirm this.

[edsantiago]

I do not understand the status of this bug. As best I can tell, #18822 changed the CI setup something something overlay/vfs/debian. Since then I have only run the buildah treadmill 2-3 times, because there have been very few buildah PRs. I have not seen these errors.

To me that means the bug has been swept under the carpet, not fixed. Opinions differ on this: some may believe the issue is fixed; some may believe that it's a difficult situation with no real good fix. I will step out of this discussion for now. If someone truly believes this is fixed, feel free to close this issue. If at all possible, please include a comment defending your actions.

[mtrmac]

My very imprecise impression, just skimming the conversations and not checking the details myself, is that Nalin diagnosed this in #18631 (comment) , and that @flouthoc fixed the c/storage behavior in containers/storage#1618 / containers/storage#1637 .

Those c/storage changes are included in Podman since e5399aa .

@flouthoc PTAL and correct me if I’m wrong.

[flouthoc]

@mtrmac Yes, regarding #18822 we had a discussion in a watercooler that it is hard to ensure this compat for older version between (new) podman <-> (old) buildah hence we decided to create a workaround for debian #18822 . For newer version this should be fixed via containers/storage#1618 / containers/storage#1637 .

[edsantiago]

#13808 is passing again. Closing.