Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

/usr/share/rpm in quay.io/coreos-assembler/fcos:testing-devel is empty #1258

Closed
cgwalters opened this issue Jul 15, 2022 · 12 comments
Closed

/usr/share/rpm in quay.io/coreos-assembler/fcos:testing-devel is empty #1258

cgwalters opened this issue Jul 15, 2022 · 12 comments

Comments

@cgwalters
Copy link
Member

ci/test-container.sh suddenly fell over with:

+ rpm-ostree ex rebuild
...
error: The transaction was empty

This needs investigation; either we merged a change without waiting for CI, or somehow things changed underneath us (shouldn't be possible?)
@lucab lucab mentioned this issue Jul 15, 2022
Merged
@cgwalters
Copy link
Member Author

cgwalters commented Jul 15, 2022 •
edited

Hmm, very interesting. So simply doing

$ podman run --rm -ti quay.io/coreos-assembler/fcos:testing-devel rpm-ostree ex rebuild
...
error: The transaction was empty

gives an error, but

$ podman run --rm -ti quay.io/coreos-assembler/fcos:stable rpm-ostree ex rebuild
...
Empty transaction
No changes to apply.

succeeds.

Ugh but wait...

$ for tag in stable testing-devel; do    echo -n $tag:; podman run --rm -ti quay.io/coreos-assembler/fcos:$tag rpm -q rpm-ostree; done
stable:rpm-ostree-2022.10-2.fc36.x86_64
testing-devel:package rpm-ostree is not installed
$

@lucab
Copy link
Contributor

lucab commented Jul 15, 2022

It looks like CI turned red when we merged coreos/rpm-ostree#3851, however the GH job on the PR itself was green.

@cgwalters
Copy link
Member Author

This is unrelated to rpm-ostree git main, it's a behavior difference triggered by us somehow missing the rpmdb in the latest testing-devel container, see above ⬆️

@cgwalters cgwalters transferred this issue from coreos/rpm-ostree Jul 15, 2022
@cgwalters cgwalters changed the title ci/test-container fell over /usr/share/rpm in quay.io/coreos-assembler/fcos:testing-devel is empty Jul 15, 2022
@cgwalters
Copy link
Member Author

Transferred this to the FCOS tracker; may be a bug in rpm-ostree still, but regardless we need to track getting the FCOS build fixed.

@cgwalters cgwalters mentioned this issue Jul 15, 2022
Closed
@cgwalters
Copy link
Member Author

OK this also affects current rhcos builds:

$ podman run --rm -ti registry.ci.openshift.org/rhcos-devel/rhel-coreos:latest ls -l /usr/share/rpm
total 0

so it must be related to an rpm-ostree build side change.

@cgwalters
Copy link
Member Author

Oh man I think this is a regression from ostreedev/ostree-rs-ext#331

The content is there in the ostree commit, but not in the tar stream. Argh.

@cgwalters
Copy link
Member Author

I think this may have something to do with hardlinked content, I do see the files in /usr/lib/sysimage/rpm-ostree-base-db/

@cgwalters
Copy link
Member Author

cgwalters commented Jul 15, 2022 •
edited

Ahhh it's even more subtle than that. The bug is when we have identical directory tree objects - which I doubt happens with anything except our hardlinked /usr/share/rpm and /usr/lib/sysimage/rpm-ostree-base-db directories.

cgwalters added a commit to cgwalters/ostree-rs-ext that referenced this issue Jul 15, 2022
@cgwalters
tests: Add two subdirectories with identical content
c156d70 
Prep for fixing coreos/fedora-coreos-tracker#1258
@dustymabe
Copy link
Member

Is this a blocker for next week's testing and next releases?

@dustymabe dustymabe mentioned this issue Jul 15, 2022
Closed
34 tasks
@jlebon
Copy link
Member

jlebon commented Jul 18, 2022

Is this a blocker for next week's testing and next releases?

It'd be nice to fix, but I wouldn't block releases on it. These containers are still experimental and we need to discuss the path to making them release blocking (which would include tests running in the pipeline).

cgwalters added a commit to cgwalters/ostree-rs-ext that referenced this issue Jul 18, 2022
@cgwalters
tests: Add two subdirectories with identical content
3463c52 
Prep for fixing coreos/fedora-coreos-tracker#1258
This was referenced Jul 18, 2022
Merged
Closed
@cgwalters
Copy link
Member Author

Filed ostreedev/ostree-rs-ext#339 to track the fix to ostree-ext; this issue will track shipping the fix for FCOS.

cgwalters added a commit to cgwalters/ostree-rs-ext that referenced this issue Jul 18, 2022
@cgwalters
tests: Add two subdirectories with identical content
862aa98 
Prep for fixing coreos/fedora-coreos-tracker#1258
cgwalters added a commit to cgwalters/ostree-rs-ext that referenced this issue Jul 18, 2022
@cgwalters
tests: Add two subdirectories with identical content
0c51650 
Prep for fixing coreos/fedora-coreos-tracker#1258
cgwalters added a commit to cgwalters/ostree-rs-ext that referenced this issue Jul 18, 2022
@cgwalters
chunking: Recurse on duplicate directory trees
ee74845 
See coreos/fedora-coreos-tracker#1258

Regression from ostreedev#331

Currently rpm-ostree emits two identical subdirectories in
`/usr/lib/sysimage/rpm-ostree-base-db` and `/usr/share/rpm`, and the
chunking export skips emitting this incorrectly.

Closes: ostreedev#339
This was referenced Jul 18, 2022
Merged
Merged
Closed
cgwalters added a commit to cgwalters/fedora-coreos-config that referenced this issue Jul 20, 2022
@cgwalters
tests: New ostree-containers/rpmdb test
069ad36 
This builds on coreos/coreos-assembler#2999
to test that `rpm -q` works inside the container, which would
have caught coreos/fedora-coreos-tracker#1258
@cgwalters cgwalters mentioned this issue Jul 20, 2022
Closed
@cgwalters
Copy link
Member Author

This is fixed

$ podman run --rm -ti quay.io/coreos-assembler/fcos:testing-devel rpm -q systemd
systemd-250.8-1.fc36.x86_64

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lucab @cgwalters @dustymabe @jlebon