Composes sometimes hitting error: fstatat(<checksum>.filez): Cannot allocate memory or error: openat(<checksum>.filez): Invalid argument
#594
Assignees
Labels
Comments
cgwalters
changed the title
error: fstatat(92/4682a3540a7d5f834f5159c024f412ff528600f8a683116ce99a1832251b67.filez): Cannot allocate memoryerror: fstatat(<checksum>.filez): Cannot allocate memory
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this issue
Jul 26, 2021
We're getting a weird OOM issue in RHCOS builds: openshift/os#594 which looks like: `fstatat(92/4682a3540a7d5f834f5159c024f412ff528600f8a683116ce99a1832251b67.filez): Cannot allocate memory` Reading some code here I noticed that the "open directory" call could follow symlinks when it shouldn't. Also explicitly only check for the size of regular files. I'm pretty sure this is a no-op, but let's harden the code.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this issue
Jul 26, 2021
We're getting a weird OOM issue in RHCOS builds: openshift/os#594 which looks like: `fstatat(92/4682a3540a7d5f834f5159c024f412ff528600f8a683116ce99a1832251b67.filez): Cannot allocate memory` Reading some code here I noticed that the "open directory" call could follow symlinks when it shouldn't. Let's just RIIR and make this code more explicit - e.g. we ignore symlinks. I'm pretty sure this is a no-op, but let's harden the code.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this issue
Jul 26, 2021
We're getting a weird OOM issue in RHCOS builds: openshift/os#594 which looks like: `fstatat(92/4682a3540a7d5f834f5159c024f412ff528600f8a683116ce99a1832251b67.filez): Cannot allocate memory` Reading some code here I noticed that the "open directory" call could follow symlinks when it shouldn't. Let's just RIIR and make this code more explicit - e.g. we ignore symlinks. I'm pretty sure this is a no-op, but let's harden the code.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this issue
Jul 26, 2021
We're getting a weird OOM issue in RHCOS builds: openshift/os#594 which looks like: `fstatat(92/4682a3540a7d5f834f5159c024f412ff528600f8a683116ce99a1832251b67.filez): Cannot allocate memory` Reading some code here I noticed that the "open directory" call could follow symlinks when it shouldn't. Let's just RIIR and make this code more explicit - e.g. we ignore symlinks. I'm pretty sure this is a no-op, but let's harden the code.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this issue
Jul 26, 2021
This ends up doing some nontrivial work in e.g. `db diff` and I want to see if an error trace is happening from there. xref openshift/os#594
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this issue
Jul 26, 2021
Also add a missing `:` in the editor code. xref openshift/os#594
This was referenced Jul 26, 2021
|
Edit: not related, see #594 (comment) |
|
OK to debug this I pushed openshift/release#20634 |
@miabbott I think you have an old cosa missing coreos/coreos-assembler#2293 ? |
Probably...doing |
Yup, got further this time. Sorry for the noise! |
|
Slight progress: That helps. It's probably the Will add more debugging. |
|
when doing a build on Power with 4.9 cosa and latest redhat-coreos , i see: is it related ? |
|
Yes, definitely. Can you reproduce this reliably? Can you reproduce outside of supermin? |
|
i can reproduce reliably but as of now only through running cosa build, so inside supermin. i will try to. |
|
Can you try e.g. wedging |
|
after running i'll try to attach gdb |
|
Ooh. I bet this is a new bug in the Fedora kernel - something like a memory leak in 9p. That would match the symptoms. Does downgrading the kernel fix it? |
|
I am so sure this was a kernel memory leak, but downgrading the kernel didn't fix it. That's odd. We bumped the supermin VM memory (on power from 4096 to 8192) and that did work. So...looking over ostree changes, I wonder if somehow the repo locking changes are triggering a 9p bug. If downgrading ostree to v2021.2 fixes it that would be a strong indication. |
|
Another thing I find odd here is it certainly looks like |
|
so as Colin suspected this only happens for RHCOS and not FCOS. FCOS builds fine and this might just be because RHCOS has more content than FCOS. In which case maybe increasing memory for rhcos is the only option? |
jlebon
pushed a commit
to coreos/coreos-assembler
that referenced
this issue
Jun 2, 2022
This is a forward cherry-pick from #2699. The RHCOS 4.10 Power pipeline is hitting the second variant in openshift/os#594 (comment). Let's try to bump the memory to see if it helps.
|
We saw this at least twice in some rpm-ostree upstream CI runs, so it looks like this is also affecting x86_64 FCOS now. Edit for more info: the failure in this case is variant 1 of #594 (comment): |
|
Yeah I think there's some sort of further regression in 9p going on here. |
jlebon
added a commit
to jlebon/coreos-assembler
that referenced
this issue
Jun 23, 2022
Something has changed recently which causes us to hit the ENOMEM issue more easily now: openshift/os#594 (comment) Mid-term, we could rework the compose so that only the OCI archive is pulled through 9p rather than a full `pull-local`. Long-term, the fix is to stop using 9p. But for now to unblock CI, let's just bump the VM memory to 3G which should help.
|
We have reports of this happening in more places now. I can reproduce it locally with diff --git a/src/cmdlib.sh b/src/cmdlib.sh
index 2fbd40eaa..f9cd2911c 100755
--- a/src/cmdlib.sh
+++ b/src/cmdlib.sh
@@ -651,7 +651,7 @@ EOF
# There seems to be some false positives in shellcheck
# https://github.com/koalaman/shellcheck/issues/2217
- memory_default=2048
+ memory_default=1024
# shellcheck disable=2031
case $arch in
# Power 8 page faults with 2G of memory in rpm-ostreeI think short-term let's just bump the memory again: coreos/coreos-assembler#2940. |
|
Remember BTW to anyone hitting this, you can also bump the memory using the |
jlebon
added a commit
to jlebon/coreos-assembler
that referenced
this issue
Jun 23, 2022
Something has changed recently which causes us to hit the ENOMEM issue more easily now: openshift/os#594 (comment) Mid-term, we could rework the compose so that only the OCI archive is pulled through 9p rather than a full `pull-local`. Long-term, the fix is to stop using 9p. But for now to unblock CI, let's just bump the VM memory to 4G which should help.
jlebon
added a commit
to jlebon/coreos-assembler
that referenced
this issue
Jun 24, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball.
jlebon
added a commit
to jlebon/coreos-assembler
that referenced
this issue
Jun 24, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball.
|
I believe coreos/coreos-assembler#2946 will fix this in the rpm-ostree compose path. |
jlebon
added a commit
to jlebon/coreos-assembler
that referenced
this issue
Jun 24, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball.
cgwalters
pushed a commit
to coreos/coreos-assembler
that referenced
this issue
Jun 26, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball.
travier
pushed a commit
to travier/coreos-assembler
that referenced
this issue
Jul 7, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball.
cgwalters
pushed a commit
to coreos/coreos-assembler
that referenced
this issue
Jul 7, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball.
dustymabe
pushed a commit
to dustymabe/coreos-assembler
that referenced
this issue
Nov 5, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball. (cherry picked from commit 269aa38)
dustymabe
pushed a commit
to dustymabe/coreos-assembler
that referenced
this issue
Nov 7, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball. (cherry picked from commit 269aa38)
dustymabe
pushed a commit
to dustymabe/coreos-assembler
that referenced
this issue
Nov 8, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball. (cherry picked from commit 269aa38)
dustymabe
pushed a commit
to coreos/coreos-assembler
that referenced
this issue
Nov 8, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball. (cherry picked from commit 269aa38)
dustymabe
pushed a commit
to dustymabe/coreos-assembler
that referenced
this issue
Nov 10, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball. (cherry picked from commit 269aa38)
dustymabe
pushed a commit
to coreos/coreos-assembler
that referenced
this issue
Nov 15, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball. (cherry picked from commit 269aa38)
jlebon
added a commit
to coreos/coreos-assembler
that referenced
this issue
Dec 2, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball. (cherry picked from commit 269aa38)
jlebon
added a commit
to coreos/coreos-assembler
that referenced
this issue
Dec 2, 2022
Instead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball. (cherry picked from commit 269aa38)
jlebon
pushed a commit
to jlebon/coreos-assembler
that referenced
this issue
Dec 6, 2022
This should fix openshift/os#594 Basically I think 9p has a bug where under memory pressure, having it *free* an inode requires allocation, which can fail. This works around that bug by pulling the ostree-container archive instead, which is a single big file. Note that this code path does *not* change the semantics at all for the generated disk. The information about the pulled container is discarded and lost. Actually making use of the container bits natively is the `deploy-via-container` image option which is still experimental, but will be used when we progress the ostree native containers work. (cherry picked from commit 0bba897)
jlebon
pushed a commit
to jlebon/coreos-assembler
that referenced
this issue
Dec 6, 2022
This should fix openshift/os#594 Basically I think 9p has a bug where under memory pressure, having it *free* an inode requires allocation, which can fail. This works around that bug by pulling the ostree-container archive instead, which is a single big file. Note that this code path does *not* change the semantics at all for the generated disk. The information about the pulled container is discarded and lost. Actually making use of the container bits natively is the `deploy-via-container` image option which is still experimental, but will be used when we progress the ostree native containers work. (cherry picked from commit 0bba897) jlebon: Minor conflict resolution. When that commit was made, cosa was on f35, which matches the Fedora version this branch is on.
jlebon
pushed a commit
to coreos/coreos-assembler
that referenced
this issue
Dec 6, 2022
This should fix openshift/os#594 Basically I think 9p has a bug where under memory pressure, having it *free* an inode requires allocation, which can fail. This works around that bug by pulling the ostree-container archive instead, which is a single big file. Note that this code path does *not* change the semantics at all for the generated disk. The information about the pulled container is discarded and lost. Actually making use of the container bits natively is the `deploy-via-container` image option which is still experimental, but will be used when we progress the ostree native containers work. (cherry picked from commit 0bba897) jlebon: Minor conflict resolution. When that commit was made, cosa was on f35, which matches the Fedora version this branch is on.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
CI is failing right now on this very reliably.
Some sort of regression? Somehow related to the size of the content? I think we're missing some error prefixing in rpm-ostree here too.
The text was updated successfully, but these errors were encountered: