util.c Unable to mount tmpfs failed restore on AmazonLinux2 ami #2384
Comments
@bsmithai Does enabling cc @Snorch |
|
Yes enabling MtnsCompatMode seemed to have been a temporary fix to the issue. |
|
In combination with Kubernetes I also had problems with the v2 mount code (#2023). In the end it was not a error in the v2 code, but just something not correctly set up. I would also be curious if a newer kernel might have additional fixes which might be missing in your 5.15. |
|
@adrianreber is infrastructure container in that issue the same thing as a pause container? |
Yes. |
The code might be a bit hard to read, but you did a great job at narrowing down the problematic stack. First let me explain how pre_create_mount_namespaces works:
note: if we change
So that looks really strange that you hit problem on make_yard in the loop over ns_ids. I would recommend adding explicit debug messages everywhere to re-confirm: Snorch@7c6d053 |
|
@Snorch This makes a lot of sense! Thank you for the break down. And yes, I can add some more explicit debug messages now that I understand the idea. I'll get back to you with those statements sometime this week! |
|
@Snorch Oh I guess a few questions I had are,
Sorry if you have to reiterate some of what you've already said |
you can read
because on each iteration we create a new copy of the empty mntns with it's own private mount with own tmpts in mnt_roots and we need to return to empty mntns to make next copy
create one more mntns
yes, nowdays tons of apps create their own mount namespaces here and there =) (basically we support nested containerization for mount namespaces) |
@Snorch I added the debug statements: The only one that trips is |
|
Ok, that is really strange, next steps to debug:
(it will print catched pid for convenience) 1') Add sleep(100) (remove old sleep) to pre_create_mount_namespaces to make_yard error path before Maybe we will understand some more with this information... |
|
@Snorch Here are the results sleep in get_empty_mntns sleep in pre_create_mount_namespaces in make_yard error |
and should not be happening at the same time, that is just impossible... Didn't you change something else since #2384 (comment) which leads to zero mnt_roots? The function cr_pivot_root explicitly does chdir to new root if 1+2 - Function get_empty_mntns is not ready for zero mnt_roots and obviously will do something completely undefined in this case. AFAICS create_mnt_roots should always be called before get_empty_mntns(), so either we'd failed earlier or mnt_roots is non zero. It should be in empty mntns copy at this point having something similar to: Sorry, but I don't understand anything now. |
|
I am still getting this result: at one point I changed and went back to mountv1 using the mntnscompat criu opt but I changed back to mountv2 for this. Both are failing with similar issues. I just did it again with the sleep after mkdirpat and the mnt_roots is null but when I pr_debug it's not? The resulting restore log updates though and has |
It's really strange... maybe my script Also can you check Also maybe it's worth trying on top of |
|
@Snorch just for some more info - we had no issues with this about two weeks ago. I'm not sure how relevant it is that between now and then AWS auto-updated the cluster. Here's the most recent patch: https://github.com/aws/eks-distro/releases/tag/v1-29-eks-10 Unfortunately there's no easy way to use an older version to test the regression hypothesis. FWIW, this same container and workload checkpoint/restores fine on our regular machines and inside a virtualized k3s cluster. |
|
jfyi: I tried to build and install amazonlinux kernel on my node and all zdtm pass fine on it. But maybe you just have a different config. You can try running zdtm in your amazon environment: to see if it passes mount tests or not... |
|
I am getting the following error: |
|
Maybe just need to rebuild everything clenanly |
Description
Dumped a simple count runc container on an AmazonLinux2 ec2 instance (uname -r 5.15.148-97.158.amzn2.x86_64). Dump was successful, but on restore I get the following error:
I added some extra debug statements, I apologize for that portion that may be confusing. But from my understanding there is a function in mountv2
pre_create_mount_namespaces(void)which callsget_empty_mntns(). Inget_empty_mntns()make_yard(path)is called which mounts a /tmp.mnt directory of type tmpfs and then makes it private. But then later inpre_create_mount_namespaces(void),make_yardis called again getting passed the same value for path, but this time errors out saying that the path DNE.What I don't understand is the portions between the start of
get_empty_mntns()to when we iterate over the ns_id list, filter for mount ns_ids and then we join the namespace created fromget_empty_mntns()to then call unshare(CLONE_NEWNS) again and to finally callmake_yardwhich then errors out saying that the path DNE which is the same path from what was passed in toget_empty_mntns().I also had a sleep call in the
make_yardfunction to verify that the tmp directory exists, it did both timesmake_yardwas called but I am wondering if the directory is somehow not visible to the calling process of unshare.Steps to reproduce the issue:
criu leave running on
8. Restore with default opts excluding the runc.conf from above
9. Error
Describe the results you received:
Error in CRIU restore saying that a file directory DNE
Describe the results you expected:
A successful restore of a runc container given the chrooted environment
Additional information you deem important (e.g. issue happens only occasionally):
CRIU logs and information:
CRIU full dump/restore logs:
Output of `criu --version`:
Output of `criu check --all`:
Additional environment details:
The text was updated successfully, but these errors were encountered: