mnt-v2 errors with Kubernetes

[adrianreber]

@Snorch Some time ago we talked about this in the chat. Trying to restore a container with Kubernetes fails with:

(00.029005)      1: Error (criu/mount-v2.c:891): mnt-v2: Failed to copy sharing from -1:/var/lib/containers/storage/overlay-containers/e80ac5757f21caec6cb74bf628a39aa47fa86cf7c9c49a361711ed0abf711c7b/userdata/.containerenv to 11: Invalid argument
(00.029028)      1: Error (criu/mount-v2.c:958): mnt-v2: Failed to copy sharing from source /var/lib/containers/storage/overlay-containers/e80ac5757f21caec6cb74bf628a39aa47fa86cf7c9c49a361711ed0abf711c7b/userdata/.containerenv to 656
(00.029821) Error (criu/mount.c:3674): mnt: Can't remove the directory /tmp/.criu.mntns.kzWYEa: Device or resource busy
(00.029843) Error (criu/cr-restore.c:2536): Restoring FAILED.

Using mntns-compat-mode in the configuration file during restore I can restore my container.

You said you needed a trace with https://github.com/Snorch/linux-helpers/blob/master/gftrace.sh:

perf probe -f 'do_set_group%return $retval'
./gftrace do_set_group

And finally here is the trace:

# tracer: function_graph
#
#     TIME        CPU  DURATION                  FUNCTION CALLS
#      |          |     |   |                     |   |   |   |
13105.460415 |   1)               |  do_set_group() {
13105.460417 |   1)               |    down_write() {
13105.460417 |   1)   0.250 us    |      __cond_resched();
13105.460417 |   1)   0.802 us    |    } /* down_write */
13105.460417 |   1)               |    ns_capable() {
13105.460418 |   1)               |      security_capable() {
13105.460418 |   1)   0.190 us    |        cap_capable();
13105.460418 |   1)               |        selinux_capable() {
13105.460418 |   1)   0.521 us    |          cred_has_capability.isra.0();
13105.460419 |   1)   0.862 us    |        } /* selinux_capable */
13105.460419 |   1)   0.200 us    |        bpf_lsm_capable();
13105.460420 |   1)   1.953 us    |      } /* security_capable */
13105.460420 |   1)   2.294 us    |    } /* ns_capable */
13105.460420 |   1)               |    ns_capable() {
13105.460420 |   1)               |      security_capable() {
13105.460420 |   1)   0.180 us    |        cap_capable();
13105.460421 |   1)               |        selinux_capable() {
13105.460421 |   1)   0.490 us    |          cred_has_capability.isra.0();
13105.460421 |   1)   0.822 us    |        } /* selinux_capable */
13105.460422 |   1)   0.191 us    |        bpf_lsm_capable();
13105.460422 |   1)   1.884 us    |      } /* security_capable */
13105.460422 |   1)   2.224 us    |    } /* ns_capable */
13105.460422 |   1)               |    namespace_unlock() {
13105.460423 |   1)   0.190 us    |      up_write();
13105.460423 |   1)   0.201 us    |      shrink_dentry_list();
13105.460423 |   1)   0.942 us    |    } /* namespace_unlock */
13105.460423 |   1)   8.547 us    |  } /* do_set_group */
13105.460426 |   1)               |  /* do_set_group__return: (__x64_sys_move_mount+0x114/0x180 <- do_set_group) arg1=0xffffffea */

This happens with criu 3.17.1 and 6.0.11-300.fc37.x86_64

Let me know if you need more debug data.

[Snorch]

Thanks! Here is what we might also need to understand the problem:

1. Looking on criu/mount-v2.c:958 we see that a most likely cause of error here is a private external source:

                        /*
                         * External slavery. We rely on the user to give us the
                         * right source for external mount with all proper
                         * sharing options setup (it should be either shared
                         * or non-shared slave). If source is a private mount
                         * we would fail.
                         */
                        if (move_mount_set_group(-1, sg->source, target->mnt_fd_id)) {
                                pr_err("Failed to copy sharing from source %s to %d\n", sg->source, target->mnt_id);
                                close(target_fd);
                                return -1;
                        }

Can you please look on /var/lib/containers/storage/overlay-containers/e80ac5757f21caec6cb74bf628a39aa47fa86cf7c9c49a361711ed0abf711c7b/userdata/.containerenv mountpoint in host /proc/self/mountinfo to see it has sharing options? (Probably it would only appear for a short interval of container restore/creation so it can be tricky to catch this mountinfo, doing that in loop greping for "containerenv" may help, though full mountinfo is preferable.)

2. From gftrace we can see that we've passed at least to this line https://github.com/torvalds/linux/blob/e2ca6ba6ba0152361aa4fcbf6067db71b2c7a770/fs/namespace.c#L2879 in do_set_group.

Can you please check on that system if later functions called from do_set_group are inlined or not, so we can say for sure if we didn't reach them or they were just not visible to gftrace:

grep "is_subdir\|has_locked_children\|set_mnt_shared\|lock_mount_hash" /sys/kernel/debug/tracing/available_filter_functions

3. Can you please give more info about our mounts, we copy external master from external source .containerenv to 656 mount, can you at least grep -a5 656 ... in dump/restore logs to see all info we have about them, full dump/restore logs would likely be more useful as nearby log lines may contain some insights.

---

For now I can assume (see // comments inline):

static int do_set_group(struct path *from_path, struct path *to_path)
{
        struct mount *from, *to;
        int err;

        from = real_mount(from_path->mnt);
        to = real_mount(to_path->mnt);

        namespace_lock();

        err = -EINVAL;
        /* To and From must be mounted */
        if (!is_mounted(&from->mnt))
                goto out;
        if (!is_mounted(&to->mnt))
                goto out;

        err = -EPERM;
        /* We should be allowed to modify mount namespaces of both mounts */
        if (!ns_capable(from->mnt_ns->user_ns, CAP_SYS_ADMIN))
                goto out;
        if (!ns_capable(to->mnt_ns->user_ns, CAP_SYS_ADMIN))
                goto out;

        // we've passed at least until here according to gftrace
        err = -EINVAL;
        /* To and From paths should be mount roots */
        if (from_path->dentry != from_path->mnt->mnt_root)
                goto out;
        if (to_path->dentry != to_path->mnt->mnt_root)
                goto out;
         // I doubt that we didn't get here as mount-v2 criu code always  gives mountpoints to kernel 

        /* Setting sharing groups is only allowed across same superblock */
        if (from->mnt.mnt_sb != to->mnt.mnt_sb)
                goto out;
        // we can check if we passed this one by comparing mountinfo for .containerenv on host and mountinfo for 656 mount in images

        /* From mount root should be wider than To mount root */
        if (!is_subdir(to->mnt.mnt_root, from->mnt.mnt_root))
                goto out;
        // for this one we need to check roots of the mounts in mountinfo

        /* From mount should not have locked children in place of To's root */
        if (has_locked_children(from, to->mnt.mnt_root))
                goto out;
        // I doubt that the file mount which .containerenv seems to be has locked children

        /* Setting sharing groups is only allowed on private mounts */
        if (IS_MNT_SHARED(to) || IS_MNT_SLAVE(to))
                goto out;
        // This one should not be a problem as we've just created mount 656 private

        /* From should not be private */
        if (!IS_MNT_SHARED(from) && !IS_MNT_SLAVE(from))
                goto out;
        // These error path is most likely, this would mean that container environment made a bad setup of external mounts, if external mount in container is a slave and there is no master in ct (meaning there is master outside ct), the source mount given to criu for restoring this slave mount should have sharing).

        if (IS_MNT_SLAVE(from)) {
                struct mount *m = from->mnt_master;

                list_add(&to->mnt_slave, &m->mnt_slave_list);
                to->mnt_master = m;
        }

        if (IS_MNT_SHARED(from)) {
                to->mnt_group_id = from->mnt_group_id;
                list_add(&to->mnt_share, &from->mnt_share);
                lock_mount_hash();
                set_mnt_shared(to);
                unlock_mount_hash();
        }

        err = 0;
out:
        namespace_unlock();
        return err;
}

[adrianreber]

Thanks! Here is what we might also need to understand the problem:

1. Looking on criu/mount-v2.c:958 we see that a most likely cause of error here is a private external source:

                        /*
                         * External slavery. We rely on the user to give us the
                         * right source for external mount with all proper
                         * sharing options setup (it should be either shared
                         * or non-shared slave). If source is a private mount
                         * we would fail.
                         */
                        if (move_mount_set_group(-1, sg->source, target->mnt_fd_id)) {
                                pr_err("Failed to copy sharing from source %s to %d\n", sg->source, target->mnt_id);
                                close(target_fd);
                                return -1;
                        }

Can you please look on /var/lib/containers/storage/overlay-containers/e80ac5757f21caec6cb74bf628a39aa47fa86cf7c9c49a361711ed0abf711c7b/userdata/.containerenv mountpoint in host /proc/self/mountinfo to see it has sharing options? (Probably it would only appear for a short interval of container restore/creation so it can be tricky to catch this mountinfo, doing that in loop greping for "containerenv" may help, though full mountinfo is preferable.)

571 557 8:1 /var/lib/containers/storage/overlay-containers/bd2b0db91a38663ed6e13aeb15709be5d9598f95d17aa8904ce8ad1d2f8bc564/userdata/.containerenv /var/lib/containers/storage/overlay/07ceaa7a4f21108524bac9a3ae93db03eb0a6407eb707e37230373329428275f/merged/run/.containerenv rw,relatime shared:1 - ext4 /dev/sda1 rw,seclabel

restore.log

2. From gftrace we can see that we've passed at least to this line https://github.com/torvalds/linux/blob/e2ca6ba6ba0152361aa4fcbf6067db71b2c7a770/fs/namespace.c#L2879 in do_set_group.

Can you please check on that system if later functions called from do_set_group are inlined or not, so we can say for sure if we didn't reach them or they were just not visible to gftrace:

grep "is_subdir\|has_locked_children\|set_mnt_shared\|lock_mount_hash" /sys/kernel/debug/tracing/available_filter_functions

$ grep "is_subdir\|has_locked_children\|set_mnt_shared\|lock_mount_hash" /sys/kernel/debug/tracing/available_filter_functions
is_subdir

3. Can you please give more info about our mounts, we copy external master from external source `.containerenv` to 656 mount, can you at least `grep -a5 656 ...` in dump/restore logs to see all info we have about them, full dump/restore logs would likely be more useful as nearby log lines may contain some insights.

The whole setup looks like this. Initially we have a pod with two containers. One container is the so-called infrastructure container and the other container is being checkpointed. The infrastructure container and the target container both bind mount the .containerenv file into their own mount namespace from the host.

During restore a new pod is created with a new infrastructure container which bind mounts the new .containerenv file. For the restored container I am changing the source of the .containerenv mount to point to the new file and CRIU should remount that new file to the old location. This is why the things are set up the way they are. Changing the source of a bind mount happens a lot during container restore for Kubernetes as there are multiple mount points which are mounted from the host into the container which are pod specific. As the restore is done into a new pod all these bind mounts are changed. I hope that makes some sense.

For now I can assume (see // comments inline):

static int do_set_group(struct path *from_path, struct path *to_path)
{
        struct mount *from, *to;
        int err;

        from = real_mount(from_path->mnt);
        to = real_mount(to_path->mnt);

        namespace_lock();

        err = -EINVAL;
        /* To and From must be mounted */
        if (!is_mounted(&from->mnt))
                goto out;
        if (!is_mounted(&to->mnt))
                goto out;

        err = -EPERM;
        /* We should be allowed to modify mount namespaces of both mounts */
        if (!ns_capable(from->mnt_ns->user_ns, CAP_SYS_ADMIN))
                goto out;
        if (!ns_capable(to->mnt_ns->user_ns, CAP_SYS_ADMIN))
                goto out;

        // we've passed at least until here according to gftrace
        err = -EINVAL;
        /* To and From paths should be mount roots */
        if (from_path->dentry != from_path->mnt->mnt_root)
                goto out;
        if (to_path->dentry != to_path->mnt->mnt_root)
                goto out;
         // I doubt that we didn't get here as mount-v2 criu code always  gives mountpoints to kernel 

        /* Setting sharing groups is only allowed across same superblock */
        if (from->mnt.mnt_sb != to->mnt.mnt_sb)
                goto out;
        // we can check if we passed this one by comparing mountinfo for .containerenv on host and mountinfo for 656 mount in images

        /* From mount root should be wider than To mount root */
        if (!is_subdir(to->mnt.mnt_root, from->mnt.mnt_root))
                goto out;
        // for this one we need to check roots of the mounts in mountinfo

        /* From mount should not have locked children in place of To's root */
        if (has_locked_children(from, to->mnt.mnt_root))
                goto out;
        // I doubt that the file mount which .containerenv seems to be has locked children

        /* Setting sharing groups is only allowed on private mounts */
        if (IS_MNT_SHARED(to) || IS_MNT_SLAVE(to))
                goto out;
        // This one should not be a problem as we've just created mount 656 private

        /* From should not be private */
        if (!IS_MNT_SHARED(from) && !IS_MNT_SLAVE(from))
                goto out;
        // These error path is most likely, this would mean that container environment made a bad setup of external mounts, if external mount in container is a slave and there is no master in ct (meaning there is master outside ct), the source mount given to criu for restoring this slave mount should have sharing).

        if (IS_MNT_SLAVE(from)) {
                struct mount *m = from->mnt_master;

                list_add(&to->mnt_slave, &m->mnt_slave_list);
                to->mnt_master = m;
        }

        if (IS_MNT_SHARED(from)) {
                to->mnt_group_id = from->mnt_group_id;
                list_add(&to->mnt_share, &from->mnt_share);
                lock_mount_hash();
                set_mnt_shared(to);
                unlock_mount_hash();
        }

        err = 0;
out:
        namespace_unlock();
        return err;
}

[Snorch]

1. Mount has sharing shared:1, so my initial guess was wrong.
2. This means that we didn't get to is_subdir check after two ns_capable checks, so we've failed somewhere in those three places:

        err = -EPERM;
        /* We should be allowed to modify mount namespaces of both mounts */
        if (!ns_capable(from->mnt_ns->user_ns, CAP_SYS_ADMIN))
                goto out;
        if (!ns_capable(to->mnt_ns->user_ns, CAP_SYS_ADMIN))
                goto out;
 
        err = -EINVAL;
        /* To and From paths should be mount roots */
        if (from_path->dentry != from_path->mnt->mnt_root)
>                goto out;
        if (to_path->dentry != to_path->mnt->mnt_root)
>                goto out;
 
        /* Setting sharing groups is only allowed across same superblock */
        if (from->mnt.mnt_sb != to->mnt.mnt_sb)
>                goto out;
 
        /* From mount root should be wider than To mount root */
        if (!is_subdir(to->mnt.mnt_root, from->mnt.mnt_root))
                goto out;
 

I don't think it's first two, so the most promising is superblock check fail.
3)

During restore a new pod is created with a new infrastructure container which
bind mounts the new .containerenv file. For the restored container I am
changing the source of the .containerenv mount to point to the new file and
CRIU should remount that new file to the old location. This is why the things
are set up the way they are. Changing the source of a bind mount happens a lot
during container restore for Kubernetes as there are multiple mount points
which are mounted from the host into the container which are pod specific. As
the restore is done into a new pod all these bind mounts are changed. I hope
that makes some sense.

From what you are explaining and what I see in logs:

(00.032358)      1: Error (criu/mount-v2.c:958): mnt-v2: Failed to copy sharing from source /var/lib/containers/storage/overlay-containers/bd2b0db91a38663ed6e13aeb15709be5d9598f95d17aa8904ce8ad1d2f8bc564/userdata/.containerenv to 887

mount-v2 engine is trying to copy sharing from external source path to mount 887

(00.003323) mnt:                Will mount 887 @ /tmp/.criu.mntns.yYgXnV/mnt-0000000887 /etc/hostname

where mount 887 has mountpoint /etc/hostname in container. Here we see inconsistency, why from '.containerenv' to 'hostname'?

From these other lines I can assume that same file was bindmounted in both places on dump:

(00.003818) mnt:        The mount 887 is bind for 886 (@/etc/hostname -> @/etc/resolv.conf)
(00.003820) mnt:        The mount 888 is bind for 886 (@/run/.containerenv -> @/etc/resolv.conf)
(00.003863) mnt-v2: Inspecting sharing on 887 shared_id 0 master_id 24 (@/etc/hostname)
(00.003865) mnt-v2: Inspecting sharing on 888 shared_id 0 master_id 24 (@/run/.containerenv)

That's why copying sharing between '.containerenv' and 'hostname' is reasonable and should've worked if external mounts were provided right to CRIU.

But it seems that on restore you try to put different files to them -> don't do that. /run/containers/storage/overlay-containers/bd2b0db91a38663ed6e13aeb15709be5d9598f95d17aa8904ce8ad1d2f8bc564/userdata/hostname and /var/lib/containers/storage/overlay-containers/bd2b0db91a38663ed6e13aeb15709be5d9598f95d17aa8904ce8ad1d2f8bc564/userdata/.containerenv should be same file in same path in it's filesystem. If one file was externaly mounted in several places in CT on restore external options should point all to the same file from the same filesystem, with same inode number else even file restore code would fail.

[adrianreber]

Thanks for your analysis. Looks like I might be doing something wrong. I will take a look at the higher level code and report back here.

[adrianreber]

It looks like the container definition I am creating is correctly setup:

                {
                        "destination": "/etc/resolv.conf",
                        "type": "bind",
                        "source": "/run/containers/storage/overlay-containers/f3eea75d926796a9d76a2fbe4c1a3b6632d3b06c9caeca015fc44c8f05080205/userdata/resolv.conf",
                        "options": [
                                "rw",
                                "bind",
                                "nodev",
                                "nosuid",
                                "noexec"
                        ]
                },
                {
                        "destination": "/etc/hostname",
                        "type": "bind",
                        "source": "/run/containers/storage/overlay-containers/f3eea75d926796a9d76a2fbe4c1a3b6632d3b06c9caeca015fc44c8f05080205/userdata/hostname",
                        "options": [
                                "rw",
                                "bind"
                        ]
                },
                {
                        "destination": "/run/.containerenv",
                        "type": "bind",
                        "source": "/var/lib/containers/storage/overlay-containers/f3eea75d926796a9d76a2fbe4c1a3b6632d3b06c9caeca015fc44c8f05080205/userdata/.containerenv",
                        "options": [
                                "rw",
                                "bind"
                        ]
                },

and the restore.log says:

(00.003793) mnt:                Will mount 888 from /var/lib/containers/storage/overlay-containers/f3eea75d926796a9d76a2fbe4c1a3b6632d3b06c9caeca015fc44c8f05080205/userdata/.containerenv (E)
(00.003795) mnt:                Will mount 888 @ /tmp/.criu.mntns.4dVqUG/mnt-0000000888 /run/.containerenv
(00.003797) mnt:        Read 888 mp @ /run/.containerenv
(00.003800) mnt:                Will mount 887 from /run/containers/storage/overlay-containers/f3eea75d926796a9d76a2fbe4c1a3b6632d3b06c9caeca015fc44c8f05080205/userdata/hostname (E)
(00.003802) mnt:                Will mount 887 @ /tmp/.criu.mntns.4dVqUG/mnt-0000000887 /etc/hostname
(00.003803) mnt:        Read 887 mp @ /etc/hostname
(00.003807) mnt:                Will mount 886 from /run/containers/storage/overlay-containers/f3eea75d926796a9d76a2fbe4c1a3b6632d3b06c9caeca015fc44c8f05080205/userdata/resolv.conf (E)
(00.003810) mnt:                Will mount 886 @ /tmp/.criu.mntns.4dVqUG/mnt-0000000886 /etc/resolv.conf
(00.003811) mnt:        Read 886 mp @ /etc/resolv.conf

restore.log

So it seems I am telling runc the right new bind mount mappings and CRIU also lists them in the log file.

All three files exist on the file system:

-rw-r--r--. 1 root root 13 Dec 15 12:06 /run/containers/storage/overlay-containers/f3eea75d926796a9d76a2fbe4c1a3b6632d3b06c9caeca015fc44c8f05080205/userdata/hostname
-rw-r--r--. 1 root root 39 Dec 15 12:06 /run/containers/storage/overlay-containers/f3eea75d926796a9d76a2fbe4c1a3b6632d3b06c9caeca015fc44c8f05080205/userdata/resolv.conf
-rw-r--r--. 1 root root  0 Dec 15 12:06 /var/lib/containers/storage/overlay-containers/f3eea75d926796a9d76a2fbe4c1a3b6632d3b06c9caeca015fc44c8f05080205/userdata/.containerenv

But I do not understand the part about the sharing. What does this mean:

(00.004225) mnt:        The mount 887 is bind for 886 (@/etc/hostname -> @/etc/resolv.conf)
(00.004227) mnt:        The mount 888 is bind for 886 (@/run/.containerenv -> @/etc/resolv.conf)
(00.004229) mnt:        The mount 892 is bind for 886 (@/run/secrets -> @/etc/resolv.conf)

what does it mean pointing to /etc/resolv.conf. I will try to collect a log from the dump.

[adrianreber]

I tried to collect some more data and now I found something interesting. If I restore the container in Kubernetes it works if I restore the container just in CRI-O it fails. Attached are both log files.

fail.log
success.log

Do you see why one works and the other doesn't. Maybe I am not correctly setting up a resource, mount or directory. Let me know if you see any differences.

[hesch]

I encountered the same error while trying to restore with Kubernetes. My mountinfo while restoring shows different file systems for the different files:

3751 3718 0:25 /containers/storage/overlay-containers/bcbc79e3652e8a3b32a0ee8c64c1429b3089ccf799d552ec269ad036f7a0dec0/userdata/hostname /var/lib/containers/storage/overlay/205218eb45caa22473e9ca2fa57fd2d342f0ac4e92414b2bf76de232a3460e8c/merged/etc/hostname rw,nosuid,nodev,noexec,relatime shared:5 - tmpfs tmpfs rw,size=6577812k,mode=755,inode64
3752 3718 8:8 /var/lib/containers/storage/overlay-containers/bcbc79e3652e8a3b32a0ee8c64c1429b3089ccf799d552ec269ad036f7a0dec0/userdata/.containerenv /var/lib/containers/storage/overlay/205218eb45caa22473e9ca2fa57fd2d342f0ac4e92414b2bf76de232a3460e8c/merged/run/.containerenv rw,relatime shared:1 - ext4 /dev/sda8 rw,errors=remount-ro

If I understand @Snorch correctly, this is the cause of the failure. If I look into mountinfo of the running container(switching to the mount namespace with nsenter) it looks like this:

3575 3567 0:25 /containers/storage/overlay-containers/5b61d8067e08066d9bb92df2dcd20ee6765abdb30160e152c8635f1b3dfd9a57/userdata/resolv.conf /etc/resolv.conf rw,nosuid,nodev,noexec,relatime master:5 - tmpfs tmpfs rw,size=6577812k,mode=755,inode64
3576 3567 0:25 /containers/storage/overlay-containers/5b61d8067e08066d9bb92df2dcd20ee6765abdb30160e152c8635f1b3dfd9a57/userdata/hostname /etc/hostname rw,nosuid,nodev,noexec,relatime master:5 - tmpfs tmpfs rw,size=6577812k,mode=755,inode64
3577 3567 0:25 /containers/storage/overlay-containers/5b61d8067e08066d9bb92df2dcd20ee6765abdb30160e152c8635f1b3dfd9a57/userdata/.containerenv /run/.containerenv rw,nosuid,nodev,noexec,relatime master:5 - tmpfs tmpfs rw,size=6577812k,mode=755,inode64

So they all come from the same tmpfs, but on restore the mount setup is somehow different. Is this actually a bug in runc?

[adrianreber]

So they all come from the same tmpfs, but on restore the mount setup is somehow different.

Thanks for highlighting this. That helps.

Is this actually a bug in runc?

No, not really. I need to think about this a bit.

[adrianreber]

So this seems to be related to pods with or without infrastructure containers.

If you checkpoint a container out of a pod without an infrastructure container you have to restore the container in a pod without an infrastructure container. Or if the original pod had an infrastructure container then the restored containers also needs to be in a pod with an infrastructure container.

It is not clear why the location of /run/.containerenv changes but it seems to change currently in CRI-O. A better error message in CRI-O would be nice.

@Snorch thanks for your time looking at this. @hesch thanks for chiming in.

Closing as it does not seem to be a CRIU problem.