Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mount: access to automounted volumes can lock up #15221

Closed
taskset opened this issue Mar 25, 2020 · 8 comments
Closed

mount: access to automounted volumes can lock up #15221

taskset opened this issue Mar 25, 2020 · 8 comments
Labels
Milestone

Comments

@taskset
Copy link
Contributor

taskset commented Mar 25, 2020

systemd version the issue has been seen with

rhel-7 systemd , rhel-8 systemd

The maximum unit_name currently supported by systemd is only 256 bytes,
but the name of the mount is generated by the path, and the maximum path can be 4k bytes.
This may cause systemd to fail to setup mount units.

Also due to #10874 , a mount point fails, the error will be passed upward from mount_setup_unit(),
and eventually mount_shutdown () will be called, and the monitored /proc/1/mountinfo file will be closed.

So, we may trigger a denial of service failure similar to CVE-2018-1049, as follows:

# mkdir -p /run/kata-containers/shared/sandboxes/f0ea3efdb417f442128830e86118cf216d1d236d6f970205a680972bcd062f74/f0ea3efdb417f442128830e86118cf216d1d236d6f970205a680972bcd062f74-a1bed3c11a474518-aaaaaa_xxxx_mix_xxxx_container_role_20200310112829109807.yyyy_container_role_20200310112829109807_15_81

# mkdir -p /tmp/test

# mount --bind /tmp/test   /run/kata-containers/shared/sandboxes/f0ea3efdb417f442128830e86118cf216d1d236d6f970205a680972bcd062f74/f0ea3efdb417f442128830e86118cf216d1d236d6f970205a680972bcd062f74-a1bed3c11a474518-aaaaaa_xxxx_mix_xxxx_container_role_20200310112829109807.yyyy_container_role_20200310112829109807_15_81 

# systemctl daemon-reload

We can see these errors in the logs:
Mar 23 22:53:48 iZuf6h1kfgutxc3el68z2lZ systemd[1]: Failed to set up mount unit: Invalid argument
Mar 23 22:53:48 iZuf6h1kfgutxc3el68z2lZ systemd[1]: Failed to reload: Invalid argument

We can also detect that / proc/1/mountinfo monitored by sysmted has been closed:
# ls -l /proc/1/fd | grep mount
# 

# systemctl start proc-sys-fs-binfmt_misc.automount
# systemctl start proc-sys-fs-binfmt_misc.mount

# umount  /proc/sys/fs/binfmt_misc

At this point, because systemd cannot listen to /proc/1/mountinfo, the status of the unit  is still * *active*.


If we visit binfmt_misc/, it will hang:
# stat /proc/sys/fs/binfmt_misc/

crash> bt
PID: 43378  TASK: ffff885042922140  CPU: 86  COMMAND: "stat"
 #0 [ffffc900575b7a28] __schedule at ffffffff81746d4b
 #1 [ffffc900575b7ab0] schedule at ffffffff817472c6
 #2 [ffffc900575b7ac8] autofs4_wait at ffffffff812e9482
 #3 [ffffc900575b7b80] autofs4_mount_wait at ffffffff812e7632
 #4 [ffffc900575b7bb0] autofs4_d_automount at ffffffff812e84c1
 #5 [ffffc900575b7be8] follow_managed at ffffffff8125e9b1
 #6 [ffffc900575b7c30] lookup_fast at ffffffff8125ef14
 #7 [ffffc900575b7c88] walk_component at ffffffff8125fda7
 #8 [ffffc900575b7ce0] path_lookupat at ffffffff81261047
 #9 [ffffc900575b7d08] filename_lookup at ffffffff81263ca1
#10 [ffffc900575b7e18] user_path_at_empty at ffffffff81263e46
#11 [ffffc900575b7e40] vfs_fstatat at ffffffff812580f6
#12 [ffffc900575b7e90] SYSC_newlstat at ffffffff812586b1
#13 [ffffc900575b7f18] sys_newlstat at ffffffff812587ee
#14 [ffffc900575b7f28] do_syscall_64 at ffffffff81003c04


Finally, the df command or any command that accesses /proc/sys/fs/binfmt_misc will get stuck:
# df 
# sysctl -a
…

This patch may fix this DoS error:
commit ba0d56f ("mount: don't propagate errors from mount_setup_unit() further up")

However, in a container environment, some mount paths are indeed very long,
so the limit of 256 bytes of the unit name may not be appropriate,
and we may also need to expand the unit name to 4k.

@dtardon
Copy link
Collaborator

dtardon commented Mar 26, 2020

A duplicate of #14294?

@poettering
Copy link
Member

Is this reproducible on current systemd? rhel7/rhel8 is very old, and we fixed a lot of stuff since then. We only track issues in current systemd here...

@poettering poettering added this to the v246 milestone Mar 30, 2020
@ret2libc
Copy link
Contributor

ret2libc commented Apr 6, 2020

@w-simon is it possible to reproduce the issue from regular user (non root)?

@taskset
Copy link
Contributor Author

taskset commented Apr 9, 2020

@w-simon is it possible to reproduce the issue from regular user (non root)?

Hello, we found this problem in some kata container environment (they have long mount paths), and then found that it can trigger a DoS. We have not yet tried whether non-root users can produce similar failures. However, nfs supports non-root user mounting, and similar scenarios may be constructed.
Thanks.

@ret2libc
Copy link
Contributor

ret2libc commented Apr 9, 2020

@w-simon is it possible to reproduce the issue from regular user (non root)?

Hello, we found this problem in some kata container environment (they have long mount paths), and then found that it can trigger a DoS.

You mean in the kernel inside the VM which runs the container?

We have not yet tried whether non-root users can produce similar failures. However, nfs supports non-root user mounting, and similar scenarios may be constructed.

You would still need a privileged user to mark the mount entry in /etc/fstab, wouldn't you?

@ret2libc
Copy link
Contributor

@w-simon Hi! Just wanted to ping you for #15221 (comment) . So far it does not seem there is a clear way for unprivileged users to trigger this, so I don't think it deserves a CVE (unless proven otherwise of course). If you feel otherwise, please provide examples so we can reproduce as unprivileged user or show there is a cross of privileged boundaries. In any case since the issue is now public it would be better to request the CVE to MITRE, if needed.

@taskset
Copy link
Contributor Author

taskset commented Apr 21, 2020

@w-simon Hi! Just wanted to ping you for #15221 (comment) . So far it does not seem there is a clear way for unprivileged users to trigger this, so I don't think it deserves a CVE (unless proven otherwise of course). If you feel otherwise, please provide examples so we can reproduce as unprivileged user or show there is a cross of privileged boundaries. In any case since the issue is now public it would be better to request the CVE to MITRE, if needed.

Hi, thank you very much for your comments. Sorry for the delay.

We found that this issue occurs on some servers, many kata containers running on these servers, and these containers have some very long path, such as (/run/kata-containers/shared/sandboxes/f0ea3efdb417f442128830e86118cf216d1d236d6f970205a680972bcd062f74/f0ea3efdb417f442128830e86118cf216d1d236d6f970205a680972bcd062f74-a1bed3c11a474518-xxxxxx_yyyyy_mix) .

We are not security professionals, and have not found a way for unprivileged users to trigger it.
And we also reported it to redhat:
https://bugzilla.redhat.com/show_bug.cgi?id=1819293

poettering added a commit to poettering/systemd that referenced this issue May 25, 2020
Whenever we pick up a new line in /proc/self/mountinfo and want to
synthesize a new mount unit from it, let's say which one it is.
Moreover, downgrade the log message when we encounter a mount point with
an overly long name to LOG_WARNING, since it's generally fine to ignore
such mount points.

Also, attach a catalog entry to explain the situation further.

Prompted-By: systemd#15221
@poettering
Copy link
Member

So, this has long been fixed by ba0d56f and is a duplicate of #10874. Hence let's close this.

That said, the logging in this case is less than ideal. Hence I prepped a PR in #15911 that improves that.

poettering added a commit to poettering/systemd that referenced this issue May 26, 2020
Whenever we pick up a new line in /proc/self/mountinfo and want to
synthesize a new mount unit from it, let's say which one it is.
Moreover, downgrade the log message when we encounter a mount point with
an overly long name to LOG_WARNING, since it's generally fine to ignore
such mount points.

Also, attach a catalog entry to explain the situation further.

Prompted-By: systemd#15221
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

5 participants