Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix race between explicit mount and handling automount request from kernel #5916

Merged
merged 1 commit into from May 10, 2017

Conversation

AnchorCat
Copy link
Contributor

@AnchorCat AnchorCat commented May 9, 2017

If a process accesses an autofs filesystem while systemd is in the middle of starting the mount unit on top of it, it is possible for the autofs_ptype_missing_direct request from the kernel to be received after
the mount unit has been fully started:

          P1                              P2
systemd forks and execs mount             ...
          ...                     access autofs, blocks
mount exits                               ...
systemd receives SIGCHLD                  ...
          ...                     kernel sends request
systemd receives request                  ...

systemd needs to respond to this request, otherwise the kernel will continue to block access to the mount point.


While I have only encountered this bug during boot on systems with automounted network filesystems, it can be triggered reasonably reliable with an artificial test case as follows:

  1. Prepare an automount and mount:
# mkdir /tmp/source /tmp/target
# echo contents >/tmp/source/file
# echo '/tmp/source /tmp/target none bind,noauto,x-systemd.automount 0 0' >>/etc/fstab
# systemctl daemon-reload
# systemctl start tmp-target.automount
  1. Raise systemd's rate limits:
# echo 'DefaultStartLimitBurst=10000' >>/etc/systemd/system.conf
# systemctl daemon-reexec 
  1. Start a loop that repeatedly races systemd mounting the unit against the automount request from the kernel:
# while sleep 0.2; do systemctl stop tmp-target.mount; systemctl start --no-block tmp-target.mount & sleep 0.006; cat /tmp/target/file; wait; done

The delay in the sleep 0.006 command needs to be tweaked so that the automount request is sometimes, but not always, the trigger for systemd to mount the unit, and such that "Automount point already active?" only occasionally appears in the journal. Eventually the "Got automount request" message will appear after "Mounted /tmp/target", showing that the autofs request was received too late:

May 09 11:38:28 hostname systemd[1]: Unmounting /tmp/target...
May 09 11:38:28 hostname systemd[1]: Unmounted /tmp/target.
May 09 11:38:28 hostname systemd[1]: Mounting /tmp/target...
May 09 11:38:28 hostname systemd[1]: tmp-target.automount: Got automount request for /tmp/target, triggered by 22427 (cat)
May 09 11:38:28 hostname systemd[1]: Mounted /tmp/target.
May 09 11:38:29 hostname systemd[1]: Unmounting /tmp/target...
May 09 11:38:29 hostname systemd[1]: Unmounted /tmp/target.
May 09 11:38:29 hostname systemd[1]: Mounting /tmp/target...
May 09 11:38:29 hostname systemd[1]: Mounted /tmp/target.
May 09 11:38:29 hostname systemd[1]: tmp-target.automount: Got automount request for /tmp/target, triggered by 22438 (cat)
May 09 11:38:29 hostname systemd[1]: tmp-target.automount: Automount point already active?

When this happens the cat command will hang, even though the /tmp/target filesystem was successfully mounted. Anything else that tries to access /tmp/target will also hang until it is unmounted (via umount or systemctl stop).

@poettering
Copy link
Member

Patch looks good, but could you add a short comment about this issue to the if block you are changing? Otherwise the next one looking at these sources or changing them might break this again. Something brief suffices, possibly containing a link to this issue containing the longer explanation...

Otherwirse looks excellent! thanks for spending the time to track this down, much appreciated.

@poettering poettering added bug 🐛 Programming errors, that need preferential fixing pid1 reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks labels May 9, 2017
If a process accesses an autofs filesystem while systemd is in the
middle of starting the mount unit on top of it, it is possible for the
autofs_ptype_missing_direct request from the kernel to be received after
the mount unit has been fully started:

  systemd forks and execs mount             ...
            ...                     access autofs, blocks
  mount exits                               ...
  systemd receives SIGCHLD                  ...
            ...                     kernel sends request
  systemd receives request                  ...

systemd needs to respond to this request, otherwise the kernel will
continue to block access to the mount point.
@AnchorCat
Copy link
Contributor Author

No problem. I've added a brief comment.

@poettering poettering merged commit e7d54bf into systemd:master May 10, 2017
@poettering
Copy link
Member

thanks!

@poettering poettering removed the reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks label May 10, 2017
euank pushed a commit to euank/systemd that referenced this pull request Jun 29, 2017
…5916)

If a process accesses an autofs filesystem while systemd is in the
middle of starting the mount unit on top of it, it is possible for the
autofs_ptype_missing_direct request from the kernel to be received after
the mount unit has been fully started:

  systemd forks and execs mount             ...
            ...                     access autofs, blocks
  mount exits                               ...
  systemd receives SIGCHLD                  ...
            ...                     kernel sends request
  systemd receives request                  ...

systemd needs to respond to this request, otherwise the kernel will
continue to block access to the mount point.
Werkov pushed a commit to Werkov/systemd that referenced this pull request Nov 27, 2018
…5916)

If a process accesses an autofs filesystem while systemd is in the
middle of starting the mount unit on top of it, it is possible for the
autofs_ptype_missing_direct request from the kernel to be received after
the mount unit has been fully started:

  systemd forks and execs mount             ...
            ...                     access autofs, blocks
  mount exits                               ...
  systemd receives SIGCHLD                  ...
            ...                     kernel sends request
  systemd receives request                  ...

systemd needs to respond to this request, otherwise the kernel will
continue to block access to the mount point.

(cherry picked from commit e7d54bf)

[fbui: adjust context]
[fbui: fixes bsc#1076308]
[fbui: fixes CVE-2018-1049]
mikhailnov pushed a commit to mikhailnov/systemd that referenced this pull request Aug 16, 2019
…5916)

If a process accesses an autofs filesystem while systemd is in the
middle of starting the mount unit on top of it, it is possible for the
autofs_ptype_missing_direct request from the kernel to be received after
the mount unit has been fully started:

  systemd forks and execs mount             ...
            ...                     access autofs, blocks
  mount exits                               ...
  systemd receives SIGCHLD                  ...
            ...                     kernel sends request
  systemd receives request                  ...

systemd needs to respond to this request, otherwise the kernel will
continue to block access to the mount point.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐛 Programming errors, that need preferential fixing pid1
Development

Successfully merging this pull request may close these issues.

None yet

2 participants