New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't mount tmpfs in userspace if SELinux engine not initialized #447

Closed
antoineco opened this Issue Aug 22, 2015 · 15 comments

Comments

Projects
None yet
7 participants
@antoineco

antoineco commented Aug 22, 2015

After upgrading to v779.0.0 I can't mount tmpfs volumes in userspace anymore. dmesg shows errors about a not initialized SELinux server.

SELinux: Unable to set superblock options before the security server is initialized

This prevents kubelet, for example, to mount volumes into pods:

kubelet[1154]: E0824 07:30:36.940461    1154 mount_linux.go:103] Mount failed: exit status 32
kubelet[1154]: Mounting arguments: tmpfs /var/lib/kubelet/pods/eb5fa7f8-4a31-11e5-aa17-0a2b0caa98ef/volumes/kubernetes.io~secret/default-token-8sfkr tmpfs [rootcontext="unlabeled"]
kubelet[1154]: Output: mount: wrong fs type, bad option, bad superblock on tmpfs,
kubelet[1154]: missing codepage or helper program, or other error
kubelet[1154]: In some cases useful info is found in syslog - try
kubelet[1154]: dmesg | tail or so.
@mjg59

This comment has been minimized.

mjg59 commented Aug 24, 2015

selinux configuration files were being installed in /etc, which isn't updated over upgrades. I'm fixing this up now.

mjg59 added a commit to mjg59/coreos-overlay that referenced this issue Aug 24, 2015

Fix selinux configuration file location
We were installing selinux configuration files in /etc which caused problems
on upgrades. Move them into /usr and ensure that systemd sets up appropriate
temporary files. Fixes coreos/bugs#447
@mjg59

This comment has been minimized.

mjg59 commented Aug 24, 2015

@mjg59 mjg59 reopened this Aug 24, 2015

@mjg59

This comment has been minimized.

mjg59 commented Aug 24, 2015

This isn't quite fixed yet - the tmpfile creation is happening after selinux attempts to load the config

@mjg59

This comment has been minimized.

mjg59 commented Aug 24, 2015

But I'd also note that nothing should assume that merely because the system has selinux enabled it has a loaded policy - I'll also try to fix up bits of userspace that are broken by this.

@mjg59

This comment has been minimized.

mjg59 commented Aug 24, 2015

@tg90nor

This comment has been minimized.

tg90nor commented Sep 28, 2015

@mjg59 i am seeing the same symptoms on v815.0.0, upgraded from v723.3.0. I am unable to create kubernetes pods with volumes.

journalctl:

kubelet[12431]: E0928 06:45:19.529254   12431 mount_linux.go:103] Mount failed: exit status 32
kubelet[12431]: Mounting arguments: tmpfs /var/lib/kubelet/pods/6a20bce0-65ac-11e5-8a6a-fa163e81f069/volumes/kubernetes.io~secret/default-token-tkn7d tmpfs [rootcontext="unlabeled"]
kubelet[12431]: Output: mount: wrong fs type, bad option, bad superblock on tmpfs,
kubelet[12431]: missing codepage or helper program, or other error
kubelet[12431]: In some cases useful info is found in syslog - try
kubelet[12431]: dmesg | tail or so.
kernel: SELinux: Unable to set superblock options before the security server is initialized

$ ls -al /etc/selinux/

total 8
drwxr-xr-x  2 root root 4096 Sep 25 10:19 .
drwxr-xr-x 30 root root 4096 Sep 25 10:27 ..
lrwxrwxrwx  1 root root   28 Sep 25 10:19 config -> ../../usr/lib/selinux/config
lrwxrwxrwx  1 root root   25 Sep 25 10:19 mcs -> ../../usr/lib/selinux/mcs
lrwxrwxrwx  1 root root   35 Sep 25 10:19 semanage.conf -> ../../usr/lib/selinux/semanage.conf
@tg90nor

This comment has been minimized.

tg90nor commented Sep 28, 2015

Creating symlinks and rebooting seems to have solved the problem.

core@kubernetes-node ~ $ cd /etc/selinux/
core@kubernetes-node /etc/selinux $ sudo ln -s ../../usr/lib/selinux/mls ./
core@kubernetes-node /etc/selinux $ sudo ln -s ../../usr/lib/selinux/targeted ./
@antoineco

This comment has been minimized.

antoineco commented Oct 2, 2015

I confirm, on fresh 815.0.0 nodes SELinux is throwing the same errors as @tg90nor mentioned.
It doesn't happen on nodes upgraded from 808.0.0 though.

@kayrus

This comment has been minimized.

kayrus commented Oct 6, 2015

Have just upgraded CoreOS form 723.3.0 to 815.0.0:

CoreOS alpha (815.0.0)
Failed Units: 1
  initrd-setup-root.service
localhost ~ # systemctl status -l initrd-setup-root.service
● initrd-setup-root.service
   Loaded: not-found (Reason: No such file or directory)
   Active: failed (Result: exit-code) since Tue 2015-10-06 11:00:52 UTC; 55s ago
 Main PID: 302 (code=exited, status=1/FAILURE)

Oct 06 11:00:52 localhost systemd[1]: Starting Root filesystem setup...
Oct 06 11:00:52 localhost systemd-tmpfiles[306]: /sysroot/etc/selinux does not exist and cannot be created as the file system is read-only.
Oct 06 11:00:52 localhost systemd-tmpfiles[306]: symlink(../../usr/lib/selinux/semanage.conf, /sysroot/etc/selinux/semanage.conf) failed: No such file or directory
Oct 06 11:00:52 localhost systemd-tmpfiles[306]: symlink(../../usr/lib/selinux/config, /sysroot/etc/selinux/config) failed: No such file or directory
Oct 06 11:00:52 localhost systemd-tmpfiles[306]: symlink(../../usr/lib/selinux/mcs, /sysroot/etc/selinux/mcs) failed: No such file or directory
Oct 06 11:00:52 localhost systemd[1]: initrd-setup-root.service: Main process exited, code=exited, status=1/FAILURE
Oct 06 11:00:52 localhost systemd[1]: Failed to start Root filesystem setup.
Oct 06 11:00:52 localhost systemd[1]: initrd-setup-root.service: Unit entered failed state.
Oct 06 11:00:52 localhost systemd[1]: initrd-setup-root.service: Failed with result 'exit-code'.
@marineam

This comment has been minimized.

marineam commented Oct 6, 2015

Yeah, either we are missing the proper tmpfiles rules to set up upgraded systems or the existing rules are incorrect.

@marineam

This comment has been minimized.

marineam commented Oct 6, 2015

From the error probably just needs a d /etc/selinux line

@marineam

This comment has been minimized.

marineam commented Oct 7, 2015

er, the tmpfiles config is fine, I just screwed up the kernel options.

@orbistertius

This comment has been minimized.

orbistertius commented Oct 13, 2015

I have similar error messages as @kayrus in PXE-booted nodes with CoreOS alpha 833.0.0 and 829.0.0:

CoreOS stable (833.0.0)
Update Strategy: No Reboots
Failed Units: 1
  initrd-setup-root.service
core@nuc0 ~ $ sudo systemctl status -l initrd-setup-root.service
● initrd-setup-root.service
   Loaded: not-found (Reason: No such file or directory)
   Active: failed (Result: exit-code) since Tue 2015-10-13 12:29:47 UTC; 1min 9s ago
 Main PID: 348 (code=exited, status=1/FAILURE)


Oct 13 12:42:44 localhost systemd[1]: Starting Root filesystem setup...
Oct 13 12:42:44 localhost systemd-tmpfiles[360]: symlink(../../usr/lib/selinux/semanage.conf, /sysroot/etc/selinux/semanage.conf) failed: No such file or directory
Oct 13 12:42:44 localhost systemd[1]: initrd-setup-root.service: Main process exited, code=exited, status=1/FAILURE
Oct 13 12:42:44 localhost systemd[1]: Failed to start Root filesystem setup.
Oct 13 12:42:44 localhost systemd[1]: initrd-setup-root.service: Unit entered failed state.
Oct 13 12:42:44 localhost systemd[1]: initrd-setup-root.service: Failed with result 'exit-code'.
CoreOS stable (829.0.0)
Update Strategy: No Reboots
Failed Units: 1
  initrd-setup-root.service
core@nuc0 ~ $ sudo systemctl status -l initrd-setup-root.service
● initrd-setup-root.service
   Loaded: not-found (Reason: No such file or directory)
   Active: failed (Result: exit-code) since Tue 2015-10-13 12:27:35 UTC; 27s ago
 Main PID: 314 (code=exited, status=1/FAILURE)


Oct 13 12:46:49 localhost systemd[1]: Starting Root filesystem setup...
Oct 13 12:46:49 localhost systemd-tmpfiles[361]: symlink(../../usr/lib/selinux/semanage.conf, /sysroot/etc/selinux/semanage.conf) failed: No such file or directory
Oct 13 12:46:49 localhost systemd[1]: initrd-setup-root.service: Main process exited, code=exited, status=1/FAILURE
Oct 13 12:46:49 localhost systemd[1]: Failed to start Root filesystem setup.
Oct 13 12:46:49 localhost systemd[1]: initrd-setup-root.service: Unit entered failed state.
Oct 13 12:46:49 localhost systemd[1]: initrd-setup-root.service: Failed with result 'exit-code'.
@marineam

This comment has been minimized.

marineam commented Oct 13, 2015

Hm, must be ordering the semanage.conf symlink before the selinux dir for some reason

@orbistertius

This comment has been minimized.

orbistertius commented Oct 16, 2015

Thanks @marineam , it looks ok now

CoreOS stable (835.0.0)
Update Strategy: No Reboots
core@nuc2 ~ $ ls -la /etc/selinux/semanage.conf
lrwxrwxrwx 1 root root 35 Oct 16 14:12 /etc/selinux/semanage.conf -> ../../usr/lib/selinux/semanage.conf 
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment