Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

entries in fstab disappear from mounted filesystems #595

Closed
dlangille opened this issue Aug 10, 2018 · 11 comments

Comments

@dlangille
Copy link
Contributor

commented Aug 10, 2018

$ iocage --version
Version 0.9.10 2017/12/22

On FreeBSD 11.2

Mount points in fstab which have been working for many months suddenly disappeared today.

[dan@x8dtu:/var/log] $ sudo iocage fstab -l x8dtu-ingress01
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| INDEX |                                                                                  FSTAB ENTRY                                                                                  |
+=======+===============================================================================================================================================================================+
| 0     |                                                                                                                                                                               |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 1     | /iocage/jails/x8dtu-ingress01/root/bin                          /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/bin           nullfs  ro,nosuid         0   0 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2     | none                                                            /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/dev           devfs   rw                0   0 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 3     | /iocage/jails/x8dtu-ingress01/root/lib                          /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/lib           nullfs  ro,nosuid         0   0 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 4     | /iocage/jails/x8dtu-ingress01/root/libexec                      /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/libexec       nullfs  ro,nosuid         0   0 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 5     | /iocage/jails/x8dtu-ingress01/root/sbin                         /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/sbin          nullfs  ro,nosuid         0   0 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 6     | /iocage/jails/x8dtu-ingress01/root/usr/share/mk                 /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/share/mk  nullfs  ro,nosuid,noexec  0   0 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 7     | /iocage/jails/x8dtu-ingress01/root/usr/sbin                     /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/sbin      nullfs  ro,nosuid         0   0 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 8     | /iocage/jails/x8dtu-ingress01/root/usr/bin                      /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/bin       nullfs  ro,nosuid         0   0 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 9     | /iocage/jails/x8dtu-ingress01/root/usr/lib                      /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/lib       nullfs  ro,nosuid         0   0 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 10    | /iocage/jails/x8dtu-ingress01/root/usr/libexec                  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/libexec   nullfs  ro,nosuid         0   0 |

If I restart the jail, these mount points exist and are working:

[dan@x8dtu:/var/log] $ mount | wc -l
      92

A short while later:

[dan@x8dtu:/var/log] $ mount | wc -l
      82

No idea why mounted filesystems are being unmounted. Nothing mentioned in /var/log/messages or /var/log/iocage

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 10, 2018

After copying the entries to /etc/fstab and issuing mount -a, the mounts came back.

A short while later, they were gone again.

Perhaps this is not an iocage issue at all.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 10, 2018

perhaps it is an icage issue:

[dan@x8dtu:~] $ sudo dtrace -n 'fbt::nullfs_unmount:entry {printf("%s (ppid %d)", curpsinfo->pr_psargs, curpsinfo->pr_ppid);}'
dtrace: description 'fbt::nullfs_unmount:entry ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  6  66394             nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
  6  66394             nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
  6  66394             nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
  6  66394             nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
  6  66394             nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
  6  66394             nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
  6  66394             nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
  6  66394             nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
  6  66394             nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
 13  66394             nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-nginx01/fstab (ppid 23492)
 13  66394             nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-nginx01/fstab (ppid 23492)

A short while later, this same thing occurred again. See https://gist.github.com/dlangille/43faf72d6bae645c4aee49e785244e7b for updates

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 10, 2018

More to follow as I track this issue down. It may be related to iocage console

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 10, 2018

Why is 'iocage chroot' doing a umount? dtrace output follows

  \-+= 27481 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-ingress01 file /bin/sh
    \-+= 27491 0.0 mount -a -F /iocage/jails/x8dtu-ingress01/fstab
      \-+= 27499 0.0 mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/usr/lib /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/lib
@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 10, 2018

This is the script I'm running. I thought adding the sleep might help. It does not. The umounts occur.

#!/bin/sh

HOSTVERSION=`/usr/bin/file /bin/sh`

JAILS=`/usr/local/bin/iocage list -H | /usr/bin/cut -f 2 -w`
ERRORS=''
for jail in ${JAILS}
do
  JAILVERSION=`/usr/local/bin/iocage chroot ${jail} 'file /bin/sh'`
  if [ "${JAILVERSION}" != "${HOSTVERSION}" ]
  then
    ERRORS="jail '${jail}' is ${JAILVERSION}"
  fi
  sleep 4
done

if [ "${ERRORS}" == "" ]
then
  echo 'All jails match the host'
  exit 0
else
  echo "WARNING: HOST and jails are NOT in sync: host = ${HOSTVERSION} but ${ERRORS}"
  exit 2
fi
@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 10, 2018

Ahh. A newer version of this script uses exec not chroot.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 10, 2018

I suspect the script shown above works fine with an older version of iocage.

Reckon?

@skarekrow

This comment has been minimized.

Copy link
Member

commented Aug 11, 2018

Likely, why do that at all? Why not use the release property for the jail?

@skarekrow skarekrow added the invalid label Aug 11, 2018

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 12, 2018

At first I wasn't sure why you mentioned that, but now I see:

$ sudo iocage get release x8dtu-ingress01
11.2-RELEASE-p1

Then compare that to the following?

 $ uname -r
11.2-RELEASE-p1

Allan Jude and I came up with the file /bin/sh method for some reason. I'll check back with him.

'file /bin/sh' will be jail manager independent, despite my script being iocage specific.

@jsegaert

This comment has been minimized.

Copy link
Contributor

commented Aug 12, 2018

Unless you're running really old (EOL) versions, you can use freebsd-version(1) that was added in FreeBSD 10.

@skarekrow

This comment has been minimized.

Copy link
Member

commented Aug 15, 2018

That is also what the release property should contain, so they should be equal in most circumstances.

@skarekrow skarekrow closed this Aug 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.