Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd incorrectly unmounts a reused mount point after a device removal #8596

Closed
amg1127 opened this Issue Mar 27, 2018 · 5 comments

Comments

4 participants
@amg1127
Copy link

amg1127 commented Mar 27, 2018

Submission type

  • Bug report

systemd version the issue has been seen with

systemd 238
+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN +PCRE2 default-hierarchy=hybrid

Used distribution

Arch Linux

In case of bug report: Expected behaviour you didn't see

systemd should not unmount anything.

In case of bug report: Unexpected behaviour you saw

After finishing a filesystem migration from an old LVM2 logical volume to a new one, I:

  • Renamed the old logical volume from something to something-OLD;
  • Renamed the new logical volume from something-NEW to something
  • Removed the old logical volume named something-OLD

Then, just after issuing a lvremove on the old logical volume, I saw systemd unmounting the directory in which the new logical volume was mounted.

In case of bug report: Steps to reproduce the problem

The steps to reproduce the problem are:

  • Have an old LVM2 logical volume automatically mounted on a target directory;
  • Create a new LVM2 logical volume;
  • Unmount the target directory;
  • Mount the new LVM2 logical volume on the same directory;
  • Rename the old LVM2 device to something else;
  • Rename the new LVM2 device so that it uses the previous name of the old LVM2 device;
  • Remove the old LVM2 device.

The following code snippet can trigger the bug:

#!/bin/bash

set -x

# Have an old LVM2 logical volume automatically mounted on a target directory
truncate -s 1G /pvol1.dat
losetup /dev/loop1 /pvol1.dat
pvcreate /dev/loop1
vgcreate volg /dev/loop1
lvcreate -l '100%VG' -n lvol volg
udevadm settle
mkfs.ext4 /dev/volg/lvol
mkdir -p /mntvol
echo '/dev/volg/lvol /mntvol ext4 defaults 0 0' >> /etc/fstab
systemctl daemon-reload
mount /mntvol

# Create a new LVM2 logical volume;
truncate -s 1G /pvol2.dat
losetup /dev/loop2 /pvol2.dat
pvcreate /dev/loop2
vgcreate volg-NEW /dev/loop2
lvcreate -l '100%VG' -n lvol-NEW volg-NEW
udevadm settle
mkfs.ext4 /dev/volg-NEW/lvol-NEW
mkdir -p /mntvol-NEW
mount /dev/volg-NEW/lvol-NEW /mntvol-NEW
echo FOO > /mntvol-NEW/file_exists.txt
sleep 1

# Unmount the target directory
umount /mntvol

# Mount the new LVM2 logical volume on the same directory
mount --rbind /mntvol-NEW /mntvol
cd /mntvol
cat /mntvol/file_exists.txt

# Rename the old LVM2 device to something else
vgrename volg volg-OLD
lvrename volg-OLD lvol lvol-OLD

# Rename the new LVM2 device so that it uses the previous name of the old LVM2 device
vgrename volg-NEW volg
lvrename volg lvol-NEW lvol
udevadm settle
cat /mntvol/file_exists.txt

# Remove the old LVM2 device
lvremove --force /dev/volg-OLD/lvol-OLD
vgremove --force volg-OLD
pvremove /dev/loop1
udevadm settle
sleep 10

# The bug happens now
cat /mntvol/file_exists.txt
journalctl --since -30s _PID=1 -f

@amg1127

This comment has been minimized.

Copy link
Author

amg1127 commented Mar 27, 2018

There is a downstream bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1494014

I brought the bug upstream because I can reproduce the behavior on a fully-updated Arch Linux installation.

@zdzichu

This comment has been minimized.

Copy link
Contributor

zdzichu commented Mar 28, 2018

Are your configurations files (/etc/fstab) fully up-to-date and have you issued daemon-reload after changing the configuration?

@amg1127

This comment has been minimized.

Copy link
Author

amg1127 commented Mar 30, 2018

Yes, my configuration files were up to date when the bug hit me. No, I haven't issued a systemctl daemon-reload nor modified my /etc/fstab file during the filesystem migration, because neither the device path nor the mount point nor filesystem options referred by /etc/fstab was changed. Besides, when I used to manage sysvinit, I had never executed init q after filesystem migrations between LVM2 devices and had never seen the init process unmounting devices mounted manually.

My bug trigger script actually modifies /etc/fstab file. However, the change is for demonstration purposes and the script issues a systemctl daemon-reload immediately after that change.

I figured that a second systemctl daemon-reload invocation before removal of the old LVM2 device avoids the bug. However, I am still wondering why it is needed, because no configuration files related to systemd is being changed.

@poettering

This comment has been minimized.

Copy link
Member

poettering commented Apr 10, 2018

I am sorry, but if you have issues with LVM2, please first contact your downstreams and the LVM community. We have no experience with this upstream, and cannot really help you with that.

If there's an issue you can reproduce without LVM involved, please reopen. But until then this is simply not the right audience for these issues, we can't help you...

Sorry!

@poettering poettering closed this Apr 10, 2018

@rmetrich

This comment has been minimized.

Copy link
Contributor

rmetrich commented Jul 6, 2018

@poettering Hello, please re-open this issue. This has nothing to do with LVM.

We have a Red Hat BZ associated already.

I can confirm that this issue has nothing to do with LVM, I can reproduce with iscsi (see below for reproducer).

Also, the issue only reproduces if filesystem is in /etc/fstab AND there ISN'T the "noauto" flag.

I can reproduce on RHEL7 and Fedora 27.

Reproducer using iscsi, with 2 Targets, 1 Lun in each, so that a target can be deleted to mimic switching between targets (e.g. Disaster Recovery scenario or migration).

Server (was using RHEL7):

  • copy saveconfig.json to /etc/target

  • create disks:

    truncate -s 200M /tmp/disk1.img
    truncate -s 200M /tmp/disk2.img

  • restart "target" service:

    systemctl restart target

  • stop firewall for convenience:

    systemctl stop firewall

Client (Fedora 27 or RHEL7):

  • copy initiatorname.iscsi to /etc/iscsi/

  • discover the targets (XXX == iscsi server):

    iscsiadm -m discovery -t st -p XXX

  • attach the targets (replace XXX by what was found above):

    iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.XXX.x8664:sn.c0e14b4d5602 -l
    iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.XXX.x8664:sn.f6f77528fc0b -l

  • format luns:

    mkfs.xfs -L DISK1 /dev/disk/by-id/scsi-3600140529f7fada8dfd43babba397b96
    mkfs.xfs -L DISK2 /dev/disk/by-id/scsi-360014051f1fbab5955b4facafb2a36fc

Now, run the usual scenario. For convenience, I did the following:

  1. Edit /etc/fstab and add line:

    LABEL=DISK1 /mnt xfs defaults 0 0

  2. Fake a normal boot

    systemctl daemon-reload; mount /mnt

  3. Unmount /mnt

    umount /mnt

  4. Edit /etc/fstab and change /mnt line to use second disk:

    LABEL=DISK2 /mnt xfs defaults 0 0

  5. Detach the 1st target with lun DISK0 (replace XXX by what was found above):

    iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.XXX.x8664:sn.c0e14b4d5602 -u

  6. Check that "systemctl status mnt.mount" complains about reloading (don't reload of course)

    systemctl status mnt.mount | grep Warning
    Warning: mnt.mount changed on disk. Run 'systemctl daemon-reload' to reload units.

  7. Mount the new disk (success):

    mount /mnt

Mount point gets automatically unmounted until systemctl daemon-reload is performed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.