Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Stopped (with error) /dev/dm-1 #1620
Comments
and others
|
You could please boot with "systemd.log_level=debug", then reboot and shutdown, and provide us with a longer excerpt of the logs this generates, around the issue you are trying to debug? This should provide us with more information about what's precisely failing there. |
poettering
added
pid1
needs-reporter-feedback
labels
Oct 20, 2015
rumpelsepp
commented
Oct 20, 2015
|
sure, will look if I have a few minutes time for that this afternoon. |
rumpelsepp
commented
Oct 20, 2015
|
Enough? |
Devices do not even have stop method so failure is pretty much expected here; but why it attempts to stop device in the first place? Can you make full log available? |
rumpelsepp
commented
Oct 20, 2015
|
I prefer to send it to an email address, but yeah, I can do. |
rumpelsepp
referenced this issue
Jan 18, 2016
Closed
[dm-crypt with LUKS] error when unmounting on shutdown #2351
igalic
commented
Mar 31, 2016
|
i just did a reboot with debug enabled. here are 12k lines of glorious log (it starts from logind noticing that we're rebooting, but i can give you more if necessary): https://gist.github.com/e032493159fbc8cda780f773c261dc03 |
poettering
added
the
cryptsetup
label
Apr 18, 2016
db0451
commented
May 30, 2016
•
Edited 1 time
-
db0451
May 30, 2016
-
May 30, 2016
|
I get this too, for dm-0 via ecryptfs-setup-swap, on Debian unstable. Am I correct to infer that this is harmless and the fix is to deactivate a nonsensical 'stop device' service somewhere? How would I do that? Should we file bugs somewhere else against this? thanks |
djgera
commented
Oct 14, 2016
|
I get this "Stopped (with error) /dev/mapper/home" if fstab line uses UUID=, otherwise no error if /dev/mapper/home is used. |
saintger
commented
Oct 21, 2016
|
On Debian Stable, with systemd 230-7~bpo8+2, I got this error as well on a /home in RAID1 configuration on top on dm-crypt/luks. Using only UUID designations in /etc/fstab and /etc/crypttab I got the errors: So for me the workaround doesn't work. |
denibertovic
commented
Oct 26, 2016
@db0451 did you find out anything regarding your question? I'm also interested if this is harmless of in fact something that I should look into ASAP? |
db0451
commented
Oct 26, 2016
|
@denibertovic The only info I have on this issue is this thread. This continues to occur for me, though admittedly my system seems to work OK anyway. But it is always worrying to see red ERROR text anywhere, so if this is ultimately a non-issue and can/should be worked around, then I wish it would be. |
iKarith
commented
Oct 26, 2016
|
I did not find it harmless. I found my disks would be dirty on the next boot often because they did not unmount cleanly, with errors introduced that required fixing. And this is really difficult to fix on systemd systems because even in "single user" mode, you can't really remount your root directory ro. And that's when I determined that LUKS + bcache + systemd were not, for the moment, something I should be trying to use on the same system. |
saintger
commented
Oct 26, 2016
|
@iKarith Thanks for the info. As a workaround, I will therefore unmount manually, before shutting down (or include an unmount script somewhere). |
ledoge
commented
Oct 28, 2016
|
I also encounter |
camoz
commented
Nov 18, 2016
|
I too have that issue since one year. |
urbie-mk2
commented
Nov 27, 2016
|
I have the same use case as @ledoge has. I already temporary tried to use device names (sda5) in the fstab / crypttab files. The same error message appears. Here is the log regarding this error during shutdown. |
mnfdev
commented
Nov 28, 2016
|
I also had the "Stopped (with error) /dev/dm.." problem with encrypted swap, then found that "generator" doesn't add deps to swap unit in /var/run/systemd/generator. Instead of crypttab/fstab entries I just wrote simple .service and .swap units and now it stops w/out error. Examples from my local computer (HITACHI-SWAP - LVM volume, HITACHI-SWAP-OPEN - encrypted swap on HITACHI-SWAP): /etc/systemd/system/systemd-cryptsetup@HITACHI\x2dSWAP\x2dOPEN.service: /etc/systemd/system/dev-mapper-HITACHI\x2dSWAP\x2dOPEN.swap: (added to the default swap.target with |
siwyd
commented
Nov 30, 2016
|
Same issue here with an encrypted swap partition. |
RalfJung
commented
Dec 6, 2016
|
I am also seeing these errors on Debian testing on two systems with root and /home on an encrypted LVM. One also has swap on that LVM, one has no swap. @poettering Why is the tag "needs-reporter-feedback" still set? Which feedback is missing? I'd be happy to provide more data if helpful. |
pdefreitas
commented
Dec 6, 2016
•
Edited 1 time
-
pdefreitas
Dec 6, 2016
-
Dec 6, 2016
|
same issue here with encrypted swap partition, ubuntu 16.04. |
marchom
commented
Dec 6, 2016
|
same on ubuntu 16.10 |
mayosemmel
commented
Dec 10, 2016
|
same error on Debian 8.6 |
sky-lake
commented
Dec 15, 2016
|
Also having this issue on ubuntu 16.10 / systemd 231 / LVM on LUKS on mdadm/Mirrored Raid 1 Can a project member please remove the "needs-reporter-feedback" tag? Not sure if this tag is preventing developers from getting started on this bug, but I believe all the necessary info has already been supplied. If not, please let us know how we can help. |
|
Well, we do not know what info was requested ... :) As far as I can tell, the reason for this error message is the fact, that crypto container device |
DzamoNorton
commented
Dec 15, 2016
|
@arvidjaar, given your above diagnosis, do you know how we can fix or work around this problem? |
Vrihub
commented
Jan 15, 2017
|
FYI: I'm having the same problem on all my LUKS encrypted devices (the underlying devices are LVM logical volumes), on Debian stretch, systemd version 232-8. I don't have UUIDs in /etc/fstab, so the workaround discussed above doesn't work in my case. |
eturner
commented
Feb 1, 2017
•
Edited 1 time
-
eturner
Feb 1, 2017
-
Feb 1, 2017
|
This is a brand new error for me. I just got done moving everything around, and ended up with three LUKS volumes on top of LVM for /boot, /, and swap, so I can use the lvmcache on / without having everything encrypted twice as with two LUKS volumes. |
dejadejoder
commented
Feb 6, 2017
|
Same error on a fresh Debian Stretch install and a crypted swap... |
|
@poettering Seems I can reliably reproduce the issue in a VM which was setup with a default LVM+luks-cryptsetup configuration generated by the Debian installer. It uses two partitions: /dev/vda1 →/boot Then LVM on top From the shutdown log, systemd tries to stop the cryptsetup device /dev/dm-0 for some reason. This is bound to fail obviously, as it contains all the still active file systems. I'm happy to provide more targetted information. |
This cannot be right. You cannot (try to) stop crypto container before unmounting and stopping all filesystems and logical volumes on top of it. Some dependency is apparently missing here. Could you provide |
|
@arvidjaar Attached the |
wavexx
commented
Feb 12, 2017
|
On Sun, Feb 12 2017, Andrei Borzenkov wrote:
[ OK ] Stopped target Local File Systems.
Unmounting /var...
Unmounting /boot...
Unmounting /tmp...
Unmounting /home...
...
***@***.***_crypt.service: Control process exited, code=exited status=1
[ OK ] Stopped Cryptography Setup for vda5_crypt.
***@***.***_crypt.service: Unit entered failed state.
***@***.***_crypt.service: Failed with result 'exit-code'.
```
This cannot be right. You cannot (try to) stop crypto container before
unmounting and stopping all filesystems and logical volumes on top of
it. Some dependency is apparently missing here.
Michael: is this sequencing similar also in your test case?
Could you provide `systemctl show *`?
|
This looks truncated in some strange way. |
OK, so my best guess is - you call But for all I can tell, devices do not have At least this is the only code path I can find that can result in |
wavexx
commented
Feb 12, 2017
|
On Sun, Feb 12 2017, Andrei Borzenkov wrote:
https://gemex.eurac.edu/dl/?t=91da60f4461ff75f0821f31256dc7136
This looks truncated in some strange way.
In which way? I checked again and that's all I get from 'show *'.
|
And it is trivial to reproduce now by doing And, BTW, |
It contains too little information, some information does not belong to any unit, etc. |
wavexx
commented
Feb 12, 2017
|
On Sun, Feb 12 2017, Andrei Borzenkov wrote:
In which way?
It contains too little information, some information does not belong to any unit, etc.
Well, sounds like systemd is in an inconsistent state then?
Can you point out something that I can debug?
|
wavexx
commented
Feb 12, 2017
|
On Sun, Feb 12 2017, Andrei Borzenkov wrote:
In which way?
It contains too little information, some information does not belong to any unit, etc.
Ok, I rebooted the vm and did 'show *' again.
https://gemex.eurac.edu/dl/?t=d9eaebc024a3e8f4944c68fac642dff1
If you find something incomplete, could you please point out one
example?
|
3point2
commented
Feb 12, 2017
|
I am running Arch Linux with an encrypted swap. My issue is exactly as described by @rumpelsepp's inital report and I've been following this issue for a while. I don't see "Reloading" in my logs as @mbiebl does. The relevant lines around the error message on shutdown are here: shutdown-swap.txt and the full logs from boot to shutdown are here: boot-shutdown.txt (both with My setup: Finally, here is the output of |
|
@evverx the Reloading bit is a red herring from what I can tell. It's caused by an if-up.d hook which is run when you run ifdown for a network interface. |
|
Sorry, this was supposed to go to @arvidjaar |
|
shmerl
commented
Mar 6, 2017
|
I'm affected by this issue as well (current Debian testing). What is the safe way of shutting down / restarting to work around this bug? I don't want to potentially lose any data on each reboot. |
db0451
commented
Mar 6, 2017
|
I wasn't aware there was any evidence of this issue having any possible risk to data. Rather, it seems to be a purely cosmetic/semantic problem of systemd trying to stop a device, which does not make sense. |
shmerl
commented
Mar 6, 2017
•
Edited 1 time
-
shmerl
Mar 6, 2017
-
Mar 6, 2017
|
@db0451 : See this comment. From it, it appears that there can be potential data loss. Can anyone please confirm whether these errors are safe or not? |
jodumont
commented
Mar 13, 2017
•
Edited 1 time
-
jodumont
Mar 13, 2017
-
Mar 13, 2017
|
I have 2systems running with Debian testing For the one with the issue:
4. But wait: At the beginning I didn't had this issue it must be a services.
So what the difference ?What I understand is it was unmounting / before /usr and /var and didn't like it. So far so good I don't have more explanation and hope it will help you to resolve your issue. |
TomAshley303
commented
Apr 8, 2017
|
I have this error on each shutdown with my m.2 drive. I think it's causing my m.2 to show unsafe shutdowns. smartmontools shows: Unsafe Shutdowns: 156. |
Olotila
commented
Apr 23, 2017
•
Edited 1 time
-
Olotila
Apr 23, 2017
-
Apr 23, 2017
|
I get this on Ubuntu Gnome 17.04 with encrypted home folder, after shutdown. I get pass this only if I press Ctrl + Alt +F7: |
shahn
commented
Apr 24, 2017
|
@poettering please remove the needs-reporter-feedback label on this or state which kind of additional feedback is needed. |
teddiegp
commented
Apr 25, 2017
|
I'm affected by this too, running Kubuntu 17.04 (Linux 4.10.0-19-generic). As I've got 16GB of RAM and don't use hibernation, I don't have any swap activated or any partition for it at all. My fstab already uses /dev/mapper names and no UUIDs for the "normal" partitions / and /home, which are both encrypted (but it does use UUIDs for /boot and /boot/efi). Is there any other workaround on this? At all, it seems this has potential for loss of data. Removing encryption shouldn't be the only way to ensure that disks unmount properly and won't crash. Especially not if this is known for about a year and a half. :/ |
joachimneu
commented
Apr 27, 2017
|
Same problem here, with Arch linux. I have the following setup: harddrive -> crypto -> filesystem, so no LVM and the like. Interestingly, the problem did not occur up to recently, where I replaced the drive and went from ext4 to btrfs. Everything else is exactly the same setup (UUID/no-UUID in fstab/crypttab, ...). Also, looking at my logs, I see that the commands are in fact issued in the right order (first unmount filesystems, then close crypto device), but closing the crypto device is attempted before unmount succeeded. So maybe this is a race condition like issue? |
Saroumane
commented
Jun 6, 2017
|
Same problem here, Ubuntu 17.04, LUKS, no UUID used. Can someone confirm that it is only cosmetic ? No risk of data loss ? |
wavexx
commented
Jun 7, 2017
|
On Tue, Jun 06 2017, Saroumane wrote:
Same problem here, Ubuntu 17.04, LUKS, no UUID used.
Can someone confirm that it is only cosmetic ? No risk of data loss ?
As I wrote, the filesystem is *not* unmounted cleanly in my case.
The issue is by far not cosmetic.
|
Saroumane
commented
Jun 7, 2017
|
Thanks for the reply. |
ledoge
commented
Jun 8, 2017
•
Edited 1 time
-
ledoge
Jun 8, 2017
-
Jun 8, 2017
|
I do not get the error anymore using systemd 233. |
wavexx
commented
Jun 8, 2017
|
On Thu, Jun 08 2017, ledoge wrote:
I do not get the error anymore in systemd 233.
I'm on systemd 233 (233-8 from debian unstable) and I still have the
issue. And it's pretty obvious as well, since the crypto block device is
stopped before unmounting the filesystem.
The sequencing of the units is unchanged to the previous versions.
I posted the console log and journal already.
But I have a very hard time reproducing the issue reliably. On some
systems I reformatted with the same disk layout I don't have the issue.
I suspect that many people simply do not notice, causing the issue to be
underreported. As a journaling FS such as ext4 or xfs will recover
quickly, often too quickly to be spotted during boot, so you probably
won't notice until a file is zeroed, lost or corrupted.
The most noticeable issue is that the journal is not synced and becomes
corrupted on shutdown, making debugging even harder on physical
hardware. I simply lost the patience to debug this.
I currently run a sync(1) job before shutdown.target.
|
ioguix
commented
Jun 8, 2017
Oh, thank you, I had such a corruption some time ago in some local files, but I totally forgot about this issue and did not thought it could come from there... I will probably not be able to investigate on this old corruption now though :/ |
Saroumane
commented
Jun 8, 2017
Could you please describe this workaround ? Which script do you create/modify ? |
wavexx
commented
Jun 12, 2017
|
On Fri, Jun 09 2017, Saroumane wrote:
Could you please describe this workaround ? Which script do you create/modify ?
Create a unit which has at least the following:
DefaultDependencies=no
Requires=shutdown.target final.target
After=shutdown.target
Before=final.target
Type=oneshot
ExecStart=/bin/sync
although you might still need tweak the dependencies (specially After)
in order to delay the invocation as much as possible. Since
umount.target doesn't actually unmount the filesystem in this case, you
don't need to worry about the filesystem not being available (if it
isn't, then umount succeeded and sync is useless).
Please bear in mind this is not a true workaround: you're just calling
sync to flush the buffers as late as possible. The filesystem will
*still* not be cleanly unmounted, the journal will *still* appear as
corrupted.
|
saintger
commented
Jun 12, 2017
|
For me I'm using the following workaround service with my encrypted /home: It works except if someone is accessing /home (tracker for instance). I may have to somehow force the unmount if I don't find another solution. |
clehner
commented
Jun 16, 2017
•
Edited 1 time
-
clehner
Jun 21, 2017
-
Jun 21, 2017
|
I have been experiencing this bug for some time on Raspbian 9 with encrypted root partition. Every time I boot the machine, i have to wait to for it to fsck the root partition that got uncleanly unmounted during shutdown. This is regardless of using UUID or the device name in fstab. I am going to try using a workaround service like @saintger suggested, to try to remount the partition as read-only ("mount -o remount,ro /") before the machine powers off. Edit: it didn't work. |
rahmnathan
commented
Jun 19, 2017
|
Same problem here since updating to Ubuntu Gnome 17.04. Full-disk encryption on 3 separate machines and it happens on all of them. |
Saroumane
commented
Jun 28, 2017
|
It seems that Debian 9 has same problem, but Fedora 25 is unaffected. |
petvoigt
commented
Jun 28, 2017
|
I am using Debian 9 now for about one week on various machines including virtual ones. Three of the machines have an encrypted swap partition and they all do show the described issue on shutdown while unmounting swap: "Stopped (with error) /dev/dm-0". Besides that those three machines have an encrypted home partiton either a regular partiton or a raid1. So far only one of them shows a related error on shutdown when /home umounts. Some background: All machines use GPT partition scheme and dm-crypt/LUKS encrypted home is always mounted manually on the command line. During the next weekend I am going to upgrade my last physical machine from Jessie to Stretch and I am interested to know how this machine will behave. I will further investigate the issue and will report back here. |
shmerl
commented
Jun 29, 2017
|
This issue is gone for me (current Debian testing). |
denibertovic
commented
Jul 4, 2017
|
I'm not seeing the issue anymore. |
petvoigt
commented
Jul 5, 2017
Current versions of systemd and kernel of Debian Stretch are: Linux kirk 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux Where did you install your systemd from? |
shmerl
commented
Jul 5, 2017
|
Debian testing has systemd 233-9 and kernel 4.11.6. |
petvoigt
commented
Jul 5, 2017
|
@shmerl: Thanks, I will try kernel and systemd from testing in a VM. But I hope, the Debian Stretch team will either fix systemd232-25 or upgrade it. |
Grinderz
commented
Jul 5, 2017
|
error gone on Arch |
|
See my comment in #5527 (comment) Posting it here, just in case someone else runs into this problem as well" Why exactly this fixes those errors on shutdown is unclear to me. |
wavexx
commented
Jul 5, 2017
|
On Wed, Jul 05 2017, Michael Biebl wrote:
Why exactly this fixes those errors on shutdown is unclear to me. As
for Debian Stretch and v232: This patch seems too invasive to be
backported for stretch
I wrote it before, I'm already on v233, but the filesystem is still not
unmounted cleanly in my case.
|
zifxify
commented
Jul 6, 2017
|
@mbiebl |
keszybz
removed
the
needs-reporter-feedback
label
Jul 9, 2017
|
@mbiebl I tried reproducing the bug using your recipe from #1620 (comment). I used the normal stretch amd64 installer, and got the same partition layout that you did, except with sda instead of vda, but I doubt that matters. It has systemd-232-5 and shuts down cleanly. Can you maybe upload the image that exhibits the issue somewhere where I can download it? -- Looking at the dependency graph, systemd-cryptsetup@vda5_crypt.service has I did Essentially, everything fails to stop, because stuff is mounted and devices are used, but systemd doesn't seem to know that. |
stop job for device just waits for device to "disappear" so unless timeouts are set very low or cryptsetup takes very long it should not matter in this case. This error is returned when "stop" job for device completes without timeout but device state at time of completion is not "stopped" (or device is still present in system from systemd PoV). One possible cause is systemd reload as I have demonstrated earlier. This is a bug that needs fixing. It is possible that some device event may cause premature job completion, I could not find it by code review. All those countless "me too"s do not help - we are facing race condition so unless there is clearly understood root cause with deterministic reproducer it is impossible to claim it has been fixed. Relative timings may change from boot to boot, not to mention from version to version. |
added a commit
to keszybz/systemd
that referenced
this issue
Jul 10, 2017
|
@arvidjaar it's likely that we're dealing with more than one bug here. I started working on patch to test my theory about missing Before= dep, but now I see that won't work, because Before=*.device deps are not allowed. But I think I might be onto something — I don't see how systemd should know that the systemd-cryptsetup@.service should stay around until all devices it creates are unmounted. Afaics, there's no ordering that goes |
|
@keszybz I can no longer reproduce it |
This issue is about quite specific error message. Users see this error message in conjunction with cryptsetup simply because cryptsetup is one of very few cases when device is (attempted to be) actively stopped. But cryptsetup itself does not cause this error. I completely agree that we have rather bad situation with proper ordering of units here but this needs its own report that makes it obvious, not buried in this hundred pages long one. @mbiebl I can still easily trigger this error using 233 on openSUSE Tumbleweed. If you mean you do not see it during real shutdown - well, that's the nature of all races ... :) |
rumpelsepp
commented
Jul 19, 2017
•
Edited 1 time
-
rumpelsepp
Jul 19, 2017
-
Jul 19, 2017
Problem seems to be gone now, at least on my machine. Since some update the error disappeared. I am on debian testing/sid: |
DarkCaster
commented
Aug 26, 2017
|
I have this issue with fresh installation of openSUSE 42.3 in my production servers. |
jugovic
commented
Oct 12, 2017
•
Edited 1 time
-
jugovic
Oct 12, 2017
-
Oct 12, 2017
|
I have the same issue, using Ubuntu Gnome 17.04 .................. Absolutely same thing. Crypted Partition and Swap issue..... Is there any solution (REAL SOLUTION) for this problem?? I will gladly pay someone to make it... if someone would like to do so. All the best & please help folks |
zifxify
commented
Oct 14, 2017
|
Tried systemd 234-3~bpo9+1 and 232-25+deb9u1, same results |
vsespb
commented
Nov 5, 2017
|
happens in ubuntu 16.04 (only visible in logs!) and debian 9 (either delay 90 seconds or red lines in logs). is there any workaround to minimize chance of data loss ? i.e. sync disks? |
wavexx
commented
Nov 5, 2017
|
On Sun, Nov 05 2017, Victor Efimov wrote:
happens in ubuntu 16.04 (only visible in logs!) and debian 9 (either
delay 90 seconds or red lines in logs).
I discovered while testing that when using dracut instead of
initramfs-tools, the fsck output is completely hidden from the console.
Lovely.
|
vsespb
commented
Nov 6, 2017
|
At least is there way to setup new system with LUKS, with workaround for the issue making it safe against data-loss: for example maybe use separate partitions for root, /var and /tmp, and remount root as readonly (or it's already remounted?) before shutdown? |
|
I'm not aware that this is actually causing data loss |
wavexx
commented
Nov 6, 2017
|
On Mon, Nov 06 2017, Michael Biebl wrote:
I'm not aware that this is actually causing data loss
Isn't that implicit when the FS in not unmounted cleanly?
Sure, the FS is journaled and recovers quickly, but data which has not
been synced will get lost. The last journal is regularly moved away
because of this, with last entries being missing.
With recent systemd versions there's no error message anymore in the
console output. However, the problem persists: the FS is not unmounted
or remounted=ro before shutdown.
It's still a mystery to me why some systems that I've setup with
identical cryptsetup and partition layout exhibit this problem, while
others don't.
|
|
@wavexx where exactly das systemd fail to unmount a filesystem or remount it ro? |
wavexx
commented
Nov 6, 2017
|
On Mon, Nov 06 2017, Michael Biebl wrote:
@wavexx where exactly das systemd fail to unmount a filesystem or
remount it ro?
See my old journal/log that I've posted as well the old posts.
Aside from the fact that the error is gone from the console, the issue
still persist today.
The net result is that during boot the filesystem is found dirty just
after cryptsetup mounts it. I wrote already earlier that since the fsck
is quick and the output can be hidden by dracut as well, many people
will *not* notice that this is happening at all.
|
|
@wavexx which log shows that the file system is not unmounted? I don't see a complete log anywhere which shows that the file system was not unmounted |
wavexx
commented
Nov 7, 2017
|
On Mon, Nov 06 2017, Michael Biebl wrote:
@wavexx which log shows that the file system is not unmounted? I don't
see a complete log anywhere which shows that the file system was not
unmounted
The console doesn't show any error that this is happening, except the
mount error on the next boot.
I couldn't manage to save the journal on a remote system where where
this is happening all the time (the system is under a different
network), and on the system itself the journal is corrupted.
Turning on debug on the console doesn't show anything useful either.
There's no dump of commands as being executed, which would be incredibly
useful.
|
|
@wavexx please follow the instructions at https://freedesktop.org/wiki/Software/systemd/Debugging |
rumpelsepp commentedOct 20, 2015
I am using full disk encryption and an encrypted swap partition. On every shutdown I get the following Error: