New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suspend/resume breaks chroot when running off external SDC card #288

Open
stsquad opened this Issue Jul 28, 2013 · 138 comments

Comments

Projects
None yet
@stsquad
Contributor

stsquad commented Jul 28, 2013

On my Chromebook Pixel it looks like a suspend/resume cycle leaves the chroot in an odd state.

04:14 alex@localhost/x86_64 [] >echo "before suspend"
before suspend
04:14 alex@localhost/x86_64 [
] >echo "after suspend"
after suspend
-su: history: /home/alex/.bash_history: cannot create: Input/output error
11:15 alex@localhost/x86_64 [~] >

Looking at the dmesg it looks like there are ext4 errors on the SDC card which forces a re-mount breaking the chroot (or possibly the SDC is just unmounted as part of the suspend cycle).

ss Generic STORAGE DEVICE 0207 PQ: 0 ANSI: 0
[15015.434666] sd 12:0:0:0: [sdf] 125827072 512-byte logical blocks: (64.4 GB/59.9 GiB)
[15015.435935] sd 12:0:0:0: [sdf] Write Protect is off
[15015.435946] sd 12:0:0:0: [sdf] Mode Sense: 0b 00 00 08
[15015.437146] sd 12:0:0:0: [sdf] No Caching mode page present
[15015.437156] sd 12:0:0:0: [sdf] Assuming drive cache: write through
[15015.441164] sd 12:0:0:0: [sdf] No Caching mode page present
[15015.441174] sd 12:0:0:0: [sdf] Assuming drive cache: write through
[15015.448779] sdf: sdf1
[15015.452132] sd 12:0:0:0: [sdf] No Caching mode page present
[15015.452142] sd 12:0:0:0: [sdf] Assuming drive cache: write through
[15015.452151] sd 12:0:0:0: [sdf] Attached SCSI removable disk
[15016.731447] EXT4-fs (sdf1): recovery complete
[15016.736860] EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts:
[15024.898066] wlan0: no IPv6 routers present
[15029.857907] EXT4-fs error (device sde1): ext4_find_entry:935: inode #3020490: comm bash: reading directory lblock 0
[15029.857941] EXT4-fs error (device sde1): ext4_read_inode_bitmap:171: comm bash: Cannot read inode bitmap - block_group = 368, inode_bitma
p = 12058640
[15029.857954] EXT4-fs error (device sde1) in ext4_new_inode:895: IO failure
[15029.857993] EXT4-fs error (device sde1): ext4_find_entry:935: inode #2884078: comm bash: reading directory lblock 0
[15029.858071] EXT4-fs error (device sde1): ext4_find_entry:935: inode #2883586: comm bash: reading directory lblock 0
[15029.858090] EXT4-fs error (device sde1): ext4_find_entry:935: inode #2883586: comm bash: reading directory lblock 0
[15035.044833] Aborting journal on device sde1-8.
[15035.044845] Buffer I/O error on device sde1, logical block 7372800
[15035.044851] lost page write due to I/O error on sde1
[15035.044857] JBD2: Error -5 detected when updating journal superblock for sde1-8.
[15092.013341] EXT4-fs error (device sde1): ext4_find_entry:935: inode #3020490: comm bash: reading directory lblock 0
[15092.013363] EXT4-fs error (device sde1): ext4_find_entry:935: inode #2884078: comm bash: reading directory lblock 0
[15092.013395] EXT4-fs error (device sde1): ext4_find_entry:935: inode #2883587: comm bash: reading directory lblock 0
[15092.013410] EXT4-fs error (device sde1): ext4_find_entry:935: inode #2884078: comm bash: reading directory lblock 0
[15092.014147] EXT4-fs error (device sde1): ext4_journal_start_sb:328: Detected aborted journal
[15092.014160] EXT4-fs (sde1): Remounting filesystem read-only
[15094.359077] EXT4-fs error (device sde1): ext4_find_entry:935: inode #3020490: comm bash: reading directory lblock 0

@stsquad

This comment has been minimized.

Contributor

stsquad commented Jul 29, 2013

Looking at enter-chroot I see it does attempt to prevent powerd from persisting usb mounts. I wonder if it's the same sort of thing?

@dnschneid

This comment has been minimized.

Owner

dnschneid commented Jul 29, 2013

Yeah, enter-chroot tries to prevent powerd from disabling usb mount persistence, but it only seems to work part of the time for me.

Is it failing if you let the system idle suspend, or when you close the lid, or both?

@stsquad

This comment has been minimized.

Contributor

stsquad commented Jul 30, 2013

Both as far as I can tell - certainly the file-browser keeps popping up after both. Looking at the sed statement I don't think anything is being patched now, possibly the powerd script has been updated?

@stsquad

This comment has been minimized.

Contributor

stsquad commented Jul 31, 2013

@dnschneid

This comment has been minimized.

Owner

dnschneid commented Jul 31, 2013

Cool, then that should be fixable by adding the line to the script in a reasonable location.

@stsquad

This comment has been minimized.

Contributor

stsquad commented Jul 31, 2013

Even simpler I think we can just get away with enabling USB persist as we enter the chroot and skip the copy/mount powerd stuff as it's not going to attempt to turn it off anyway. I'll have a play.

@dnschneid

This comment has been minimized.

Owner

dnschneid commented Jul 31, 2013

Good call. We just need to make sure that patch has landed in the stable channel for all platforms before removing the copy/mount hack stuff.

@dnschneid

This comment has been minimized.

Owner

dnschneid commented Jul 31, 2013

Yep, looks like that patch has landed in stable.

stsquad added a commit to stsquad/crouton that referenced this issue Jul 31, 2013

Change handling of USB persistence (fixes dnschneid#288)
In the old way of doing things ChromeOS would turn off device
persistence to save a few milliseconds of suspend time. Crouton had to
jump though hoops to stop it doing this if the chroot was on a USB
device.

Since upstream merged commit: 0007a546853a529064772b1b96bd9164dece8c46
into power_manager.git and defaulted the kernel to have USB persistence
off all we now need to do is turn it on when we detect a chroot in
/media (where USB devices are mounted).

Arguably you could be smarter about which USB devices to enable but at
least by doing them all you ensure nothing breaks and it works.

@dnschneid dnschneid closed this in 112caeb Aug 1, 2013

@dnschneid

This comment has been minimized.

Owner

dnschneid commented Nov 8, 2013

This fix may have stopped working.

@stsquad

This comment has been minimized.

Contributor

stsquad commented Nov 8, 2013

I've noticed file-manger popping up (a sign the remount has been detected). I've yet to notice a crouton session getting killed.

@mark0978

This comment has been minimized.

mark0978 commented Nov 8, 2013

On my Acer C710, its happening all the time right now. The only way my chroot lives across sleep/resume is on the internal storage. This effectively renders the SD Card useless.

@phoenix00a

This comment has been minimized.

phoenix00a commented Nov 12, 2013

So, I've got to forget about using my SD Card and install crouton internally? Yikes. I've only got 16GB on my HP Pavilion 14 Chromebook..

@mark0978

This comment has been minimized.

mark0978 commented Nov 12, 2013

Yea, it sucks Same on the Acert C710

@dnschneid

This comment has been minimized.

Owner

dnschneid commented Nov 12, 2013

Yeah, major suck. I've looked through kconfig commit logs to see if anything other than the default persistence setting was changed, but I didn't find much, assuming this isn't buggy. I'll need to ask one of the kernel guys what's stopping USB persist from working.

Of course, I've done this checking without actually confirming that the crouton code is still doing what it's supposed to be doing. If anyone wants to check it before me, make sure the echo 1 line in enter-chroot is getting called, and the echo 0 line in unmount-chroot isn't being called at an incorrect time.

@dnschneid

This comment has been minimized.

Owner

dnschneid commented Nov 13, 2013

Try the latest crouton; I restructured the mount system to fix another issue, and I think it might have resolved this as well.

@dnschneid

This comment has been minimized.

Owner

dnschneid commented Nov 13, 2013

Hmm, it looks like it's persisting, but there are journal errors upon resume so it gets remounted RO...unless that's just my SD card.

@dnschneid

This comment has been minimized.

Owner

dnschneid commented Nov 13, 2013

Okay, something's wrong with superblock access which is causing the device to be error-mounted RO when replaying the journal. I'll investigate what is causing this in Chromium OS. In the meantime, you have two options:

  1. Set the partition to ignore errors and mount anyway. This is such a terrible idea that I'm not even going to post the command here.
  2. Disable journaling. This significantly increases the risk of data loss when things go poorly, so it's still a pretty bad idea. sudo umount /dev/sdb1 && sudo tune2fs -O ^has_journal /dev/sdb1 and then unplug and re-plug the SD card. You can re-enable journaling at a future date.
@dnschneid

This comment has been minimized.

Owner

dnschneid commented Nov 13, 2013

Relevant bug. Star it if you're affected.

@tocker

This comment has been minimized.

tocker commented Jan 28, 2014

Is it possible to manually remount the sdcard as RW?
I tried a few variations of mount but none worked.

@mark0978

This comment has been minimized.

mark0978 commented Jan 28, 2014

Yes you can remount it. But the suspend/remount destroys any hope of
recovery from sleep.
On Jan 28, 2014 8:59 AM, "tocker" notifications@github.com wrote:

Is it possible to manually remount the sdcard as RW?
I tried a few variations of mount but none worked.

Reply to this email directly or view it on GitHubhttps://github.com//issues/288#issuecomment-33479939
.

@NullVoxPopuli

This comment has been minimized.

NullVoxPopuli commented Mar 10, 2014

I am experiencing this issue on the HP11.

Other issues I've noticed that I think may be related:

  • When the USB drive does remount, I can't re-enter chroot. It says that the chroot is invalid, and tries to kill some processes.
  • I cannot get back in to chroot until I have restarted my computer.

my uname -a from chrome OS:
Linux localhost 3.4.0 #1 SMP Tue Feb 18 23:28:43 PST 2014 armv7l ARMv7 Processor rev 4 (v7l) SAMSUNG EXYNOS5 (Flattened Device Tree) GNU/Linux

@dnschneid

This comment has been minimized.

Owner

dnschneid commented Mar 27, 2014

The relevant crbug has a request for someone to reproduce and provide an event log. Any takers?

@DennisLfromGA

This comment has been minimized.

Collaborator

DennisLfromGA commented Jan 20, 2017

There's crbug.com/434372 that's related to this issue, Commenter#41 mentions it affecting 'crouton'.

@choikwa

This comment has been minimized.

choikwa commented Apr 10, 2017

I'm seeing same problem as @MarcSabatella where upon suspend and resume, my SD card would be mounted with noexec. My trivial workaround is just running sudo mount -o remount,exec /media/removable/<SD>, but I would like if SD could be mounted as exec in the first place (or at least sticky this command somehow upon resume). Is this problem to be fixed in Chromium upstream and if so, beyond this GH issue?

@darren3

This comment has been minimized.

darren3 commented May 20, 2017

How do you install the above scripts?

@ChrisHodgesUK

This comment has been minimized.

ChrisHodgesUK commented May 26, 2017

I'm getting this too on a C201 (xenial and chrome 60). I'm running off a SD card as I don't have much internal storage. It's new in one of the recent chrome updates, but I'm not sure which one as chrome broke my previous chroot (also on the SD card) and I've been trying alternative approaches like dual-booting. Sometimes it remounts my SD card -- but at /media/removable SD Card 1 instead of the usual /media/removable SD Card

@ChrisHodgesUK

This comment has been minimized.

ChrisHodgesUK commented May 26, 2017

@tedm by what mechanism would flash memory fail faster if the chip is in a card rather than soldered to the motherboard?

I'd rather burn up write cycles on something cheap and replaceable than on the main storage.

@tedm

This comment has been minimized.

Contributor

tedm commented May 26, 2017

@ChrisHodgesUK I don't really know for sure. I suspect it might be that many of the SD and USB flash cards I've seen fail when running 24x7 for weeks and months were of lower quality than internal memory cells, or perhaps the fact that it's going through more interfaces (usb, pci, etc.) causes /proc, /sys reads and writes more?

@jmaris

This comment has been minimized.

jmaris commented May 29, 2017

do we have any news on this upstream ?

@andersan

This comment has been minimized.

andersan commented Jun 15, 2017

been using crouton for over a year on an external drive - have always had this issue

@xzhao025

This comment has been minimized.

xzhao025 commented Jul 28, 2017

I am using thinkpad13 and have installed crouton on sd card. I was struggling with the suspend/resume crouton not working issue. but it seems got fixed after I manually mount my /dev/mmcblk1 under a mounting point under my home folder, say ~/crouton, instead of /media/removable/xxx. I am not 100% sure but it seems chrome os is only messing up with those device mounted under removable. Hope this might help someone.

@armyofpunk

This comment has been minimized.

armyofpunk commented Aug 7, 2017

I am using thinkpad13 and have installed crouton on sd card. I was struggling with the suspend/resume crouton not working issue. but it seems got fixed after I manually mount my /dev/mmcblk1 under a mounting point under my home folder, say ~/crouton, instead of /media/removable/xxx. I am not 100% sure but it seems chrome os is only messing up with those device mounted under removable. Hope this might help someone.

sorry xzhao025 could you explain more how to mount differently? do you mean on install you installed it in a different place? thanks.

@xzhao025

This comment has been minimized.

xzhao025 commented Aug 7, 2017

hope this alias that I use to start crouton will make more sense out of my comment
alias cli='sudo umount /dev/mmcblk1 ; sudo mount /dev/mmcblk1 ~/crouton ; sudo sh ~/crouton/bin/startcli'

@DennisLfromGA

This comment has been minimized.

Collaborator

DennisLfromGA commented Aug 7, 2017

Hopefully, these new features coming to Chrome OS will help to alleviate some or all of these problems.

-DennisL

@addrummond

This comment has been minimized.

addrummond commented Aug 28, 2017

xzhao025's solution works for me. Unmount the SD card from /media/removable and mount it somewhere in your home dir. Suspend/resume then appears to work as expected.

@davidbrewster

This comment has been minimized.

davidbrewster commented Sep 4, 2017

Just wanted to also add that xzhao025's solution worked for me too on Toshiba Chromebook 2. For me the relevent commands were just slightly different:
sudo mount /dev/mmcblk1p1 ~/crouton; sudo sh ~/crouton/bin/enter-chroot
Although obviously you can use startxfce4 or whatever instead of enter-chroot.
So, thanks very much xzhao025! The problem with hanging/loss of mount after suspend had been a real pain in the neck.

@G3zz

This comment has been minimized.

G3zz commented Sep 8, 2017

@davidbrewster did you have to move or copy your chroot using edit-chroot -m?
I can mount /dev/mmcblk1p1 at ~/crouton but when I try and execute enter-chroot I am getting No chroots found in /mnt/stateful_partition/crouton/chroots.

@davidbrewster

This comment has been minimized.

davidbrewster commented Sep 8, 2017

@G3zz I'm not with my chromebook at the moment, but IIRC I may have just been running ~/crouton/bin/enter-chroot directly (if that's the right path) - [edit: In fact thats exactly what I did if I look at the second part of the command I posted earlier]

@evansharp

This comment has been minimized.

evansharp commented Oct 8, 2017

Does anyone have any recovery advice?
I've got a large Lexar SD card that was holding a chroot for my Toshiba CR2, but now it won't mount at all. I can't see the mmcblk1 items in /dev and my other linux laptop can't see it either. Really hoping this $60 card isn't bricked.
Thanks for any thoughts!

@DanPete

This comment has been minimized.

DanPete commented Oct 10, 2017

Also wanted to add that @xzhao025's solution also works if the crouton is on a USB drive too. this is the command I ended up using

sudo umount /dev/sda1
sudo mount /dev/sda1 ~/crouton
sudo sh ~/crouton/bin/startunity

I also created a folder at ~/ called crouton, but I am not sure if that was necessary or not. But, it is working very well now, and happy that I can close my Chromebook without the USB unmounting and crouton crashing or hanging.

Thanks @xzhao025!

@xzhao025

This comment has been minimized.

xzhao025 commented Oct 10, 2017

glad my solution could help! Thanks again for crouton project to make chromebook more than a browser!

@cefn

This comment has been minimized.

cefn commented Oct 10, 2017

@DanPete Can your solution be used for mapping specific directories from /dev/sda1 into a specific chroot, or can it only be used to map the common parent of all chroots from /dev/sda1? I would like to have the responsiveness and avoidance of wear by keeping the crouton chroot on my Toshiba Chromebook 2, but with /usr and /home mounted from an ext4 partition on a USB drive to provide for expansion of installed packages and documents beyond the 16G limit.

@zhaofengli

This comment has been minimized.

zhaofengli commented Oct 19, 2017

Is it necessary to unmount the drive from the original ChromeOS mountpoint for this to work? In my limited testing (tried closing the lid once), my own mountpoint isn't affected when the original ChromeOS mount is kept intact.

@Lucaacer

This comment has been minimized.

Lucaacer commented Oct 28, 2017

I have just moved crouton to an ext SD card, which I had ext4 formatted, and I can confirm that creating a crouton folder in home and mounting there the ext SD card seems to work like a charm.

chronos@localhost / $ sudo mkdir ~/crouton
chronos@localhost / $ sudo mount /dev/mmcblk1p1 ~/crouton
chronos@localhost / $ sudo sh ~/crouton/bin/startxfce4

The only caveat is that chrome OS, after suspending, cannot see the SD card any longer, but the xiwi windows is still there and the linux distro works just the same.

Update:

Tested and confirmed as working after my Lenovo thinkpad yoga 3rd gen slept 3 hours straight.

@ipstone

This comment has been minimized.

ipstone commented Nov 23, 2017

@Lucaacer

Hi Lucaacer: I am trying to do the same on the external SD card, but I found the SD card is quite slow - is that normal? I followed the instruction to format sd card to ext4 format. The same SD card in FAT format is quite fast with 50MB/sec read speed etc.

@Lucaacer

This comment has been minimized.

Lucaacer commented Nov 23, 2017

Hi, sadly my Chromebook was stolen... so I can't compare the speed! Anyway, the Linux chroot was working well and wasn't sluggish. But it was just my impression and nothing else, sorry.

@Bappy1988

This comment has been minimized.

Bappy1988 commented Mar 21, 2018

So, best way I've found to do this (on my c100p) is install in the default SD Card mount location;

sh ~/crouton -t xfce,xiwi,kbd,touch -p /media/removable/SD\ Card

let it go through and install everything (takes AGES)

when it has finished, create a new mount point:

mkdir ~/SDCARD

create a launch script. I went with ~/ubuntu, I did this with vi as that's included in chromeos' shell

#!/bin/sh
sudo umount /dev/mmcblk1*
sudo mount /dev/mmcblk1p1 ~/SDCARD
sudo sh ~/SDCARD/bin/startxfce4 -b

now, when I start up my chromebook, I launch a terminal, type shell, then

sh ~/ubuntu

it asks for my sudo password, then once typed in, it sorts the mountpoint, launches the chroot in the background and I can close the terminal and enjoy coding is vscode on ubuntu 16.04 in a xiwi window, I can close the lid of my chromebook or just leave it to go to sleep mode and not worry about my ubuntu session

@guyman70718

This comment has been minimized.

guyman70718 commented Aug 25, 2018

Is this issue still being worked on? I'm sorry for the major necro but i'm still having this issue.

@CINJ

This comment has been minimized.

CINJ commented Oct 1, 2018

Still having a similar version of the same issue: XFCE (no Xiwi) will allow sleep, however, the USB device disappears from ChromeOS as soon as XFCE is logged out and there doesn't seem any way to get the USB device back. Pull out the USB stick and ChromeOS gives a WARNING about removing the USB device! So it's still there somehow! After plugging the USB back in, the chroot is left in an invalid state and after you enter-chroot again it cleans up the chroot and you need to enter it AGAIN and then it enters the chroot as expected.

@JRHLaw

This comment has been minimized.

JRHLaw commented Oct 19, 2018

Still having a similar version of the same issue: XFCE (no Xiwi) will allow sleep, however, the USB device disappears from ChromeOS as soon as XFCE is logged out and there doesn't seem any way to get the USB device back. Pull out the USB stick and ChromeOS gives a WARNING about removing the USB device! So it's still there somehow! After plugging the USB back in, the chroot is left in an invalid state and after you enter-chroot again it cleans up the chroot and you need to enter it AGAIN and then it enters the chroot as expected.

Have also just experienced this when running the chroot off a MicroSD card -- see crosspost at https://www.reddit.com/r/Crouton/comments/9pcfa2/chroot_on_sd_card_still_works_even_when_unmounted.

How odd...

@TriVoxel

This comment has been minimized.

TriVoxel commented Nov 7, 2018

I'm having this issue now as well. I cannot find anything to fix it other than to go into the ChromeOS settings and disable sleep on lid close and suspending after inactivity. Not too good for the battery though. Even worse, if your Chromebook sleeps and you shut down Crouton it cannot access the exit-chroot script and it will inevitably lead to a corrupted installation. I had KDE and after about the tenth time plasmashell could not load anymore and eventually the whole system was unusable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment