one ZFS file system always starts with readonly=on temporary on boot

[timothyjlaurent]

I am running Ubuntu 12.04.
So I have created a zpool and put my /home directory in a zfs called volume/home.

I discovered after not being able to log in that my home directory was readonly.

Look here:

root@supaserva:/home/dadmin# zfs get all | grep readonly
volume              readonly              off                    default
volume/home         readonly              on                     temporary
volume/movies       readonly              off                    default
volume/music        readonly              off                    default
volume/pictures     readonly              off                    default
volume/vault        readonly              off                    default

So if I do zfs set readonly=off volume/home I am able to log in and everything is good until I reboot and the volume/home mounts as readonly again, but the listing in zfs get all remains the same.

Can anyone help figure out how to fix this issue?

[behlendorf]

According to the output your default value of off has been overridden by a temporary value on. Is it possible this filesystem is being mounted through fstab and the 'ro' option is being passed?

[timothyjlaurent]

I just double checked the fstab and /home isn't mounted there at all.

Another thing : I noticed that my other ZFS's weren't mounted ... had to zfs mount -a

Any thoughts on how I can fix this?

[timothyjlaurent]

After manually zfs set readonly=off <volume> and zfs mount -a at every boot for several weeks, I now had a boot without any issues.. issue closed

[timothyjlaurent]

Gonna reopen this.

I was only able to log in one time without manually setting readonly=off .

What I have tried:

putting sleep 60 before the mount command in /etc/init/mountall.conf

Adding the set readonly off after the mount command in /etc/init/mountall.conf

script
    . /etc/default/rcS
    [ -f /forcefsck ] && force_fsck="--force-fsck"
    [ "$FSCKFIX" = "yes" ] && fsck_fix="--fsck-fix"

    # set $LANG so that messages appearing in plymouth are translated
    if [ -r /etc/default/locale ]; then
        . /etc/default/locale
        export LANG LANGUAGE LC_MESSAGES LC_ALL
    fi
    exec mountall --daemon $force_fsck $fsck_fix
    zfs set readonly=off volume/home
end script

None of these helps.

One other problem : My volume resilvers at every boot, but another zpool does not.

Another thingL I have a similar setup at work 12.04 Ubuntu running zfs from ppa, with with a zfs mounted at /home ..This setup also mounts with readonly=on for the /home partition.

Maybe will try mounting with the legacy options and see if this helps.

[timothyjlaurent]

It also occurs to me that the exec mountall may be forking and the zfs set readonly... never getting called... I changed this to no avail...

this bug is super annoying.

There should be a warning "do not make your /home a zfs mount on ubuntu"

[ank]

I can confirm similar behavior with ubuntu 12.04 and zfs /home. This is after a reboot.

zfs get all | grep readonly
tank                   readonly              off                            default
tank/home              readonly              on                             temporary
tank/timemachine       readonly              off                            default

/etc/fstab

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/sda1 during installation
UUID=c072c0e6-3a25-46ea-b1aa-2f30eb614ab8 /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=553b9524-eae6-4510-8c9c-5f7f5077080f none            swap    sw              0       0

[hydro-b]

I'm experiencing the same read-only behaviour. I'm using zol since Ubuntu 12.04, issues still there with Ubuntu 14.04. No entries in /etc/fstab. This only happens for one out of three filesystems though, the other 2 have readonly off.

# zfs get readonly hydro-b/music

NAME           PROPERTY  VALUE   SOURCE
hydro-b/music  readonly  on      temporary

[DarwinAwardWinner]

I'm seeing similar behavior with /opt on my Ubuntu 14.04 system:

$ zfs get all | grep readonly
rpool                                                             readonly                    off                           default
rpool/TimeCapsule                                                 readonly                    off                           default
rpool/home                                                        readonly                    off                           default
rpool/home/ryan                                                   readonly                    off                           default
rpool/home/ryan/BTSync                                            readonly                    off                           default
rpool/home/ryan/Downloads                                         readonly                    off                           default
rpool/opt                                                         readonly                    on                            temporary
rpool/root-backup                                                 readonly                    off                           default
rpool/seafile                                                     readonly                    off                           default
rpool/transmission-daemon                                         readonly                    off                           default
$ mount
/dev/mapper/mneme-mneme on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
rpool on /rpool type zfs (rw)
rpool/TimeCapsule on /mnt/TimeCapsule type zfs (rw)
rpool/home on /home type zfs (rw)
rpool/home/ryan on /home/ryan type zfs (rw)
rpool/home/ryan/BTSync on /home/ryan/BTSync type zfs (rw)
rpool/home/ryan/Downloads on /home/ryan/Downloads type zfs (rw)
rpool/opt on /opt type zfs (ro)
rpool/root-backup on /rpool/root-backup type zfs (rw)
rpool/seafile on /srv/seafile type zfs (rw)
rpool/transmission-daemon on /var/lib/transmission-daemon type zfs (rw)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/mapper/mneme-mneme /               ext4    errors=remount-ro 0       1
# swap was on /dev/sdb1 during installation
UUID=6aeb2b6d-b6b9-48fa-8836-e4e25ce6e7f9 none            swap    sw,pri=0              0       0
UUID=dc89622c-71b8-4731-9d71-4566b91be76a none            swap    sw,pri=0              0       0
UUID=d9097ff2-e1b4-43da-b145-38d67551afdf none            swap    sw,pri=0              0       0
$ uname -a
Linux mneme.homenet.org 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.2 LTS
Release:        14.04
Codename:       trusty

Edit: In my case it just happened once and rebooting fixed it.

[UltraSalem]

Same issue for me on 14.04.02 64bit. Using the zfs in-built mounting, not fstab. Was working fine until I did a dist-upgrade a few weeks or so ago. As per OP above, if I manually set readonly=off, I can write to the pool, but the property still shows as readonly=on

$ sudo zfs get all | grep readonly
zdata readonly on temporary

$ sudo aptitude versions zfs
Package libzfs-dev:
p 0.6.3-5~trusty trusty 1001

Package libzfs-dev:i386:
p 0.6.3-5~trusty trusty 1001

Package libzfs1:
p 0.6.2-1saucy 100
p 0.6.2-2trusty trusty 1001

Package libzfs1-dbg:
p 0.6.2-2~trusty trusty 1001

Package libzfs1-dbg:i386:
p 0.6.2-2~trusty trusty 1001

Package libzfs1:i386:
p 0.6.2-2~trusty trusty 1001

Package libzfs2:
i A 0.6.3-5~trusty trusty 1001

Package libzfs2-dbg:
p 0.6.3-5~trusty trusty 1001

Package libzfs2-dbg:i386:
p 0.6.3-5~trusty trusty 1001

Package libzfs2:i386:
p 0.6.3-5~trusty trusty 1001

Package ubuntu-zfs:
i 8~trusty trusty 1001

Package ubuntu-zfs:i386:
p 8~trusty trusty 1001

Package zfs-auto-snapshot:
p 1.1.0-0ubuntu1~trusty trusty 1001

Package zfs-dkms:
i 0.6.3-5~trusty trusty 1001

Package zfs-dkms:i386:
p 0.6.3-5~trusty trusty 1001

Package zfs-doc:
i A 0.6.3-5~trusty trusty 1001

Package zfs-doc:i386:
p 0.6.3-5~trusty trusty 1001

Package zfs-fuse:
p 0.7.0-10.1 trusty 500

Package zfs-fuse:i386:
p 0.7.0-10.1 trusty 500

Package zfs-initramfs:
p 0.6.3-5~trusty trusty 1001

Package zfs-initramfs:i386:
p 0.6.3-5~trusty trusty 1001

Package zfsutils:
i 0.6.3-5~trusty trusty 1001

Package zfsutils-dbg:
p 0.6.3-5~trusty trusty 1001

Package zfsutils-dbg:i386:
p 0.6.3-5~trusty trusty 1001

Package zfsutils:i386:
p 0.6.3-5~trusty trusty 1001

[myii]

@timothyjlaurent @ank @hydro-b @DarwinAwardWinner @UltraSalem

I've been using ZFS-based home directories on a number of Ubuntu computers (10.04, 12.04 & 14.04) for around 2 years and haven't faced this problem - until yesterday.

In my case, this wasn't a bug in ZFS but rather an issue in my startup configuration - as an example, I was able to resolve the situation in the following way:

• After reboot, immediately opened up a virtual terminal (i.e. Ctrl-Alt-F1)
• Logged in as root
• Identified all the readonly datasets; in my case, $POOL/home and $POOL/home/$USER
• Attempted to unmount: zfs unmount $POOL/home/$USER - this informed me that the dataset was in use
• As suggested, used lsof /home/$USER - and this found the offending cause (which is specific to my startup):
  • There were a number of gunicorn processes running - I had recently been testing an Upstart configuration
  • After killing the processes, was able to unmount, first the $POOL/home/$USER dataset and then $POOL/home
  • Immediately mounted both datasets again (obviously home, then home/$USER) and neither was readonly
• So to test this properly, I removed the Upstart job and then rebooted the computer - no problem
• Just to make sure, have rebooted 3 times - no issues since

In essence, my suggestion is to find out if you have any startup processes that are causing this problem. The procedure above may help to identify those.

[gdevenyi]

I just ran into this today with 0.6.5.7-1~trusty

[phiresky]

I had the same issue with my root partition always mounting read only with "temporary". I don't have zfs in my fstab and haven't overridden anything, just following the instructions in the ZFS on root partition arch wiki page.

My kernel commandline was just cryptdevice=... zfs=bootfs.

Turns out the linux kernel defaults to having the root partition read only, and zfs marks this as "temporary". So it was fixed by changing the commandline to cryptdevice=... zfs=bootfs rw