ReaR Recover Issue - Unrecognized mount option, wrong fs type, bad option #555
Comments
|
@bbeaver Could you attach the |
|
Yes, thanks - please see disklayout.conf below. You can see it was created with an invalid "data" option within the mount_options: |
|
@bbeaver ok thanks - and could you also post the |
|
Sure - plesae see mount output below: mount/dev/mapper/vg_centos02-LogVol00 on / type ext4 (rw) |
The difference is coming from: before applying the patch of @jsmeix we just used Alright, between |
|
I manually edited the restore script in both my cases where rear recover failed. On the CentOS box related to the info above, I changed the mount command from: "mount -o rw,relatime,barrier=1,data /dev/mapper/vg_centos02-LogVol00 /mnt/local/" to: "mount -o rw,relatime,barrier=1 /dev/mapper/vg_centos02-LogVol00 /mnt/local/" On the RHEL system, I basically did the same thing, but removed a "strip" option instead of the "data" option. After doing this, I chose to continue the recovery, and it succeeded. |
|
What are the exact outputs for your CentOS6.6 and RHEL6.5 systems for each of the commands findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs versus mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs |
|
at a first glange I currenly guess that the findmnt output that results entries like fs ... options=rw,relatime,barrier=1,data=ordered is probably correct but I guess some other script might fail to correctly process that kind of options value (in particular options of the form keyword=value)? I guess this because bbeaver reported mount -o rw,relatime,barrier=1,data /dev/... which shows that the mount options have been clipped at the last '='. I assume with the whole mount options as reported by findmnt mount -o rw,relatime,barrier=1,data=ordered /dev/... mounting would have worked. |
|
Phew! |
|
I think in 13_include_mount_filesystem_code.sh the following is wrong: # Extract mount options.
local option mountopts
for option in $options ; do
name=${option%%=*} # options can contain more '=' signs
value=${option#*=}
because: $ options=rw,relatime,barrier=1,data=ordered ; echo $options ; for option in $options ; do name=${option%%=*} ; value=${option#*=} ; echo name=$name value=$value ; done
outputs rw,relatime,barrier=1,data=ordered name=rw,relatime,barrier value=1,data=ordered |
|
Gdha is exactly right, the "=" is getting clipped out. Here is the output from findmnt and mount: [root@centos02]]# findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs [root@centos02]# mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs |
|
My previous comment is wrong because the options entry is not rw,relatime,barrier=1,data=ordered but instead it is options=rw,relatime,barrier=1,data=ordered and with the latter it works because $ options="options=rw,relatime,barrier=1,data=ordered" ; echo $options ; for option in $options ; do name=${option%%=*} ; value=${option#*=} ; echo name=$name value=$value ; done
outputs options=rw,relatime,barrier=1,data=ordered name=options value=rw,relatime,barrier=1,data=ordered Somewhere else it happens that the mount options are clipped at the last '='. |
|
@bbeaver could you run |
|
The "savelayout" shows proper mount options, but running "mkbackup" does not, and cuts off the string. Here's the output, First, run "rear -vD savelayout" and check disklayout.conf: Then, run "rear -v mkbackup" and check disklayout.conf: |
|
@bbeaver that is interesting, but I don't get it as the disklayout.conf file is written only once (during the savelayout workflow) |
|
The dksklayout.conf file definitely gets updated when running the "rear mkbackup" command. Please see below. Anything specific you are wanting to see from the rear.log? Thanks for your help [root@centos02 tmp]# ls -ltr /var/lib/rear/layout [root@centos02 tmp]# date root@centos02 tmp]# nohup rear -vD mkbackup & [root@centos02 tmp]# ls -ltr /var/lib/rear/layout |
|
@bbeaver that is normal as |
|
Sure - looking to see how I upload the file... |
|
Please see if this gets you there - full rear-centos02.log debug: |
|
@bbeaver thanks for the logging: I'll dig into the code tomorrow and see what it is causing it. |
|
At #545 I found OS_VENDOR=RedHatEnterpriseServer OS_VERSION=6.4 and +++ mount -o rw,relatime,seclabel,barrier=1,dat /dev/mapper/vg_lsrs0vio-lv_root /mnt/local/ mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg_lsrs0vio-lv_root, It is the same issue there: Because there it is cut in between the "data=ordered" (and not at the '=' character) it now seems there is really some obscure length limit somewhere. But if it is a length limit it is not a fixed length limit. Very strange. |
|
do the 1/10 bad happen also with "rear savelayout" or do the 1/10 bad happen only with "rear mkbackup"? |
|
I'm seeing RedHat/Fedora/CentOS servers behave the same with this issue. Not too surprising, since these distros are all RedHat based. |
|
Only a blind guess: Perhaps it is not rear that sometimes cuts the mount options. Perhaps it is the findmnt command itself that sometimes reports clipped mount options? On my SLES12 system "man findmnt" reads in particular: -u, --notruncate Do not truncate text in columns. The default is to not truncate the TARGET, SOURCE, UUID, LABEL, PARTUUID, PARTLABEL columns. This option disables text truncation also in all other columns. and 23_filesystem_layout.sh contains read_filesystems_command="$findmnt_command -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS ... wich means without '-u' the FSTYPE and OPTIONS could be truncated. |
|
That might explain why I just started seeing this issue recently. A few months ago I had not come accross this problem. I understand that ReaR recently changed from using "mount" to using "findmnt". If there are problems with "findmnt", then ReaR is just now seeing those issues. |
|
I've made it back as far as Fedora 14, (which supports findmnt), but don't have easy access to a box with an older rhel5'ish o/s. I will continue looking though. In the meantime, will this "-u" option to the findmnt command be added to the latest rear code? |
…ere options was sometimes truncated on older RHEL 5/6
|
Thank you much for your help and quick response |
|
On RHEL 5.1 there is no findmnt command (at least not by default): # cat /etc/redhat-release Red Hat Enterprise Linux Server release 5.1 (Tikanga) # findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -bash: findmnt: command not found # type -a findmnt -bash: type: findmnt: not found In this case rear falls back using "mount" in 23_filesystem_layout.sh as follows: # mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' | cut -d ' ' -f 1,3,5,6 /dev/mapper/VolGroup00-LogVol00 / ext3 (rw) /dev/hda1 /boot ext3 (rw) |
|
The we should probably use this fallback code for all cases where Are there any downsides to using mount instead of findmount? If yes, then WARNING: On your system ReaR cannot determine XYZ because findmnt is not On 5 March 2015 at 11:48, Johannes Meixner notifications@github.com wrote:
|
|
One downside to using mount instead of findmount is documented in 23_filesystem_layout.sh: # If the findmnt command is available use it instead of the traditional mount command # because (since SLE12) "man 8 mount" reads: # The listing mode is maintained for backward compatibility only. # For more robust and customizable output use findmnt(8), especially in your scripts. Another downside seems to be shown here in this issue: Finally findmnt is basically required for btrfs subvolumes to get the mounted btrfs subvolume path (i.e. the subvolume name), see "Mounted btrfs subvolumes" in 23_filesystem_layout.sh - if findmnt is not available it falls back using awkward hacks to determine the mounted btrfs subvolume path. In practice I think on old systems without findmnt there will be no btrfs used and then the old systems get recreated as all the time before by plain "mount". |
|
On RHEL 5.9 it is (at least by default) the same as on RHEL 5.1: # cat /etc/redhat-release Red Hat Enterprise Linux Server release 5.9 (Tikanga) # findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -bash: findmnt: command not found # type -a findmnt -bash: type: findmnt: not found # mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' | cut -d ' ' -f 1,3,5,6 /dev/mapper/VolGroup00-LogVol00 / ext3 (rw) /dev/hda1 /boot ext3 (rw) |
|
I guess systems without findmnt won't notice much difference then? On 5 March 2015 at 12:31, Johannes Meixner notifications@github.com wrote:
|
|
I did my changes in 23_filesystem_layout.sh so that on systems without findmnt everything should work as before. But on systems with findmnt it now uses findmnt and then issues like this one appear. |
|
That is why I suggest that we use findmnt only if it supports the -u option. On 5 March 2015 at 12:41, Johannes Meixner notifications@github.com wrote:
|
|
@schlomo |
|
No, but I find it simpler to require that findmnt can handle -u than to Actually, before this thread I was not consciously aware of findmnt, so On 5 March 2015 at 12:49, Johannes Meixner notifications@github.com wrote:
|
|
@schlomo I am currently installing RHEL 6 to find out about findmnt there... FYI, regarding findmnt in general: |
|
On RHEL 6.5: # cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.5 (Santiago) # findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' /dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered /dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered # findmnt -alnuv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' /dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered /dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered # mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' | cut -d ' ' -f 1,3,5,6 /dev/mapper/VolGroup-lv_root / ext4 (rw) /dev/sda1 /boot ext4 (rw) Currently I don't know how to trigger that findmnt truncates FSTYPE or OPTIONS output. |
|
It depends on the width of the terminal wherein findmnt is run whether or not it truncates output. On RHEL 6.5 in a smaller-width xterm: # findmnt -alnv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' /dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=o /dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=o # findmnt -alnuv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' /dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered /dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered # findmnt -alnuv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | tr -s '[:blank:]' ' ' | od -a 0000000 / d e v / m a p p e r / V o l G 0000020 r o u p - l v _ r o o t nl sp / sp 0000040 e x t 4 sp r w , r e l a t i m e 0000060 , s e c l a b e l , b a r r i e 0000100 r = 1 , d a t a = o r d e r e d 0000120 nl / d e v / s d a 1 sp / b o o t 0000140 sp e x t 4 sp r w , r e l a t i m 0000160 e , s e c l a b e l , b a r r i 0000200 e r = 1 , d a t a = o r d e r e 0000220 d nl 0000222 Scaring! With '-u' findmnt inserts a newline character when the output is too long for one line. |
|
@gdha |
|
I think using the '-r' option alone helps - at least on RHEL 6.5: # findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs /dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered /dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered |
|
Also on SLE12 "findmnt -nrv -o .." seems to output the right thing: # findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs /dev/sda2 / btrfs rw,relatime,space_cache /dev/sda2 /.snapshots btrfs rw,relatime,space_cache /dev/sda2 /var/tmp btrfs rw,relatime,space_cache /dev/sda2 /var/spool btrfs rw,relatime,space_cache /dev/sda2 /var/opt btrfs rw,relatime,space_cache /dev/sda2 /var/log btrfs rw,relatime,space_cache /dev/sda2 /var/lib/named btrfs rw,relatime,space_cache /dev/sda2 /var/lib/pgsql btrfs rw,relatime,space_cache /dev/sda2 /var/lib/mailman btrfs rw,relatime,space_cache /dev/sda2 /usr/local btrfs rw,relatime,space_cache /dev/sda2 /var/crash btrfs rw,relatime,space_cache /dev/sda2 /opt btrfs rw,relatime,space_cache /dev/sda2 /tmp btrfs rw,relatime,space_cache /dev/sda2 /srv btrfs rw,relatime,space_cache /dev/sda2 /boot/grub2/x86_64-efi btrfs rw,relatime,space_cache /dev/sda2 /home btrfs rw,relatime,space_cache /dev/sda2 /boot/grub2/i386-pc btrfs rw,relatime,space_cache # findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs /dev/sda2 / rw,relatime,space_cache /@ /dev/sda2 /.snapshots rw,relatime,space_cache /@/.snapshots /dev/sda2 /var/tmp rw,relatime,space_cache /@/var/tmp /dev/sda2 /var/spool rw,relatime,space_cache /@/var/spool /dev/sda2 /var/opt rw,relatime,space_cache /@/var/opt /dev/sda2 /var/log rw,relatime,space_cache /@/var/log /dev/sda2 /var/lib/named rw,relatime,space_cache /@/var/lib/named /dev/sda2 /var/lib/pgsql rw,relatime,space_cache /@/var/lib/pgsql /dev/sda2 /var/lib/mailman rw,relatime,space_cache /@/var/lib/mailman /dev/sda2 /usr/local rw,relatime,space_cache /@/usr/local /dev/sda2 /var/crash rw,relatime,space_cache /@/var/crash /dev/sda2 /opt rw,relatime,space_cache /@/opt /dev/sda2 /tmp rw,relatime,space_cache /@/tmp /dev/sda2 /srv rw,relatime,space_cache /@/srv /dev/sda2 /boot/grub2/x86_64-efi rw,relatime,space_cache /@/boot/grub2/x86_64-efi /dev/sda2 /home rw,relatime,space_cache /@/home /dev/sda2 /boot/grub2/i386-pc rw,relatime,space_cache /@/boot/grub2/i386-pc |
|
On RHEL 6.0 there is no findmnt (in contrast to RHEL 6.5) - at least not by default: # cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.0 (Santiago) # type -a findmnt -bash: type: findmnt: not found # mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | cut -d ' ' -f 1,3,5,6 /dev/mapper/VolGroup-lv_root / ext4 (rw) /dev/sda1 /boot ext4 (rw) |
|
On RHEL 6.x findmnt was added to the util-linux-ng RPM # rpm -q --changelog util-linux-ng . . . * Thu Jan 27 2011 Karel Zak 2.17.2-8 ... - fix #616325 - Include findmnt in util-linux-ng package |
|
findmnt is available since RHEL 6.1 and works there as needed: # cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.1 (Santiago) # findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs /dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered /dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered # mount -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs | cut -d ' ' -f 1,3,5,6 /dev/mapper/VolGroup-lv_root / ext4 (rw) /dev/sda1 /boot ext4 (rw) |
|
I submitted #559 that should hopefully fix it. |
|
@bbeaver @jsmeix and what if we do the following: |
|
With '-r' findmnt results the right output regardless of the terminal width. I had tried "COLUMNS=900" before but that did not make a difference for me. I think the whole issue is that by default findmnt tries (too?) hard to make the output nice looking for interactive usage and only with '-r' one gets the real hard facts for real hard men ;-) With RHEL 6.1 in a smaller xterm: # COLUMNS=254 findmnt -alnuv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/mapper/VolGroup-lv_root
/ ext4 rw,relatime,seclabel,barrier=1,data=ordered
/dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered
# findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs
/dev/mapper/VolGroup-lv_root / ext4 rw,relatime,seclabel,barrier=1,data=ordered
/dev/sda1 /boot ext4 rw,relatime,seclabel,barrier=1,data=ordered
Note the additional newline in the first findmnt output that breaks what belongs to /dev/mapper/VolGroup-lv_root |
Hi All:
Running current rear-1.16.1-1.git201502200825, and had both a Centos6.6 and RHEL6.5 box experience recover problems that had to be manually corrected. Both had very similar problems, but with different invalid options built within a mount command. My Centos6.6 box attempted a 'mount -o rw,relatime,barrier=1,data /dev/mapper/vg_centos02-LogVol00 /mnt/local/" command, and my RHEL6.5 box attempted a "mount -o rw,relatime,barrier=1,strip /dev/mapper/vg00-LogVol00 /mnt/local/" command. Both failed and had to be manually tweaked using the "Edit restore script" option. In the Centos case, had to remove the "data" option that had been added, in the case of RHEL had to remove the "strip" option that had been added. How are these invalid mount options landing in the restore script? Thanks for your help.
The text was updated successfully, but these errors were encountered: