LVM: How to deal with orphaned PVs like 'lvmdev /dev/#orphans_lvm2'

[mreubold]

• ReaR version 2.5

• SuSE Enterprise 11.4 on Power8 (ppc64)

• PoverVM LPAR

• System architecture PPC64

• Storage SAN (FC)and/or multipath (DM)

• Description of the issue (ideally so that others can reproduce it):

We've installed the latest master.zip file on the partition.
Once starting usr/sbin/rear mkbackup
we get this error messages :

svpsgsap22:~/ReaR/rear-master # usr/sbin/rear mkbackup
LVM no 'lvmgrp /dev/#orphans_lvm2' for 'lvmdev /dev/#orphans_lvm2'
LVM no 'lvmvol /dev/#orphans_lvm2' for 'lvmdev /dev/#orphans_lvm2'
ERROR:
====================
BUG in /root/ReaR/rear-master/usr/share/rear/layout/save/default/950_verify_disklayout_file.sh line 254:
'Entries in /root/ReaR/rear-master/var/lib/rear/layout/disklayout.conf are broken ('rear recover' would fail)'
--------------------
Please report this issue at https://github.com/rear/rear/issues
and include the relevant parts from /root/ReaR/rear-master/var/log/rear/rear-svpsgsap22.log
preferably with full debug information via 'rear -D mkbackup'

• Workaround, if any: no

• Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
  svpsgsap22rearlog.txt

Any idea what happens ?
Many thanks
Regards,
Martin

[jsmeix]

@mreubold
in your https://github.com/rear/rear/files/4072247/svpsgsap22rearlog.txt
there is (excerpts):

+ source /root/ReaR/rear-master/usr/share/rear/layout/save/GNU/Linux/220_lvm_layout.sh
++ echo 'lvmdev /dev/system /dev/mapper/3600507638081001a480000000000009d_part3 dLRKGv-c6qU-ycWp-EVwH-333U-Jhpn-XWhgRT 41535488'
++ echo 'lvmdev /dev/system /dev/mapper/3600507638081001a480000000000009d_part4 bOsEUv-9kAM-jx0l-U9HE-lQoj-Mvlr-jLyiUK 399360'
++ echo 'lvmdev /dev/sap_DS1_vg /dev/mapper/3600507638081001a480000000000009e nG1ft9-2NPt-YnrT-J0N7-1qGm-uzbO-0LEwQY 209715200'
++ echo 'lvmdev /dev/#orphans_lvm2 /dev/mapper/3600507638081001a480000000000009d_part2 AT1inR-9ss0-vVPi-N6zs-afDm-8RyH-y5J3LD 83458048'
++ echo 'lvmgrp /dev/system 4096 5118 20963328'
++ echo 'lvmgrp /dev/sap_DS1_vg 4096 25599 104853504'
++ echo 'lvmvol /dev/sap_DS1_vg sap_DS1_lv 53687091200b linear stripes:1'
++ echo 'lvmvol /dev/sap_DS1_vg sapmnt_DS1_lv 53682896896b linear stripes:1'
++ echo 'lvmvol /dev/system root 10737418240b linear stripes:1'
++ echo 'lvmvol /dev/system swap 2147483648b linear stripes:1'

So as far as I see from those log excerpts
you should have those LVM entries in your
var/lib/rear/layout/disklayout.conf file

lvmdev /dev/system /dev/mapper/3600507638081001a480000000000009d_part3 dLRKGv-c6qU-ycWp-EVwH-333U-Jhpn-XWhgRT 41535488
lvmdev /dev/system /dev/mapper/3600507638081001a480000000000009d_part4 bOsEUv-9kAM-jx0l-U9HE-lQoj-Mvlr-jLyiUK 399360
lvmdev /dev/sap_DS1_vg /dev/mapper/3600507638081001a480000000000009e nG1ft9-2NPt-YnrT-J0N7-1qGm-uzbO-0LEwQY 209715200
lvmdev /dev/#orphans_lvm2 /dev/mapper/3600507638081001a480000000000009d_part2 AT1inR-9ss0-vVPi-N6zs-afDm-8RyH-y5J3LD 83458048
lvmgrp /dev/system 4096 5118 20963328
lvmgrp /dev/sap_DS1_vg 4096 25599 104853504
lvmvol /dev/sap_DS1_vg sap_DS1_lv 53687091200b linear stripes:1
lvmvol /dev/sap_DS1_vg sapmnt_DS1_lv 53682896896b linear stripes:1
lvmvol /dev/system root 10737418240b linear stripes:1
lvmvol /dev/system swap 2147483648b linear stripes:1

Please attach your var/lib/rear/layout/disklayout.conf file
if the LVM entries in your file are any different
so I could see what there actually is in your disklayout.conf file.

So it seems you got a lvmdev /dev/#orphans_lvm2 entry
in disklayout.conf that has neither a matching lvmgrp
nor a lvmvol entry.

This case is currently considered that something has gone wrong
while usr/share/rear/layout/save/GNU/Linux/220_lvm_layout.sh
saved your LVM layout into the disklayout.conf file, cf.
27e2f0c

For the resoning behind see
#2259
#2291

I am not really a LVM expert and I never experienced
such a #orphans_lvm thingy in my simple LVM setups.

By Googling for orphans_lvm the only thing I found was
https://www.redhat.com/archives/linux-lvm/2010-April/msg00026.html
which reads (excerpt)

Cc: LVM general discussion and development <linux-lvm redhat com>
Subject: Re: [linux-lvm] what does 'orphan vg' mean?(global vg?)
...
> >       While I was reading the lvm source code, I can't understand what
> > 'is_orphan' means(like 'is_orphan_vg' function). what's the different
> > between orphan and global vg?
> 
> Orphan PV is device with PV label which is not attached to any Volume Group.
> (IOW it is handled by LVM, but space on it is not yet allocatable.)
> 
> There is no such thing like Global VG in LVM2, only "global lock".
> It is internal lock used to avoid parallel scanning of all devices.
> (you will see it in vgscan command for example).

This description matches what there is in your
https://github.com/rear/rear/files/4072247/svpsgsap22rearlog.txt
(excerpts):

+ source /root/ReaR/rear-master/usr/share/rear/layout/save/GNU/Linux/220_lvm_layout.sh
...
++ lvm pvdisplay -c
...
+++ echo /dev/mapper/3600507638081001a480000000000009d_part2:#orphans_lvm2:83458048:-1:8:8:-1:0:0:0:0:AT1inR-9ss0-vVPi-N6zs-afDm-8RyH-y5J3LD
...
++ vgrp='#orphans_lvm2'
...
++ echo 'lvmdev /dev/#orphans_lvm2 /dev/mapper/3600507638081001a480000000000009d_part2 AT1inR-9ss0-vVPi-N6zs-afDm-8RyH-y5J3LD 83458048'

So it seems my simple test that I added to
layout/save/default/950_verify_disklayout_file.sh
to verify that the 'lvm...' entries in disklayout.conf
look syntactically correct is currently too strict because
it causes false alarm for orphans_lvm entries.

Or alternatively perhaps Orphan PV devices should be already
handled in layout/save/GNU/Linux/220_lvm_layout.sh
so that they appear only as comments in disklayout.conf
if it would be "the right thing" to do so with orphaned PVs
(i.e. orphaned PVs would not be recreated during "rear recover").

@rmetrich @pcahyna
could you please tell me what you think is best
how to deal with /dev/#orphans_lvm2 things?

Should orphaned PVs be listed as active entries in disklayout.conf like

lvmdev /dev/#orphans_lvm2 /dev/...

and ignored by the test in 950_verify_disklayout_file.sh
OR
should they be only shown as comments in disklayout.conf like

# lvmdev /dev/#orphans_lvm2 /dev/...

if orphaned PVs should not be recreated during "rear recover" ?

[jsmeix]

@mreubold
to test if "rear recover" works at all
when there are active lvmdev /dev/#orphans_lvm2 /dev...
entries for orphaned PVs in disklayout.conf
please disable the LVM test in
usr/share/rear/layout/save/default/950_verify_disklayout_file.sh
e.g. just disable that whole script for the test by adding a line

return 0

at its very beginning, cf.
https://github.com/rear/rear/blob/master/doc/rear-release-notes.txt#L247
and then try out if "rear -D recover" works for you even with the active

lvmdev /dev/#orphans_lvm2 /dev/mapper/3600507638081001a480000000000009d_part2 AT1inR-9ss0-vVPi-N6zs-afDm-8RyH-y5J3LD 83458048

entry in your disklayout.conf file.

[mreubold]

Hello,
I attach the files in var/lib/rear/layout/.

In var/lib/rear/output there is no file. Hence I guess this issue blocks the creation of iso file and so can't do any recovery test.
Please let me know,
Thanks,
Regards,
Martin

df.txt

diskdeps.txt
disklayout.txt
disktodo.txt
system.txt
sap_DS1_vg.txt

[mreubold]

Hello,
in addition to this I tried usr/sbin/rear checklayout with following result :

svpsgsap22:~/ReaR/rear-master # usr/sbin/rear checklayout
LVM no 'lvmgrp /dev/#orphans_lvm2' for 'lvmdev /dev/#orphans_lvm2'
LVM no 'lvmvol /dev/#orphans_lvm2' for 'lvmdev /dev/#orphans_lvm2'
ERROR:
====================
BUG in /root/ReaR/rear-master/usr/share/rear/layout/save/default/950_verify_disklayout_file.sh line 254:
'Entries in /tmp/rear.WJdfMvGTurxs83k/tmp/checklayout.conf are broken ('rear recover' would fail)'
--------------------
Please report this issue at https://github.com/rear/rear/issues
and include the relevant parts from /root/ReaR/rear-master/var/log/rear/rear-svpsgsap22.log.lockless
preferably with full debug information via 'rear -D checklayout'
====================
Some latest log messages since the last called script 950_verify_disklayout_file.sh:
  2020-01-20 14:23:25.476379194 Including layout/save/default/950_verify_disklayout_file.sh
  2020-01-20 14:23:25.477491546 Verifying that the entries in /tmp/rear.WJdfMvGTurxs83k/tmp/checklayout.conf are correct ...
  2020-01-20 14:23:25.478665735 Verifying that the 'disk' entries in /tmp/rear.WJdfMvGTurxs83k/tmp/checklayout.conf are correct
  2020-01-20 14:23:25.481878315 Verifying that the 'lvm...' entries in /tmp/rear.WJdfMvGTurxs83k/tmp/checklayout.conf are correct
  2020-01-20 14:23:25.498903703 LVM no 'lvmgrp /dev/#orphans_lvm2' for 'lvmdev /dev/#orphans_lvm2'
  2020-01-20 14:23:25.501121038 LVM no 'lvmvol /dev/#orphans_lvm2' for 'lvmdev /dev/#orphans_lvm2'
Aborting due to an error, check /root/ReaR/rear-master/var/log/rear/rear-svpsgsap22.log.lockless for details

I attach debug log file for checklayout
rear-svpsgsap22.log.lockless.txt

[jsmeix]

@mreubold
as long as you don't disable that BugError
in the 950_verify_disklayout_file.sh script
you won't get an ISO, cf.
#2315 (comment)

[mreubold]

Hello

backup is running with 950_verify_disklayout_file.sh "disabled" and the disklayout.conf file has been populated (as previewed) with :
lvmdev /dev/#orphans_lvm2 /dev/mapper/3600507638081001a480000000000009d_part2 AT1inR-9ss0-vVPi-N6zs-afDm-8RyH-y5J3LD 8345804

I've to ask if I can restore this lpar (it is currently in use) and let you know.
Thank you
Regards,
Martin

[gdha]

@mreubold @jsmeix Strange that script /usr/share/rear/layout/save/GNU/Linux/340_false_blacklisted.sh did not recognized this orphans_lvm2? This was first seen in issue #227

[github-actions]

Stale issue message