Support by-path PVs

[schlomo]

ReaR 2.4 on SLES12 SP3 with local disk in VMware VM with LVM

Problem
PVs are addressed by /dev/disk/by-path/ like this:

# pvs
  PV                                                           VG                     Fmt  Attr PSize   PFree
  /dev/disk/by-id/dm-name-local-lvm-modbtsbl-114v_vol00_dg-001 modbtsbl-114v_vol00_dg lvm2 a--  550.10g 104.00m
  /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part2        storage                lvm2 a--   19.99g   1.49g

# pvdisplay -c
/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part2:storage:41924608:-1:8:8:-1:4096:5117:381:4736:HcNHiV-SSq8-yvzK-3q21-p8ND-qGah-J1eb6J
/dev/disk/by-id/dm-name-local-lvm-modbtsbl-114v_vol00_dg-001:modbtsbl-114v_vol00_dg:1153654784:-1:8:8:-1:4096:140826:26:140800:JDWfR1-feeC-Eznp-x1Qo-7TAb-j5KK-JMc1Bj

The resulting disklayout.conf contains broken lines like this:

lvmdev /dev/03 /dev/disk/by-path/pci-0000 -1 00.0-scsi-0

And consequently ReaR doesn't work.

Cause
Root cause is 220_lvm_layout.sh using : to separate the output. : also appears as part of the PV physical block device name so that our code breaks.

Solution
Don't use : as a separator in stuff that contains file names because : is valid and used in file names.

Further TODO
This fix alone is not enough because ReaR does not recognize the dependency between the PV and the /dev/sda disk and therefore excludes that disk from formatting and recovery.

[schlomo]

@skrueger8 FYI

[jsmeix]

I added @gdha because according to the

git log -p --follow usr/share/rear/layout/save/GNU/Linux/220_lvm_layout.sh

output layout/save/GNU/Linux/220_lvm_layout.sh was initially creted by
@gdha via
c7b79dd
that already uses : as filed separator but the commit message
'added patch of Jeroen to check/save-layout workflows'
indicates it originates from @jhoekx

I think the 'pvdisplay -c' output is the root cause because

dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part2:storage:...

is already broken because the 'pvdisplay -c' output uses : as field separator
but that character appears in the first field value in this case.

[jsmeix]

I need to set up a test system with LVM to be able to actually work on it...

[schlomo]

@skrueger8 can you please check that the fix in ea2b700 reflects what we tested?

[schlomo]

@jsmeix for testing some more info:

• GPT disk with BIOS boot
• 1st partition is bios_grub
• 2nd partition is PV on which resides the LV with the btrfs volume for the system
• /boot is on the btrfs volume that has /
• PV is defined via its by-path block device file

Please add additional fixes to the issue-1958 branch and we will also try them out.

[jsmeix]

@schlomo
there are so many ways how to setup storage in YaST
that I need more details how to setup what you have
so that I can reproduce their issue.

What is the output of commands like

# parted -s /dev/sda unit MiB print

# lsblk -i -p -o NAME,KNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT /dev/sda

on a relatively small system that has this issue?

FYI
what I have on my SLES12 test system with the default LVM setup that one gets
when one selects during installation in YaST the "LVM-based Proposal"
(but not the "Encrypted LVM-based Proposal")

# parted -s /dev/sda unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 20480MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start    End       Size      Type     File system  Flags
 1      1.00MiB  20480MiB  20479MiB  primary               boot, lvm, type=8e

# lsblk -i -p -o NAME,KNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT /dev/sda
NAME                        KNAME     TRAN TYPE FSTYPE       SIZE MOUNTPOINT
/dev/sda                    /dev/sda  ata  disk               20G 
`-/dev/sda1                 /dev/sda1      part LVM2_member   20G 
  |-/dev/mapper/system-swap /dev/dm-0      lvm  swap         1.4G [SWAP]
  `-/dev/mapper/system-root /dev/dm-1      lvm  btrfs       18.6G /

Meanwhile I found out how to enforce GPT in YaST (somewhat hidden) and
how to manually create a BIOS Grub partition in YaST (Googling helped me)
perhaps I should even read some SUSE manuals (my very last resort ;-) ...

[jsmeix]

I get closer but up to now I failed to setup by-path PVs.
What I have now via manual setup in YaST:

# parted -s /dev/sda unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 20480MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot
Number  Start    End       Size      File system  Name  Flags
 1      1.00MiB  16.0MiB   15.0MiB                      bios_grub
 2      16.0MiB  19469MiB  19453MiB                     lvm, legacy_boot

# lsblk -i -p -o NAME,KNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT
NAME                                KNAME     TRAN TYPE FSTYPE       SIZE MOUNTPOINT
/dev/sda                            /dev/sda  ata  disk               20G 
|-/dev/sda1                         /dev/sda1      part               15M 
`-/dev/sda2                         /dev/sda2      part LVM2_member   19G 
  |-/dev/mapper/system-system--swap /dev/dm-0      lvm  swap           1G [SWAP]
  `-/dev/mapper/system-system--root /dev/dm-1      lvm  btrfs         18G /
/dev/sr0                            /dev/sr0  ata  rom              1024M 

# pvs
  PV         VG     Fmt  Attr PSize  PFree
  /dev/sda2  system lvm2 a--  19.00g 4.00m

# pvdisplay -c
  /dev/sda2:system:39839744:-1:8:8:-1:4096:4863:1:4862:7guAfh-09wj-3ivj-oH4K-kvqE-NUjJ-r9sPSu

# pvdisplay --columns --separator "|"
  PV|VG|Fmt|Attr|PSize|PFree
  /dev/sda2|system|lvm2|a--|19.00g|4.00m

[skrueger8]

@schlomo
there are so many ways how to setup storage in YaST
that I need more details how to setup what you have
so that I can reproduce their issue.

What is the output of commands like

# parted -s /dev/sda unit MiB print

# lsblk -i -p -o NAME,KNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT /dev/sda
on a relatively small that has this issue?

Hi, here's the output you requested:

# parted -s /dev/sda unit MiB print
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 20480MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start    End       Size      File system  Name     Flags
 1      1.00MiB  8.00MiB   7.00MiB                primary  bios_grub
 2      8.00MiB  20479MiB  20471MiB               primary  lvm, legacy_boot

# lsblk -i -p -o NAME,KNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT /dev/sda
NAME                             KNAME      TRAN TYPE FSTYPE       SIZE MOUNTPOINT
/dev/sda                         /dev/sda        disk               20G
|-/dev/sda1                      /dev/sda1       part                7M
`-/dev/sda2                      /dev/sda2       part LVM2_member   20G
  |-/dev/mapper/storage-root     /dev/dm-0       lvm  btrfs         12G /
  |-/dev/mapper/storage-home     /dev/dm-8       lvm  xfs            5G /home
  |-/dev/mapper/storage-varcrash /dev/dm-9       lvm  xfs            1G /var/crash
  `-/dev/mapper/storage-varlog   /dev/dm-10      lvm  xfs          512M /var/log

[skrueger8]

@skrueger8 can you please check that the fix in ea2b700 reflects what we tested?

Yup, that's the fix we tested yesterday.

[jsmeix]

@skrueger8
thanks for the info!

@skrueger8 @schlomo
do you perhaps know how to install a SLES12 system so that it uses by-path PVs?
I mean what I would have to do in YaST during installation to get by-path PVs?
Or do you know a URL to documentation where that is described?
(I can find that out on my own but I don't know how long that may take...)

Or can you describe me how you installed your system so that it uses by-path PVs
so that I can relatively easily reproduce it?

I guess if you have many such systems you probably don't install each
of them one by one with YaST but e.g. perhaps with AutoYaST (or whatever
other deployment tool you may use) perhaps together with even more
sophisticated software on top of that like SUSE Manager (or anything else)
but that would be something that I cannot "relatively easily reproduce".

On the other hand - I guess - when you use AutoYaST for installation
you may provide me your AutoYaST autoinst.xml file for one of your systems
(ideally for an as simple and small as possible system that uses by-path PVs)
pereferably attach your AutoYaST autoinst.xml file to this issue here.
If you provide me your AutoYaST autoinst.xml file ensure it does
no longer contain secrets like root password or things like that.
Then - I hope - I can relatively easily reproduce your system.

[schlomo]

https://www.suse.com/documentation/sles11/book_autoyast/data/createprofile_partitioning.html suggests to use

<mountby config:type="symbol">path</mountby>

We will provide you with the relevant information

[jsmeix]

I am afraid, today I cannot work on this issue,
I have some Ghostscript security related things to do, cf.
https://ghostscript.com/pipermail/gs-devel/2018-November/010126.html

Hopefully next week...

[jsmeix]

"needs sponsorship" => for a "ReaR future" milestone

[jsmeix]

I think calling the 'lvm' commands with --separator ' | ' --aligned
(for better human readability) or plain --separator '|' options
could be a (hopefully) sufficiently fail-safe way because at least on
all my systems find /dev | grep '|' results nothing so that it seems
the | character does not appear in device nodes or device symlinks
...
of course except those users who use the | character in labels
so that it appears in /dev/disk/by-label/ e.g. something like

/dev/disk/by-label/my|stuff

[github-actions]

Stale issue message

[github-actions]

Stale issue message

[pcahyna]

I believe #2903 addresses all the issues:

This fix alone is not enough because ReaR does not recognize the dependency between the PV and the /dev/sda disk and therefore excludes that disk from formatting and recovery.

this is not an issue for some reason. Here is the layout on a two-disk system configured to show /dev/disk/by-path PVs:
/var/lib/rear/layout/disklayout.conf

# Disk layout dated 20230102095535 (YYYYmmddHHMMSS)
# NAME                                         KNAME     PKNAME    TRAN   TYPE FSTYPE      LABEL   SIZE MOUNTPOINT UUID                                   WWN
# /dev/sda                                     /dev/sda            sata   disk                   931.5G                                                   0x5000c5007971fc8c
# |-/dev/sda1                                  /dev/sda1 /dev/sda         part vfat                600M /boot/efi  C7D3-F00C                              0x5000c5007971fc8c
# |-/dev/sda2                                  /dev/sda2 /dev/sda         part xfs                   1G /boot      a8512f5b-ebc6-460b-8f5e-508df68c185f   0x5000c5007971fc8c
# `-/dev/sda3                                  /dev/sda3 /dev/sda         part LVM2_member       929.9G            2z2Dm7-R1JN-eCjn-arTJ-rQDR-YQz8-meSfPN 0x5000c5007971fc8c
#   |-/dev/mapper/rhel_hpe--dl120gen9--01-root /dev/dm-0 /dev/sda3        lvm  xfs                  70G /          89705cff-2ff7-4b45-90be-1cc292c0ef20   
#   |-/dev/mapper/rhel_hpe--dl120gen9--01-swap /dev/dm-1 /dev/sda3        lvm  swap                7.8G [SWAP]     d5b1cc1d-f7ab-4f25-bccf-e8fb73750cb5   
#   `-/dev/mapper/rhel_hpe--dl120gen9--01-home /dev/dm-2 /dev/sda3        lvm  xfs                 1.8T /home      77b0e7ae-92c7-4137-842b-f06bdadda278   
# /dev/sdb                                     /dev/sdb            sata   disk                   931.5G                                                   0x5000c500798fc371
# `-/dev/sdb1                                  /dev/sdb1 /dev/sdb         part LVM2_member       931.5G            Vg2HFj-EZwJ-k1D9-UjKv-Gwkr-6dhx-d0O7mc 0x5000c500798fc371
#   `-/dev/mapper/rhel_hpe--dl120gen9--01-home /dev/dm-2 /dev/sdb1        lvm  xfs                 1.8T /home      77b0e7ae-92c7-4137-842b-f06bdadda278   
# Disk /dev/sda
# Format: disk <devname> <size(bytes)> <partition label type>
disk /dev/sda 1000204886016 gpt
# Partitions on /dev/sda
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
part /dev/sda 629145600 1048576 EFI%20System%20Partition boot,esp /dev/sda1
part /dev/sda 1073741824 630194176 rear-noname none /dev/sda2
part /dev/sda 998500204544 1703936000 rear-noname lvm /dev/sda3
# Disk /dev/sdb
# Format: disk <devname> <size(bytes)> <partition label type>
disk /dev/sdb 1000204886016 gpt
# Partitions on /dev/sdb
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
part /dev/sdb 1000203091968 1048576 rear-noname lvm /dev/sdb1
# Format for LVM PVs
# lvmdev <volume_group> <device> [<uuid>] [<size(bytes)>]
lvmdev /dev/rhel_hpe-dl120gen9-01 /dev/sda3 2z2Dm7-R1JN-eCjn-arTJ-rQDR-YQz8-meSfPN 998496010240
lvmdev /dev/rhel_hpe-dl120gen9-01 /dev/sdb1 Vg2HFj-EZwJ-k1D9-UjKv-Gwkr-6dhx-d0O7mc 1000198897664
# Format for LVM VGs
# lvmgrp <volume_group> <extentsize> [<size(extents)>] [<size(bytes)>]
lvmgrp /dev/rhel_hpe-dl120gen9-01 4096 476526 1951850496
# Format for LVM LVs
# lvmvol <volume_group> <name> <size(bytes)> <layout> [key:value ...]
lvmvol /dev/rhel_hpe-dl120gen9-01 swap 8338276352b linear 
lvmvol /dev/rhel_hpe-dl120gen9-01 root 75161927680b linear 
# Volume rhel_hpe-dl120gen9-01/home has multiple segments. Recreating it by 'lvcreate' will not preserve segments and properties of the other segments as well
lvmvol /dev/rhel_hpe-dl120gen9-01 home 1915194703872b linear 
# Extra parameters for the 'lvmvol /dev/rhel_hpe-dl120gen9-01 home' line above not taken into account when restoring using 'lvcreate': segmentsize:1000198897664b
#lvmvol /dev/rhel_hpe-dl120gen9-01 home 1915194703872b linear 
# Extra parameters for the '#lvmvol /dev/rhel_hpe-dl120gen9-01 home' line above not taken into account when restoring using 'lvcreate': segmentsize:914995806208b
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/mapper/rhel_hpe--dl120gen9--01-home /home xfs uuid=77b0e7ae-92c7-4137-842b-f06bdadda278 label=  options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
fs /dev/mapper/rhel_hpe--dl120gen9--01-root / xfs uuid=89705cff-2ff7-4b45-90be-1cc292c0ef20 label=  options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
fs /dev/sda1 /boot/efi vfat uuid=C7D3-F00C label= options=rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro
fs /dev/sda2 /boot xfs uuid=a8512f5b-ebc6-460b-8f5e-508df68c185f label=  options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
swap /dev/mapper/rhel_hpe--dl120gen9--01-swap uuid=d5b1cc1d-f7ab-4f25-bccf-e8fb73750cb5 label=

/var/lib/rear/layout/diskdeps.conf

/dev/sda1 /dev/sda
/dev/sda2 /dev/sda
/dev/sda3 /dev/sda
/dev/sdb1 /dev/sdb
/dev/rhel_hpe-dl120gen9-01 pv:/dev/sda3
pv:/dev/sda3 /dev/sda3
/dev/rhel_hpe-dl120gen9-01 pv:/dev/sdb1
pv:/dev/sdb1 /dev/sdb1
/dev/mapper/rhel_hpe--dl120gen9--01-swap /dev/rhel_hpe-dl120gen9-01
/dev/mapper/rhel_hpe--dl120gen9--01-root /dev/rhel_hpe-dl120gen9-01
/dev/mapper/rhel_hpe--dl120gen9--01-home /dev/rhel_hpe-dl120gen9-01
fs:/home /dev/mapper/rhel_hpe--dl120gen9--01-home
fs:/home fs:/
fs:/ /dev/mapper/rhel_hpe--dl120gen9--01-root
fs:/boot/efi /dev/sda1
fs:/boot/efi fs:/
fs:/boot/efi fs:/boot
fs:/boot /dev/sda2
fs:/boot fs:/
swap:/dev/mapper/rhel_hpe--dl120gen9--01-swap /dev/mapper/rhel_hpe--dl120gen9--01-swap

/var/lib/rear/layout/disktodo.conf

todo /dev/sda disk
todo /dev/sda1 part
todo /dev/sda2 part
todo /dev/sda3 part
todo /dev/sdb disk
todo /dev/sdb1 part
todo pv:/dev/sda3 lvmdev
todo pv:/dev/sdb1 lvmdev
todo /dev/rhel_hpe-dl120gen9-01 lvmgrp
todo /dev/mapper/rhel_hpe--dl120gen9--01-swap lvmvol
todo /dev/mapper/rhel_hpe--dl120gen9--01-root lvmvol
todo /dev/mapper/rhel_hpe--dl120gen9--01-home lvmvol
todo fs:/home fs
todo fs:/ fs
todo fs:/boot/efi fs
todo fs:/boot fs
todo swap:/dev/mapper/rhel_hpe--dl120gen9--01-swap swap

lvm pvdisplay -c

  /dev/disk/by-path/pci-0000:00:11.4-ata-1-part3:rhel_hpe-dl120gen9-01:1950195712:-1:8:8:-1:4096:238060:0:238060:2z2Dm7-R1JN-eCjn-arTJ-rQDR-YQz8-meSfPN
  /dev/disk/by-path/pci-0000:00:11.4-ata-2-part1:rhel_hpe-dl120gen9-01:1953521664:-1:8:8:-1:4096:238466:0:238466:Vg2HFj-EZwJ-k1D9-UjKv-Gwkr-6dhx-d0O7mc

So, /dev/disk/by-path/* does not appear in the layout, only /dev/sd* does, and dependencies seem to be properly recorded. This is despite pvdisplay showing /dev/disk/by-path/* paths. I believe the layout code performs enough symlink resolution now.

Or can you describe me how you installed your system so that it uses by-path PVs
so that I can relatively easily reproduce it?

LVM configuration described in #2903: add

filter = [ "r|/dev/disk/by-path/.*-usb-|", "a|/dev/disk/by-path/pci-.*-nvme-|", "a|/dev/disk/by-path/pci-.*-scsi-|", "a|/dev/disk/by-path/pci-.*-ata-|", "a|/dev/disk/by-path/pci-.*-sas-|", "a|loop|", "r|.*|" ]

to /etc/lvm/lvm.conf.

ea2b700#commitcomment-31237351

PV size is human readable and not in bytes :(

solved by --units=b.