Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docker hosts would better exclude the file systems under control by docker #1749

Closed
gdha opened this issue Mar 8, 2018 · 10 comments
Closed

docker hosts would better exclude the file systems under control by docker #1749

gdha opened this issue Mar 8, 2018 · 10 comments
Assignees
@gdha @rmetrich
Labels
bug The code does not do what it is meant to do enhancement Adaptions and new features fixed / solved / done
Milestone
ReaR v2.6

Comments

@gdha
Copy link
Member

gdha commented Mar 8, 2018

  • rear version (/usr/sbin/rear -V): rear-2.00 and higher
  • OS version (cat /etc/rear/os.conf or lsb_release -a): rhel 7 (but probably others as well)
  • rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf): empty
  • Are you using legacy BIOS or UEFI boot? BIOS
  • Brief description of the issue: checking the savelayout only
#-> df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda2      16764908 4288520  12476388  26% /
devtmpfs        32871068       0  32871068   0% /dev
tmpfs           32896424       8  32896416   1% /dev/shm
tmpfs           32896424  239080  32657344   1% /run
tmpfs           32896424       0  32896424   0% /sys/fs/cgroup
/dev/xvdf      103081248 2790172  96015008   3% /app
/dev/dm-3       10473472  260024  10213448   3% /app/docker/devicemapper/mnt/d21a02d860b705dccf3b79c57ffcf0a863d13ddfe061d40a0d967a4f5bc2169e
shm                65536       0     65536   0% /app/docker/containers/4c2331930fcefd69f5c9dd7e401cd678c24c89450ac6d9185135dfc40d747e37/shm
shm                65536       0     65536   0% /app/docker/containers/99a4b36771e0db20fc1360da3072d826e9f78faaa640720c85b90c0a6c1db33e/shm
/dev/dm-5       10473472  278428  10195044   3% /app/docker/devicemapper/mnt/c369cf523f69144ec83d35ed9ae49f02cb088927a5d19824cd7733706199d4ea
shm                65536       0     65536   0% /app/docker/containers/1a0e1c0452ff3762c5c2d3daca9a744f154a91b49884c774eb978b22b7707e76/shm
/dev/dm-7       10473472  294072  10179400   3% /app/docker/devicemapper/mnt/903969c07c44bbf949ef7b4b24fbd6fcf1ebbcfbf798c79ad415b03bd1eb1a27
shm                65536       0     65536   0% /app/docker/containers/e253881cf82cde743cb13653de1a886bdfe1b5830feeb1793694ea5fa41a1469/shm
/dev/dm-8       10473472  410340  10063132   4% /app/docker/devicemapper/mnt/a238e6a06ab4820134e95a6924eb5a8ff93fbdc1be9cd7bb9f01b6814f147c6d
shm                65536       0     65536   0% /app/docker/containers/bc25d0d82a98ad55a04d9d409fdd7c1ab271f010ecec6261c1fba5fee553836c/shm
/dev/dm-9       10473472  267064  10206408   3% /app/docker/devicemapper/mnt/731c75288780b5cd959925096d8cfe2114029091637266b39cabe170605c7223
shm                65536       0     65536   0% /app/docker/containers/8855132be26f5043630d073e64938330340e22060a2dd45e8d2fadee44d0b067/shm
/dev/dm-10      10473472  275788  10197684   3% /app/docker/devicemapper/mnt/029be07c5533645e1d2177b32383386b8a418947cf2f577183e5cd8fbf3e6811
shm                65536       0     65536   0% /app/docker/containers/ec5abed6e23f3b05c6d10cb26e2d9e132ec4b16cf374ad45a82a71163d38f22b/shm
/dev/dm-11      10473472  294332  10179140   3% /app/docker/devicemapper/mnt/73dc5c60598e25b2e13539e816fe747ef18b7797fe0187e5049a9fea68ab90ac
shm                65536       0     65536   0% /app/docker/containers/d704d9391a673166819f27bd8b9e294697c9dd8283f60035919f1fa2fda50d54/shm
/dev/dm-14      10473472   81860  10391612   1% /app/docker/devicemapper/mnt/3d40fc33d3ab1fb42c2f8df9f74b5154951d765e8ba070e654b01c7f1bc656db
shm                65536       0     65536   0% /app/docker/containers/ccb098bab9cb8a5907785bfe741b79445dae5113ee61cc3d339e4d07d43fa761/shm
tmpfs            6579288       0   6579288   0% /run/user/0
/dev/dm-6       10473472 2626616   7846856  26% /app/docker/devicemapper/mnt/3da983fbe101aa59943b9c2d69c28538a08dcb886b8fd779852910e7d5404e0a
shm                65536       0     65536   0% /app/docker/containers/be09c2017aa71a6bda025e4709751958c42abce165624f4a30c723632600ec64/shm
/dev/dm-12      10473472  193356  10280116   2% /app/docker/devicemapper/mnt/3d527d49b0b344c26f38af5c567e01678493b2936ed71be66e12d23383bc150c
shm                65536       0     65536   0% /app/docker/containers/a1c56fa2ca4779529765f3a703264240e898b87b5ca02a5717abf72be819db15/shm
/dev/dm-13      10473472 1007124   9466348  10% /app/docker/devicemapper/mnt/b23bc01d6b87dfe523aa92cbf822e9c111f1610665b122291b3f884ef6a2f4e8
shm                65536       0     65536   0% /app/docker/containers/2af9689dfec03b79f2697b42de58cd34cd760d6ef1141d6af0e7e65657f8df51/shm
tmpfs            6579288       0   6579288   0% /run/user/17010
tmpfs            6579288       0   6579288   0% /run/user/11787

#-> lsblk
NAME                                                                                        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
xvda                                                                                        202:0    0    16G  0 disk
├─xvda1                                                                                     202:1    0     1M  0 part
└─xvda2                                                                                     202:2    0    16G  0 part /
xvdb                                                                                        202:16   0   200G  0 disk
├─vg--docker-vg--thinpool_tmeta                                                             253:0    0   100M  0 lvm
│ └─vg--docker-vg--thinpool                                                                 253:2    0 199.8G  0 lvm
│   ├─docker-202:80-720905-d21a02d860b705dccf3b79c57ffcf0a863d13ddfe061d40a0d967a4f5bc2169e 253:3    0    10G  0 dm   /app/docker/devicemapper/mnt/d21
│   ├─docker-202:80-720905-8f2d7f8d1d402ec145dc9dbe6d9b38b137f3b16c8b3dc2ba54c978e8ec46452c 253:4    0    10G  0 dm
│   ├─docker-202:80-720905-c369cf523f69144ec83d35ed9ae49f02cb088927a5d19824cd7733706199d4ea 253:5    0    10G  0 dm   /app/docker/devicemapper/mnt/c36
│   ├─docker-202:80-720905-3da983fbe101aa59943b9c2d69c28538a08dcb886b8fd779852910e7d5404e0a 253:6    0    10G  0 dm   /app/docker/devicemapper/mnt/3da
│   ├─docker-202:80-720905-903969c07c44bbf949ef7b4b24fbd6fcf1ebbcfbf798c79ad415b03bd1eb1a27 253:7    0    10G  0 dm   /app/docker/devicemapper/mnt/903
│   ├─docker-202:80-720905-a238e6a06ab4820134e95a6924eb5a8ff93fbdc1be9cd7bb9f01b6814f147c6d 253:8    0    10G  0 dm   /app/docker/devicemapper/mnt/a23
│   ├─docker-202:80-720905-731c75288780b5cd959925096d8cfe2114029091637266b39cabe170605c7223 253:9    0    10G  0 dm   /app/docker/devicemapper/mnt/731
│   ├─docker-202:80-720905-029be07c5533645e1d2177b32383386b8a418947cf2f577183e5cd8fbf3e6811 253:10   0    10G  0 dm   /app/docker/devicemapper/mnt/029
│   ├─docker-202:80-720905-73dc5c60598e25b2e13539e816fe747ef18b7797fe0187e5049a9fea68ab90ac 253:11   0    10G  0 dm   /app/docker/devicemapper/mnt/73d
│   ├─docker-202:80-720905-3d527d49b0b344c26f38af5c567e01678493b2936ed71be66e12d23383bc150c 253:12   0    10G  0 dm   /app/docker/devicemapper/mnt/3d5
│   ├─docker-202:80-720905-b23bc01d6b87dfe523aa92cbf822e9c111f1610665b122291b3f884ef6a2f4e8 253:13   0    10G  0 dm   /app/docker/devicemapper/mnt/b23
│   └─docker-202:80-720905-3d40fc33d3ab1fb42c2f8df9f74b5154951d765e8ba070e654b01c7f1bc656db 253:14   0    10G  0 dm   /app/docker/devicemapper/mnt/3d4
└─vg--docker-vg--thinpool_tdata                                                             253:1    0 199.8G  0 lvm
  └─vg--docker-vg--thinpool                                                                 253:2    0 199.8G  0 lvm
    ├─docker-202:80-720905-d21a02d860b705dccf3b79c57ffcf0a863d13ddfe061d40a0d967a4f5bc2169e 253:3    0    10G  0 dm   /app/docker/devicemapper/mnt/d21
    ├─docker-202:80-720905-8f2d7f8d1d402ec145dc9dbe6d9b38b137f3b16c8b3dc2ba54c978e8ec46452c 253:4    0    10G  0 dm
    ├─docker-202:80-720905-c369cf523f69144ec83d35ed9ae49f02cb088927a5d19824cd7733706199d4ea 253:5    0    10G  0 dm   /app/docker/devicemapper/mnt/c36
    ├─docker-202:80-720905-3da983fbe101aa59943b9c2d69c28538a08dcb886b8fd779852910e7d5404e0a 253:6    0    10G  0 dm   /app/docker/devicemapper/mnt/3da
    ├─docker-202:80-720905-903969c07c44bbf949ef7b4b24fbd6fcf1ebbcfbf798c79ad415b03bd1eb1a27 253:7    0    10G  0 dm   /app/docker/devicemapper/mnt/903
    ├─docker-202:80-720905-a238e6a06ab4820134e95a6924eb5a8ff93fbdc1be9cd7bb9f01b6814f147c6d 253:8    0    10G  0 dm   /app/docker/devicemapper/mnt/a23
    ├─docker-202:80-720905-731c75288780b5cd959925096d8cfe2114029091637266b39cabe170605c7223 253:9    0    10G  0 dm   /app/docker/devicemapper/mnt/731
    ├─docker-202:80-720905-029be07c5533645e1d2177b32383386b8a418947cf2f577183e5cd8fbf3e6811 253:10   0    10G  0 dm   /app/docker/devicemapper/mnt/029
    ├─docker-202:80-720905-73dc5c60598e25b2e13539e816fe747ef18b7797fe0187e5049a9fea68ab90ac 253:11   0    10G  0 dm   /app/docker/devicemapper/mnt/73d
    ├─docker-202:80-720905-3d527d49b0b344c26f38af5c567e01678493b2936ed71be66e12d23383bc150c 253:12   0    10G  0 dm   /app/docker/devicemapper/mnt/3d5
    ├─docker-202:80-720905-b23bc01d6b87dfe523aa92cbf822e9c111f1610665b122291b3f884ef6a2f4e8 253:13   0    10G  0 dm   /app/docker/devicemapper/mnt/b23
    └─docker-202:80-720905-3d40fc33d3ab1fb42c2f8df9f74b5154951d765e8ba070e654b01c7f1bc656db 253:14   0    10G  0 dm   /app/docker/devicemapper/mnt/3d4
xvdf                                                                                        202:80   0   100G  0 disk /app

The docker device /dev/xvdb is not mounted by the system, but is under control by the docker application. Therefore all thin-pool mounted file systems should not be included in savelayout.conf file:

#-> cat /var/lib/rear/layout/disklayout.conf
# Disk /dev/xvda
# Format: disk <devname> <size(bytes)> <partition label type>
disk /dev/xvda 17179869184 gpt
# Partitions on /dev/xvda
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
part /dev/xvda 1048576 1048576 rear-noname bios_grub /dev/xvda1
part /dev/xvda 17177755136 2097152 rear-noname none /dev/xvda2
# Disk /dev/xvdb
# Format: disk <devname> <size(bytes)> <partition label type>
#disk /dev/xvdb 214748364800 unknown
# Partitions on /dev/xvdb
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
# Disk /dev/xvdf
# Format: disk <devname> <size(bytes)> <partition label type>
disk /dev/xvdf 107374182400 loop
# Partitions on /dev/xvdf
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
#lvmdev /dev/vg-docker /dev/xvdb qVorYA-oIev-qunB-hscU-nJH4-t2c5-8zvfK5 419430400
#lvmgrp /dev/vg-docker 4096 51199 209711104
#lvmvol /dev/vg-docker vg-thinpool 51149 419012608
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/mapper/docker-202:80-720905-029be07c5533645e1d2177b32383386b8a418947cf2f577183e5cd8fbf3e6811 /app/docker/devicemapper/mnt/029be07c5533645e1d2177b32383386b8a418947cf2f577183e5cd8fbf3e6811 xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-3d40fc33d3ab1fb42c2f8df9f74b5154951d765e8ba070e654b01c7f1bc656db /app/docker/devicemapper/mnt/3d40fc33d3ab1fb42c2f8df9f74b5154951d765e8ba070e654b01c7f1bc656db xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-3d527d49b0b344c26f38af5c567e01678493b2936ed71be66e12d23383bc150c /app/docker/devicemapper/mnt/3d527d49b0b344c26f38af5c567e01678493b2936ed71be66e12d23383bc150c xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-3da983fbe101aa59943b9c2d69c28538a08dcb886b8fd779852910e7d5404e0a /app/docker/devicemapper/mnt/3da983fbe101aa59943b9c2d69c28538a08dcb886b8fd779852910e7d5404e0a xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-731c75288780b5cd959925096d8cfe2114029091637266b39cabe170605c7223 /app/docker/devicemapper/mnt/731c75288780b5cd959925096d8cfe2114029091637266b39cabe170605c7223 xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-73dc5c60598e25b2e13539e816fe747ef18b7797fe0187e5049a9fea68ab90ac /app/docker/devicemapper/mnt/73dc5c60598e25b2e13539e816fe747ef18b7797fe0187e5049a9fea68ab90ac xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-903969c07c44bbf949ef7b4b24fbd6fcf1ebbcfbf798c79ad415b03bd1eb1a27 /app/docker/devicemapper/mnt/903969c07c44bbf949ef7b4b24fbd6fcf1ebbcfbf798c79ad415b03bd1eb1a27 xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-a238e6a06ab4820134e95a6924eb5a8ff93fbdc1be9cd7bb9f01b6814f147c6d /app/docker/devicemapper/mnt/a238e6a06ab4820134e95a6924eb5a8ff93fbdc1be9cd7bb9f01b6814f147c6d xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-b23bc01d6b87dfe523aa92cbf822e9c111f1610665b122291b3f884ef6a2f4e8 /app/docker/devicemapper/mnt/b23bc01d6b87dfe523aa92cbf822e9c111f1610665b122291b3f884ef6a2f4e8 xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-c369cf523f69144ec83d35ed9ae49f02cb088927a5d19824cd7733706199d4ea /app/docker/devicemapper/mnt/c369cf523f69144ec83d35ed9ae49f02cb088927a5d19824cd7733706199d4ea xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/mapper/docker-202:80-720905-d21a02d860b705dccf3b79c57ffcf0a863d13ddfe061d40a0d967a4f5bc2169e /app/docker/devicemapper/mnt/d21a02d860b705dccf3b79c57ffcf0a863d13ddfe061d40a0d967a4f5bc2169e xfs uuid=b382d6a5-9147-47b2-8c3a-6ef7c33751dc label=  options=rw,relatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=256,noquota
fs /dev/xvda2 / xfs uuid=de4def96-ff72-4eb9-ad5e-0847257d1866 label=  options=rw,relatime,attr2,inode64,noquota
fs /dev/xvdf /app ext3 uuid=86eaf035-485c-483d-a44e-cf8c4c8a0868 label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
  • Work-around, if any:
@gdha gdha added enhancement Adaptions and new features bug The code does not do what it is meant to do labels Mar 8, 2018
@gdha gdha self-assigned this Mar 8, 2018
@jsmeix
Copy link
Member

jsmeix commented Mar 8, 2018

Only FYI:
We had already #1496
Error during layout recreation with btrfs and docker
where docker stuff appears as lots of btrfs subvolumes
under /var/lib/docker/btrfs/subvolumes/...
which was fixed by @OliverO2 via
#1497

@gdha gdha added this to the ReaR v2.4 milestone Mar 13, 2018
@gdha gdha mentioned this issue Mar 14, 2018
Merged
gdha added a commit that referenced this issue Mar 16, 2018
@gdha
Merge pull request #1757 from gdha/master
5f93dff 
exclude docker FS from layout #1749
@jsmeix
Copy link
Member

jsmeix commented Mar 19, 2018

I assume since #1757 is merged
this issue is "fixed/solved/done" and can be closed.
@gdha if not just reopen it.

@jsmeix jsmeix closed this as completed Mar 19, 2018
@gdha
Copy link
Member Author

gdha commented Apr 14, 2020

We still see messages like:

2020-04-14 16:21:23.170257675 /dev/mapper/vg00-lv_audit is mounted below /app/gtsc/docker (mount point /var/log/audit is under docker control), skipping.

but which are incorrect. The savelayout does contain:

lvmvol /dev/vg00 lv_audit 4294967296b linear
fs /dev/mapper/vg00-lv_audit /var/log/audit ext3 uuid=8e92b10f-1065-46ff-ae41-eb9dcf594a25 label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16384 default_mount_options=user_xattr,acl options=rw,relatime,data=ordered

The wrong assumption happens in script /usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh

        if is_true $docker_is_running ; then
            # docker daemon/service is running
            docker_root_dir=$( docker info 2>/dev/null | grep 'Docker Root Dir' | awk '{print $4}' )
            # If $docker_root_dir is in the beginning of the $mountpoint string then FS is under docker control
            # and we better exclude from saving the layout,
            # see https://github.com/rear/rear/issues/1749
            Log "$device is mounted below $docker_root_dir (mount point $mountpoint is under docker control), skipping."
            echo "$mountpoint" | grep -q "^$docker_root_dir" && continue
        fi

The Log should only happen when we find a hit on docker_root_dir.
ReaR 2.5 has the issue, but also the official RHEL rear-2.4-10.el7_7.x86_64 RPM has been bitten by it.
@rmetrich is this something you could fix in the rear-2.4 branch of RHEL?

@gdha gdha reopened this Apr 14, 2020
@gdha gdha removed this from the ReaR v2.4 milestone Apr 14, 2020
@rmetrich
Copy link
Contributor

@pcahyna I'm sorry I'm under the sea since a few months. Could you have a look at what is reported here?

@gdha
Copy link
Member Author

gdha commented Apr 14, 2020

@rmetrich Under the sea? What do you mean (just for my knowledge if I can still bother you or not)?

@rmetrich
Copy link
Contributor

Well I'm in a deep dive under the water :-) and there is lots of water

@pcahyna
Copy link
Member

pcahyna commented Apr 14, 2020

Is it related to #1989 (fixed by #2021)? A fix for that one was released with RHEL 7.8 two weeks ago. @yontalcar can you please have a look?

@gdha gdha added this to the ReaR v2.6 milestone Apr 16, 2020
@gdha
Copy link
Member Author

gdha commented Apr 16, 2020

Code is fixed in master branch.
@pcahyna @yontalcar The only remaining question is now will RedHat back-port this to RHEL 7.7 as well?

gdha added a commit to gdha/rear that referenced this issue Apr 20, 2020
@gdha
docker_root_dir itself may not be excluded, all what is beneath should …
4000549 
…rear#1749 rear#2365

Signed-off-by: Gratien D'haese <gratien.dhaese@gmail.com>
@gdha
Copy link
Member Author

gdha commented Apr 20, 2020 •
edited

@pcahyna @yontalcar Found a minor bug in the script 230_filesystem_layout.sh which also exclude the mount point docker_root_dir itself, which should not be the case but only the mountpoints beneath docker_root_dir.
I think this fix should be inlcuded in your update to the rear package in rhel 7.7, 7.8 and 8.x

@jsmeix
Copy link
Member

jsmeix commented Apr 29, 2020

I think this isue is fixed by
#2373
so that it can be closed.
Feel free to reopen it if I am wrong.

@jsmeix jsmeix closed this as completed Apr 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gdha gdha

@rmetrich rmetrich

Labels
bug The code does not do what it is meant to do enhancement Adaptions and new features fixed / solved / done
Projects
None yet
Milestone
ReaR v2.6
Development

No branches or pull requests
4 participants
@gdha @rmetrich @jsmeix @pcahyna