subdir mount X-mount.subdir= option do not work on bcachefs

[detiam]

I plan to mount a bcachefs snapshot as root today, so I ran:

# mount -v -o X-mount.subdir=.snapshots/1 /dev/nvme0n1p2 /mnt/
INFO - bcachefs::commands::cmd_mount: mounting with params: device: /dev/nvme0n1p2, target: /run/mount/tmptgt, options: rw
DEBUG - bcachefs::commands::cmd_mount: parsing mount options: rw
INFO - bcachefs::commands::cmd_mount: mounting bcachefs filesystem, /run/mount/tmptgt
INFO - bcachefs::commands::cmd_mount: mounting filesystem
INFO - bcachefs::commands::cmd_mount: Successfully mounted

the command exit with 0, but /mnt/isn't mount, nothing change.
I have test X-mount.subdir on my system with ext4, btrfs, these can mount folder in target filesystem successfully.

---

os: archlinux rolling
ver: 2.39.3-2 in archlinux core repo

[dblaber]

@detiam I am seeing same issue behavior, latest bcachefs tools in arch (1.6), util-linux. By any chance, are you using an encrypted volume? I am trying to narrow it down, as I am trying to use encrypted volume + separate subvol as root (using X-mount.subdir). Trying to narrow down issue before I open it on bcachefs side.

[detiam]

@dblaber no I only enabled lz4 background compress

[dblaber]

@dblaber no I only enabled lz4 background compress

Hmm, very weird then... weirdly some folks on reddit say that mount is working. Which kernel version are you using? (I am still on latest 6.7x in arch, although tempted to try 6.8 just to see if it fixes)

[detiam]

@dblaber no I only enabled lz4 background compress

Hmm, very weird then... weirdly some folks on reddit say that mount is working. Which kernel version are you using? (I am still on latest 6.7x in arch, although tempted to try 6.8 just to see if it fixes)

Linux Tinkling-PC 6.7.8-zen1-1.1-zen #1 ZEN SMP PREEMPT_DYNAMIC Sun, 03 Mar 2024 17:02:55 +0000 x86_64 GNU/Linux

[yemouu]

I'm getting the same issue, I can get the subdir mounted if i have the environment variable LIBMOUNT_FORCE_MOUNT2=always

I can also replicate this with NTFS where X-mount.subdir only works if the above environment variable is set. So maybe this is an issue with filesystems that have mount helpers?

When running without the above environment variable and with LIBMOUNT_DEBUG=all I get the following:

90115: libmount:     INIT: library debug mask: 0xffffff
90115: libmount:     INIT: library version: 2.39.3
90115: libmount:     INIT:     feature: btrfs
90115: libmount:     INIT:     feature: namespaces
90115: libmount:     INIT:     feature: idmapping
90115: libmount:     INIT:     feature: fd-based-mount
90115: libmount:     INIT:     feature: statx
90115: libmount:     INIT:     feature: assert
90115: libmount:     INIT:     feature: debug
Available "LIBMOUNT_DEBUG=<name>[,...]|<mask>" debug masks:
   all      [0xffffff] : info about all subsystems
   cache    [0x000004] : paths and tags cache
   cxt      [0x000200] : library context (handler)
   diff     [0x000400] : mountinfo changes tracking
   fs       [0x000040] : FS abstraction
   help     [0x000001] : this help
   hook     [0x008000] : hooks functionality
   locks    [0x000010] : mtab and utab locking
   loop     [0x002000] : loop devices routines
   options  [0x000008] : mount options parsing
   optlist  [0x010000] : mount options container
   tab      [0x000020] : fstab, mtab, mountinfo routines
   update   [0x000080] : mtab, utab updates
   utils    [0x000100] : misc library utils
   monitor  [0x000800] : mount tables monitor
   btrfs    [0x001000] : btrfs specific routines
   verity   [0x004000] : verity specific routines
90115: libmount:      CXT: [0x234d680]: ----> allocate 
90115: libmount:  OPTLIST: [0x234d7e0]: alloc
90115: libmount:  OPTLIST: [0x234d7e0]: registr map 0x7f458c01aec0
90115: libmount:  OPTLIST: [0x234d7e0]: registr map 0x7f458c01acc0
90115: libmount:  OPTLIST: [0x234d7e0]: append X-mount.subdir=etc
90115: libmount:  OPTLIST: [0x234d7e0]:  added X-mount.subdir [id=0x00020000 map=0x7f458c01acc0]
90115: libmount:       FS: [0x234da00]: alloc
90115: libmount:      CXT: [0x234d680]: mount: preparing
90115: libmount:      CXT: [0x234d680]: use default optsmode
90115: libmount:  OPTLIST: [0x234d7e0]: return flags 0x00000000 [map=0x7f458c01aec0]
90115: libmount:      CXT: [0x234d680]: OPTSMODE (file-part): force=0, fstab=1, mtab=1
90115: libmount:      CXT: [0x234d680]: fstab not required -- skip
90115: libmount:  OPTLIST: [0x234d7e0]: merging
90115: libmount:      CXT: [0x234d680]: mount: evaluating permissions
90115: libmount:  OPTLIST: [0x234d7e0]: return flags 0x00020000 [map=0x7f458c01acc0]
90115: libmount:      CXT: [0x234d680]: perms: superuser [rc=0]
90115: libmount:      CXT: [0x234d680]: --> preparing options
90115: libmount:      CXT: [0x234d680]: ---> stage:prep-options
90115: libmount:      CXT: [0x234d680]: calling __idmap [first]
90115: libmount:      CXT: [0x234d680]: calling __owner [first]
90115: libmount:      CXT: [0x234d680]: <--- stage:prep-options [rc=0 status=1]
90115: libmount:      CXT: [0x234d680]: <-- preparing options done [rc=0]
90115: libmount:      CXT: [0x234d680]: --> preparing source path
90115: libmount:      CXT: [0x234d680]: srcpath '/dev/nvme0n1p2'
90115: libmount:    CACHE: [0x234db30]: alloc
90115: libmount:    CACHE: [0x234db30]: canonicalize path /dev/nvme0n1p2
90115: libmount:    CACHE: [0x234db30]: add entry [ 1] (path): /dev/nvme0n1p2: /dev/nvme0n1p2
90115: libmount:      CXT: [0x234d680]: ---> stage:prep-source
90115: libmount:      CXT: [0x234d680]: calling __loopdev [first]
90115: libmount:  OPTLIST: [0x234d7e0]: return flags 0x00020000 [map=0x7f458c01acc0]
90115: libmount:      CXT: [0x234d680]: <--- stage:prep-source [rc=0 status=1]
90115: libmount:      CXT: [0x234d680]: final srcpath '/dev/nvme0n1p2'
90115: libmount:      CXT: [0x234d680]: --> preparing fstype
90115: libmount:    CACHE: [0x234db30]: get /dev/nvme0n1p2 FS type
90115: libmount:    CACHE: [0x234db30]: tags for /dev/nvme0n1p2 requested
90115: libmount:    CACHE: [0x234db30]: reading tags for: /dev/nvme0n1p2
90115: libmount:    CACHE: [0x234db30]: add entry [ 2] (tag): /dev/nvme0n1p2: UUID
90115: libmount:    CACHE: [0x234db30]: add entry [ 3] (tag): /dev/nvme0n1p2: TYPE
90115: libmount:    CACHE: [0x234db30]: add entry [ 4] (tag): /dev/nvme0n1p2: PARTUUID
90115: libmount:    CACHE: [0x234db30]: 	read 3 tags
90115: libmount:      CXT: [0x234d680]: FS type: bcachefs [rc=0]
90115: libmount:      CXT: [0x234d680]: --> preparing target path
90115: libmount:    CACHE: [0x234db30]: canonicalize path /mnt
90115: libmount:    CACHE: [0x234db30]: add entry [ 5] (path): /mnt: /mnt
90115: libmount:      CXT: [0x234d680]: ---> stage:prep-target
90115: libmount:      CXT: [0x234d680]: calling __mkdir [first]
90115: libmount:      CXT: [0x234d680]: calling __subdir [first]
90115: libmount:      CXT: [0x234d680]:  alloc '__subdir' data
90115: libmount:     HOOK: [0x7f458c01b820]: subdir etc wanted
90115: libmount:      CXT: [0x234d680]:  appending pre-mount hook from __subdir
90115: libmount:      CXT: [0x234d680]: <--- stage:prep-target [rc=0 status=1]
90115: libmount:      CXT: [0x234d680]: final target '/mnt' [rc=0]
90115: libmount:      CXT: [0x234d680]: checking for helper
90115: libmount:      CXT: [0x234d680]: /run/wrappers/bin/mount.bcachefs ... not found
90115: libmount:      CXT: [0x234d680]: /run/current-system/sw/bin/mount.bcachefs ... found
90115: libmount:      CXT: [0x234d680]: ---> stage:prep
90115: libmount:      CXT: [0x234d680]: calling __mount [first]
90115: libmount:     HOOK: [0x7f458c01b7c0]: prepare mount
90115: libmount:  OPTLIST: [0x234d7e0]: return flags 0x00000000 [map=0x7f458c01aec0]
90115: libmount:  OPTLIST: [0x234d7e0]: return attrs set=0x00000000, clr=0x00000000 
90115: libmount:     HOOK: [0x7f458c01b7c0]: initialize API fds
90115: libmount:      CXT: [0x234d680]:  alloc '__mount' data
90115: libmount:     HOOK: [0x7f458c01b7c0]: prepare mount done [rc=0]
90115: libmount:      CXT: [0x234d680]: calling __legacy-mount [first]
90115: libmount:  OPTLIST: [0x234d7e0]: return flags 0x00000000 [map=0x7f458c01aec0]
90115: libmount:      CXT: [0x234d680]: <--- stage:prep [rc=0 status=1]
90115: libmount:      CXT: [0x234d680]: --> prepare update
90115: libmount:      CXT: [0x234d680]: utab path initialized to: /run/mount/utab
90115: libmount:      CXT: [0x234d680]: checking for writable tab files
90115: libmount:    UTILS: utab: /run/mount/utab
90115: libmount:    UTILS: try write /run/mount/utab dir: (null)
90115: libmount:    UTILS:  access OK
90115: libmount:   UPDATE: [0x23550e0]: allocate
90115: libmount:  OPTLIST: [0x234d7e0]: return flags 0x00000000 [map=0x7f458c01aec0]
90115: libmount:   UPDATE: [0x23550e0]: resetting FS [target=(null), flags=0x00000000]
90115: libmount:   UPDATE: [0x23550e0]: FS template:
90115: libmount:   UPDATE: 90115: libmount:       FS: [0x234da00]: synced: vfs: 'rw' fs: '(null)' user: 'X-mount.subdir=etc', optstr: 'rw,X-mount.subdir=etc'
------ fs:
source: /dev/nvme0n1p2
target: /mnt
fstype: bcachefs
optstr: rw,X-mount.subdir=etc
VFS-optstr: rw
user-optstr: X-mount.subdir=etc
90115: libmount:   UPDATE: prepare utab entry
90115: libmount:   UPDATE: utab entry unnecessary (no options)
90115: libmount:      CXT: [0x234d680]: mount: do mount
90115: libmount:      CXT: [0x234d680]: ---> stage:pre-mount
90115: libmount:      CXT: [0x234d680]: calling __subdir [active]
90115: libmount:    UTILS: /run/mount/tmptgt unshared
90115: libmount:      CXT: [0x234d680]:  appending post-mount hook from __subdir
90115: libmount:     HOOK: [0x7f458c01b820]: unshared tmp target /run/mount/tmptgt [rc=0]
90115: libmount:      CXT: [0x234d680]: <--- stage:pre-mount [rc=0 status=1]
90115: libmount:      CXT: [0x234d680]: mount: executing helper /run/current-system/sw/bin/mount.bcachefs
90116: libmount:      CXT: [0x234d680]: argv[0] = "/run/current-system/sw/bin/mount.bcachefs"
90116: libmount:      CXT: [0x234d680]: argv[1] = "/dev/nvme0n1p2"
90116: libmount:      CXT: [0x234d680]: argv[2] = "/run/mount/tmptgt"
90116: libmount:      CXT: [0x234d680]: argv[3] = "-o"
90116: libmount:      CXT: [0x234d680]: argv[4] = "rw"
90115: libmount:      CXT: [0x234d680]: /run/current-system/sw/bin/mount.bcachefs executed [status=0, rc=0]
90115: libmount:      CXT: [0x234d680]: ---> stage:post-mount
90115: libmount:      CXT: [0x234d680]: calling __subdir [active]
90115: libmount:     HOOK: attach subdir  etc
90115: libmount:      CXT: syscall 'open_tree' [Bad file descriptor]
90115: libmount:      CXT: [0x234d680]: <--- stage:post-mount [rc=-9 status=-9]
90115: libmount:     HOOK: [0x7f458c01b880]: deinit '__loopdev'
90115: libmount:     HOOK: [0x7f458c01b800]: deinit '__mkdir'
90115: libmount:     HOOK: [0x7f458c01b820]: deinit '__subdir'
90115: libmount:      CXT: [0x234d680]:  removing pre-mount hook from __subdir
90115: libmount:      CXT: [0x234d680]:  removing post-mount hook from __subdir
90115: libmount:    UTILS: /run/mount/tmptgt cleanup done
90115: libmount:      CXT: [0x234d680]:  free '__subdir' data
90115: libmount:     HOOK: [0x7f458c01b7c0]: deinit '__mount'
90115: libmount:      CXT: [0x234d680]:  free '__mount' data
90115: libmount:     HOOK: [0x7f458c01b7e0]: deinit '__legacy-mount'
90115: libmount:     HOOK: [0x7f458c01b860]: deinit '__idmap'
90115: libmount:     HOOK: [0x7f458c01b840]: deinit '__owner'
90115: libmount:      CXT: [0x234d680]: mnt_context_mount() done [rc=-9]
90115: libmount:      CXT: [0x234d680]: excode: rc=0 message=""
90115: libmount:      CXT: [0x234d680]: <---- reset [status=1] ---->
90115: libmount:       FS: [0x234da00]: free [refcount=0]
90115: libmount:  OPTLIST: [0x234d7e0]:  remove X-mount.subdir
90115: libmount:    CACHE: [0x234db30]: free [refcount=0]
90115: libmount:   UPDATE: [0x23550e0]: free
90115: libmount:      CXT: [0x234d680]: Setting (null) as target namespace
90115: libmount:      CXT: [0x234d680]: free

[t-8ch]

@yemouu Do you have a strace log to the libmount log?

[yemouu]

@t-8ch here is the strace output

execve("/run/wrappers/bin/mount", ["mount", "/dev/nvme0n1p2", "/mnt", "-o", "X-mount.subdir=etc"], 0x7ffda2596f50 /* 41 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x40e2b8)       = 0
set_tid_address(0x40e3f0)               = 94803
getxattr("/proc/self/exe", "security.capability", 0x7ffe63e14ed0, 24) = -1 ENODATA (No data available)
execve("/nix/store/sjd7437h8vhn8vdrrgz0dmcbdgzh972m-util-linux-2.39.3-bin/bin/mount", ["mount", "/dev/nvme0n1p2", "/mnt", "-o", "X-mount.subdir=etc"], 0x7ffe63e14fb8 /* 39 vars */) = 0
brk(NULL)                               = 0xe29000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffe0dbf6a70) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8a0a571000
access("/etc/ld-nix.so.preload", R_OK)  = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/c1zmd6rnss7nzhddzxax983nn7ksriv9-util-linux-2.39.3-lib/lib/glibc-hwcaps/x86-64-v3/libmount.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/c1zmd6rnss7nzhddzxax983nn7ksriv9-util-linux-2.39.3-lib/lib/glibc-hwcaps/x86-64-v3/", 0x7ffe0dbf5ca0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/c1zmd6rnss7nzhddzxax983nn7ksriv9-util-linux-2.39.3-lib/lib/glibc-hwcaps/x86-64-v2/libmount.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/c1zmd6rnss7nzhddzxax983nn7ksriv9-util-linux-2.39.3-lib/lib/glibc-hwcaps/x86-64-v2/", 0x7ffe0dbf5ca0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/c1zmd6rnss7nzhddzxax983nn7ksriv9-util-linux-2.39.3-lib/lib/libmount.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0555, st_size=515160, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 459336, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f8a0a500000
mmap(0x7f8a0a50d000, 299008, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd000) = 0x7f8a0a50d000
mmap(0x7f8a0a556000, 94208, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x56000) = 0x7f8a0a556000
mmap(0x7f8a0a56d000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6c000) = 0x7f8a0a56d000
close(3)                                = 0
openat(AT_FDCWD, "/nix/store/c1zmd6rnss7nzhddzxax983nn7ksriv9-util-linux-2.39.3-lib/lib/libblkid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0555, st_size=425680, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 373696, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f8a0a4a4000
mmap(0x7f8a0a4ae000, 233472, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x7f8a0a4ae000
mmap(0x7f8a0a4e7000, 73728, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x43000) = 0x7f8a0a4e7000
mmap(0x7f8a0a4f9000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x55000) = 0x7f8a0a4f9000
close(3)                                = 0
openat(AT_FDCWD, "/nix/store/c1zmd6rnss7nzhddzxax983nn7ksriv9-util-linux-2.39.3-lib/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44/lib/glibc-hwcaps/x86-64-v3/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44/lib/glibc-hwcaps/x86-64-v3/", 0x7ffe0dbf5c60, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44/lib/glibc-hwcaps/x86-64-v2/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44/lib/glibc-hwcaps/x86-64-v2/", 0x7ffe0dbf5c60, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\202\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
newfstatat(3, "", {st_mode=S_IFREG|0555, st_size=2265064, ...}, AT_EMPTY_PATH) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 2002320, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f8a0a2bb000
mmap(0x7f8a0a2e1000, 1421312, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f8a0a2e1000
mmap(0x7f8a0a43c000, 348160, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x181000) = 0x7f8a0a43c000
mmap(0x7f8a0a491000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1d5000) = 0x7f8a0a491000
mmap(0x7f8a0a497000, 52624, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f8a0a497000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8a0a2b9000
arch_prctl(ARCH_SET_FS, 0x7f8a0a2ba080) = 0
set_tid_address(0x7f8a0a2ba350)         = 94803
set_robust_list(0x7f8a0a2ba360, 24)     = 0
rseq(0x7f8a0a2ba9a0, 0x20, 0, 0x53053053) = 0
mprotect(0x7f8a0a491000, 16384, PROT_READ) = 0
mprotect(0x7f8a0a4f9000, 24576, PROT_READ) = 0
mprotect(0x7f8a0a56d000, 12288, PROT_READ) = 0
mprotect(0x40e000, 4096, PROT_READ)     = 0
mprotect(0x7f8a0a5a4000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
getrandom("\xa7\x7d\x42\x94\xbe\x76\x09\xd7", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0xe29000
brk(0xe4a000)                           = 0xe4a000
openat(AT_FDCWD, "/run/current-system/sw/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=3058640, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 3058640, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8a09fce000
close(3)                                = 0
getuid()                                = 0
geteuid()                               = 0
readlink("/dev", 0x7ffe0dbf5390, 1023)  = -1 EINVAL (Invalid argument)
readlink("/dev/nvme0n1p2", 0x7ffe0dbf5390, 1023) = -1 EINVAL (Invalid argument)
newfstatat(AT_FDCWD, "/dev/nvme0n1p2", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x103, 0x2), ...}, 0) = 0
access("/dev/nvme0n1p2", F_OK)          = 0
openat(AT_FDCWD, "/dev/nvme0n1p2", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 3
fadvise64(3, 0, 0, POSIX_FADV_RANDOM)   = 0
newfstatat(3, "", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x103, 0x2), ...}, AT_EMPTY_PATH) = 0
ioctl(3, BLKGETSIZE64, [999666221056])  = 0
ioctl(3, FDGETFDCSTAT, 0x7ffe0dbf45e0)  = -1 ENOTTY (Inappropriate ioctl for device)
openat(AT_FDCWD, "/sys/dev/block/259:2", O_RDONLY|O_CLOEXEC) = 4
openat(4, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(4)                                = 0
openat(AT_FDCWD, "/sys/dev/block/259:2", O_RDONLY|O_CLOEXEC) = 4
faccessat2(4, "partition", F_OK, 0)     = 0
readlink("/sys/dev/block/259:2", "../../devices/pci0000:00/0000:00"..., 4095) = 79
openat(AT_FDCWD, "/sys/block/nvme0n1/dev", O_RDONLY|O_CLOEXEC) = 5
newfstatat(5, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0
read(5, "259:0\n", 4096)                = 6
close(5)                                = 0
close(4)                                = 0
ioctl(3, BLKGETZONESZ, [0])             = 0
lseek(3, 0, SEEK_SET)                   = 0
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 16384, SEEK_SET)               = 16384
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 32768, SEEK_SET)               = 32768
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 65536, SEEK_SET)               = 65536
read(3, "%U\337\32\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 131072, SEEK_SET)              = 131072
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 262144, SEEK_SET)              = 262144
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 524288, SEEK_SET)              = 524288
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 1048576, SEEK_SET)             = 1048576
read(3, "(\213\231\35\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 2097152, SEEK_SET)             = 2097152
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 4194304, SEEK_SET)             = 4194304
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 999666155520, SEEK_SET)        = 999666155520
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 64) = 64
lseek(3, 999666212864, SEEK_SET)        = 999666212864
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 256) = 256
lseek(3, 4096, SEEK_SET)                = 4096
read(3, "&T\315\207\0\0\0\0\0\0\0\0\0\0\0\0\4\4\3\4\0\0\0\0\306\205s\366f\316\220\251"..., 256) = 256
lseek(3, 999666220544, SEEK_SET)        = 999666220544
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 999666089472, SEEK_SET)        = 999666089472
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
lseek(3, 4096, SEEK_SET)                = 4096
read(3, "&T\315\207\0\0\0\0\0\0\0\0\0\0\0\0\4\4\3\4\0\0\0\0\306\205s\366f\316\220\251"..., 1024) = 1024
lseek(3, 4096, SEEK_SET)                = 4096
read(3, "&T\315\207\0\0\0\0\0\0\0\0\0\0\0\0\4\4\3\4\0\0\0\0\306\205s\366f\316\220\251"..., 4752) = 4752
lseek(3, 0, SEEK_SET)                   = 0
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(3, 999666216960, SEEK_SET)        = 999666216960
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 104) = 104
lseek(3, 999666216960, SEEK_SET)        = 999666216960
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1392) = 1392
lseek(3, 1024, SEEK_SET)                = 1024
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(3, 1048576, SEEK_SET)             = 1048576
read(3, "(\213\231\35\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(3, 3072, SEEK_SET)                = 3072
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(3, 15360, SEEK_SET)               = 15360
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(3, 31744, SEEK_SET)               = 31744
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(3, 64512, SEEK_SET)               = 64512
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(3, 0, SEEK_SET)                   = 0
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8a09f8d000
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 262144) = 262144
lseek(3, 262144, SEEK_SET)              = 262144
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8a09f4c000
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 262144) = 262144
lseek(3, 999665696768, SEEK_SET)        = 999665696768
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8a09f0b000
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 262144) = 262144
lseek(3, 999665958912, SEEK_SET)        = 999665958912
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8a09eca000
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 262144) = 262144
lseek(3, 2097152, SEEK_SET)             = 2097152
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
ioctl(3, BLKSSZGET, [512])              = 0
openat(AT_FDCWD, "/sys/dev/block/259:0", O_RDONLY|O_CLOEXEC) = 4
readlink("/sys/dev/block/259:0", "../../devices/pci0000:00/0000:00"..., 4095) = 69
newfstatat(AT_FDCWD, "/dev/nvme0n1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x103, 0), ...}, 0) = 0
close(4)                                = 0
openat(AT_FDCWD, "/dev/nvme0n1", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 4
fadvise64(4, 0, 0, POSIX_FADV_RANDOM)   = 0
newfstatat(4, "", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x103, 0), ...}, AT_EMPTY_PATH) = 0
ioctl(4, BLKGETSIZE64, [1000204886016]) = 0
ioctl(4, FDGETFDCSTAT, 0x7ffe0dbf4540)  = -1 ENOTTY (Inappropriate ioctl for device)
openat(AT_FDCWD, "/sys/dev/block/259:0", O_RDONLY|O_CLOEXEC) = 5
openat(5, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(5)                                = 0
openat(AT_FDCWD, "/sys/dev/block/259:0", O_RDONLY|O_CLOEXEC) = 5
faccessat2(5, "partition", F_OK, 0)     = -1 ENOENT (No such file or directory)
openat(5, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(5)                                = 0
ioctl(4, CDROM_GET_CAPABILITY, 0)       = -1 ENOTTY (Inappropriate ioctl for device)
ioctl(4, CDROM_LAST_WRITTEN, 0x7ffe0dbf4530) = -1 ENOTTY (Inappropriate ioctl for device)
ioctl(4, BLKGETZONESZ, [0])             = 0
lseek(4, 0, SEEK_SET)                   = 0
read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
ioctl(4, BLKSSZGET, [512])              = 0
lseek(4, 1032192, SEEK_SET)             = 1032192
read(4, "(s*\301\37\370\322\21\272K\0\240\311>\311;I\236\324dX\277AA\256c\26\21\275G\340\331"..., 16384) = 16384
openat(AT_FDCWD, "/sys/dev/block/259:2", O_RDONLY|O_CLOEXEC) = 5
openat(5, "size", O_RDONLY|O_CLOEXEC)   = 6
fcntl(6, F_GETFL)                       = 0x8000 (flags O_RDONLY|O_LARGEFILE)
newfstatat(6, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0
read(6, "1952473088\n", 4096)           = 11
close(6)                                = 0
openat(5, "start", O_RDONLY|O_CLOEXEC)  = 6
fcntl(6, F_GETFL)                       = 0x8000 (flags O_RDONLY|O_LARGEFILE)
newfstatat(6, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0
read(6, "1050624\n", 4096)              = 8
close(6)                                = 0
close(5)                                = 0
close(3)                                = 0
munmap(0x7f8a09f8d000, 266240)          = 0
munmap(0x7f8a09f4c000, 266240)          = 0
munmap(0x7f8a09f0b000, 266240)          = 0
munmap(0x7f8a09eca000, 266240)          = 0
close(4)                                = 0
readlink("/mnt", 0x7ffe0dbf53f0, 1023)  = -1 EINVAL (Invalid argument)
statx(AT_FDCWD, "/run/wrappers/bin/mount.bcachefs", AT_STATX_DONT_SYNC|AT_NO_AUTOMOUNT, STATX_TYPE|STATX_MODE|STATX_INO, 0x7ffe0dbf45e0) = -1 ENOENT (No such file or directory)
statx(AT_FDCWD, "/run/current-system/sw/bin/mount.bcachefs", AT_STATX_DONT_SYNC|AT_NO_AUTOMOUNT, STATX_TYPE|STATX_MODE|STATX_INO, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0555, stx_size=5016224, ...}) = 0
fsopen(NULL, FSOPEN_CLOEXEC)            = -1 EFAULT (Bad address)
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
newfstatat(AT_FDCWD, "/run/mount/utab", {st_mode=S_IFREG|0644, st_size=48, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/run/mount/utab", {st_mode=S_IFREG|0644, st_size=48, ...}, 0) = 0
geteuid()                               = 0
getegid()                               = 0
getuid()                                = 0
getgid()                                = 0
access("/run/mount/utab", R_OK|W_OK)    = 0
mkdir("/run", 0700)                     = -1 EEXIST (File exists)
mkdir("/run/mount", 0700)               = -1 EEXIST (File exists)
mkdir("/run/mount/tmptgt", 0700)        = -1 EEXIST (File exists)
openat(AT_FDCWD, "/proc/self/ns/mnt", O_RDONLY|O_CLOEXEC) = 3
unshare(CLONE_NEWNS)                    = 0
mount("none", "/run", NULL, MS_PRIVATE, NULL) = 0
openat(AT_FDCWD, "/proc/self/ns/mnt", O_RDONLY|O_CLOEXEC) = 4
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f8a0a2ba350) = 94804
wait4(94804, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 94804
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=94804, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
open_tree(-1, "etc", OPEN_TREE_CLONE|OPEN_TREE_CLOEXEC) = -1 EBADF (Bad file descriptor)
umount2("/run/mount/tmptgt", 0)         = 0
close(4)                                = 0
setns(3, CLONE_NEWNS)                   = 0
close(3)                                = 0
ioctl(2, TCGETS, {c_iflag=ICRNL|IXON|IUTF8, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
newfstatat(AT_FDCWD, "/run/systemd/systemd-units-load", {st_mode=S_IFREG|0444, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/etc/fstab", {st_mode=S_IFREG|0444, st_size=413, ...}, 0) = 0
dup(1)                                  = 3
close(3)                                = 0
dup(2)                                  = 3
close(3)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

[dblaber]

Just to add some additional datapoints, I found that setting LIBMOUNT_FORCE_MOUNT2 does not work in preboot/emergency shell in initrd (before root is mounted). Not sure if this is related to ash shell, or util-linux(mount2) depends on something else that needs to be in initrd env to make this work. After system is up and running, root is mounted and switched to, LIBMOUNT_FORCE_MOUNT2 works again.