Skip to content
Permalink
Browse files

Implement logical to inode v2 ioctl

The v2 LOGICAL_INO ioctl allows us to set a flag 'ignore_offset', which
results in returning the inodes that reference any part of the extent.
It also allows us to provide a larger buffer to receive results, so that
it can handle extents which have more than 2730 references.

This new ioctl will be included in linux 4.15. Relevant commits:
* commit c995ab3 "btrfs: add a flag to iterate_inodes_from_logical to
  find all extent refs for uncompressed extents"
* commit d24a67b "btrfs: add a flags argument to LOGICAL_INO and call it
  LOGICAL_INO_V2"
* commit b115e3b "btrfs: increase output size for LOGICAL_INO_V2 ioctl"

Also see the commit message on 481a4f9 here in python-btrfs, which
shows the previous limitation.

------------------------------------- 8< -------------------------------------

Here's the testing scenarios I used earlier (from btrfs mailing list):

    -# cp /boot/vmlinuz-4.14.0-rc1-zygo1 /btrfs

    -# ./show_block_groups.py /btrfs
    block group vaddr 0 length 4194304 flags SYSTEM used 16384 used_pct 0
    block group vaddr 4194304 length 8388608 flags METADATA used 131072 used_pct 2
    block group vaddr 12582912 length 8388608 flags DATA used 0 used_pct 0
    block group vaddr 20971520 length 268435456 flags METADATA used 0 used_pct 0

Using 'v1':

    -# ./show_block_group_data_extent_filenames.py 12582912 /btrfs
    block group vaddr 12582912 length 8388608 flags DATA used 4198400 used_pct 50
    extent vaddr 12582912 length 4198400 refs 1 gen 17 flags DATA
        root 5 inode 258 offset 0 path utf-8 vmlinuz-4.14.0-rc1-zygo1

Let's overwrite the first few blocks:

    -# dd if=/dev/urandom bs=16K conv=notrunc of=/btrfs/vmlinuz-4.14.0-rc1-zygo1 count=1
    1+0 records in
    1+0 records out
    16384 bytes (16 kB, 16 KiB) copied, 0.000191925 s, 85.4 MB/s

Here we see the limitation of the 'v1' ioctl. I get no name back for the
first extent any more:

    -# ./show_block_group_data_extent_filenames.py 12582912 /btrfs
    block group vaddr 12582912 length 8388608 flags DATA used 4214784 used_pct 50
    extent vaddr 12582912 length 4198400 refs 1 gen 17 flags DATA
    extent vaddr 16781312 length 16384 refs 1 gen 19 flags DATA
        root 5 inode 258 offset 0 path utf-8 vmlinuz-4.14.0-rc1-zygo1

(And to test that the 'v1' also still works...)
When I change it to use LOGICAL_INO_V2, it shows the same:

    -# ./show_block_group_data_extent_filenames2.py 12582912 /btrfs
    block group vaddr 12582912 length 8388608 flags DATA used 4214784 used_pct 50
    extent vaddr 12582912 length 4198400 refs 1 gen 17 flags DATA
    extent vaddr 16781312 length 16384 refs 1 gen 19 flags DATA
        root 5 inode 258 offset 0 path utf-8 vmlinuz-4.14.0-rc1-zygo1

When I also set the ignore_offset flag:

    -# ./show_block_group_data_extent_filenames2.py 12582912 /btrfs
    block group vaddr 12582912 length 8388608 flags DATA used 4214784 used_pct 50
    extent vaddr 12582912 length 4198400 refs 1 gen 17 flags DATA
        root 5 inode 258 offset 16384 path utf-8 vmlinuz-4.14.0-rc1-zygo1
    extent vaddr 16781312 length 16384 refs 1 gen 19 flags DATA
        root 5 inode 258 offset 0 path utf-8 vmlinuz-4.14.0-rc1-zygo1

Now, for the amount of results we can retrieve, let's cause an extent to
have more than 2730 references:

    /btrfs 2-# btrfs sub create 0
    Create subvolume './0'
    /btrfs 2-# cp /boot/vmlinuz-4.14.0-rc1-zygo1 0/0
    /btrfs 2-# for i in $(seq 1 499); do cp --reflink 0/0 0/$i; done
    /btrfs 2-# for i in $(seq 1 5); do btrfs sub snap 0 $i; done
    Create a snapshot of '0' in './1'
    Create a snapshot of '0' in './2'
    Create a snapshot of '0' in './3'
    Create a snapshot of '0' in './4'
    Create a snapshot of '0' in './5'

    -# ./show_block_groups.py /btrfs
    block group vaddr 0 length 4194304 flags SYSTEM used 16384 used_pct 0
    block group vaddr 4194304 length 8388608 flags METADATA used 507904 used_pct 6
    block group vaddr 12582912 length 8388608 flags DATA used 4198400 used_pct 50
    block group vaddr 20971520 length 268435456 flags METADATA used 0 used_pct 0

    -# ./show_block_group_contents.py 12582912 /btrfs
    block group vaddr 12582912 length 8388608 flags DATA used 4198400 used_pct 50
    extent vaddr 12582912 length 4198400 refs 500 gen 25 flags DATA
        inline extent data backref root 257 objectid 262 offset 0 count 1
        inline extent data backref root 257 objectid 277 offset 0 count 1
        inline extent data backref root 257 objectid 288 offset 0 count 1
        [...]
        extent data backref root 257 objectid 663 offset 0 count 1
        extent data backref root 257 objectid 366 offset 0 count 1
        extent data backref root 257 objectid 715 offset 0 count 1
        extent data backref root 257 objectid 306 offset 0 count 1
        extent data backref root 257 objectid 470 offset 0 count 1
        [...]

Total 500 lines, the extra 2500 files in the snapshots are hidden behind
the shared metadata refs now...

    >>> import btrfs
    >>> fs = btrfs.FileSystem('/btrfs')

Checking that 'v1' still works:

    >>> inodes, bytes_missed = btrfs.ioctl.logical_to_ino(fs.fd, 12582912, 65536)
    >>> len(inodes)
    2730
    >>> bytes_missed
    6480

Yes, we only get 2730, as expected with a 64k buffer.

v2 can do the same:

    >>> inodes, bytes_missed = btrfs.ioctl.logical_to_ino_v2(fs.fd, 12582912, 65536)
    >>> len(inodes)
    2730
    >>> bytes_missed
    6480

The bytes_missed is really useful, because it tells us the exact size of
the buf we need instead :)

    >>> inodes, bytes_missed = btrfs.ioctl.logical_to_ino_v2(fs.fd, 12582912, 65536 + 6480)
    >>> len(inodes)
    3000
    >>> bytes_missed
    0

Yay!
  • Loading branch information
knorrie committed Nov 22, 2017
1 parent 0c7d24e commit 38cd5528ff1ced0be908e1e697758c7431b92a0d
Showing with 22 additions and 3 deletions.
  1. +22 −3 btrfs/ioctl.py
@@ -287,12 +287,31 @@ def search_v2(fd, tree, min_key=None, max_key=None,


def logical_to_ino(fd, vaddr, bufsize=4096):
bufsize = min(bufsize, 65536)
return logical_to_ino_v2(fd, vaddr, bufsize, _v2=False)


ioctl_logical_ino_args_v2 = struct.Struct('=QQ24xQQ')
IOC_LOGICAL_INO_V2 = _IOWR(BTRFS_IOCTL_MAGIC, 59, ioctl_logical_ino_args)
LOGICAL_INO_ARGS_IGNORE_OFFSET = 1 << 0


def logical_to_ino_v2(fd, vaddr, bufsize=4096, ignore_offset=False, _v2=True):
if _v2:
bufsize = min(bufsize, 16777216)
else:
bufsize = min(bufsize, 65536)
inodes_buf = array.array(u'B', bytearray(bufsize))
inodes_ptr = inodes_buf.buffer_info()[0]
args = bytearray(ioctl_logical_ino_args.size)
ioctl_logical_ino_args.pack_into(args, 0, vaddr, bufsize, inodes_ptr)
fcntl.ioctl(fd, IOC_LOGICAL_INO, args)
if _v2:
flags = 0
if ignore_offset:
flags |= LOGICAL_INO_ARGS_IGNORE_OFFSET
ioctl_logical_ino_args_v2.pack_into(args, 0, vaddr, bufsize, flags, inodes_ptr)
fcntl.ioctl(fd, IOC_LOGICAL_INO_V2, args)
else:
ioctl_logical_ino_args.pack_into(args, 0, vaddr, bufsize, inodes_ptr)
fcntl.ioctl(fd, IOC_LOGICAL_INO, args)
bytes_left, bytes_missing, elem_cnt, elem_missed = data_container.unpack_from(inodes_buf, 0)
inodes = []
pos = data_container.size

0 comments on commit 38cd552

Please sign in to comment.
You can’t perform that action at this time.