knorrie / python-btrfs

* errno is a property of BalanceError, but it was not documented
* For each function that can raise BalanceError, document expected
  values of the 'state' and 'errno' property of the exception.

The README was really terse. Now at least it has some more useful info.

Wheee!

This is kind of obsolete. It's also not really an interesting example
for users to look at.

After moving a bunch of interesting example programs into the bin/
location I'm looking at the rest of the random pile of code in examples/

What about a tool that allows searching any metadata you want? And what
if it already has a bunch of presets that select ranges from trees
automatically?

    btrfs-search-metadata [--format {keys,short,long}] <preset> [<moar> ..] <path>

The first version of this program has preset search options for:
- chunks
- block groups
- device extents
- all information for an inode by pointing it to a file or directory
- all information for an inode (objectid) in any subvolume
- devices (the metadata items)
- orhpans (the orphan items from the root tree)
- dump: custom search query for a range of items in a tree
- block_group_contents: show block group contents
- block_group_free_space: decoded free space extents inside a block group

This tool will be used in the tutorial a lot, when looking around and
learning about all what's to see and how it relates together.

This is just a wrapper around ioctl.search_v2 so that we can write
simple searches with even less verbose code.

Don't bother with situations that need falling back to search v1, since
if you're running a Linux kernel < 3.16 with btrfs, then instead of
playing around with this code, you need to upgrade.

These live in the root tree. One way to spot them is to throw away a
subvolume. The tree id will show up as orphan item while it's not
completely removed in the background.

This item has no data, only an item key. For convenience reasons we put
the objectid in an attribute.

I want to add an option in the new btrfs-search-metadata that shows free
space extents, and then I mean the information with bitmaps unpacked. We
have helper functions to do this, but the FreeSpaceExtent object was not
made ready for the pretty printer code yet.

    -$ btrfs-search-metadata block_group_free_space --vaddr 895564644352 /
    free space extent vaddr 895728484352 length 4096
    free space extent vaddr 895728599040 length 4096
    free space extent vaddr 895830040576 length 102400
    free space extent vaddr 895832805376 length 274432
    free space extent vaddr 895833083904 length 4096
    free space extent vaddr 895835869184 length 16384
    free space extent vaddr 895862583296 length 4096
    [...]

    -$ btrfs-search-metadata --format=long block_group_free_space --vaddr 895564644352 /
    -
      <btrfs.free_space_tree.FreeSpaceExtent>
      vaddr: 895728484352
      length: 4096
    -
      <btrfs.free_space_tree.FreeSpaceExtent>
      vaddr: 895728599040
      length: 4096
    -
      <btrfs.free_space_tree.FreeSpaceExtent>
      vaddr: 895830040576
      length: 102400
    [...]

Note that using --format=keys will not give any output, because these
helper objects are not actual metadata ItemData.

So, we have all kinds of code to pretty print a metadata key. Now, next
step is to add the reverse, and be able to parse key text back to a Key
object.

When initializing a Key object, it's now also allowed to use text
representations of the numeric values, like using 'DEV_EXTENT' instead
of 204 as type value.

To parse entire pretty formatted text strings, there's a helper function
in utils, parse_key_string, which can handle them, e.g. '(31832
INODE_REF 31798)' as input, and do whatever is necessary to dissect the
text and return a Key object.

We can do fun stuff with keys, like this:

    >>> k = btrfs.ctree.Key(0, 0, 0) - 1
    >>> k
    Key(18446744073709551615, 255, 18446744073709551615)

However... what's the actual 136 bit key value in this case?

    >>> k.key
    -1

That's not good, because -1 is not a valid unsigned 136 bit number at
all. That should be:

    >> btrfs.ctree.Key(-1, -1, -1).key
    87112285931760246646623899502532662132735

So, when using  btrfs.ctree.Key(0, 0, 0) - 1 as max key value for a
search, only the values from the first search ioctl call result buffer
would be returned, and then the search would stop.

Same bug is present for wrapping the value the other way round:

    >>> l = btrfs.ctree.Key(18446744073709551615, 255, 18446744073709551615) + 1
    >>> l
    Key(0, 0, 0)
    >>> l.key
    87112285931760246646623899502532662132736

So, we have to manually reduce the value to 136 bits again, since python
has no idea about different sized number storage.

Fixes: 2916bbf ("Search ioctl, search key, devices, chunks and stripes")

So, in commit 606a5b5 ("Retrieve subvolume info"), there's a whole
story about multiple root tree items for a single subvolume, because I
found all kinds of documentation about that in the btrfs wiki. However,
it seems this is already outdated information since 2009.

So, simplify the code and forget about all of this, kthxbye.

Fixes: 606a5b5 ("Retrieve subvolume info")
Link: https://lore.kernel.org/linux-btrfs/f0144a34-f14a-5413-9b0c-a2ccba1a1cd9@knorrie.org/

This was in my git stash somewhere... I don't exactly remember why. I
think that I just don't want to have only some items report the binary
item length using len. If they do, it should work for all ItemData, and
then we should have __len__ in the ItemData class. However, InodeRefList
already (conveniently) use len for the list length, while they're also
ItemData. So, since this len was only used for internal purpose, remove
it.

I mixed up two fields in the dev_stats struct when initially writing this
code.

Reported-By: Kepi
Fixes: c7cb45d ("Retrieve Device Stats")
Closes: #21

This is nice for parsing command line input when a tree id or name can
be given.

Valid input is e.g.:
 - 5
 - 'fs'
 - 'fs_tree'
 - 'FS_TREE'

We can print much more, e.g. DevInfo objects from the ioctl module.

Make it look more like the __str__ of DevInfo, but without the path,
because that's not stored in the metadata of course.

Now we can simply generate a stream of dictionary objects with a little
collection of interesting stuff and it will be printed.

Yeah. It's pretty convenient to be able to access the actual subvolume
id number when you get a RootItem object to work with.

Change the example to use the newly added RootRef to make the output
list all locations of subvolumes. This is very similar to the output of
doing btrfs sub list -a.

-# ./show_subvolumes.py .
ID 259 parent 5 path /rootfs
ID 1052 parent 259 path /rootfs/var/lib/machines
ID 260 parent 5 path /home
ID 2454 parent 260 path /home/knorrie/tmp
ID 852 parent 5 path /zooi
ID 5167 parent 5 path /.snapshot/rootfs.20190624T222913+0200
ID 5168 parent 5 path /.snapshot/home.20190624T222913+0200
[...]

Like mentioned in the documentation part, this item shows us where
subvolumes are located inside another subvolume.

After releasing v11, I found out the documentation still shows v10. Doh.
Instead of having to change it in multiple places, use btrfs/version.py.

Add some hacks to read it without actually importing the module, for
setup.py and docs.

Reported by Andrei Borzenkov, the behavior was incorrect.

-$ ./btrfs-space-calculator --help
usage: btrfs-space-calculator [-h] -m METADATA -d DATA [-M] [-r RATIO]
                              [sizes [sizes ...]]

Without any disk sizes, executing the program is meaningless. So,
correct the nargs, and now at least one disk size has to be provided:

-$ ./btrfs-space-calculator -m single -d single
usage: btrfs-space-calculator [-h] -m METADATA -d DATA [-M] [-r RATIO]
                              size [size ...]
btrfs-space-calculator: error: the following arguments are required: size

Also rename the thing to 'size' instead of 'sizes', since that makes the
help text even more understandable.

Closes: #20

Ratio does not have a meaning when there are no separate data and
metadata allocations. Make this more clear.

Right... Andrei Borzenkov reported that when using raid5 in the options,
the program crashes:

  File "/home/bor/src/python-btrfs/bin/btrfs/volumes.py", line 192,
      in chunk_length_to_dev_extent_length
    dev_extent_length = raw_data_bytes // num_data_stripes
ZeroDivisionError: integer division or modulo by zero

The way this space calculator works is a little bit naughty. By
constructing some fake replacements for the official Chunk, Stripe etc
classes, we fool the usage report module and have it believe it's
looking at an actual filesystem.

In order to do so, we need to present at least one fake chunk of each
chosen profile (for data, metadata etc) to it, so it can take over from
there and do a simulation filling up the rest of the raw space.

The FakeChunk had num_stripes = 1 hardcoded, which obviously is fragile.
In the specific case of RAID5, it crashes because the nparity is
subtracted from num_stripes, which makes it 0.

Fix this a bit more properly by changing the usage reporting code to
allow passing a completely empty filesystem to it.

This means:
* Target profiles have to be set explicitly, because there's no chunk
  with highest vaddr yet to look at.
* virtual_block_group_type_usage has to deal with questions about unused
  but allocated space for a type it hasn't seen yet.

This removes a bunch of Fake classes, which is good.

Closes: #19

The space-calculator utility provides an option to set data:metadata
ratio when simulating filling up an empty filesystem. This provided
ratio was set by manipulating usage.default_data_metadata_ratio.

However, since the allocation simulation is done directly now, when
creating the FsUsage object, that's too late.

Indeed, set it using the constructor, which is better and more explicit.

I want to add an example that can dump any metadata search result range
from any tree in either str or full pretty print detail output.

So, add a streaming object printer similar to the very verbose pretty
printer, but using str() on objects.

This is a helper class to identify nested structs in ItemData objects.
An example is the Chunk object, which contains Stripe objects.

I'm going to add a streaming object printer based on the __str__
methods of objects, which can quickly display objects in a more concise
way than the very verbose pretty printer. By checking if an attribute of
an ItemData object is of type SubItem, we can decide to also print it.

The tree_block_refs and shared_block_refs attributes of the MetaDataItem
class are of type list. This means they have an append function, and not
_append.

So hitting these lines of code crash a program with "AttributeError:
'list' object has no attribute '_append'"

This made me find out that both of the filesystems that I use for
regression testing before doing a release were originally created using
Debian Jessie, and they were never converted to enable the
skinny_metadata feature, which introduces the usage of these
MetaDataItem objects.

Unfortunately this means that working with metadata related information
from the extent tree is pretty much broken for any recently created
filesystem. Also affected is the btrfs-heatmap program, which will crash
while generating pictures on extent level of metadata block groups.

Andrei Borzenkov reported this bug, and found it by using the
show_metadata_tree_sizes example.

Fixes: 899fdf4 "ctree: hide internal ref item helpers"
Closes: #17