Share usage not reported correctly while pool usage is #1412

Closed
sfranzen opened this Issue Jul 30, 2016 · 13 comments

Comments

Projects
None yet
4 participants
@sfranzen
Contributor

sfranzen commented Jul 30, 2016

I have one share on a 2-disk simple btrfs volume. In the shares view, its usage is reported as 16.00 KB, while the pools view correctly lists 12.99 GB in use.

Note: if this is relevant, the share in question is /home. I've tinkered with my setup a bit by moving /home from its default place on the USB stick with the OS, and allowing it to be shared so I can use samba's [homes] section.

Btrfs details:

[root@rockstor ~]# btrfs filesystem usage /home
Overall:
    Device size:                 298.10GiB
    Device allocated:             17.03GiB
    Device unallocated:          281.07GiB
    Device missing:                  0.00B
    Used:                         12.99GiB
    Free (estimated):            283.10GiB      (min: 283.10GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:               16.00MiB      (used: 0.00B)

Data,single: Size:15.00GiB, Used:12.96GiB
   /dev/sda        7.00GiB
   /dev/sde        8.00GiB

Metadata,single: Size:2.00GiB, Used:27.02MiB
   /dev/sda        1.00GiB
   /dev/sde        1.00GiB

System,single: Size:32.00MiB, Used:16.00KiB
   /dev/sda       32.00MiB

Unallocated:
   /dev/sda      141.02GiB
   /dev/sde      140.05GiB
@sfranzen

This comment has been minimized.

Show comment
Hide comment
@sfranzen

sfranzen Aug 2, 2016

Contributor

Did some poking around and discovered that this is due to the quirks of btrfs. In particular: I had experimented with btrfs quota as a way of putting a limit on user directory sizes. This means that my /home directories were actually subvolumes, with their own qgroup. Since rockstor currently determines a share's size by looking up its ID in the output of btrfs qgroup show <share_pool>, in this case it only reports the (negligible) usage of the /home subvolume, not of any child subvolumes.

I have resolved this for now by placing the user files back into regular directories and deleting the subvolumes, but this raises the question of how to correctly report the share usage in such a use case. Unfortunately, btrfs does not provide straightforward ways of doing so at the moment. The most robust way seems to be btrfs filesystem du -s <path>, but this will be extremely slow for shares with many files. Any thoughts?

Contributor

sfranzen commented Aug 2, 2016

Did some poking around and discovered that this is due to the quirks of btrfs. In particular: I had experimented with btrfs quota as a way of putting a limit on user directory sizes. This means that my /home directories were actually subvolumes, with their own qgroup. Since rockstor currently determines a share's size by looking up its ID in the output of btrfs qgroup show <share_pool>, in this case it only reports the (negligible) usage of the /home subvolume, not of any child subvolumes.

I have resolved this for now by placing the user files back into regular directories and deleting the subvolumes, but this raises the question of how to correctly report the share usage in such a use case. Unfortunately, btrfs does not provide straightforward ways of doing so at the moment. The most robust way seems to be btrfs filesystem du -s <path>, but this will be extremely slow for shares with many files. Any thoughts?

@schakrava schakrava added the bug label Aug 2, 2016

@schakrava schakrava added this to the Pinnacles milestone Aug 2, 2016

@sfranzen

This comment has been minimized.

Show comment
Hide comment
@sfranzen

sfranzen Aug 19, 2016

Contributor

Hmmm... Although these changes work for me in normal circumstances, I've found another (smaller) issue. Yesterday I modified my storage a little by swapping two disks between pools. This is extremely easy and painless with btrfs and a few balance operations, but afterwards , shares in the affected pools were showing a usage of 0 bytes. This issue is fixed by performing a btrfs quota rescan on the affected filesystems. It's a cheap operation that should be included somewhere in the code handling disk changes.

Contributor

sfranzen commented Aug 19, 2016

Hmmm... Although these changes work for me in normal circumstances, I've found another (smaller) issue. Yesterday I modified my storage a little by swapping two disks between pools. This is extremely easy and painless with btrfs and a few balance operations, but afterwards , shares in the affected pools were showing a usage of 0 bytes. This issue is fixed by performing a btrfs quota rescan on the affected filesystems. It's a cheap operation that should be included somewhere in the code handling disk changes.

@sfranzen

This comment has been minimized.

Show comment
Hide comment
@sfranzen

sfranzen Aug 22, 2016

Contributor

Adding some more related issues to the list:

  • Dashboard widget shows different usage than shares list view
  • Share editing view might use wrong value when trying to resize
Contributor

sfranzen commented Aug 22, 2016

Adding some more related issues to the list:

  • Dashboard widget shows different usage than shares list view
  • Share editing view might use wrong value when trying to resize

@sfranzen sfranzen closed this Aug 22, 2016

@sfranzen sfranzen reopened this Aug 22, 2016

@MFlyer

This comment has been minimized.

Show comment
Hide comment
@MFlyer

MFlyer Jan 4, 2017

Member

Hi @schakrava & @phillxnet , picking up this

Share size -> actually computed on 0/* rfer size, while our shares' size is under 2015/id (id autoincrement on each share creation)

M.

Member

MFlyer commented Jan 4, 2017

Hi @schakrava & @phillxnet , picking up this

Share size -> actually computed on 0/* rfer size, while our shares' size is under 2015/id (id autoincrement on each share creation)

M.

@phillxnet

This comment has been minimized.

Show comment
Hide comment
@phillxnet

phillxnet Jan 4, 2017

Member

@MFlyer If you are looking into this the simply unit tests I started on have a test_share_usage() which might help, there is also a test_share_id().
https://github.com/rockstor/rockstor-core/blob/master/src/rockstor/fs/tests/test_btrfs.py#L196-L200

Member

phillxnet commented Jan 4, 2017

@MFlyer If you are looking into this the simply unit tests I started on have a test_share_usage() which might help, there is also a test_share_id().
https://github.com/rockstor/rockstor-core/blob/master/src/rockstor/fs/tests/test_btrfs.py#L196-L200

@MFlyer

This comment has been minimized.

Show comment
Hide comment
@MFlyer

MFlyer Jan 5, 2017

Member

@phillxnet & @schakrava here some basic testing on shares:

  • create a 12GB Raid 1 pool (2 disks)
  • create a 1GB share named "Test" (usage 0 bytes -> OK), original qgroup 0/258 becoming child of our "fake" qgroup 2015/1 - "fake" 2015/1 is our subvols container, original qgroup 0/258 handles current real share content
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        16.00KiB     16.00KiB         none         none 2015/1  ---
2015/1       16.00KiB     16.00KiB         none         none ---     0/258
  • writing some random 10MB to share, waiting data collector syncing qgroups and got new right size, usage col on Shares list OK
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        10.02MiB     10.02MiB         none         none 2015/1  ---
2015/1       10.02MiB     10.02MiB         none         none ---     0/258
  • take a snapshot named "first" getting qgroup 0/259 and here we come with first small "discrepancy"/misunderstandig with 0.01MB
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        10.02MiB     16.00KiB         none         none 2015/1  ---
0/259        10.02MiB     16.00KiB         none         none 2015/1  ---
2015/1       10.03MiB     10.03MiB         none         none ---     0/258,0/259
  • add another 10MB file, share updating size and correctly rfer size eual file a+b size, while exclusive size ok on just 1 file size
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        20.02MiB     10.02MiB         none         none 2015/1  ---
0/259        10.02MiB     16.00KiB         none         none 2015/1  ---
2015/1       20.03MiB     20.03MiB         none         none ---     0/258,0/259
  • take "second" snapshot with qgroup 0/260 and we update our qgroups nicely
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        20.02MiB     16.00KiB         none         none 2015/1  ---
0/259        10.02MiB     16.00KiB         none         none 2015/1  ---
0/260        20.02MiB     16.00KiB         none         none 2015/1  ---
2015/1       20.05MiB     20.05MiB         none         none ---     0/258,0/259,0/260
  • delete first file "a" and here come our misunderstanding on shares "usage": original share qgroup (current share state/content) has current files size of 10MB (1 file deleted) and we report usage of 10MB of shares page, while "real" share content is 20MB (1 current 10MB + additional 10MB from deleted file, but snapshotted!)
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        10.02MiB     16.00KiB         none         none 2015/1  ---
0/259        10.02MiB     16.00KiB         none         none 2015/1  ---
0/260        20.02MiB     16.00KiB         none         none 2015/1  ---
2015/1       20.05MiB     20.05MiB         none         none ---     0/258,0/259,0/260
  • delete second file "b" and here we have our "fake" 2015/1 holding real share size and original qgroup has "user visible" share size (share actual content)
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        16.00KiB     16.00KiB         none         none 2015/1  ---
0/259        10.02MiB     16.00KiB         none         none 2015/1  ---
0/260        20.02MiB     10.02MiB         none         none 2015/1  ---
2015/1       20.05MiB     20.05MiB         none         none ---     0/258,0/259,0/260

btrfs fi usage confirms this too on Used: ( 41.47M / Data ratio for Raid1 ~20M, 2 * 10M files )

root@rockstone:~# btrfs fi usage /mnt2/Data/Test
Overall:
    Device size:                  24.00GiB
    Device allocated:              4.02GiB
    Device unallocated:           19.98GiB
    Device missing:                  0.00B
    Used:                         41.47MiB
    Free (estimated):             10.97GiB      (min: 10.97GiB)
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:               16.00MiB      (used: 0.00B)

Data,RAID1: Size:1.00GiB, Used:20.50MiB
   /dev/disk/by-id/virtio-Data-00          1.00GiB
   /dev/vdc        1.00GiB

Metadata,RAID1: Size:1.00GiB, Used:224.00KiB
   /dev/disk/by-id/virtio-Data-00          1.00GiB
   /dev/vdc        1.00GiB

System,RAID1: Size:8.00MiB, Used:16.00KiB
   /dev/disk/by-id/virtio-Data-00          8.00MiB
   /dev/vdc        8.00MiB

Unallocated:
   /dev/disk/by-id/virtio-Data-00          9.99GiB
   /dev/vdc        9.99GiB

My idea: preserve "Usage" on shares list renaming it and add another "Real used space" or similar field, one reporting original 0/258 qgroup size (actual size visible to end user), one reporting fake qgroup 2015/1 size (this size considers snapshots too, so the real used space)

M.

Member

MFlyer commented Jan 5, 2017

@phillxnet & @schakrava here some basic testing on shares:

  • create a 12GB Raid 1 pool (2 disks)
  • create a 1GB share named "Test" (usage 0 bytes -> OK), original qgroup 0/258 becoming child of our "fake" qgroup 2015/1 - "fake" 2015/1 is our subvols container, original qgroup 0/258 handles current real share content
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        16.00KiB     16.00KiB         none         none 2015/1  ---
2015/1       16.00KiB     16.00KiB         none         none ---     0/258
  • writing some random 10MB to share, waiting data collector syncing qgroups and got new right size, usage col on Shares list OK
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        10.02MiB     10.02MiB         none         none 2015/1  ---
2015/1       10.02MiB     10.02MiB         none         none ---     0/258
  • take a snapshot named "first" getting qgroup 0/259 and here we come with first small "discrepancy"/misunderstandig with 0.01MB
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        10.02MiB     16.00KiB         none         none 2015/1  ---
0/259        10.02MiB     16.00KiB         none         none 2015/1  ---
2015/1       10.03MiB     10.03MiB         none         none ---     0/258,0/259
  • add another 10MB file, share updating size and correctly rfer size eual file a+b size, while exclusive size ok on just 1 file size
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        20.02MiB     10.02MiB         none         none 2015/1  ---
0/259        10.02MiB     16.00KiB         none         none 2015/1  ---
2015/1       20.03MiB     20.03MiB         none         none ---     0/258,0/259
  • take "second" snapshot with qgroup 0/260 and we update our qgroups nicely
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        20.02MiB     16.00KiB         none         none 2015/1  ---
0/259        10.02MiB     16.00KiB         none         none 2015/1  ---
0/260        20.02MiB     16.00KiB         none         none 2015/1  ---
2015/1       20.05MiB     20.05MiB         none         none ---     0/258,0/259,0/260
  • delete first file "a" and here come our misunderstanding on shares "usage": original share qgroup (current share state/content) has current files size of 10MB (1 file deleted) and we report usage of 10MB of shares page, while "real" share content is 20MB (1 current 10MB + additional 10MB from deleted file, but snapshotted!)
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        10.02MiB     16.00KiB         none         none 2015/1  ---
0/259        10.02MiB     16.00KiB         none         none 2015/1  ---
0/260        20.02MiB     16.00KiB         none         none 2015/1  ---
2015/1       20.05MiB     20.05MiB         none         none ---     0/258,0/259,0/260
  • delete second file "b" and here we have our "fake" 2015/1 holding real share size and original qgroup has "user visible" share size (share actual content)
root@rockstone:~# btrfs qgroup show /mnt2/Data/ -recp
qgroupid         rfer         excl     max_rfer     max_excl parent  child
--------         ----         ----     --------     -------- ------  -----
0/5          16.00KiB     16.00KiB         none         none ---     ---
0/258        16.00KiB     16.00KiB         none         none 2015/1  ---
0/259        10.02MiB     16.00KiB         none         none 2015/1  ---
0/260        20.02MiB     10.02MiB         none         none 2015/1  ---
2015/1       20.05MiB     20.05MiB         none         none ---     0/258,0/259,0/260

btrfs fi usage confirms this too on Used: ( 41.47M / Data ratio for Raid1 ~20M, 2 * 10M files )

root@rockstone:~# btrfs fi usage /mnt2/Data/Test
Overall:
    Device size:                  24.00GiB
    Device allocated:              4.02GiB
    Device unallocated:           19.98GiB
    Device missing:                  0.00B
    Used:                         41.47MiB
    Free (estimated):             10.97GiB      (min: 10.97GiB)
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:               16.00MiB      (used: 0.00B)

Data,RAID1: Size:1.00GiB, Used:20.50MiB
   /dev/disk/by-id/virtio-Data-00          1.00GiB
   /dev/vdc        1.00GiB

Metadata,RAID1: Size:1.00GiB, Used:224.00KiB
   /dev/disk/by-id/virtio-Data-00          1.00GiB
   /dev/vdc        1.00GiB

System,RAID1: Size:8.00MiB, Used:16.00KiB
   /dev/disk/by-id/virtio-Data-00          8.00MiB
   /dev/vdc        8.00MiB

Unallocated:
   /dev/disk/by-id/virtio-Data-00          9.99GiB
   /dev/vdc        9.99GiB

My idea: preserve "Usage" on shares list renaming it and add another "Real used space" or similar field, one reporting original 0/258 qgroup size (actual size visible to end user), one reporting fake qgroup 2015/1 size (this size considers snapshots too, so the real used space)

M.

@MFlyer

This comment has been minimized.

Show comment
Hide comment
@MFlyer

MFlyer Jan 7, 2017

Member

Code investigation:

Share usage updates (storageadmin/views/command.py) rely on import_shares under share_helpers.py

import_shares code

def import_shares(pool, request):
    disk = Disk.objects.filter(pool=pool)[0].name
    shares = [s.name for s in Share.objects.filter(pool=pool)]
    shares_d = shares_info(pool)
    for s in shares:
        if (s not in shares_d):
            Share.objects.get(pool=pool, name=s).delete()
    for s in shares_d:
        if (s in shares):
            logger.debug(shares_d[s])
            share = Share.objects.get(name=s)
            share.qgroup = shares_d[s]
            rusage, eusage = share_usage(pool, share.qgroup)
            ts = datetime.utcnow().replace(tzinfo=utc)
            if (rusage != share.rusage or eusage != share.eusage):
                share.rusage = rusage
                share.eusage = eusage
                su = ShareUsage(name=s, r_usage=rusage, e_usage=eusage,
                                ts=ts)
                su.save()
........
        try:
            cshare = Share.objects.get(name=s)
            cshares_d = shares_info('%s%s' % (settings.MNT_PT,
                                              cshare.pool.name))
            if (s in cshares_d):
                e_msg = ('Another pool(%s) has a Share with this same '
                         'name(%s) as this pool(%s). This configuration is not supported.'
                         ' You can delete one of them manually with this command: '
                         'btrfs subvol delete %s[pool name]/%s' %
                         (cshare.pool.name, s, pool.name, settings.MNT_PT, s))
                handle_exception(Exception(e_msg), request)
            else:
                cshare.pool = pool
                cshare.qgroup = shares_d[s]
                cshare.size = pool.size
                cshare.subvol_name = s
                cshare.rusage, cshare.eusage = share_usage(pool, cshare.qgroup)
                cshare.save()

import_shares gets shares_info(pool) to list current share children (snapshots) and share itself, but shares_info differs from qgroup structure:

  • subvol structure is 0/something original share qgroup, 0/something+n with n=1 to infinite being share's snapshots
  • qgroup structure is 0/something original share qgroup becoming child of 2015/something fake Rockstor qgroup (check previous post) and all snapshots children of 2015/something, not reflecting subvolume structure
Member

MFlyer commented Jan 7, 2017

Code investigation:

Share usage updates (storageadmin/views/command.py) rely on import_shares under share_helpers.py

import_shares code

def import_shares(pool, request):
    disk = Disk.objects.filter(pool=pool)[0].name
    shares = [s.name for s in Share.objects.filter(pool=pool)]
    shares_d = shares_info(pool)
    for s in shares:
        if (s not in shares_d):
            Share.objects.get(pool=pool, name=s).delete()
    for s in shares_d:
        if (s in shares):
            logger.debug(shares_d[s])
            share = Share.objects.get(name=s)
            share.qgroup = shares_d[s]
            rusage, eusage = share_usage(pool, share.qgroup)
            ts = datetime.utcnow().replace(tzinfo=utc)
            if (rusage != share.rusage or eusage != share.eusage):
                share.rusage = rusage
                share.eusage = eusage
                su = ShareUsage(name=s, r_usage=rusage, e_usage=eusage,
                                ts=ts)
                su.save()
........
        try:
            cshare = Share.objects.get(name=s)
            cshares_d = shares_info('%s%s' % (settings.MNT_PT,
                                              cshare.pool.name))
            if (s in cshares_d):
                e_msg = ('Another pool(%s) has a Share with this same '
                         'name(%s) as this pool(%s). This configuration is not supported.'
                         ' You can delete one of them manually with this command: '
                         'btrfs subvol delete %s[pool name]/%s' %
                         (cshare.pool.name, s, pool.name, settings.MNT_PT, s))
                handle_exception(Exception(e_msg), request)
            else:
                cshare.pool = pool
                cshare.qgroup = shares_d[s]
                cshare.size = pool.size
                cshare.subvol_name = s
                cshare.rusage, cshare.eusage = share_usage(pool, cshare.qgroup)
                cshare.save()

import_shares gets shares_info(pool) to list current share children (snapshots) and share itself, but shares_info differs from qgroup structure:

  • subvol structure is 0/something original share qgroup, 0/something+n with n=1 to infinite being share's snapshots
  • qgroup structure is 0/something original share qgroup becoming child of 2015/something fake Rockstor qgroup (check previous post) and all snapshots children of 2015/something, not reflecting subvolume structure
@MFlyer

This comment has been minimized.

Show comment
Hide comment
@MFlyer

MFlyer Jan 9, 2017

Member

To @schakrava & @phillxnet, found share usage "faulty" code / something wrong how we create shares/subvols :

def share_usage(pool, share_id):
    """
    Return the sum of the qgroup sizes of this share and any child subvolumes
    N.B. qgroupid defaults to a unique identifier of the form 0/<subvolume id>
    """
    # Obtain path to share in pool
    root_pool_mnt = mount_root(pool)
    cmd = [BTRFS, 'subvolume', 'list', root_pool_mnt]
    out, err, rc = run_command(cmd, log=True)
    short_id = share_id.split('/')[1]
    share_dir = ''
    for line in out:
        fields = line.split()
        if (len(fields) > 0 and short_id in fields[1]):
            share_dir = root_pool_mnt + '/' + fields[8]
            break
    # Obtain list of child subvolume qgroups
    cmd = [BTRFS, 'subvolume', 'list', '-o', share_dir]
    out, err, rc = run_command(cmd, log=True)
    qgroups = [short_id]
    for line in out:
        fields = line.split()
        if (len(fields) > 0):
            qgroups.append(fields[1])

    # Sum qgroup sizes
    cmd = [BTRFS, 'qgroup', 'show', share_dir]
    out, err, rc = run_command(cmd, log=True)
    rusage = eusage = 0
    for line in out:
        fields = line.split()
        qgroup = []
        if (len(fields) > 0 and '/' in fields[0]):
            qgroup = fields[0].split('/')
        if (len(qgroup) > 0 and qgroup[1] in qgroups):
            rusage += convert_to_kib(fields[1])
            eusage += convert_to_kib(fields[2])
    return (rusage, eusage)

Assumptions:
every snapshot is a subvolume

  • "Data" is my Pool
  • "Test" is my Share under Pool "Data"
  • "Test" has some snapshots
  • Manually had one new subvolume (call it "one"), 1 subvolume of that subvolume (call it "two") and some snapshots

Starting from our share cmd = [BTRFS, 'subvolume', 'list', '-o', share_dir] should return all child subvolumes and same way btrfs subvolume list -s 'share_dir' its snapshots, but currently first command has no output
ex.:
having btrfs sub list - /mnt2/one output has "two" with parent set to one
taking snapshots under /mnt2/one/two then btrfs sub list -s /mnt2/one/two i get all snapshots ok with parent "two"

On Rockstor side:
we have all snapshots created for my "Test" share under "Data" pool (/mnt2/Data/.snapshots/Test/etc etc) so btrfs sobvolume list -o /mnt2/Data/Test will never compute snapshots and our share_usage will always have shares without subvols

M.

Member

MFlyer commented Jan 9, 2017

To @schakrava & @phillxnet, found share usage "faulty" code / something wrong how we create shares/subvols :

def share_usage(pool, share_id):
    """
    Return the sum of the qgroup sizes of this share and any child subvolumes
    N.B. qgroupid defaults to a unique identifier of the form 0/<subvolume id>
    """
    # Obtain path to share in pool
    root_pool_mnt = mount_root(pool)
    cmd = [BTRFS, 'subvolume', 'list', root_pool_mnt]
    out, err, rc = run_command(cmd, log=True)
    short_id = share_id.split('/')[1]
    share_dir = ''
    for line in out:
        fields = line.split()
        if (len(fields) > 0 and short_id in fields[1]):
            share_dir = root_pool_mnt + '/' + fields[8]
            break
    # Obtain list of child subvolume qgroups
    cmd = [BTRFS, 'subvolume', 'list', '-o', share_dir]
    out, err, rc = run_command(cmd, log=True)
    qgroups = [short_id]
    for line in out:
        fields = line.split()
        if (len(fields) > 0):
            qgroups.append(fields[1])

    # Sum qgroup sizes
    cmd = [BTRFS, 'qgroup', 'show', share_dir]
    out, err, rc = run_command(cmd, log=True)
    rusage = eusage = 0
    for line in out:
        fields = line.split()
        qgroup = []
        if (len(fields) > 0 and '/' in fields[0]):
            qgroup = fields[0].split('/')
        if (len(qgroup) > 0 and qgroup[1] in qgroups):
            rusage += convert_to_kib(fields[1])
            eusage += convert_to_kib(fields[2])
    return (rusage, eusage)

Assumptions:
every snapshot is a subvolume

  • "Data" is my Pool
  • "Test" is my Share under Pool "Data"
  • "Test" has some snapshots
  • Manually had one new subvolume (call it "one"), 1 subvolume of that subvolume (call it "two") and some snapshots

Starting from our share cmd = [BTRFS, 'subvolume', 'list', '-o', share_dir] should return all child subvolumes and same way btrfs subvolume list -s 'share_dir' its snapshots, but currently first command has no output
ex.:
having btrfs sub list - /mnt2/one output has "two" with parent set to one
taking snapshots under /mnt2/one/two then btrfs sub list -s /mnt2/one/two i get all snapshots ok with parent "two"

On Rockstor side:
we have all snapshots created for my "Test" share under "Data" pool (/mnt2/Data/.snapshots/Test/etc etc) so btrfs sobvolume list -o /mnt2/Data/Test will never compute snapshots and our share_usage will always have shares without subvols

M.

@MFlyer

This comment has been minimized.

Show comment
Hide comment
@MFlyer

MFlyer Jan 10, 2017

Member

While testing got another glibc nice moment (https://www.centos.org/forums/viewtopic.php?f=47&t=60880), @schakrava & @phillxnet can you confirm this? got it with a --sort too

M.

Member

MFlyer commented Jan 10, 2017

While testing got another glibc nice moment (https://www.centos.org/forums/viewtopic.php?f=47&t=60880), @schakrava & @phillxnet can you confirm this? got it with a --sort too

M.

@phillxnet

This comment has been minimized.

Show comment
Hide comment
@phillxnet

phillxnet Jan 10, 2017

Member

@MFlyer Not directly related but I've just put a fix (pr) in for the prior mentioned fs unit tests. Found a simple issue so sorted in case you needed to update them in lite of changes you may make here re share size calculation.

Member

phillxnet commented Jan 10, 2017

@MFlyer Not directly related but I've just put a fix (pr) in for the prior mentioned fs unit tests. Found a simple issue so sorted in case you needed to update them in lite of changes you may make here re share size calculation.

@MFlyer

This comment has been minimized.

Show comment
Hide comment
@MFlyer

MFlyer Jan 12, 2017

Member

Final considerations before coding
Currently we have quite right shares/snapshots qgroups structures, but to grant hidden snapshots feature we altered btrfs with an odd vol/subvols skeleton:

Normal btrfs struct has vol and subvols (snapshots are subvols too) like normal sys folders and we have snapshots being subvols of Pool vol, example:
Pool is "Data", Share is "Test", every hidden snapshot will be under /Data/.snapshots/Test/snapshotname and this is ok with qgroups (we force them), but not with btrfs vol/subvol structure (not possible to force it), so every btrfs command working on real btrfs hierarchy (btrfs subvolume list, etc) will report "wrong" data (well, actually it's right, but wrong and not useful on Rockstor point of view)

Manually adding vols and snapshots where expected (under parent vol) all btrfs tools work fine.

Hack: current share_usage running with btrfs sub list -o /shareref has always a 0 length array output (right, subvols are all under pool) so we have exclusive & refered sizes pointing only to original share qgroup (current share content, i want to use it too 😃 ), but btrfs qgroup show -pc gives us what we need to compute real rfer and excl sizes
My idea: extend shares list & details with current size (original qgroup 0/x) plus total size with snapshots via qgroups (ex.: a share having 50MB current data can be nearly full because of snapshots) and probably widgets too (sort top shares by real size)
@schakrava & @phillxnet ?

Member

MFlyer commented Jan 12, 2017

Final considerations before coding
Currently we have quite right shares/snapshots qgroups structures, but to grant hidden snapshots feature we altered btrfs with an odd vol/subvols skeleton:

Normal btrfs struct has vol and subvols (snapshots are subvols too) like normal sys folders and we have snapshots being subvols of Pool vol, example:
Pool is "Data", Share is "Test", every hidden snapshot will be under /Data/.snapshots/Test/snapshotname and this is ok with qgroups (we force them), but not with btrfs vol/subvol structure (not possible to force it), so every btrfs command working on real btrfs hierarchy (btrfs subvolume list, etc) will report "wrong" data (well, actually it's right, but wrong and not useful on Rockstor point of view)

Manually adding vols and snapshots where expected (under parent vol) all btrfs tools work fine.

Hack: current share_usage running with btrfs sub list -o /shareref has always a 0 length array output (right, subvols are all under pool) so we have exclusive & refered sizes pointing only to original share qgroup (current share content, i want to use it too 😃 ), but btrfs qgroup show -pc gives us what we need to compute real rfer and excl sizes
My idea: extend shares list & details with current size (original qgroup 0/x) plus total size with snapshots via qgroups (ex.: a share having 50MB current data can be nearly full because of snapshots) and probably widgets too (sort top shares by real size)
@schakrava & @phillxnet ?

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Jan 14, 2017

#1412 add required migration
Signed-off-by: Mirko Arena <mirko.arena@gmail.com>
@MFlyer

This comment has been minimized.

Show comment
Hide comment
@MFlyer

MFlyer Jan 15, 2017

Member

Starting with ref to related issue, adding a todo list

Member

MFlyer commented Jan 15, 2017

Starting with ref to related issue, adding a todo list

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Jan 16, 2017

#1412 add required migration
Signed-off-by: Mirko Arena <mirko.arena@gmail.com>
@MFlyer

This comment has been minimized.

Show comment
Hide comment
@MFlyer

MFlyer Jan 16, 2017

Member

share_usage requires enhancements: actually serves both shares and snapshots rusage & eusage, so a simple fix with cmd = [BTRFS, 'subvolume', 'list', '-o', share_dir] (empy list returned) to cmd = [BTRFS, 'subvolume', 'list', '-s', share_dir] not working because running on snapshots group id also other snapshots get listed, generating wrong snapshot sizes

M.

Member

MFlyer commented Jan 16, 2017

share_usage requires enhancements: actually serves both shares and snapshots rusage & eusage, so a simple fix with cmd = [BTRFS, 'subvolume', 'list', '-o', share_dir] (empy list returned) to cmd = [BTRFS, 'subvolume', 'list', '-s', share_dir] not working because running on snapshots group id also other snapshots get listed, generating wrong snapshot sizes

M.

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Jan 18, 2017

#1412 add required migration
Signed-off-by: Mirko Arena <mirko.arena@gmail.com>

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Jan 23, 2017

#1412 add new volume_usage function vs old share_usage
Signed-off-by: Mirko Arena <mirko.arena@gmail.com>

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Jan 23, 2017

volume_usage redisign to fit both shares and snapshots
On #1412 - volume_usage returns 2/4 vals, 4 vals for shares
and 2 vals for snapshots (shares have pqgroup size too)

Signed-off-by: Mirko Arena <mirko.arena@gmail.com>

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Jan 23, 2017

#1412 fixing snapshot code too
Signed-off-by: Mirko Arena <mirko.arena@gmail.com>

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Jan 23, 2017

#1412 Removing share_usage after testing new volume_usage
Signed-off-by: Mirko Arena <mirko.arena@gmail.com>

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Jan 23, 2017

#1412 Add btrfs snapshots size to dashboard widget too
Signed-off-by: Mirko Arena <mirko.arena@gmail.com>

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Jan 23, 2017

#1412 Fix regression over shares.js view
Signed-off-by: Mirko Arena <mirko.arena@gmail.com>

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Jan 24, 2017

#1412 Fixing typo
Signed-off-by: Mirko Arena <mirko.arena@gmail.com>

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Jan 24, 2017

#1412 btrfs testunits update
Signed-off-by: Mirko Arena <mirko.arena@gmail.com>

@schakrava schakrava closed this in f325a53 Feb 13, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment