Skip to content

Conversation

@mmaybee
Copy link
Contributor

@mmaybee mmaybee commented Feb 11, 2021

This commit removes the need to provide the -v flag for per-metaslab
output. For example, histograms can now be obtained with only the -mH flags.
Additionally, this commit adds "rolled up" per-vdev histogram data
when the -v flag is provided with the -H flag.

Pull request checklist

Please check if your PR fulfills the following requirements:

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been reviewed and added / updated if needed (for bug fixes / features)
  • Build was run locally and any changes were pushed
  • Lint has passed locally and any fixes were made for failures

Pull request type

Please check the type of change your PR introduces:

  • Bugfix
  • Feature
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • Documentation content changes
  • Other (please describe):

What is the current behavior?

Historgram data (-H) is only available at the pool level or metaslab level. There is no available histogram data at the vdev level.

sdb> spa -vH domain0
ADDR               NAME
------------------------------------------------------------
0xffff96101972c000 domain0
       512.0B:   1750 ************
        1.0KB:   1807 ************
        2.0KB:   4385 *****************************
        4.0KB:   4612 *******************************
        8.0KB:   4833 ********************************
       16.0KB:   4784 ********************************
       32.0KB:   6029 ****************************************
       64.0KB:   4237 ****************************
      128.0KB:   2946 ********************
      256.0KB:   2001 *************
      512.0KB:   1561 **********
        1.0MB:   1333 *********
        2.0MB:   1055 *******
        4.0MB:    651 ****
        8.0MB:    297 **
       16.0MB:     96 *
       32.0MB:     10 
       64.0MB:      1 
     Approx. Median: 4.9MB
      ADDR               STATE   AUX  DESCRIPTION
      ------------------------------------------------------------
      0xffff9610168a8000 HEALTHY NONE  root
      0xffff96105ad94000 HEALTHY NONE    /dev/disk/by-id/scsi-36000c290d3f4e58aedefb5a8d2ddb87a-part1
      0xffff96104fd7c000 HEALTHY NONE    /dev/disk/by-id/scsi-36000c294d63d9b7040357c43dea72431-part1
      0xffff961044acc000 HEALTHY NONE    /dev/disk/by-id/scsi-36000c291c4941fcfe92623085d9c2972-part1

The -m flag (request metaslab data) has no impact at the pool level unless requested with the -v flag. Requesting metaslab data at the pool level should implicitly request a vdev traversal.

sdb> spa -m domain0
ADDR               NAME
------------------------------------------------------------
0xffff96101972c000 domain0

Issue Number: N/A

What is the new behavior?

sdb> spa -vH tank
ADDR               NAME
------------------------------------------------------------
0xffff9c8628528000 tank
     seg-size   count
     --------   -----
       512.0B:     29 *****************************
        1.0KB:     23 ***********************
        2.0KB:     35 ***********************************
        4.0KB:     21 *********************
        8.0KB:      4 ****
       16.0KB:      4 ****
       32.0KB:      0 
       64.0KB:      0 
      128.0KB:      0 
      256.0KB:      0 
      512.0KB:      0 
        1.0MB:      0 
        2.0MB:      0 
        4.0MB:      0 
        8.0MB:      0 
       16.0MB:      0 
       32.0MB:      0 
       64.0MB:      0 
      128.0MB:      0 
      256.0MB:     12 ************
     Approx. Median: 384.0MB
      ADDR               STATE   AUX  DESCRIPTION
      ------------------------------------------------------------
      0xffff9c8626fc4000 HEALTHY NONE  root
      0xffff9c8625020000 HEALTHY NONE    /dev/sdb1
          seg-size   count
          --------   -----
            512.0B:      4 ****
             1.0KB:      7 *******
             2.0KB:      9 *********
             4.0KB:      8 ********
             8.0KB:      2 **
            16.0KB:      2 **
            32.0KB:      0 
            64.0KB:      0 
           128.0KB:      0 
           256.0KB:      0 
           512.0KB:      0 
             1.0MB:      0 
             2.0MB:      0 
             4.0MB:      0 
             8.0MB:      0 
            16.0MB:      0 
            32.0MB:      0 
            64.0MB:      0 
           128.0MB:      0 
           256.0MB:      4 ****
          Approx. Median: 384.0MB
      0xffff9c86259e4000 HEALTHY NONE    /dev/sdc1
          seg-size   count
          --------   -----
            512.0B:     14 **************
             1.0KB:      6 ******
             2.0KB:     16 ****************
             4.0KB:      7 *******
             8.0KB:      0 
            16.0KB:      0 
            32.0KB:      0 
            64.0KB:      0 
           128.0KB:      0 
           256.0KB:      0 
           512.0KB:      0 
             1.0MB:      0 
             2.0MB:      0 
             4.0MB:      0 
             8.0MB:      0 
            16.0MB:      0 
            32.0MB:      0 
            64.0MB:      0 
           128.0MB:      0 
           256.0MB:      4 ****
          Approx. Median: 384.0MB
      0xffff9c8627898000 HEALTHY NONE    /dev/sdd1
          seg-size   count
          --------   -----
            512.0B:     11 ***********
             1.0KB:     10 **********
             2.0KB:     10 **********
             4.0KB:      6 ******
             8.0KB:      2 **
            16.0KB:      2 **
            32.0KB:      0 
            64.0KB:      0 
           128.0KB:      0 
           256.0KB:      0 
           512.0KB:      0 
             1.0MB:      0 
             2.0MB:      0 
             4.0MB:      0 
             8.0MB:      0 
            16.0MB:      0 
            32.0MB:      0 
            64.0MB:      0 
           128.0MB:      0 
           256.0MB:      4 ****
          Approx. Median: 384.0MB
sdb> spa -m tank
ADDR               NAME
------------------------------------------------------------
0xffff9c8628528000 tank
      ADDR               STATE   AUX  DESCRIPTION
      ------------------------------------------------------------
      0xffff9c8626fc4000 HEALTHY NONE  root
      0xffff9c8625020000 HEALTHY NONE    /dev/sdb1
           ADDR                 ID           OFFSET     FREE  FRAG     UCMU
           -----------------------------------------------------------------
           0xffff9c878ef29000    0              0x0    511MB    0%      8KB
           0xffff9c878ef2c800    1       0x20000000    282MB    0%      8KB
           0xffff9c878ef2f000    2       0x40000000    511MB    0%      8KB
           0xffff9c878ef2f800    3       0x60000000    511MB    0%      8KB
           0xffff9c878ef2b000    4       0x80000000    512MB    0%       0B
           0xffff9c878ef29800    5       0xa0000000    512MB    0%       0B
           0xffff9c878ef2d800    6       0xc0000000    512MB    0%       0B
           0xffff9c878ef28800    7       0xe0000000    512MB    0%       0B
           0xffff9c878ef2b800    8      0x100000000    512MB    0%       0B
           0xffff9c878ef2d000    9      0x120000000    512MB    0%       0B
           0xffff9c878ef2a000   10      0x140000000    512MB    0%       0B
           0xffff9c8794c9d800   11      0x160000000    512MB    0%       0B
           0xffff9c8794c9e000   12      0x180000000    512MB    0%       0B
           0xffff9c8794c98000   13      0x1a0000000    512MB    0%       0B
           0xffff9c8794c9a800   14      0x1c0000000    512MB    0%       0B
      0xffff9c86259e4000 HEALTHY NONE    /dev/sdc1
           ADDR                 ID           OFFSET     FREE  FRAG     UCMU
           -----------------------------------------------------------------
           0xffff9c87953fd000    0              0x0    511MB    0%      8KB
           0xffff9c87953fd800    1       0x20000000    511MB    0%      8KB
           0xffff9c87953fc000    2       0x40000000    511MB    0%      8KB
           0xffff9c87953f9000    3       0x60000000    290MB    0%      4KB
           0xffff9c87953ff000    4       0x80000000    512MB    0%       0B
           0xffff9c87953f8000    5       0xa0000000    512MB    0%       0B
           0xffff9c87953ff800    6       0xc0000000    512MB    0%       0B
           0xffff9c878e0c4000    7       0xe0000000    512MB    0%       0B
           0xffff9c878e0c0800    8      0x100000000    512MB    0%       0B
           0xffff9c878e0c7800    9      0x120000000    512MB    0%       0B
           0xffff9c878e0c0000   10      0x140000000    512MB    0%       0B
           0xffff9c878e0c1800   11      0x160000000    512MB    0%       0B
           0xffff9c878e0c7000   12      0x180000000    512MB    0%       0B
           0xffff9c878e0c6000   13      0x1a0000000    512MB    0%       0B
           0xffff9c878e0c1000   14      0x1c0000000    512MB    0%       0B
      0xffff9c8627898000 HEALTHY NONE    /dev/sdd1
           ADDR                 ID           OFFSET     FREE  FRAG     UCMU
           -----------------------------------------------------------------
           0xffff9c878e0c2800    0              0x0    511MB    0%      4KB
           0xffff9c878e0c5000    1       0x20000000    286MB    0%      4KB
           0xffff9c878e0c5800    2       0x40000000    511MB    0%      4KB
           0xffff9c878e0c2000    3       0x60000000    511MB    0%      4KB
           0xffff9c878e0c3000    4       0x80000000    512MB    0%       0B
           0xffff9c879c706800    5       0xa0000000    512MB    0%       0B
           0xffff9c879c705000    6       0xc0000000    512MB    0%       0B
           0xffff9c879c702000    7       0xe0000000    512MB    0%       0B
           0xffff9c879c706000    8      0x100000000    512MB    0%       0B
           0xffff9c879c701800    9      0x120000000    512MB    0%       0B
           0xffff9c879c703800   10      0x140000000    512MB    0%       0B
           0xffff9c879c701000   11      0x160000000    512MB    0%       0B
           0xffff9c879c703000   12      0x180000000    512MB    0%       0B
           0xffff9c879c707000   13      0x1a0000000    512MB    0%       0B
           0xffff9c879c705800   14      0x1c0000000    512MB    0%       0B

Does this introduce a breaking change?

  • Yes
  • No

Other information

@mmaybee mmaybee requested review from ahrens and sdimitro February 11, 2021 17:51
first_time = True
histsum: List[int] = [];
for msp in metaslabs:
if msp.ms_sm == sdb.get_typed_null(msp.ms_sm.type_):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this is the canonical way of doing this, but it seems pretty tricky just to check for null. As an orthogonal cleanup, maybe we should change this helper function. It looks like all the callers of get_typed_null(type) could be changed to a simpler-to-use helper, something like:

def is_null(obj: drgn.Object) -> bool:
  return obj == drgn.NULL(prog, obj.type_)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, I will file a new bug for this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed: #258

Copy link
Contributor

@sdimitro sdimitro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some issues with pylint, yapf, and mypy as you can see in the automation. If you want to deal with those locally you can install these utilities through pip the same way we do it in the automation (ref: https://github.com/delphix/sdb/blob/master/.github/workflows/main.yml) - keep in mind that for certain utilities like mypy we use a very specific version (i.e. not the latest).

Furthermore, now that you added support for spa -mH it would be cool to add a regression test case for it too.

Finally, Matt's idea of changing the null-check wrapper is a good one but could span multiple files that are unrelated to this change. If you feel like picking it up you can do this as a separate commit, otherwise feel free to file a bug and assign it to me.

@codecov-io
Copy link

codecov-io commented Feb 12, 2021

Codecov Report

Merging #256 (b26d84f) into master (6754f05) will increase coverage by 0.09%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #256      +/-   ##
==========================================
+ Coverage   87.41%   87.51%   +0.09%     
==========================================
  Files          60       60              
  Lines        2496     2524      +28     
==========================================
+ Hits         2182     2209      +27     
- Misses        314      315       +1     
Impacted Files Coverage Δ
sdb/commands/zfs/histograms.py 98.55% <100.00%> (-1.45%) ⬇️
sdb/commands/zfs/metaslab.py 95.45% <100.00%> (-0.06%) ⬇️
sdb/commands/zfs/spa.py 97.77% <100.00%> (-0.05%) ⬇️
sdb/commands/zfs/vdev.py 92.68% <100.00%> (+3.39%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6754f05...2a8fa79. Read the comment docs.

Copy link
Contributor

@sdimitro sdimitro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

make sure to squash you commits before merging

This commit removes the need to provide the -v flag for per-metaslab
histograms. They can now be obtained with only the -mH flags.
Additionally, this commit adds "rolled up" per-vdev histogram data
when the -v flag is provided with the -H flag.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

4 participants