arc/zpool metric support on linux

[cburroughs]

The arc and ZFS metrics are burried inside a 'solaris' pmdas but zfs is now available on linux. ZoL exports a similar set of kstat like metrics.

[natoscott]

Oh that's interesting - are the metrics exported via libzfs on Linux too? If so, the right approach here might be to create a pmdazfs that can be installed on either platform.

Do you use ZFS on Linux and if so, do you want to take this work on? I'm happy to give you pointers as neede, just let me know.

[jeffpc]

AFAIK, on Solaris and Illumos systems the zfs stats are exposed via kstats - a generic stats framework used by the kernel to export all sorts of kernel stats to userspace. This is totally unrelated to libzfs. Since Linux doesn't have kstats, I don't know how these stats are exported to userspace.

On Illumos with two pools (rpool & storage):

$ kstat -p zfs:::
zfs:0:arcstats:anon_evictable_data  0
zfs:0:arcstats:anon_evictable_metadata  0
zfs:0:arcstats:anon_size    425984
zfs:0:arcstats:arc_meta_limit   2537575424
zfs:0:arcstats:arc_meta_max 222719568
zfs:0:arcstats:arc_meta_min 634393856
zfs:0:arcstats:arc_meta_used    222672800
zfs:0:arcstats:c    10150301696
zfs:0:arcstats:c_max    10150301696
zfs:0:arcstats:c_min    1268787712
zfs:0:arcstats:class    misc
zfs:0:arcstats:crtime   13.066910337
zfs:0:arcstats:data_size    732880384
zfs:0:arcstats:deleted  17
zfs:0:arcstats:demand_data_hits 204922
zfs:0:arcstats:demand_data_misses   7025
zfs:0:arcstats:demand_metadata_hits 353788
zfs:0:arcstats:demand_metadata_misses   34908
zfs:0:arcstats:duplicate_buffers    0
zfs:0:arcstats:duplicate_buffers_size   0
zfs:0:arcstats:duplicate_reads  0
zfs:0:arcstats:evict_l2_cached  0
zfs:0:arcstats:evict_l2_eligible    135680
zfs:0:arcstats:evict_l2_ineligible  2048
zfs:0:arcstats:evict_l2_skip    0
zfs:0:arcstats:evict_not_enough 0
zfs:0:arcstats:evict_skip   2375
zfs:0:arcstats:hash_chain_max   1
zfs:0:arcstats:hash_chains  127
zfs:0:arcstats:hash_collisions  206
zfs:0:arcstats:hash_elements    20890
zfs:0:arcstats:hash_elements_max    20954
zfs:0:arcstats:hdr_size 4717728
zfs:0:arcstats:hits 565560
zfs:0:arcstats:l2_abort_lowmem  0
zfs:0:arcstats:l2_asize 0
zfs:0:arcstats:l2_cdata_free_on_write   0
zfs:0:arcstats:l2_cksum_bad 0
zfs:0:arcstats:l2_compress_failures 0
zfs:0:arcstats:l2_compress_successes    0
zfs:0:arcstats:l2_compress_zeros    0
zfs:0:arcstats:l2_evict_l1cached    0
zfs:0:arcstats:l2_evict_lock_retry  0
zfs:0:arcstats:l2_evict_reading 0
zfs:0:arcstats:l2_feeds 0
zfs:0:arcstats:l2_free_on_write 0
zfs:0:arcstats:l2_hdr_size  0
zfs:0:arcstats:l2_hits  0
zfs:0:arcstats:l2_io_error  0
zfs:0:arcstats:l2_misses    0
zfs:0:arcstats:l2_read_bytes    0
zfs:0:arcstats:l2_rw_clash  0
zfs:0:arcstats:l2_size  0
zfs:0:arcstats:l2_write_bytes   0
zfs:0:arcstats:l2_writes_done   0
zfs:0:arcstats:l2_writes_error  0
zfs:0:arcstats:l2_writes_lock_retry 0
zfs:0:arcstats:l2_writes_sent   0
zfs:0:arcstats:memory_throttle_count    0
zfs:0:arcstats:metadata_size    172011520
zfs:0:arcstats:mfu_evictable_data   294233600
zfs:0:arcstats:mfu_evictable_metadata   11375104
zfs:0:arcstats:mfu_ghost_evictable_data 0
zfs:0:arcstats:mfu_ghost_evictable_metadata 0
zfs:0:arcstats:mfu_ghost_hits   0
zfs:0:arcstats:mfu_ghost_size   0
zfs:0:arcstats:mfu_hits 428190
zfs:0:arcstats:mfu_size 309649408
zfs:0:arcstats:misses   49514
zfs:0:arcstats:mru_evictable_data   438646784
zfs:0:arcstats:mru_evictable_metadata   73229824
zfs:0:arcstats:mru_ghost_evictable_data 0
zfs:0:arcstats:mru_ghost_evictable_metadata 0
zfs:0:arcstats:mru_ghost_hits   0
zfs:0:arcstats:mru_ghost_size   0
zfs:0:arcstats:mru_hits 130520
zfs:0:arcstats:mru_size 594816512
zfs:0:arcstats:mutex_miss   0
zfs:0:arcstats:other_size   45943552
zfs:0:arcstats:p    5075150848
zfs:0:arcstats:prefetch_data_hits   40
zfs:0:arcstats:prefetch_data_misses 2351
zfs:0:arcstats:prefetch_metadata_hits   6810
zfs:0:arcstats:prefetch_metadata_misses 5230
zfs:0:arcstats:size 955553184
zfs:0:arcstats:snaptime 797.095073772
zfs:0:rpool:class   disk
zfs:0:rpool:crtime  13.613081594
zfs:0:rpool:nread   300412928
zfs:0:rpool:nwritten    46036992
zfs:0:rpool:rcnt    0
zfs:0:rpool:reads   11623
zfs:0:rpool:rlastupdate 792236607667
zfs:0:rpool:rlentime    11567533720
zfs:0:rpool:rtime   4103293101
zfs:0:rpool:snaptime    797.095381801
zfs:0:rpool:wcnt    0
zfs:0:rpool:wlastupdate 792236544998
zfs:0:rpool:wlentime    95630219701
zfs:0:rpool:writes  3406
zfs:0:rpool:wtime   1361647235
zfs:0:storage:class disk
zfs:0:storage:crtime    13.773738155
zfs:0:storage:nread 114585600
zfs:0:storage:nwritten  109811200
zfs:0:storage:rcnt  0
zfs:0:storage:reads 8059
zfs:0:storage:rlastupdate   796584315130
zfs:0:storage:rlentime  4969686426
zfs:0:storage:rtime 3408945696
zfs:0:storage:snaptime  797.095431242
zfs:0:storage:wcnt  0
zfs:0:storage:wlastupdate   796584023740
zfs:0:storage:wlentime  35066891661
zfs:0:storage:writes    9412
zfs:0:storage:wtime 1092824960
zfs:0:vdev_cache_stats:class    misc
zfs:0:vdev_cache_stats:crtime   13.066934950
zfs:0:vdev_cache_stats:delegations  0
zfs:0:vdev_cache_stats:hits 0
zfs:0:vdev_cache_stats:misses   0
zfs:0:vdev_cache_stats:snaptime 797.095348176
zfs:0:xuio_stats:class  misc
zfs:0:xuio_stats:crtime 13.051310423
zfs:0:xuio_stats:onloan_read_buf    0
zfs:0:xuio_stats:onloan_write_buf   0
zfs:0:xuio_stats:read_buf_copied    0
zfs:0:xuio_stats:read_buf_nocopy    0
zfs:0:xuio_stats:snaptime   797.095024276
zfs:0:xuio_stats:write_buf_copied   0
zfs:0:xuio_stats:write_buf_nocopy   566
zfs:0:zfetchstats:bogus_streams 0
zfs:0:zfetchstats:class misc
zfs:0:zfetchstats:colinear_hits 27
zfs:0:zfetchstats:colinear_misses   474862
zfs:0:zfetchstats:crtime    13.062448407
zfs:0:zfetchstats:hits  376780
zfs:0:zfetchstats:misses    474889
zfs:0:zfetchstats:reclaim_failures  473845
zfs:0:zfetchstats:reclaim_successes 1017
zfs:0:zfetchstats:snaptime  797.095047105
zfs:0:zfetchstats:streams_noresets  3413
zfs:0:zfetchstats:streams_resets    26
zfs:0:zfetchstats:stride_hits   373367
zfs:0:zfetchstats:stride_misses 1737

[cburroughs]

I use ZFS on linux but probably do not have the time/expertise to work on this in the near future.

The kstats for the arc are exposed via /proc. Tools like arcstat and arcsummary have succesfully been ported to Linux using this.

$ cat /proc/spl/kstat/zfs/arcstats  | head
5 1 0x01 85 4080 13244929775 34208057604003643
name                            type data
hits                            4    20277124688
misses                          4    20409570979
demand_data_hits                4    18717631221
demand_data_misses              4    17805335114
demand_metadata_hits            4    347311602
demand_metadata_misses          4    3882959
prefetch_data_hits              4    262568264
prefetch_data_misses            4    2596018462

libzfs will (I think) "Just Work" (with the standard caveats about it not being intended as a stable/production interface). It looks like that is how pcp is getting the zpool topoligy information.