[performance] z_rd_int is not using all available cpu cycles during read

[mailinglists35]

I needed to compare two snapshots and I ran a "diff -qr" between them.
Noticed that z_rd_int/0 is consuming only a limited set of available resources (just 40% of cpu and only 25..28 percent of disk performance).

I have come to this conclusion by looking at the number of I/O requests in queue and they are zero, which means the drives are not fully stressed.
Then I thought: ok, this means zfs cannot process more data because is busy processing what has read so it must be fully busy with 100%, but then looking at cpu, I see only a quarter occupied by system process, another third busy on I/O and a third part idle.

Seeing this, I enable "top"-style process reporting in nmon and see that z_rd_int/0 is only taking 40%.
Instead I expected either for disks to burn in flames at 100% busy and few more I/O in "Inflight" queue, or cpu to be used at maximum.

zfs is 0.6.3, kernel is 3.16-2-amd64 on debian wheezy. notice that "diff" is taking a mere 3.9% fraction of cpu and no other processes are significantly consuming resources at that moment.

note: this happens only on datasets with lz4 compression! z_rd_int takes only few cpu percents on datasets with default compression

┌nmon─13g──────[H for help]───Hostname=xrouter───Refresh= 1secs───04:59.41───────────────────────────────────┐
│ CPU Utilisation ───────────────────────────────────────────────────────────────────────────────────────────────│
│                           +-------------------------------------------------+                                  │
│CPU  User%  Sys% Wait% Idle|0          |25         |50          |75       100|                                  │
│ 1   2.0  24.5  39.8   33.7|UssssssssssssWWWWWWWWWWWWWWWWWWW    >            |                                  │
│ 2   1.0  26.0  33.7   39.4|ssssssssssssWWWWWWWWWWWWWWWW      >              |                                  │
│                           +-------------------------------------------------+                                  │
│Avg  1.5  24.6  36.9   36.9|ssssssssssssWWWWWWWWWWWWWWWWWW     >             |                                  │
│                           +-------------------------------------------------+                                  │
│ Disk I/O ──/proc/diskstats────mostly in KB/s─────Warning:contains duplicates───────────────────────────────────│
│DiskName Busy    Read    MBite       Xfers   Size  Peak%  Peak-RW    InFlight                                   │
│sdb        0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sdb1       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sdb2       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sdb3       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sdb4       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sda        0%      0.0      0.0KB/s    0.0   0.0KB    6%     287.4KB/s   0                                      │
│sda1       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sda2       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sda3       0%      0.0      0.0KB/s    0.0   0.0KB    5%     159.7KB/s   0                                      │
│sda4       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sda5       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sda6       0%      0.0      0.0KB/s    0.0   0.0KB    0%     127.9KB/s   0                                      │
│sdc       28%  24028.5      0.0KB/s  190.4 126.2KB   40%   27346.4KB/s   0                                      │
│sdc1      28%  24028.5      0.0KB/s  190.4 126.2KB   40%   27346.4KB/s   0                                      │
│sdc9       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sdd       25%  18243.3      0.0KB/s  150.0 121.7KB   37%   28340.6KB/s   0                                      │
│sdd1      25%  18243.3      0.0KB/s  150.0 121.7KB   36%   28340.6KB/s   0                                      │
│sdd9       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sde        0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│sde1       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│zd0        0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│zd0p1      0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0                                      │
│Totals Read-MB/s=82.6     Writes-MB/s=0.0      Transfers/sec=680.7                                              │
│ Top Processes Procs=305 mode=3 (1=Basic, 3=Perf 4=Size 5=I/O)──────────────────────────────────────────────────│
│  PID    %CPU  Size   Res   Res   Res   Res Shared   Faults Command                                             │
│          Used    KB   Set  Text  Data   Lib    KB  Min  Maj                                                    │
│    1096  40.4     0     0     0     0     0     0    0    0 z_rd_int/0                                         │
│   17728   3.9 10320  2204   120     0   528  2040    0    0 diff                                               │
│    6033   3.0     0     0     0     0     0     0    0    0 kworker/1:1                                        │
│    3772   2.0 317668 18544   204     0 159220  7624    0    0 motion                                           │
│   11999   2.0     0     0     0     0     0     0    0    0 kworker/0:1                                        │
│   17673   2.0 17536  5584   100     0  7108  1876  300    0 nmon                                               │
└─────────Warning: Some Statistics may not shown─────────────────────────────────────────────────────────────────┘
0$ bash  1-$ bash  2$* bash                                                               Tue 09 Dec 2014  4:59:42

root@xrouter:/var/tmp# zfs get all xbackup/4s511
NAME              PROPERTY               VALUE                  SOURCE
xbackup/4s511  type                   filesystem             -
xbackup/4s511  creation               Sun Aug 18 10:53 2013  -
xbackup/4s511  used                   23.6G                  -
xbackup/4s511  available              31.1G                  -
xbackup/4s511  referenced             8.22G                  -
xbackup/4s511  compressratio          1.10x                  -
xbackup/4s511  mounted                yes                    -
xbackup/4s511  quota                  none                   default
xbackup/4s511  reservation            none                   default
xbackup/4s511  recordsize             128K                   default
xbackup/4s511  mountpoint             /mnt/4s511             local
xbackup/4s511  sharenfs               off                    default
xbackup/4s511  checksum               on                     default
xbackup/4s511  compression            lz4                    inherited from xbackup
xbackup/4s511  atime                  off                    inherited from xbackup
xbackup/4s511  devices                on                     default
xbackup/4s511  exec                   on                     default
xbackup/4s511  setuid                 on                     default
xbackup/4s511  readonly               off                    default
xbackup/4s511  zoned                  off                    default
xbackup/4s511  snapdir                hidden                 default
xbackup/4s511  aclinherit             restricted             default
xbackup/4s511  canmount               on                     default
xbackup/4s511  xattr                  on                     default
xbackup/4s511  copies                 1                      default
xbackup/4s511  version                5                      -
xbackup/4s511  utf8only               off                    -
xbackup/4s511  normalization          none                   -
xbackup/4s511  casesensitivity        sensitive              -
xbackup/4s511  vscan                  off                    default
xbackup/4s511  nbmand                 off                    default
xbackup/4s511  sharesmb               off                    default
xbackup/4s511  refquota               none                   default
xbackup/4s511  refreservation         none                   default
xbackup/4s511  primarycache           metadata               inherited from xbackup
xbackup/4s511  secondarycache         all                    default
xbackup/4s511  usedbysnapshots        15.4G                  -
xbackup/4s511  usedbydataset          8.22G                  -
xbackup/4s511  usedbychildren         0                      -
xbackup/4s511  usedbyrefreservation   0                      -
xbackup/4s511  logbias                throughput             inherited from xbackup
xbackup/4s511  dedup                  on                     local
xbackup/4s511  mlslabel               none                   default
xbackup/4s511  sync                   standard               default
xbackup/4s511  refcompressratio       1.16x                  -
xbackup/4s511  written                3.94G                  -
xbackup/4s511  logicalused            25.0G                  -
xbackup/4s511  logicalreferenced      8.94G                  -
xbackup/4s511  snapdev                hidden                 default
xbackup/4s511  acltype                off                    default
xbackup/4s511  context                none                   default
xbackup/4s511  fscontext              none                   default
xbackup/4s511  defcontext             none                   default
xbackup/4s511  rootcontext            none                   default
xbackup/4s511  relatime               off                    default
xbackup/4s511  com.sun:auto-snapshot  false                  inherited from xbackup

root@xrouter:/var/tmp# zpool get all xbackup
NAME        PROPERTY               VALUE                  SOURCE
xbackup  size                   928G                   -
xbackup  capacity               95%                    -
xbackup  altroot                -                      default
xbackup  health                 ONLINE                 -
xbackup  guid                   7515519392152294710    default
xbackup  version                -                      default
xbackup  bootfs                 -                      default
xbackup  delegation             on                     default
xbackup  autoreplace            off                    default
xbackup  cachefile              -                      default
xbackup  failmode               continue               local
xbackup  listsnapshots          off                    default
xbackup  autoexpand             off                    default
xbackup  dedupditto             0                      default
xbackup  dedupratio             1.16x                  -
xbackup  free                   45.7G                  -
xbackup  allocated              882G                   -
xbackup  readonly               off                    -
xbackup  ashift                 12                     local
xbackup  comment                -                      default
xbackup  expandsize             0                      -
xbackup  freeing                0                      default
xbackup  feature@async_destroy  enabled                local
xbackup  feature@empty_bpobj    active                 local
xbackup  feature@lz4_compress   active                 local

[mailinglists35]

also on datasets with gzip-9 makes a bit more of effort (10..18% cpu)