Sync 6.0 Stage with master

[brad-lewis]

This PR backports several commits from master and should effectively sync up master.

We need to coordinate with an update of bcc to work around the same issue we hit here updating master:
#36 Invalid mem errors when using latest BCC

I tested this on a 6.0/stage machine with an updated bcc that Pavel set up by running all the estat and stbtrace scripts. While not testing functionality of the scripts, this does make sure the tracing scripts compile and run. It is not clean as these known issues are present:
On 6.0.5, estat zpl and zvol subcommands are failing to compile
estat warning messages

sudo ./estat.py backend-io 1
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
1 warning generated.
WARNING: kprobe: blk_start_request - not found
10/31/20 - 16:02:22 UTC

Tracing enabled... Hit Ctrl-C to end.
microseconds write
value range count ------------- Distribution -------------
[700, 800) 2 |@@@@@@@@@
[800, 900) 3 |@@@@@@@@@@@@@@@@@

                                   iops(/s)  avg latency(us)       stddev(us)  throughput(k/s)

write 5 787 1758 68

                                   iops(/s)  throughput(k/s)

total 5 68

sudo ./estat.py iscsi 1
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
1 warning generated.
10/31/20 - 16:02:25 UTC

Tracing enabled... Hit Ctrl-C to end.
sudo ./estat.py metaslab-alloc 1
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
1 warning generated.
10/31/20 - 16:02:30 UTC

Tracing enabled... Hit Ctrl-C to end.
sudo ./estat.py nfs 1
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
1 warning generated.
10/31/20 - 16:02:37 UTC

Tracing enabled... Hit Ctrl-C to end.
sudo ./estat.py nfs-by-client 1
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
1 warning generated.
10/31/20 - 16:02:44 UTC

Tracing enabled... Hit Ctrl-C to end.
sudo ./estat.py zio 1
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
1 warning generated.
10/31/20 - 16:02:48 UTC

Tracing enabled... Hit Ctrl-C to end.
sudo ./estat.py zio-queue 1
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
1 warning generated.
10/31/20 - 16:02:52 UTC

Tracing enabled... Hit Ctrl-C to end.
sudo ./estat.py zpl 1
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
/virtual/main.c:92:2: error: unknown type name 'rangelock_t'
rangelock_t zv_rangelock; /* for range locking */
^
/virtual/main.c:193:3: warning: 'memcpy' will always overflow; destination buffer has size 5, but size argument is 6 [-Wfortify-source]
__builtin_memcpy(&axis, "sync", WRITE_LENGTH);
^
/virtual/main.c:196:3: warning: 'memcpy' will always overflow; destination buffer has size 5, but size argument is 6 [-Wfortify-source]
__builtin_memcpy(&axis, "async", WRITE_LENGTH);
^
3 warnings and 1 error generated.
Traceback (most recent call last):
File "./estat.py", line 402, in
b = BPF(text=bpf_text, cflags=cflags, debug=debug_level)
File "/usr/lib/python3/dist-packages/bcc/init.py", line 364, in init
raise Exception("Failed to compile BPF module %s" % (src_file or ""))
Exception: Failed to compile BPF module
sudo ./estat.py zil
Tracing enabled... Hit Ctrl-C to end.
^C10/31/20 - 16:03:11 UTC

sudo ./estat.py txg
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
1 warning generated.
^Csudo ./estat.py arc_prefetch
Warning: No pool filtering, unable to find zfs pool domain0
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
1 warning generated.
10/31/20 - 16:03:32 UTC

prefetch hit count : 2
10/31/20 - 16:03:37 UTC

prefetch hit count : 99
^C10/31/20 - 16:03:39 UTC

sudo ./stbtrace.py io
{"t":"1604160222", "op":"write", "device":"nvme0n1", "error":"0", "count":"4", "avgLatency":"813528", "throughput":"49152", "latency":"{800000,2},{900000,1},{1000000,1}", "size":"{16383,4}"}

{"t":"1604160223", "op":"write", "device":"nvme0n1", "error":"0", "count":"5", "avgLatency":"840022", "throughput":"61440", "latency":"{800000,3},{900000,1},{2000000,1}", "size":"{8191,1},{16383,3},{32767,1}"}

^Csudo ./stbtrace.py iscsi
^Csudo ./stbtrace.py nfs
^Csudo ./stbtrace.py vfs
{"t":"1604160235", "op":"write", "sync":"0", "cached":"-1", "count":"126", "avgLatency":"15789", "throughput":"22007", "latency":"{10000,53},{20000,62},{30000,2},{40000,1},{50000,4},{90000,2},{200000,1},{300000,1}", "size":"{63,29},{127,3},{255,63},{511,31}"}

{"t":"1604160235", "op":"read", "sync":"-1", "cached":"1", "count":"130", "avgLatency":"2206", "throughput":"93580", "latency":"{10000,130}", "size":"{31,5},{127,5},{511,15},{1023,105}"}

{"t":"1604160236", "op":"write", "sync":"0", "cached":"-1", "count":"4", "avgLatency":"44290", "throughput":"32768", "latency":"{40000,1},{50000,3}", "size":"{16383,4}"}

{"t":"1604160237", "op":"write", "sync":"0", "cached":"-1", "count":"4", "avgLatency":"41870", "throughput":"32768", "latency":"{30000,1},{50000,3}", "size":"{16383,4}"}

^Csudo ./stbtrace.py zio
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
1 warning generated.
{"t":"1604160241", "op":"write", "count":"1", "avgLatency":"1112882", "throughput":"20480", "latency":"{2000000,1}", "size":"{32767,1}"}

{"t":"1604160242", "op":"write", "count":"7", "avgLatency":"924654", "throughput":"110592", "latency":"{800000,1},{900000,3},{1000000,2},{2000000,1}", "size":"{16383,5},{32767,2}"}

{"t":"1604160243", "op":"write", "count":"145", "avgLatency":"965054", "throughput":"1545216", "latency":"{700000,15},{800000,25},{900000,40},{1000000,18},{2000000,46},{3000000,1}", "size":"{1023,10},{2047,23},{4095,24},{8191,39},{16383,24},{32767,14},{65535,8},{131071,3}"}

^Csudo ./stbtrace.py zpl
In file included from :3:
/virtual/include/bcc/helpers.h:47:9: warning: 'CC_USING_FENTRY' macro redefined [-Wmacro-redefined]
#define CC_USING_FENTRY
^
:7:9: note: previous definition is here
#define CC_USING_FENTRY 1
^
1 warning generated.
{"t":"1604160246", "op":"write", "sync":"0", "cached":"-1", "count":"4", "avgLatency":"49799", "throughput":"32768", "latency":"{50000,1},{60000,3}", "size":"{16383,4}"}

{"t":"1604160247", "op":"write", "sync":"0", "cached":"-1", "count":"16", "avgLatency":"35498", "throughput":"103034", "latency":"{20000,4},{30000,3},{40000,1},{50000,6},{60000,1},{70000,1}", "size":"{1023,2},{4095,2},{8191,1},{16383,11}"}

{"t":"1604160247", "op":"read", "sync":"-1", "cached":"1", "count":"33", "avgLatency":"5056", "throughput":"197120", "latency":"{10000,29},{20000,3},{30000,1}", "size":"{1023,3},{2047,6},{4095,1},{8191,1},{16383,22}"}

{"t":"1604160247", "op":"write", "sync":"0", "cached":"-1", "count":"3", "avgLatency":"63960", "throughput":"24576", "latency":"{60000,1},{70000,1},{80000,1}", "size":"{16383,3}"}

{"t":"1604160248", "op":"write", "sync":"0", "cached":"-1", "count":"11", "avgLatency":"95251", "throughput":"106496", "latency":"{40000,2},{50000,3},{60000,2},{70000,2},{90000,1},{600000,1}", "size":"{16383,10},{32767,1}"}

{"t":"1604160248", "op":"write", "sync":"0", "cached":"-1", "count":"3", "avgLatency":"49222", "throughput":"24576", "latency":"{50000,2},{60000,1}", "size":"{16383,3}"}

{"t":"1604160249", "op":"write", "sync":"0", "cached":"-1", "count":"11", "avgLatency":"45978", "throughput":"90112", "latency":"{30000,1},{40000,1},{50000,6},{60000,3}", "size":"{16383,11}"}

./nfs_threads.py
packets sockets threads threads metadata read read write write
arrived enqueued woken used calls iops thruput iops thruput
0 0 0 0 0 0.0 0KB 0.0 0KB

[pzakha]

Did you run analytics_positive on those changes (+ my BCC merge)? If so, could you add a link to it?

Also, you may want to escape the output in your description with ``` for easier readability.

We will need to coordinate so that we can land this together with my BCC merge (delphix/bcc#8).

[brad-lewis]

Here is an os_test run that does analytics_positive.

There is an failure with nfs_ops that I'm looking into.

[brad-lewis]

The issue with the previous os_tests was in manually selecting the the right environment to test with. Here is a run with two known failures:
DLPX-73857
DLPX-69346