Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Corrupted data using raw zfs send/recv with encryption #8852

Closed
dsnet opened this issue Jun 3, 2019 · 11 comments
Closed

Corrupted data using raw zfs send/recv with encryption #8852

dsnet opened this issue Jun 3, 2019 · 11 comments
Labels
Component: Encryption "native encryption" feature Component: Send/Recv "zfs send/recv" feature Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@dsnet
Copy link

dsnet commented Jun 3, 2019

Using zfsonlinux 0.8.

I have two machines src and dst.

On src, I have an encrypted dataset that is 77G large. The non-default properties are:

$ zfs get all tank/drive | grep -v default
NAME        PROPERTY              VALUE                  SOURCE
tank/drive  type                  filesystem             -
tank/drive  creation              Sat Jun  1 16:51 2019  -
tank/drive  used                  77.4G                  -
tank/drive  available             3.44T                  -
tank/drive  referenced            77.4G                  -
tank/drive  compressratio         1.94x                  -
tank/drive  mounted               yes                    -
tank/drive  mountpoint            /mnt/drive             local
tank/drive  compression           gzip                   local
tank/drive  atime                 off                    local
tank/drive  createtxg             235                    -
tank/drive  version               5                      -
tank/drive  utf8only              off                    -
tank/drive  normalization         none                   -
tank/drive  casesensitivity       sensitive              -
tank/drive  guid                  12695194897195142574   -
tank/drive  usedbysnapshots       264K                   -
tank/drive  usedbydataset         77.4G                  -
tank/drive  usedbychildren        0B                     -
tank/drive  usedbyrefreservation  0B                     -
tank/drive  objsetid              80                     -
tank/drive  refcompressratio      1.94x                  -
tank/drive  written               264K                   -
tank/drive  logicalused           150G                   -
tank/drive  logicalreferenced     150G                   -
tank/drive  encryption            aes-256-ccm            -
tank/drive  keylocation           prompt                 local
tank/drive  keyformat             hex                    -
tank/drive  encryptionroot        tank/drive             -
tank/drive  keystatus             available              -

I send that dataset from src to dst using (running on the dst machine):

$ ssh src zfs send --raw tank/drive@2019-06-03T07:32:00Z | zfs recv -s -o mountpoint=none -o readonly=on tank/mirror/src

In the middle of the send/recv I had a network failure and had to use the resumable send/recv feature to resume the send. At the very end, the operation completes successfully.

To test that the data really is the same between src and dst, I began hashing files on both ends to verify integrity, but ran into some input/output errors. Specifically, there are no errors on the src dataset, but there are errors on the dst machine:

# running on dst machine
$ zfs set mountpoint=/tmp/mirror-src tank/mirror/src
$ zfs mount -l tank/mirror/src  # enter the encryption key at the prompt 
$ cd <redacted> && ls -l
ls: cannot access 'Sep27_CADthru': Input/output error
ls: cannot access 'Sep28_UDF_Ex11': Input/output error
ls: cannot access 'Sep25_SCT_Basic_Cource': Input/output error
ls: cannot access 'Sep26_SCT_Geometry_Fix': Input/output error
ls: cannot access 'Sep28_ST_and_HeatPAC': Input/output error
total 346
d????????? ? ?    ?         ?            ? Sep25_SCT_Basic_Cource
d????????? ? ?    ?         ?            ? Sep26_SCT_Geometry_Fix
d????????? ? ?    ?         ?            ? Sep27_CADthru
d????????? ? ?    ?         ?            ? Sep28_ST_and_HeatPAC
d????????? ? ?    ?         ?            ? Sep28_UDF_Ex11
-rwxr----- 1 1001 1001 350208 Sep 25  2006 HowToAccessOurUsersPage.doc
-rwxr----- 1 1001 1001  21504 Sep 26  2006 Schedule.xls

This suggests that either a problem with the send or the recv.

This may not be the most helpful, but I'm happy to run more commands to help diagnose the problem.

@dsnet
Copy link
Author

dsnet commented Jun 3, 2019

After accessing the problematic directory, zpool status reports:

  pool: tank
 state: ONLINE
status: One or more devices has experienced an error resulting in data
	corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
	entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 0 days 01:52:29 with 0 errors on Fri May 31 06:06:58 2019
config:

	NAME                                             STATE     READ WRITE CKSUM
	tank                                             ONLINE       0     0     0
	  raidz1-0                                       ONLINE       0     0     0
	    ata-Hitachi_HUA723020ALA641_YGHZGVBA         ONLINE       0     0     0
	    ata-ST2000DM001-1CH164_Z1E3D0VL              ONLINE       0     0     0
	    ata-ST2000DM001-1CH164_Z1E3CKD0              ONLINE       0     0     0
	    ata-ST2000DM001-1CH164_Z1E3C1VY              ONLINE       0     0     0
	    ata-ST2000DM001-1ER164_W4Z06MQ2              ONLINE       0     0     0
	logs
	  mirror-1                                       ONLINE       0     0     0
	    ata-MKNSSDCR240GB-7_MK140611AS1405515-part4  ONLINE       0     0     0
	    ata-MKNSSDCR240GB-7_MK140611AS1405515-part5  ONLINE       0     0     0
	cache
	  ata-MKNSSDCR240GB-7_MK140611AS1405515-part3    ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        tank/mirror/src:<0x0>

@dsnet dsnet changed the title Corrupted data using raw zfs send/recv with encryption Corrupted data using resumable raw zfs send/recv with encryption Jun 3, 2019
@dsnet
Copy link
Author

dsnet commented Jun 3, 2019

@kpande, are you suggesting that this is a duplicate of #8168 or simply related? That issue is closed with code committed that is present in 0.8, so either did not fix the issue or I'm hitting a different issue.

@dsnet
Copy link
Author

dsnet commented Jun 4, 2019

Additional datapoints:

  • zpool scrub reports no issues.
  • Another send/recv pair without resuming still has corruption, suggesting that resuming is not the issue. Note that the -s flag is still specified for recv.
  • Corruption occurs on the exact same files, suggesting that the problem is deterministic.

@dsnet dsnet changed the title Corrupted data using resumable raw zfs send/recv with encryption Corrupted data using raw zfs send/recv with encryption Jun 4, 2019
@behlendorf behlendorf added Component: Encryption "native encryption" feature Component: Send/Recv "zfs send/recv" feature Type: Defect Incorrect behavior (e.g. crash, hang) labels Jun 4, 2019
@behlendorf
Copy link
Contributor

@dsnet based on your description and debugging I agree this is unlikely related to resumable sends or issue #8168 which was resolved.

Based on the errors you're seeing it sounds as if a dnode block on the receive side isn't exactly matching its counterpart from the source. This would result in authentication errors and the symptoms you're seeing. Can you run zpool events -v and verify there are authentication errors being reported.

Any hints you can provide which would help us reproduce this locally would be very helpful.

@dsnet
Copy link
Author

dsnet commented Jun 4, 2019

Not sure what to look for, but here's the full printout:

$ sudo zpool events -v
TIME                           CLASS
Jun  3 2019 15:10:40.835224357 ereport.fs.zfs.zpool
        class = "ereport.fs.zfs.zpool"
        ena = 0x3992fbba301c01
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x79cb7bbee62fca18
        (end detector)
        pool = "zfs-crypt"
        pool_guid = 0x79cb7bbee62fca18
        pool_state = 0x0
        pool_context = 0x2
        pool_failmode = "wait"
        time = 0x5cf59ae0 0x31c88325
        eid = 0x1

Jun 3 2019 15:10:44.359224224 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_internal_str = "pool version 5000; software version unknown; uts prism 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64"
history_internal_name = "open"
history_txg = 0x226338a
history_time = 0x5cf59ae4
time = 0x5cf59ae4 0x156953a0
eid = 0x2

Jun 3 2019 15:10:45.475224182 sysevent.fs.zfs.config_sync
version = 0x0
class = "sysevent.fs.zfs.config_sync"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
time = 0x5cf59ae5 0x1c535876
eid = 0x3

Jun 3 2019 15:10:45.475224182 sysevent.fs.zfs.pool_import
version = 0x0
class = "sysevent.fs.zfs.pool_import"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
time = 0x5cf59ae5 0x1c535876
eid = 0x4

Jun 3 2019 15:10:45.475224182 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_internal_str = "pool version 5000; software version unknown; uts prism 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64"
history_internal_name = "import"
history_txg = 0x226338c
history_time = 0x5cf59ae5
time = 0x5cf59ae5 0x1c535876
eid = 0x5

Jun 3 2019 15:10:46.095224159 sysevent.fs.zfs.config_sync
version = 0x0
class = "sysevent.fs.zfs.config_sync"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
time = 0x5cf59ae6 0x5ad015f
eid = 0x6

Jun 3 2019 15:19:51.706589338 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83cfae7e6001401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59d07 0x2a1db29a
eid = 0x7

Jun 3 2019 15:19:51.706589338 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83cfb3dd7e01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59d07 0x2a1db29a
eid = 0x8

Jun 3 2019 15:19:51.790591564 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83d4af070800801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59d07 0x2f1f784c
eid = 0x9

Jun 3 2019 15:19:51.790591564 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83d4b4282f00801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59d07 0x2f1f784c
eid = 0xa

Jun 3 2019 15:19:51.830592624 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83d718609f00c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d07 0x3181d670
eid = 0xb

Jun 3 2019 15:19:51.830592624 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83d721283000c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d07 0x3181d670
eid = 0xc

Jun 3 2019 15:19:51.846593048 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83d82ec3f000c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59d07 0x3275fc18
eid = 0xd

Jun 3 2019 15:19:51.846593048 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83d833d17800c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59d07 0x3275fc18
eid = 0xe

Jun 3 2019 15:19:51.874593790 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83d9d19d5800c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d07 0x34213dfe
eid = 0xf

Jun 3 2019 15:19:51.874593790 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83d9d9e95200c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d07 0x34213dfe
eid = 0x10

Jun 3 2019 15:19:51.898594426 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83db205ca300c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d07 0x358f767a
eid = 0x11

Jun 3 2019 15:19:51.898594426 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83db29993e00c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d07 0x358f767a
eid = 0x12

Jun 3 2019 15:19:51.918594956 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83dc522bd701001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d07 0x36c0a58c
eid = 0x13

Jun 3 2019 15:19:51.918594956 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83dc5bff2001001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d07 0x36c0a58c
eid = 0x14

Jun 3 2019 15:19:51.922595062 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83dc9d933f01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d07 0x36fdaef6
eid = 0x15

Jun 3 2019 15:19:51.922595062 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83dca693d401001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d07 0x36fdaef6
eid = 0x16

Jun 3 2019 15:19:51.962596122 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83defb552901001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59d07 0x39600d1a
eid = 0x17

Jun 3 2019 15:19:51.962596122 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83df00bfa501001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59d07 0x39600d1a
eid = 0x18

Jun 3 2019 15:19:52.206602587 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83ed85a7a200c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d08 0xc50815b
eid = 0x19

Jun 3 2019 15:19:52.206602587 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83ed8efc6c00c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d08 0xc50815b
eid = 0x1a

Jun 3 2019 15:19:52.222603011 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83ee6a170700c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59d08 0xd44a703
eid = 0x1b

Jun 3 2019 15:19:52.222603011 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83ee6f836000c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59d08 0xd44a703
eid = 0x1c

Jun 3 2019 15:19:52.242603541 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83efb5cba500801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d08 0xe75d615
eid = 0x1d

Jun 3 2019 15:19:52.242603541 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x83efbb711500801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59d08 0xe75d615
eid = 0x1e

Jun 3 2019 15:19:52.758617213 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x840e81eac101001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d08 0x2d37947d
eid = 0x1f

Jun 3 2019 15:19:52.758617213 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x840e86bf7a01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d08 0x2d37947d
eid = 0x20

Jun 3 2019 15:19:52.758617213 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x840e8d2fd401001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d08 0x2d37947d
eid = 0x21

Jun 3 2019 15:19:52.758617213 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x840e92d3d601001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d08 0x2d37947d
eid = 0x22

Jun 3 2019 15:19:52.762617319 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x840e98e0b601001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d08 0x2d749de7
eid = 0x23

Jun 3 2019 15:19:52.762617319 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x840e9eccea01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d08 0x2d749de7
eid = 0x24

Jun 3 2019 15:19:52.762617319 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x840ea44dda01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d08 0x2d749de7
eid = 0x25

Jun 3 2019 15:19:52.762617319 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x840ea8f56601001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d08 0x2d749de7
eid = 0x26

Jun 3 2019 15:19:52.762617319 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x840eae2a8201001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d08 0x2d749de7
eid = 0x27

Jun 3 2019 15:19:52.762617319 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x840eb2d34a01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d08 0x2d749de7
eid = 0x28

Jun 3 2019 15:20:09.667064932 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x87fe90eca601401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d19 0x27c29a64
eid = 0x29

Jun 3 2019 15:20:09.667064932 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x87fe9a6b7601401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d19 0x27c29a64
eid = 0x2a

Jun 3 2019 15:20:09.671065037 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x87fea0bbd201401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d19 0x27ffa3cd
eid = 0x2b

Jun 3 2019 15:20:09.671065037 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x87fea655f101401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d19 0x27ffa3cd
eid = 0x2c

Jun 3 2019 15:20:09.671065037 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x87feab810b01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d19 0x27ffa3cd
eid = 0x2d

Jun 3 2019 15:22:06.542151634 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xa3348151ff00401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d8e 0x205093d2
eid = 0x2e

Jun 3 2019 15:22:06.546151739 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xa3348ce63400401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d8e 0x208d9d3b
eid = 0x2f

Jun 3 2019 15:22:06.546151739 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xa334945b8c00401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d8e 0x208d9d3b
eid = 0x30

Jun 3 2019 15:22:06.546151739 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xa3349c3a2100401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d8e 0x208d9d3b
eid = 0x31

Jun 3 2019 15:22:06.546151739 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xa334a3cf5400401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59d8e 0x208d9d3b
eid = 0x32

Jun 3 2019 15:24:52.470527212 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xc9d63dbc0c00c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59e34 0x1c0bacec
eid = 0x33

Jun 3 2019 15:24:52.470527212 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xc9d64ab34300c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59e34 0x1c0bacec
eid = 0x34

Jun 3 2019 15:24:52.470527212 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xc9d653144a00c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59e34 0x1c0bacec
eid = 0x35

Jun 3 2019 15:24:52.474527318 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xc9d65c32c400c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59e34 0x1c48b656
eid = 0x36

Jun 3 2019 15:24:52.474527318 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xc9d6663b9100c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59e34 0x1c48b656
eid = 0x37

Jun 3 2019 15:29:07.073239802 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051d7d740000801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59f33 0x45d8cfa
eid = 0x38

Jun 3 2019 15:29:07.073239802 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051d857a3b00801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59f33 0x45d8cfa
eid = 0x39

Jun 3 2019 15:29:07.077239907 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051d99a55f00801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59f33 0x49a9663
eid = 0x3a

Jun 3 2019 15:29:07.077239907 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051da9df8001001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59f33 0x49a9663
eid = 0x3b

Jun 3 2019 15:29:07.077239907 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051db5e7cf01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x49a9663
eid = 0x3c

Jun 3 2019 15:29:07.077239907 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051dbe812e01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x49a9663
eid = 0x3d

Jun 3 2019 15:29:07.081240013 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051dc6b05701001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59f33 0x4d79fcd
eid = 0x3e

Jun 3 2019 15:29:07.081240013 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051dcb6af001001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59f33 0x4d79fcd
eid = 0x3f

Jun 3 2019 15:29:07.081240013 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051dd2d04001001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x4d79fcd
eid = 0x40

Jun 3 2019 15:29:07.081240013 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051dd7a8b501001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x4d79fcd
eid = 0x41

Jun 3 2019 15:29:07.081240013 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051ddd7f3b01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x4d79fcd
eid = 0x42

Jun 3 2019 15:29:07.081240013 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051de233a101001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x4d79fcd
eid = 0x43

Jun 3 2019 15:29:07.081240013 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051de80dae01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x4d79fcd
eid = 0x44

Jun 3 2019 15:29:07.081240013 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051decb91501001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x4d79fcd
eid = 0x45

Jun 3 2019 15:29:07.081240013 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051df2ac3c01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x4d79fcd
eid = 0x46

Jun 3 2019 15:29:07.081240013 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051df82c2901001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x4d79fcd
eid = 0x47

Jun 3 2019 15:29:07.081240013 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051dff6cf201001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59f33 0x4d79fcd
eid = 0x48

Jun 3 2019 15:29:07.085240118 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e0464df01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59f33 0x514a936
eid = 0x49

Jun 3 2019 15:29:07.085240118 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e1209ea01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x514a936
eid = 0x4a

Jun 3 2019 15:29:07.085240118 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e16c42c01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x514a936
eid = 0x4b

Jun 3 2019 15:29:07.085240118 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e1d06f101001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59f33 0x514a936
eid = 0x4c

Jun 3 2019 15:29:07.085240118 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e21ac1201001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf59f33 0x514a936
eid = 0x4d

Jun 3 2019 15:29:07.085240118 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e27a65a01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x514a936
eid = 0x4e

Jun 3 2019 15:29:07.085240118 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e2c4c9c01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf59f33 0x514a936
eid = 0x4f

Jun 3 2019 15:29:07.089240224 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e4ff67801001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59f33 0x551b2a0
eid = 0x50

Jun 3 2019 15:29:07.089240224 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e581e8d01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59f33 0x551b2a0
eid = 0x51

Jun 3 2019 15:29:07.089240224 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e608e6901001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59f33 0x551b2a0
eid = 0x52

Jun 3 2019 15:29:07.089240224 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e68074a01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59f33 0x551b2a0
eid = 0x53

Jun 3 2019 15:29:07.089240224 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e6fb9cb01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59f33 0x551b2a0
eid = 0x54

Jun 3 2019 15:29:07.089240224 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e767f2701001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59f33 0x551b2a0
eid = 0x55

Jun 3 2019 15:29:07.093240329 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e7e403101001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59f33 0x58ebc09
eid = 0x56

Jun 3 2019 15:29:07.093240329 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e84fe3f01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59f33 0x58ebc09
eid = 0x57

Jun 3 2019 15:29:07.093240329 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e8cc73801001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59f33 0x58ebc09
eid = 0x58

Jun 3 2019 15:29:07.093240329 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x1051e9390ce01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf59f33 0x58ebc09
eid = 0x59

Jun 3 2019 15:29:10.389327229 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x105e327d23c01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x75
time = 0x5cf59f36 0x1734a97d
eid = 0x5a

Jun 3 2019 15:29:10.389327229 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x105e32ca30801401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x75
time = 0x5cf59f36 0x1734a97d
eid = 0x5b

Jun 3 2019 15:29:10.409327757 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x105e4247dbc01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x73
time = 0x5cf59f36 0x1865d88d
eid = 0x5c

Jun 3 2019 15:29:10.409327757 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0x105e42d08ab01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x73
time = 0x5cf59f36 0x1865d88d
eid = 0x5d

Jun 3 2019 15:29:25.961737791 sysevent.fs.zfs.scrub_start
version = 0x0
class = "sysevent.fs.zfs.scrub_start"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
time = 0x5cf59f45 0x3952f43f
eid = 0x5e

Jun 3 2019 15:29:25.961737791 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_internal_str = "func=1 mintxg=0 maxtxg=36058216"
history_internal_name = "scan setup"
history_txg = 0x2263468
history_time = 0x5cf59f45
time = 0x5cf59f45 0x3952f43f
eid = 0x5f

Jun 3 2019 15:39:49.446175755 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove"
history_internal_str = "mountpoint=none"
history_internal_name = "set"
history_dsid = 0x1b7
history_txg = 0x22634ba
history_time = 0x5cf5a1b5
time = 0x5cf5a1b5 0x1a981a0b
eid = 0x60

Jun 3 2019 15:47:36.883307728 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-retry"
history_internal_str = ""
history_internal_name = "receive"
history_dsid = 0x1cf
history_txg = 0x22634f2
history_time = 0x5cf5a388
time = 0x5cf5a388 0x34a634d0
eid = 0x61

Jun 3 2019 15:47:39.891190362 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-retry"
history_internal_str = "keylocation=prompt"
history_internal_name = "set"
history_dsid = 0x1cf
history_txg = 0x22634f4
history_time = 0x5cf5a38b
time = 0x5cf5a38b 0x351e7c5a
eid = 0x62

Jun 3 2019 17:49:25.115410746 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_internal_str = "errors=8"
history_internal_name = "scan done"
history_txg = 0x2263a03
history_time = 0x5cf5c015
time = 0x5cf5c015 0x6e1073a
eid = 0x63

Jun 3 2019 17:49:25.123410785 sysevent.fs.zfs.scrub_finish
version = 0x0
class = "sysevent.fs.zfs.scrub_finish"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
time = 0x5cf5c015 0x75b1961
eid = 0x64

Jun 3 2019 18:50:08.999122284 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-retry"
history_internal_str = "snap=2019-06-03T07:32:00Z"
history_internal_name = "finish receiving"
history_dsid = 0x1cf
history_txg = 0x2263ccb
history_time = 0x5cf5ce50
time = 0x5cf5ce50 0x3b8d656c
eid = 0x65

Jun 3 2019 18:50:08.999122284 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-retry@2019-06-03T07:32:00Z"
history_internal_str = ""
history_internal_name = "snapshot"
history_dsid = 0x1ca
history_txg = 0x2263ccb
history_time = 0x5cf5ce50
time = 0x5cf5ce50 0x3b8d656c
eid = 0x66

Jun 3 2019 19:02:02.540073681 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-retry"
history_internal_str = "mountpoint=/tmp/scotgrove"
history_internal_name = "set"
history_dsid = 0x1cf
history_txg = 0x2263d57
history_time = 0x5cf5d11a
time = 0x5cf5d11a 0x2030ded1
eid = 0x67

Jun 3 2019 19:02:07.328106803 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-retry"
history_internal_str = "readonly=1"
history_internal_name = "set"
history_dsid = 0x1cf
history_txg = 0x2263d58
history_time = 0x5cf5d11f
time = 0x5cf5d11f 0x138e8333
eid = 0x68

Jun 3 2019 19:03:10.028540475 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb355870c9601801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d15e 0x1b37e3b
eid = 0x69

Jun 3 2019 19:03:10.032540502 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb3558f985d01801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d15e 0x1f08756
eid = 0x6a

Jun 3 2019 19:03:10.032540502 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb355c51adc00001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d15e 0x1f08756
eid = 0x6b

Jun 3 2019 19:03:10.032540502 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb355c9e99100001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d15e 0x1f08756
eid = 0x6c

Jun 3 2019 19:03:10.044540585 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb3565f721401401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x2a7a2a9
eid = 0x6d

Jun 3 2019 19:03:10.044540585 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35664c0b601401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x2a7a2a9
eid = 0x6e

Jun 3 2019 19:03:10.044540585 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35673522401801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d15e 0x2a7a2a9
eid = 0x6f

Jun 3 2019 19:03:10.044540585 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb3567861b401801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d15e 0x2a7a2a9
eid = 0x70

Jun 3 2019 19:03:10.048540613 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35690b59a00801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x2e4abc5
eid = 0x71

Jun 3 2019 19:03:10.048540613 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb3569669ea00801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x2e4abc5
eid = 0x72

Jun 3 2019 19:03:10.048540613 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb356a32f8a00801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x2e4abc5
eid = 0x73

Jun 3 2019 19:03:10.048540613 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb356a8000700801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x2e4abc5
eid = 0x74

Jun 3 2019 19:03:10.048540613 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb356b4d00000801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x2e4abc5
eid = 0x75

Jun 3 2019 19:03:10.048540613 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb356b986d900801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x2e4abc5
eid = 0x76

Jun 3 2019 19:03:10.052540641 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb356c56f3600801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x321b4e1
eid = 0x77

Jun 3 2019 19:03:10.052540641 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb356ca9af700801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x321b4e1
eid = 0x78

Jun 3 2019 19:03:10.052540641 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb356e0e38200c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d15e 0x321b4e1
eid = 0x79

Jun 3 2019 19:03:10.052540641 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb356e9046300c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d15e 0x321b4e1
eid = 0x7a

Jun 3 2019 19:03:10.064540724 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb3577885f401801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x3d8d034
eid = 0x7b

Jun 3 2019 19:03:10.064540724 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb3577df99f01801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x3d8d034
eid = 0x7c

Jun 3 2019 19:03:10.064540724 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb3578c504901801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d15e 0x3d8d034
eid = 0x7d

Jun 3 2019 19:03:10.064540724 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35792028701801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d15e 0x3d8d034
eid = 0x7e

Jun 3 2019 19:03:10.064540724 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb3579cfdcf01801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x3d8d034
eid = 0x7f

Jun 3 2019 19:03:10.064540724 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb357a2ae2001801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d15e 0x3d8d034
eid = 0x80

Jun 3 2019 19:03:10.116541083 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35abfb9ba01801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d15e 0x6f2469b
eid = 0x81

Jun 3 2019 19:03:10.116541083 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35ac8b8fe01801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d15e 0x6f2469b
eid = 0x82

Jun 3 2019 19:03:10.120541112 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35ad34ec301801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d15e 0x72f4fb8
eid = 0x83

Jun 3 2019 19:03:10.120541112 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35adc17e101801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d15e 0x72f4fb8
eid = 0x84

Jun 3 2019 19:03:10.120541112 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35ae5453801801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d15e 0x72f4fb8
eid = 0x85

Jun 3 2019 19:03:10.120541112 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35aed99a701801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d15e 0x72f4fb8
eid = 0x86

Jun 3 2019 19:03:10.120541112 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35af6a00501801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d15e 0x72f4fb8
eid = 0x87

Jun 3 2019 19:03:10.120541112 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35afcc5bc01801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d15e 0x72f4fb8
eid = 0x88

Jun 3 2019 19:03:10.120541112 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35b02ad4f01801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d15e 0x72f4fb8
eid = 0x89

Jun 3 2019 19:03:10.120541112 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb35b07658f01801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d15e 0x72f4fb8
eid = 0x8a

Jun 3 2019 19:03:11.144548193 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb397faaafd01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x75
time = 0x5cf5d15f 0x89da161
eid = 0x8b

Jun 3 2019 19:03:11.144548193 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb397ff731901001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x75
time = 0x5cf5d15f 0x89da161
eid = 0x8c

Jun 3 2019 19:03:11.144548193 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb3980f2e6701001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x73
time = 0x5cf5d15f 0x89da161
eid = 0x8d

Jun 3 2019 19:03:11.144548193 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xcb39815757e01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1cf
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x73
time = 0x5cf5d15f 0x89da161
eid = 0x8e

Jun 3 2019 19:10:36.735627327 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-retry@2019-06-03T07:32:00Z"
history_internal_str = ""
history_internal_name = "destroy"
history_dsid = 0x1ca
history_txg = 0x2263dbc
history_time = 0x5cf5d31c
time = 0x5cf5d31c 0x2bd8c83f
eid = 0x8f

Jun 3 2019 19:10:41.203658184 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-retry"
history_internal_str = ""
history_internal_name = "destroy"
history_dsid = 0x1cf
history_txg = 0x2263dbe
history_time = 0x5cf5d321
time = 0x5cf5d321 0xc2393c8
eid = 0x90

Jun 3 2019 19:11:15.955898183 sysevent.fs.zfs.config_sync
version = 0x0
class = "sysevent.fs.zfs.config_sync"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
time = 0x5cf5d343 0x38f9d947
eid = 0x91

Jun 3 2019 19:11:24.263955556 sysevent.fs.zfs.config_sync
version = 0x0
class = "sysevent.fs.zfs.config_sync"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
time = 0x5cf5d34c 0xfbba464
eid = 0x92

Jun 3 2019 19:18:04.798720771 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove"
history_internal_str = "mountpoint=/tmp/scotgrove"
history_internal_name = "set"
history_dsid = 0x1b7
history_txg = 0x2263e22
history_time = 0x5cf5d4dc
time = 0x5cf5d4dc 0x2f9b8303
eid = 0x93

Jun 3 2019 19:18:30.798900232 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89b776035a01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d4f6 0x2f9e4008
eid = 0x94

Jun 3 2019 19:18:30.802900261 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89b77ba61001401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d4f6 0x2fdb4925
eid = 0x95

Jun 3 2019 19:18:30.882900812 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89bc42008d00001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d4f6 0x349fff4c
eid = 0x96

Jun 3 2019 19:18:30.882900812 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89bc46d9a400001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d4f6 0x349fff4c
eid = 0x97

Jun 3 2019 19:18:30.926901116 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89bf11e7c601001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f6 0x373f637c
eid = 0x98

Jun 3 2019 19:18:30.930901143 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89bf1a7b5e01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f6 0x377c6c97
eid = 0x99

Jun 3 2019 19:18:30.946901254 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c02b222c01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d4f6 0x38709106
eid = 0x9a

Jun 3 2019 19:18:30.946901254 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c032c5f801401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d4f6 0x38709106
eid = 0x9b

Jun 3 2019 19:18:30.974901447 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c1d98d7a01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f6 0x3a1bd0c7
eid = 0x9c

Jun 3 2019 19:18:30.974901447 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c1e21d2f01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f6 0x3a1bd0c7
eid = 0x9d

Jun 3 2019 19:18:30.994901586 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c312569a01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f6 0x3b4cfe52
eid = 0x9e

Jun 3 2019 19:18:30.994901586 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c31af34c01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f6 0x3b4cfe52
eid = 0x9f

Jun 3 2019 19:18:31.014901724 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c42940b501401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f7 0xe361dc
eid = 0xa0

Jun 3 2019 19:18:31.014901724 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c4318cdf01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f7 0xe361dc
eid = 0xa1

Jun 3 2019 19:18:31.018901751 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c48e0b7301401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f7 0x1206af7
eid = 0xa2

Jun 3 2019 19:18:31.018901751 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c492c0ee01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f7 0x1206af7
eid = 0xa3

Jun 3 2019 19:18:31.062902054 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c70cae6d00801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d4f7 0x3bfcf26
eid = 0xa4

Jun 3 2019 19:18:31.062902054 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89c7119a1e00801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d4f7 0x3bfcf26
eid = 0xa5

Jun 3 2019 19:18:31.282903573 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89d44b301600801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f7 0x10dcc415
eid = 0xa6

Jun 3 2019 19:18:31.286903602 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89d451e11500801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f7 0x1119cd32
eid = 0xa7

Jun 3 2019 19:18:31.302903711 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89d563604f00801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d4f7 0x120df19f
eid = 0xa8

Jun 3 2019 19:18:31.302903711 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89d5682ebb00801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x12f
time = 0x5cf5d4f7 0x120df19f
eid = 0xa9

Jun 3 2019 19:18:31.318903822 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89d648d0d700801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f7 0x1302160e
eid = 0xaa

Jun 3 2019 19:18:31.318903822 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89d64db11f00801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x130
time = 0x5cf5d4f7 0x1302160e
eid = 0xab

Jun 3 2019 19:18:31.838907411 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89f56c3c8801001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d4f7 0x3200b613
eid = 0xac

Jun 3 2019 19:18:31.838907411 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89f5744e4601001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d4f7 0x3200b613
eid = 0xad

Jun 3 2019 19:18:31.842907439 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89f57d956d01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d4f7 0x323dbf2f
eid = 0xae

Jun 3 2019 19:18:31.842907439 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89f584d9c401001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d4f7 0x323dbf2f
eid = 0xaf

Jun 3 2019 19:18:31.842907439 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89f58ccc7201001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d4f7 0x323dbf2f
eid = 0xb0

Jun 3 2019 19:18:31.842907439 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89f59420c301001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d4f7 0x323dbf2f
eid = 0xb1

Jun 3 2019 19:18:31.842907439 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89f59bff5f01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d4f7 0x323dbf2f
eid = 0xb2

Jun 3 2019 19:18:31.842907439 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89f5a3613101001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d4f7 0x323dbf2f
eid = 0xb3

Jun 3 2019 19:18:31.842907439 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89f5ac1d4401001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d4f7 0x323dbf2f
eid = 0xb4

Jun 3 2019 19:18:31.846907467 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd89f5b4009e01001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x120
time = 0x5cf5d4f7 0x327ac84b
eid = 0xb5

Jun 3 2019 19:18:40.078964286 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd8be06f6a4101c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x75
time = 0x5cf5d500 0x4b4e63e
eid = 0xb6

Jun 3 2019 19:18:40.078964286 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd8be07a096d01c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x75
time = 0x5cf5d500 0x4b4e63e
eid = 0xb7

Jun 3 2019 19:18:40.094964396 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd8be173788e01c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x73
time = 0x5cf5d500 0x5a90aac
eid = 0xb8

Jun 3 2019 19:18:40.094964396 ereport.fs.zfs.authentication
class = "ereport.fs.zfs.authentication"
ena = 0xd8be17c2d7701c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x7af0b8b413816ff7
(end detector)
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_objset = 0x1b7
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x73
time = 0x5cf5d500 0x5a90aac
eid = 0xb9

Jun 3 2019 19:27:53.226712118 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror"
history_internal_str = "ul$1000 receive"
history_internal_name = "permission update"
history_dsid = 0x106
history_txg = 0x2263e95
history_time = 0x5cf5d729
time = 0x5cf5d729 0xd835a36
eid = 0xba

Jun 3 2019 19:27:53.226712118 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror"
history_internal_str = "ul$1000 create"
history_internal_name = "permission update"
history_dsid = 0x106
history_txg = 0x2263e95
history_time = 0x5cf5d729
time = 0x5cf5d729 0xd835a36
eid = 0xbb

Jun 3 2019 19:27:53.226712118 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror"
history_internal_str = "ul$1000 mount"
history_internal_name = "permission update"
history_dsid = 0x106
history_txg = 0x2263e95
history_time = 0x5cf5d729
time = 0x5cf5d729 0xd835a36
eid = 0xbc

Jun 3 2019 19:27:53.226712118 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror"
history_internal_str = "ul$1000 mountpoint"
history_internal_name = "permission update"
history_dsid = 0x106
history_txg = 0x2263e95
history_time = 0x5cf5d729
time = 0x5cf5d729 0xd835a36
eid = 0xbd

Jun 3 2019 19:27:53.226712118 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror"
history_internal_str = "ul$1000 readonly"
history_internal_name = "permission update"
history_dsid = 0x106
history_txg = 0x2263e95
history_time = 0x5cf5d729
time = 0x5cf5d729 0xd835a36
eid = 0xbe

Jun 3 2019 19:27:53.226712118 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror"
history_internal_str = "ud$1000 receive"
history_internal_name = "permission update"
history_dsid = 0x106
history_txg = 0x2263e95
history_time = 0x5cf5d729
time = 0x5cf5d729 0xd835a36
eid = 0xbf

Jun 3 2019 19:27:53.226712118 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror"
history_internal_str = "ud$1000 create"
history_internal_name = "permission update"
history_dsid = 0x106
history_txg = 0x2263e95
history_time = 0x5cf5d729
time = 0x5cf5d729 0xd835a36
eid = 0xc0

Jun 3 2019 19:27:53.226712118 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror"
history_internal_str = "ud$1000 mount"
history_internal_name = "permission update"
history_dsid = 0x106
history_txg = 0x2263e95
history_time = 0x5cf5d729
time = 0x5cf5d729 0xd835a36
eid = 0xc1

Jun 3 2019 19:27:53.226712118 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror"
history_internal_str = "ud$1000 mountpoint"
history_internal_name = "permission update"
history_dsid = 0x106
history_txg = 0x2263e95
history_time = 0x5cf5d729
time = 0x5cf5d729 0xd835a36
eid = 0xc2

Jun 3 2019 19:27:53.226712118 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror"
history_internal_str = "ud$1000 readonly"
history_internal_name = "permission update"
history_dsid = 0x106
history_txg = 0x2263e95
history_time = 0x5cf5d729
time = 0x5cf5d729 0xd835a36
eid = 0xc3

Jun 3 2019 19:33:41.200772945 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-repro"
history_internal_str = ""
history_internal_name = "receive"
history_dsid = 0x1da
history_txg = 0x2263ed9
history_time = 0x5cf5d885
time = 0x5cf5d885 0xbf78d51
eid = 0xc4

Jun 3 2019 19:33:41.556775138 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-repro"
history_internal_str = "mountpoint=none"
history_internal_name = "set"
history_dsid = 0x1da
history_txg = 0x2263eda
history_time = 0x5cf5d885
time = 0x5cf5d885 0x212fb6e2
eid = 0xc5

Jun 3 2019 19:33:41.556775138 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-repro"
history_internal_str = "readonly=1"
history_internal_name = "set"
history_dsid = 0x1da
history_txg = 0x2263eda
history_time = 0x5cf5d885
time = 0x5cf5d885 0x212fb6e2
eid = 0xc6

Jun 3 2019 19:33:41.832776837 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-repro"
history_internal_str = "keylocation=prompt"
history_internal_name = "set"
history_dsid = 0x1da
history_txg = 0x2263edb
history_time = 0x5cf5d885
time = 0x5cf5d885 0x31a32a85
eid = 0xc7

Jun 3 2019 19:39:41.211044975 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-repro"
history_internal_str = "snap=now"
history_internal_name = "finish receiving"
history_dsid = 0x1da
history_txg = 0x2263f22
history_time = 0x5cf5d9ed
time = 0x5cf5d9ed 0xc944a6f
eid = 0xc8

Jun 3 2019 19:39:41.211044975 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-repro@now"
history_internal_str = ""
history_internal_name = "snapshot"
history_dsid = 0x1dd
history_txg = 0x2263f22
history_time = 0x5cf5d9ed
time = 0x5cf5d9ed 0xc944a6f
eid = 0xc9

Jun 3 2019 19:40:08.363219822 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-repro"
history_internal_str = "mountpoint=/tmp/scotgrove-repro"
history_internal_name = "set"
history_dsid = 0x1da
history_txg = 0x2263f28
history_time = 0x5cf5da08
time = 0x5cf5da08 0x15a64b6e
eid = 0xca

Jun 3 2019 19:51:01.827504904 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-repro@now"
history_internal_str = ""
history_internal_name = "destroy"
history_dsid = 0x1dd
history_txg = 0x2263fa8
history_time = 0x5cf5dc95
time = 0x5cf5dc95 0x3152b908
eid = 0xcb

Jun 3 2019 19:51:02.383508590 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/mirror/scotgrove-repro"
history_internal_str = ""
history_internal_name = "destroy"
history_dsid = 0x1da
history_txg = 0x2263faa
history_time = 0x5cf5dc96
time = 0x5cf5dc96 0x16dbe06e
eid = 0xcc

Jun 3 2019 21:15:52.559915879 sysevent.fs.zfs.config_sync
version = 0x0
class = "sysevent.fs.zfs.config_sync"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
time = 0x5cf5f078 0x215fa367
eid = 0xcd

Jun 3 2019 21:15:53.535717716 sysevent.fs.zfs.config_sync
version = 0x0
class = "sysevent.fs.zfs.config_sync"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
time = 0x5cf5f079 0x1fee6754
eid = 0xce

Jun 3 2019 22:45:01.391635204 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid"
history_internal_str = "readonly=0"
history_internal_name = "set"
history_dsid = 0x15
history_txg = 0x22647a7
history_time = 0x5cf6055d
time = 0x5cf6055d 0x1757e104
eid = 0xcf

Jun 3 2019 22:45:01.655639690 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/archive"
history_internal_str = "readonly=0"
history_internal_name = "set"
history_dsid = 0x59
history_txg = 0x22647a8
history_time = 0x5cf6055d
time = 0x5cf6055d 0x2714448a
eid = 0xd0

Jun 3 2019 22:45:10.107782887 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid"
history_internal_str = "readonly=1"
history_internal_name = "set"
history_dsid = 0x15
history_txg = 0x22647aa
history_time = 0x5cf60566
time = 0x5cf60566 0x66ca2e7
eid = 0xd1

Jun 3 2019 22:45:10.399787820 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "zfs-raid"
pool_guid = 0x7af0b8b413816ff7
pool_state = 0x0
pool_context = 0x0
history_hostname = "prism"
history_dsname = "zfs-raid/archive"
history_internal_str = "readonly=1"
history_internal_name = "set"
history_dsid = 0x59
history_txg = 0x22647ab
history_time = 0x5cf60566
time = 0x5cf60566 0x17d4472c
eid = 0xd2

I didn't bother mangling my dataset names like I did before. The pool with issues is called zfs-raid, the dataset with IO errors is called zfs-raid/mirror/scotgrove.

@behlendorf behlendorf added this to To do in 0.8-release Jun 5, 2019
@tcaputi
Copy link
Contributor

tcaputi commented Jun 5, 2019

@dsnet can you try adding this patch? #8857 If that doesn't fix things I will probably need to give you a few more debugging instructions. You will need to redo the receive.

@dsnet
Copy link
Author

dsnet commented Jun 6, 2019

I confirm that the patch removes the IO errors. I'm hashing the contents of everything to make sure it checks out.

EDIT: Hashing all contents reports that the received snapshot matches the source.

@dsnet
Copy link
Author

dsnet commented Jun 6, 2019

Thank you for the quick fix! Once I update to a hypothetical 0.8.1, how would I clear the errors that zpool status -v is reported? Do I have to rebuild the pool from scratch?

@behlendorf
Copy link
Contributor

@dsnet you'll need to run zpool clear on the destination pool, and you'll need to resend the datasets from the source to the destination. I'm working on putting together an 0.8.1 with the fix.

behlendorf pushed a commit that referenced this issue Jun 7, 2019
This patch re-adds a check that was removed in 369aa50. The check
confirms that a raw receive is not occuring before truncating an
object's dn_maxblkid. At the time, it was believed that all cases
that would hit this code path would be handled in other places,
but that was not the case.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8852 
Closes #8857
@behlendorf behlendorf removed this from To do in 0.8-release Jun 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Encryption "native encryption" feature Component: Send/Recv "zfs send/recv" feature Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Development

No branches or pull requests

4 participants
@behlendorf @tcaputi @dsnet and others