Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OS-8136 Add DISCARD/TRIM support to bhyve #268

Merged
merged 4 commits into from
Mar 18, 2020
Merged

Conversation

jasonbking
Copy link

@jasonbking jasonbking commented Mar 17, 2020

This is the first of two PRs to implement the DISCARD/TRIM functionality. This one is largely Allan Jude's original work that we're picking up. I made a few tweaks to his code based on the last bits of FreeBSD phrabicator feedback, as well as added a 'nodelete' option for blockif to be able to explicitly disable the feature for a device.

A second PR w/ the necessary blockif changes for illumos as well as the needed bhyve zone updates (to wire up the 'nodelete' feature), will follow this PR.

@sjorge
Copy link

sjorge commented Mar 17, 2020

This will allow freeing of space on the zvol's backing the bhyve disks right?

Edit: If they are virtio-blk

I'll see if I can pull this in on a local PI somewhere today, with bhyve_bar, bhyve_fbuf and this patch, building now so should take a few hours.

@jasonbking
Copy link
Author

You will need the second PR I mentioned to get the functionality working on illumos. It's separate since this change is almost all Allan's work (so easier to preserve proper attribution), and would be what is upstreamed, while the blockif and bhyve zone changes (which is what will be in the second PR) are all illumos specific.

@jasonbking
Copy link
Author

All of this work combined is on the 'bhyve-unmap' branch if you want to see/try that.

usr/src/cmd/bhyve/pci_virtio_block.c Outdated Show resolved Hide resolved
usr/src/cmd/bhyve/pci_virtio_block.c Outdated Show resolved Hide resolved
usr/src/cmd/bhyve/pci_virtio_block.c Show resolved Hide resolved
usr/src/cmd/bhyve/pci_virtio_block.c Outdated Show resolved Hide resolved
usr/src/cmd/bhyve/pci_virtio_block.c Outdated Show resolved Hide resolved
usr/src/cmd/bhyve/pci_virtio_block.c Outdated Show resolved Hide resolved
usr/src/cmd/bhyve/block_if.c Show resolved Hide resolved
@papertigers papertigers self-requested a review March 17, 2020 16:30
usr/src/cmd/bhyve/pci_virtio_block.c Show resolved Hide resolved
usr/src/cmd/bhyve/pci_virtio_block.c Show resolved Hide resolved
* An arbitrary limit to prevent excessive latency due to large
* delete requests.
*/
#define VTBLK_MAX_DISCARD_SECT ((16 << 20) / VTBLK_BSIZE) /* 16 MiB */

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know how Allan ever arrived at 16MiB? I know the comment says arbitrary, but might be something we want to experiment with down the road.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No -- he never responded on that. You have to advertise some limit, so it's as good as any I suppose. IIRC, one of the reasons zfs trim support took so long was issuing too many/too large TRIM requests would cause pretty noticeable drops in performance while it was freeing the space, which is why a more elaborate mechanism was required, so it does seem prudent to start out relatively conservatively. No guest should be making assumptions on the sizes advertised by the host, so we should be able to change this value as needed, or maybe even turn it into a tunable (though if we decided to do that, I'd rather do it as a separate change).

usr/src/cmd/bhyve/pci_virtio_block.c Show resolved Hide resolved
usr/src/cmd/bhyve/pci_virtio_block.c Show resolved Hide resolved
usr/src/cmd/bhyve/pci_virtio_block.c Show resolved Hide resolved
@sjorge
Copy link

sjorge commented Mar 18, 2020

I did some testing with this (well the whole patch set for illumos) The only VM I got working was Windows 10...

[root@boron ~]# zfs list -t all -r -o space zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b
NAME                                                           AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b                     9.99G  66.9G         0   5.75M            10G      56.9G
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b@vmsnap-drvup            -      0         -       -              -          -
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b/disk0               38.9G  56.9G      100M   25.9G          30.8G          0
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b/disk0@vmsnap-drvup      -   100M         -       -              -          -
[root@boron ~]# zfs list -t all -r -o space zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b
NAME                                                           AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b                     9.99G  66.9G      200K   5.88M          10.0G      56.9G
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b@vmsnap-drvup            -   200K         -       -              -          -
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b/disk0               37.2G  56.9G     11.7G   16.0G          29.2G          0
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b/disk0@vmsnap-drvup      -  11.7G         -       -              -          -

You an see the space after creating a post driver update snapshot, then again after trimming.
Windows claimed 10G trimmed, which seems to match with the snapshot size growth!

scrshot

Note I had to use latest fedora drivers, the stable drivers or the recommended (very old now) joyent drivers are too old.

https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/virtio-win.iso

From the changelog

* Wed Sep 04 2019 Cole Robinson <crobinso@redhat.com> - 0.1.172-1
- Update to virtio-win-prewhql-0.1-172
- netkvm: Increase timeout of control command completion
- viostor: add preliminary support for discard (unmap) command
- vioinput: Fix mistakes in commit 4c9942754912
- netkvm: RSC support according to virtio spec

@sjorge
Copy link

sjorge commented Mar 18, 2020

As per IRC the output of dtrace

[root@boron ~]# dtrace -n 'pid$target::ioctl:entry /arg1==1074/ {}' -p 8294
dtrace: description 'pid$target::ioctl:entry ' matched 2 probes
CPU     ID                    FUNCTION:NAME
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  7  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 14  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  3  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 11  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 11  88919                      ioctl:entry
 11  88919                      ioctl:entry
  2  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
  0  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
  7  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  9  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  9  88919                      ioctl:entry
  9  88919                      ioctl:entry
  9  88919                      ioctl:entry
  9  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
  9  88919                      ioctl:entry
  9  88919                      ioctl:entry
  9  88919                      ioctl:entry
  9  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
  3  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
 15  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
 14  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  3  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
  2  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  0  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  0  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  0  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
  4  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 11  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 10  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  4  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
  3  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 14  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
 12  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
  8  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  7  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  1  88919                      ioctl:entry
  9  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
  5  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
 13  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  1  88919                      ioctl:entry
  9  88919                      ioctl:entry
  9  88919                      ioctl:entry
  9  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
 10  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  6  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
  8  88919                      ioctl:entry
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
                                                                                                                        Try the new cross-platform PowerShell https://aka.ms/pscore6                                                                                                                                                                                    PS C:\WINDOWS\system32> Optimize-Volume -DriveLetter C -ReTrim -Verbose                                                 VERBOSE: Invoking retrim on (C:)...                                                                                     VERBOSE: Performing pass 1:                                                                                             VERBOSE: Retrim:  0% complete...                                                                                        VERBOSE: Retrim:  83% complete...                                                                                       VERBOSE: Retrim:  86% complete...                                                                                       VERBOSE: Retrim:  100% complete.
VERBOSE:
Post Defragmentation Report:
VERBOSE:
 Volume Information:
VERBOSE:   Volume size                 = 28.95 GB
VERBOSE:   Cluster size                = 4 KB
VERBOSE:   Used space                  = 15.58 GB
VERBOSE:   Free space                  = 13.36 GB
VERBOSE:
 Retrim:
VERBOSE:   Backed allocations          = 28
VERBOSE:   Allocations trimmed         = 793
VERBOSE:   Total space trimmed         = 11.83 GB
PS C:\WINDOWS\system32>
[root@boron ~]# zfs list -t all -r -o space zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b
NAME                                                             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b                       9.99G  66.3G         0   5.89M            10G      56.3G
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b@vmsnap-pretrim            -      0         -       -              -          -
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b/disk0                 39.6G  56.3G     12.6M   25.3G          30.9G          0
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b/disk0@vmsnap-pretrim      -  12.6M         -       -              -          -
[root@boron ~]# zfs list -t all -r -o space zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b
NAME                                                             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b                       9.99G  66.3G         0   5.89M            10G      56.3G
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b@vmsnap-pretrim            -      0         -       -              -          -
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b/disk0                 39.5G  56.3G     11.2G   14.2G          30.9G          0
zones/eceec5eb-22c7-e65b-fd6b-81ebdb36406b/disk0@vmsnap-pretrim      -  11.2G         -       -              -          -

@jasonbking
Copy link
Author

Note that @sjorge tested with the second part of this PR (which includes the illumos-specific bits) -- unfortunately unlike Gerrit, with the repo config, you can't do a series of dependent changes (e.g. 'review+1' depends on changes in 'review') to review at the same time and then have them separate commits (at least with how illumos-joyent appears to be configured), so the illumos bits will follow after this one is integrated.

@mgerdts
Copy link

mgerdts commented Mar 18, 2020

Note that @sjorge tested with the second part of this PR (which includes the illumos-specific bits) -- unfortunately unlike Gerrit, with the repo config, you can't do a series of dependent changes (e.g. 'review+1' depends on changes in 'review') to review at the same time and then have them separate commits (at least with how illumos-joyent appears to be configured), so the illumos bits will follow after this one is integrated.

Can you point to a branch where the follow-on work is at?

@jasonbking
Copy link
Author

jasonbking commented Mar 18, 2020

The bhyve-unmap branch https://github.com/joyent/illumos-joyent/tree/bhyve-unmap

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants