zstdmt --adapt is MUCH SLOWER (for me) than pzstd

[xrobau]

I noticed that the default --compress option is now zstdmt, so I thought I would give that a try.

Connectivity is store1 <--> store2 via bonded 10gb, 9100 byte MTU.

Using zstdmt --adapt, the sender was using 16 cores and zfs recv was reporting 20-30MB/s transfers.

Using pzstd, transfer speeds are reported as 300-400MB/s, with 200-300% cpu usage.

This may be just because the data I'm transferring is not that compressible (vmware vmdks), but I think that an order of magnitude difference is pretty significant, and --adapt is MEANT to handle that sensibly.

OS is Ubuntu 21.04, version reported is

*** zstd command line interface 64-bits v1.4.8, by Yann Collet ***

[psy0rz]

Can someone else confirm/try this? @bagbag ?

[bagbag]

After some research I still don't get what the difference between pzstd and zstdmt (same as zstd -T0) is. Do you know that @xrobau?

But yes, I can confirm this issue. And GitHub knows about it too: facebook/zstd#2200.

--adapt starts fast and then slows down pretty fast. pzstd and zstdmt performs the same. Sometimes A is faster, sometimes B (I guess because I have some other load in the background).

But it should still be better for lower bandwith (like 1 Gbit/s) links, as it is limiting factor in this case. I think we should simply add a warning/hint to not use zstd-adapt for very fast links to --help and leave zstd-adapt the default, as I guess that the majority don't have 10 Gbit/s links.

$ cat /dev/zero | pv | pzstd > /dev/null
^C,1GiB 0:00:05 [4,10GiB/s] [    <=>                                           ]

$ cat /dev/zero | pv | zstdmt > /dev/null
^C,5GiB 0:00:08 [3,65GiB/s] [       <=>                                        ]

$ cat /dev/zero | pv | zstdmt --adapt > /dev/null
^C58GiB 0:00:10 [ 704MiB/s] [         <=>                                      ]

[psy0rz]

But @xrobau reported only 20-30MB/s with 16 cores, which seems way too low. Maybe he can also do the same test you did on his machine.

Also testing compression with zeros isnt a very realistic test. Maybe create a binary file with /dev/urandom output and test with that. (/dev/urandom is too slow to use realtime)

[bagbag]

Fair point. I tested that again with a 8 GB HTML dump from Wikipedia. --adapt was fast for the first few hundred MBs and then started slowing down a lot. So my assumption about slower links was kind of wrong. I never noticed such a slow speed when I used --adapt (which was not that often).

Well. You should set the default back to zstd-fast (but keep zstd-adapt for future improvements in zstd).

$ pv wikipedia-de-html.tar.small | pzstd > /dev/null
7,81GiB 0:00:08 [ 980MiB/s] [===================================================] 100%            

$ pv wikipedia-de-html.tar.small | zstdmt > /dev/null
7,81GiB 0:00:07 [1001MiB/s] [===================================================] 100%            

$ pv wikipedia-de-html.tar.small | zstdmt --adapt > /dev/null
^C1,63GiB 0:00:37 [7,60MiB/s] [================>                                ] 20%                       

[psy0rz]

Yes i'll change it back then.