Add Zstd/LZ4/(GZIP write) support for region files

[egg82]

Using zstd-jni: https://github.com/luben/zstd-jni
There's a new option in PaperConfig for this which defaults to false.
Since the patch hooks Mojang's versioning system for new compression methods, it
should be internally consistent, easily-updatable, and reversible with a force-upgrade.

This will likely conflict with plugins and programs expecting region files to be compressed using the current standard.

[egg82]

LZ4 PR here: #2376
Note that Zstd is optimized for compression, where LZ4 is optimized for decompression.

[hugmanrique]

Related #416

[PaulBGD]

As a user/developer, I don't like this. This option generates worlds that look like regular Minecraft worlds, but can never be loaded in vanilla without a conversion program. I feel like a major change to the filetype should result in a different filetype, even if that means just changing the extension of the files.

[wizjany]

To the contrary, the file format specifies a compression type. The fact that the vanilla implementation doesn't recognize this new type doesn't mean the file is not-to-spec. Given that this isn't meant to be a default, I don't think admins using this option will expect it to load in vanilla.

[VADemon]

@PaulBGD I wanted to complain that a standalone tool is missing, this is the downside. But you can repack the whole world using Paper again as egg82 said:

reversible with a --force-upgrade

[astei]

Personally, I see no reason not to include this. If the user decides to enable zstandard or LZ4 compression, they must understand that their setup will be non-standard and tools that manipulate the world data directly will not work.

[egg82]

Stats for the different algorithms:
Zlib: https://docs.google.com/spreadsheets/d/1rf_srqWDvEfBDR9OmcFofkcuqdqV5ypCZb8ImQ4L1FE/edit?usp=sharing
LZ4: https://docs.google.com/spreadsheets/d/1iNcqo07ySM-r2-1DTBOmd5vl809u97AmacdxSOMiPCQ/edit?usp=sharing
Zstd: https://docs.google.com/spreadsheets/d/1mGqXu2fAgQXiFgiQdE830TJayZwWbyz2yqYT4T4KRfo/edit?usp=sharing

Test bench used: https://github.com/egg82/CompressionTests
Training/test data used: https://drive.google.com/open?id=1egzZKkV305LX7jt2Xxe0Y-DfYkkozDa_

[ghost]

Thank you for making this default to false.

[egg82]

Thank you for making this default to false.

There's no way anyone here would want the default to be something other than what Mojang uses. It's far too risky with external tools and plugins.

[LoneDev6]

Do you think it will conflict with https://www.spigotmc.org/resources/regionerator.12219/ ?

[egg82]

Do you think it will conflict with https://www.spigotmc.org/resources/regionerator.12219/ ?

Almost certainly, yes. Plugins that read files directly (such as Regionerator, Dynmap, etc) won't be able to read the chunk. Since the patch uses the standard file format to determine what compression algorithm is used, some will (hopefully) fail silently. If not, contact the author. Plugins will have access to the Zstd dictionary in Paper, and the patch is MIT.