New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zstd to tiled format and compression ratio #1886

Open
alphaonex86 opened this Issue Feb 19, 2018 · 14 comments

Comments

Projects
None yet
3 participants
@alphaonex86

alphaonex86 commented Feb 19, 2018

Hi, can you add at Tile Layer format the entry Base64 (Zstandard compression) and the level of compression?

  • It very more fast on compression and decompression for same ratio than Zlib
  • Very more fast at decompression for higher ratio than Zlib
  • Compression similar to Lzma, Xz then very better than zlib at higher ratio
  • Add option to choose the compression level
  • Default to this new entry

I can do the code or donation for this

Cheers, it's a great software.

@bjorn

This comment has been minimized.

Owner

bjorn commented Feb 20, 2018

It definitely looks like zstd would be interesting to support, however did you actually measure the map loading time and is the decompression really a significant part of it?

And if loading times are such a concern, did you consider writing a specialized binary format to minimize processing time? Or would it alternatively be interesting to enable the chunked layer data format for non-infinite maps? This would significantly reduce the data that needs to be processed of mostly empty tile layers.

I can do the code or donation for this

You're welcome to work on this and/or to make a donation! :-)

@alphaonex86

This comment has been minimized.

alphaonex86 commented Feb 20, 2018

Hi,
On 2 map set (1972 maps and 171 maps), the decompression speed is 3-6x more faster with zstd.
Maybe a specialized binary format to minimize processing time can be interesting, but for now I prefer the compression.
I will make donation. I don't known if I have time to work on this.

@alphaonex86

This comment has been minimized.

alphaonex86 commented Feb 20, 2018

PS: I think it's more logic split map into small chunk than just the layer data. That's split all the overhead to parse the map format too
EDIT: And the compression ratio is -40% on average map at best compression ratio, without impact on speed decompression

@bjorn

This comment has been minimized.

Owner

bjorn commented Feb 20, 2018

PS: I think it's more logic split map into small chunk than just the layer data. That's split all the overhead to parse the map format too

It's not just splitting the layer data in chunks. It's also leaving out empty chunks entirely. That makes a big difference if you have maps with tile layers that are mostly empty, which is not uncommon.

@alphaonex86

This comment has been minimized.

alphaonex86 commented Feb 20, 2018

@bjorn bjorn added the feature label Feb 20, 2018

@alphaonex86

This comment has been minimized.

alphaonex86 commented Feb 20, 2018

Zstandard done, I failed to compression level option, can you help me?

@bjorn

This comment has been minimized.

Owner

bjorn commented Feb 20, 2018

Zstandard done, I failed to compression level option, can you help me?

Please open a pull request, then I can help with review and additional changes.

@alphaonex86

This comment has been minimized.

alphaonex86 commented Feb 20, 2018

Done #1888

@Seanba

This comment has been minimized.

Contributor

Seanba commented Feb 20, 2018

Could we put off making this the default option for now? People building tech off of Tiled may need some time to support this.

@alphaonex86

This comment has been minimized.

alphaonex86 commented Feb 20, 2018

Here I have put default arguments to transparent usage. I think is good the new map is directly created with new format.

@bjorn

This comment has been minimized.

Owner

bjorn commented Feb 20, 2018

For sure the default will stay on Zlib, no worries @Seanba! We can't have the default require people to rely on custom extraction code, and zlib is still the most commonly available format in frameworks and standard libraries.

@alphaonex86 The Zstandard format doesn't need to be the default. Once somebody switches to it, Tiled will remember that choice so you can use it without needing to choose it again for each new map.

@alphaonex86

This comment has been minimized.

alphaonex86 commented May 30, 2018

Part of my mapper community is asking to get zstd into .tmx format for mi game.

When it will be into official code and binary?

@bjorn

This comment has been minimized.

Owner

bjorn commented May 30, 2018

@alphaonex86 I'll have another look at your PR #1888 tomorrow.

@alphaonex86

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment