Make osmflat more compact (especially when compressed)

[VeaaC]

• Store granularity explicitly instead of pre-multiplied numbers
• Move Ids to separate optional sub-archive
• Reduce bits needed for coordinates from 40 to 32
• Remove unused header information

Comparison:

Comparing:

• Compressed PBF (internal zlib compression)
• Unpacked osmflat with and without optional Ids (only new version has them optional)
• Compression with pzstd level 3
• Compression with shuffly ( https://github.com/VeaaC/shuffly ) + pzstd level 3

Before:

Dataset	PBF (zlib)	osmflat w Ids	zstd + osmflat w Ids	shuffly + zstd + osmflat w Ids
Berlin	70M	227M	94M	58M
Europe	26G	89G	41G	23G
Planet	66G	223G	102G	57G

After:

Dataset	PBF (zlib)	osmflat w Ids	osmflat w/o Ids	zstd + osmflat w Ids	zstd + osmflat w/o Ids	shuffly + zstd + osmflat w Ids	shuffly + zstd + osmflat w/o Ids
Berlin	70M	214M	176M	83M	69M	53M	49M
Europe	26G	84G	68G	36G	30G	20G	19G
Planet	66G	208G	164G	97G	76G	48G	47G

Observations:

• More compact in all scenarios.
• Using shuffly is still worth it (Ids have a compression ratio of > factor 20), but a bit less so due to rearranged data, and granularity.
• Shuffly compressed version is smaller than compressed PBF (with and without Ids)
• Most people might want to use a version without ids since it saves disk space
• Ids are almost for free if compressed with shuffly (due to data being sorted by id)

[hallahan]

Such a big difference! I wonder how much better it would be if you reduce integer sizes to 38 bytes instead of 40?

[VeaaC]

Not much / anything at all: Most structures would only be a few bit smaller, and flatdata rounds up to the next byte. It would only help if a structure had 4 references, each saving 2 bits.

[hallahan]

How do you do your size benchmarking? Is there a script somewhere?

[boxdot]

How do you do your size benchmarking? Is there a script somewhere?

+1 for documenting the command lines which produced the above numbers