Add a method to get the cluster size of a ZIM file

[kelson42]

zimdiff/zimpatch must be able to rebuild exactly the same file like the original one. Do achieve this, they need to know what was the cluster size used by the zimwriter. The zimlib should implement a solution to deliver this information.

Moved from https://phabricator.wikimedia.org/T54082

[kelson42]

@mgautierfr Might be interesting as your are polishing zimmerge/zimpatch Kiran's code

[kelson42]

@mgautierfr The problem here is that this is not written in the ZIM. Could we add it as a new header?

[mgautierfr]

No :)

It would be a change (potentially incompatible) in the zim format.
And cluster is a concept internal to the zim format (and the libzim). A user code as no access to the cluster. There is no api to access it.
So, knowing its size is useless.

[kelson42]

Yes, I ask for a new API. Considering that the Header can be extended without breaking the format, a default value shoudl just set.

[mgautierfr]

There is no way to access cluster outside of libzim. And I want it to stay that way. (The same with dirent) Internal things must stay internal.

What is your use case ? How this would help you ?

[kelson42]

@mgautierfr zimrecreate and zimpatch/zimdiff

[mgautierfr]

How they could use it ?
zimrecreate do not need it at all.

[kelson42]

These tools should should be able to know what is the ZIM cluster size to be able to then recreate one with the same clustersize.

[mgautierfr]

If you provide the same content and compress the same way, the cluster should have the same size.

[kelson42]

@mgautierfr How do I guaranty "the same way" if I ignore the cluster size? Considering the cluster is a variable in the compression process.

[mgautierfr]

I cannot tell you how to avoid to use the cluster size if you don't tell me how you would use it.

---

Honestly, I don't think is is possible to zimpatch to guaranty this "same way" using libzim. I've already change the way zim's creator decides (and when) to close a cluster and this is totally internal. There is no way to force a specific set of articles to go in the same cluster.
We have one naive heuristic to determine which articles go in which cluster but this may change in the future. We may want to regroup articles per mimetype (because compressing content of the same type may be more efficient), or try different order of the article and take the best compression ratio, or regroup css and js together because there is a lot of chance that they are shared by all articles, or....

We cannot guaranty that articles will be written in the same clusters the same way the x years old implementation was doing it.
And I don't want to guaranty that.

[mgautierfr]

We don't have the information about a cluster size (except by decompressing it and see how much have been read).
libzim, even internally, never try to get the size of a cluster. So we will not provide a public API to get it.