Compressing parts folder of pool

[fbruetting]

v1.0.0

The official pool file count is extremely large (17.326 objects, with 15.734 residing in parts which only are 14,3 MB). This lengthens copy/delete/move/backup actions pretty much (they say some people still don’t have SSDs today). Can you thus compress the parts folder in e.g. a TXZ/TGZ/TAR archive?

[atoav]

Just as an aside, there also has been discussion regarding size in the issue 131 of the horizon-pool. The work branch of my own fork of the pool has 31710 objects, which is a glimpse of the future.. So looking at ways of deal with this more efficiently are certainly something that might become important sooner or later.

[carrotIndustries]

Just compressing the files isn't feasible since we want them to be plain text/json for convenient version control. Having a file for each item is a feature.

[fbruetting]

Maybe for developers who work on those files, but this creates a LOT of overhead for every single user out there. Maybe these can be compressed by default, but with a “developer mode” switch made available as regular files?

If users modify or add some parts, these could be excluded/diffed from (or ignored in) the compressed archive and be available as plain text files next to that archive. Would that be a feasible solution? Then users would see just the modified and self-created parts in their pool folder in plain text, which makes own parts pretty obvious – and everything they don’t care about is out of the way in an archive (which still is pretty easy accessible). 😊 (Modified parts then either would be removed from the archive or better: be logically overwritten by plain-text files.)

Some file managers also are able to open archives like regular folders, not sure which programs could do this with which formats though…

[atoav]

I think pretty much any file manager nowadays can browse the contents of a zip file, but I imagine there are other issues as well. Pool updates (and if you create parts and contribute them to the pool) are managed via git. And I am not sure if git repositories can work inside a compressed file..?

[fbruetting]

Yeah, I meant that the parts you actively work on could be “checked out” – i.e. put as plaintext files in a folder like currently done. But all the other files (so usually 99%) could stay in such an archive.

Alternatively mounts could maybe be used. Those mounts are easily accessible, but on the disk they reside as just one big chunk.

[katyo]

Because git uses zlib to compress data, as an option pool manager can extract data directly from bare git repo using libgit2. In my opinion this is quite easy to implement.