LFS ?

[philn]

This repo is gigantic... Has anything been investigated to make it easier to clone (yes i know --depth 1)? Would LFS be useful?

[philn]

OK I give up, I wanted to submit a PR fixing the frecuency typo in the wktesthunter script, line 162, but cloning this repo puts my laptop on its knees.

[clopez]

You need at least ~40GB of free disk space to use this.
The problem is not because of git, but because of the checked out files in disk with the results: there are thousands of them.

$ du -hs .git jsonresults/
2.4G	.git
37G	jsonresults/

So git itself does store the json files very efficiently, but not the filesystem.

In that regard, i don't think LFS will be helpful here

I guess that the solution would be to use a database instead of the filesystem to store the data.

[philn]

Do you really need all these files in git?

Maybe hosting them on an HTTP server or similar could be explored.

[clopez]

There is clearly room for optimization here.

Maybe we could explore uploading all this files to a remote server and offering a HTTP API endpoint to do queries to them, but then I'm a bit worried about performance. I think that would likely make the experience of using this slower because then you have to wait for the remote service to process all the data for your query and reply. Notice that this program spends lot of CPU time searching for the test data on those files, if that computation is offloaded to a remote server then the problem is worse if several people is using it at the same time or the server has a slower/busy CPU.

Another possible idea is to keep storing the files locally but store them compressed in tarballs which can be decompressed on the fly with python to search for the data. The problem then is that such compressed format (binary) doesn't play well with git, so perhaps that compressed files would have to be moved to LFS.