Potentially move away from git LFS

[maartenweyns]

Judging by the CH GitHub billing and quota tab, this website is growing to reach the limits of what we're allowed to use (at least bandwidth-wise):

So maybe git LFS is not a long-term solution to store the archives. We could move to a google cloud bucket? Or maybe even a persistent volume in the cluster. Something to look at!

[verwoerd]

This bandwidth problem is caused by the builds AFAIK. Every PR and following commits in the PR and pushes on main trigger a build. We already tried to tackle this by using the nschloe/action-cached-lfs-checkout@v1, but there seem to be an issue when any LFS data is added, the cache is invalidated. We could attempt a direct solution as proposed here.

I think a k8s PV is in my experience not easily accessible (only through kubectl and their derivatives afaik). No experience with Google cloud buckets, but i fear the access for uploading and link generation can be tricky to maintain long term. It is desirable that the new entries in the archive are published within a hour after the results are known (and there is no embargo).

[maartenweyns]

No experience with Google cloud buckets, but i fear the access for uploading and link generation can be tricky to maintain long term.

We could potentially make a public bucket in the CHipCie Google Cloud project, so that CHipCie members can add files to this bucket and get URLs. I understand that LFS is easier, but having version control and performing workflows over the entire CHipCie archive maybe seems a bit overkill?

[praseodym]

I've added another Git LFS data pack because we nearly ran out of bandwidth and have a 100% off coupon on our account anyway: