dist-git: provide a way to request cleanup on per-package basis (front-end config)

[fedora-copr-github-bot]

Original issue: https://pagure.io/copr/copr/issue/276
Opened: 2018-03-23 06:34:29
Opened by: praiskup

Even though I'm sure we should protect the dist-git feature in copr --
there are coprs like https://copr.fedorainfracloud.org/coprs/praiskup/distgen-ci/ where we can expect a lot of different tarballs imported into dist-git (multiple builds for each upstream pull request) - and nobody will ever look at those tarballs (nor we can expect resubmit, etc.).

Generally, we should have an option (can by default on) to request dist-git lookaside cache cleanup, thus keep only the git changes persistent (so the git history for spec files (those are often generated) is persistently presented in cgit).

I know that implementing https://bugzilla.redhat.com/show_bug.cgi?id=1427431 wouldn't be trivial, and that it is not easy to detect that some file (candidate for removal) is not referenced by some other package in distgit (and thus remove anything). So the proposal to make it easy to implement would be:

web-ui frontend checkbox for Package:
[ ] persistent storage for dist-git lookaside cache

dist-git should have two lookaside cache paths, the second one where ..

• we can safely garbage collected files (e.g. all files older than 24 hours)
• we don't have to serve the re-submit functionality
• each build has it's own directory in lookaside storage, so that multiple builds do not share any imported file

---

clime commented at 2018-03-23 07:50:38:

Even though I'm sure we should protect the dist-git feature in copr --
there are coprs like https://copr.fedorainfracloud.org/coprs/praiskup/distgen-ci/ where we can expect > a lot of different tarballs imported into dist-git (multiple builds for each upstream pull request) - and > nobody will ever look at those tarballs (nor we can expect resubmit, etc.).

We won't be ultimately importing those dynamically built tarballs into copr-dist-git. Instead they will be just stored in srpms on backend for some time and then probably automatically cleaned up by prunerepo. We might make it possible to enable storing them somewhere else by some optional hook in some deployments.

---

clime commented at 2018-03-23 07:52:25:

I think there already was more than enough discussion about this, so closing. Feel free to reopen.

---

praiskup commented at 2018-03-23 08:11:37:

We won't be ultimately importing those dynamically built tarballs into copr-dist-git.

It's sometimes useful, and you are too definitive here.

I think there already was more than enough discussion about this,

Obviously not.

Feel free to reopen.

Taking the liberty.

---

praiskup commented at 2018-03-23 08:22:40:

Obviously not.

Btw., nobody did the research about "how the dist git is useful" in copr in reality, and
whether automatic imports can co-exist with "writeable dist-git" proposal. I know @clime you claimed you'll do the research before even thinking about changing how copr dist-git works (but you consistently keep definitively saying how "we" will change this..; you're not the only one here, really).

I'm ok to setup some survey, should I?

---

clime commented at 2018-03-23 08:45:46:

Obviously not.

Btw., nobody did the research about "how the dist git is useful" in copr in reality, and
whether automatic imports can co-exist with "writeable dist-git" proposal. I know @clime you claimed you'll do the research before even thinking about changing how copr dist-git works (but you consistently keep definitively saying how "we" will change this..; you're not the only one here, really).
I'm ok to setup some survey, should I?

well, sure, you can. But I would wait until we actually get the point where we need to implement this. Right now, we have an active survey about Github<->COPR CI so I am not sure if we should put up more of them right now (we can pace it a bit).

[praiskup]

Triage mtg: The problem now is reversed, we have automatic cleanup for every project. And opt-out would require a customer.