Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to handle "leftover" projects that built in rolling chroots #2933

Closed
praiskup opened this issue Sep 27, 2023 · 7 comments · Fixed by #3265
Closed

How to handle "leftover" projects that built in rolling chroots #2933

praiskup opened this issue Sep 27, 2023 · 7 comments · Fixed by #3265
Assignees
Labels

Comments

@praiskup
Copy link
Member

The problem with fedora-rawhide-* chroots is that we never make them EOL. So far there's no automatic removal other storage consumption optimization other than prunerepo that keeps at most one version of each package in one repository.

This is not enough; some projects doing mass rebuilds are kept lingering around and consume a nontrivial amount of storage. For example some projects in this 140GB namespace inactive for 2+ years. Some of those projects seem to be created for mass rebuilds, but the user is no longer active in Fedora for us to ask. Well actually asking is a bit sub-optimal for us, it would be better to have some automation.. but dunno.

I'm not inundated with ideas, but could we e.g. take a look at RPMs in the directory, check the "buildtime" baked into RPMs, and if all the RPMs are older than say 2 years mark the project "staled" in DB? I'm not trying to propose a policy for removals right now -> it is a hard decision to remove something when it still might be used by someone. But for us administrators, it is hard to identify such projects to at least start some conversation about possible removals.

@praiskup
Copy link
Member Author

Let's do the discussion in the next 3 months.

@FrostyX
Copy link
Member

FrostyX commented Sep 27, 2023

but could we e.g. take a look at RPMs in the directory, check the "buildtime" baked into RPMs, and if all the RPMs are older than say 2 years mark the project "staled" in DB? I'm not trying to propose a policy for removals right now

I had a similar idea but I didn't have time to think it through yet.

If I understand correctly, the issue is only with Rawhide chroots. So I was thinking about not marking the project as stale but rather doing something with Rawhide packages that were built many releases ago. For example, Rawhide is currently F40, so considering doing something with Rawhide results from the era when Rawhide was F35 and lower. They will most likely not work today.

@kwk
Copy link
Contributor

kwk commented Sep 28, 2023

@praiskup I guess this is somewhat related to our internal discussion on storage size, correct? If you come up with ideas and if needed, feel free to ping us and brainstorm. We're happy to help if we can. Your team tremendously helped us in the past.

@praiskup
Copy link
Member Author

praiskup commented Oct 2, 2023

Thank you for the nice words, @kwk, this ticket is related to our discussion "by accident" -> I realized that your team doesn't keep data in stale/useless projects, I just needed to confirm that fact :-) I think the point of this ticket will be to identify such projects or prevent them from occurring. We'll see, your ideas are definitely welcome.

@praiskup
Copy link
Member Author

Moving priority to 6 to handle the discussion early.

@praiskup praiskup self-assigned this Apr 8, 2024
@praiskup
Copy link
Member Author

The proposal for this issue: fedora-copr/debate#8

@praiskup
Copy link
Member Author

In progress - local commit prepared - db migration prepared.

praiskup added a commit to praiskup/copr that referenced this issue May 20, 2024
When rolling policy finishes, we want to remove the data, but we also
want to mark the CoprChroot.deleted (disable it from the project)
because the corresponding mock chroot is still active in Copr.

When user does a new build, we reset the CoprChroot
delete_after/notification state, always.  This is ok because during the
normal MockChroot EOL policy, we do not allow new builds.

Implements: https://github.com/fedora-copr/debate/blob/main/2024-04-23-rawhide-chroots-eol.md
Fixes: fedora-copr#2933
praiskup added a commit to praiskup/copr that referenced this issue May 20, 2024
When rolling policy finishes, we want to remove the data, but we also
want to mark the CoprChroot.deleted (disable it from the project)
because the corresponding mock chroot is still active in Copr.

When user does a new build, we reset the CoprChroot
delete_after/notification state, always.  This is ok because during the
normal MockChroot EOL policy, we do not allow new builds.

Implements: https://github.com/fedora-copr/debate/blob/main/2024-04-23-rawhide-chroots-eol.md
Fixes: fedora-copr#2933
@praiskup praiskup removed the research label May 20, 2024
praiskup added a commit to praiskup/copr that referenced this issue May 21, 2024
When rolling policy finishes, we want to remove the data, but we also
want to mark the CoprChroot.deleted (disable it from the project)
because the corresponding mock chroot is still active in Copr.

When user does a new build, we reset the CoprChroot
delete_after/notification state, always.  This is ok because during the
normal MockChroot EOL policy, we do not allow new builds.

Implements: https://github.com/fedora-copr/debate/blob/main/2024-04-23-rawhide-chroots-eol.md
Fixes: fedora-copr#2933
@praiskup praiskup changed the title How to handle "leftover" projects that built against Fedora Rawhide? How to handle "leftover" projects that built in rolling chroots May 22, 2024
praiskup added a commit to praiskup/copr that referenced this issue Jun 21, 2024
When rolling policy finishes, we want to remove the data, but we also
want to mark the CoprChroot.deleted (disable it from the project)
because the corresponding mock chroot is still active in Copr.

When user does a new build, we reset the CoprChroot
delete_after/notification state, always.  This is ok because during the
normal MockChroot EOL policy, we do not allow new builds.

Implements: https://github.com/fedora-copr/debate/blob/main/2024-04-23-rawhide-chroots-eol.md
Fixes: fedora-copr#2933
praiskup added a commit to praiskup/copr that referenced this issue Jun 25, 2024
When rolling policy finishes, we want to remove the data, but we also
want to mark the CoprChroot.deleted (disable it from the project)
because the corresponding mock chroot is still active in Copr.

When user does a new build, we reset the CoprChroot
delete_after/notification state, always.  This is ok because during the
normal MockChroot EOL policy, we do not allow new builds.

Implements: https://github.com/fedora-copr/debate/blob/main/2024-04-23-rawhide-chroots-eol.md
Fixes: fedora-copr#2933
praiskup added a commit that referenced this issue Jun 25, 2024
When rolling policy finishes, we want to remove the data, but we also
want to mark the CoprChroot.deleted (disable it from the project)
because the corresponding mock chroot is still active in Copr.

When user does a new build, we reset the CoprChroot
delete_after/notification state, always.  This is ok because during the
normal MockChroot EOL policy, we do not allow new builds.

Implements: https://github.com/fedora-copr/debate/blob/main/2024-04-23-rawhide-chroots-eol.md
Fixes: #2933
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants