-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
APIs for git-gc and git-reflog #3247
Comments
Similarly for |
I'm actually not looking for the pack functionalities of |
There is no functionality to delete objects or perform the repack necessary for that. |
Do we consider add this ability? It's common to get unreachable objects in a repo so we need a way to remove them while |
As I mentioned above, I don't think there is much of an advantage, if at all, of having a libgit2-implemented repacker or the other house-keeping operations. Not having git itself but having libgit2 on such a system is an edge case which I don't believe we should base our goals on. |
Any updates? When used on a server, I wish it could do background garbage collection regularly to save disk space, so I think it's a very useful functionally. Currently I have to call |
The goals of the project haven't changed, and neither have the tradeoffs. If one were to rewrite git-gc on top of libgit2, the best-case scenario is ending up with what we already had. If you want to use regular maintenance on some repostories, use |
I keep using libgit2 maintaining a repo, and the repo eventually run into
Git gc will solve the issue. |
Question: wouldn't be running a local clone of the repo in |
@eaigner: while this would seem like an easy solution, it's unfortunately not equivalent to git-gc. git-gc has some important aspects that we definitely have to honor if we were to implement it in libgit2. Most importantly, a git-gc will by default not purge unreachable objects unless they haven't been touched in at least |
@pks-t true, but may be enough for some peoples use case. And as you know, I'm all about thoes half-assed-stop-gap solutions 😆 |
Haha. Well, yeah, it would be a possible way to implement it for users if they know to have a tight grip on the environment. Most importantly, it needs to be known that no other process will access the repo at the same point in time, which is a guarantee that cannot be made in most environments as users are always free to use e.g. the IDE (which may regularly execute git commands in a repo without the user doing anything) and the command line at the same time. |
@pks-t Any ideas how one would go about implementing garbage collection for a custom odb backend without cloning/rebuilding from scratch? With my project, I'm using an OO database as the backend, so I've no way of using standard git tools to clean-up any old garbage. In my database environment, there is such things as single-user & offline modes, and there's other locking strategies I could use while online, which would guarantee it's safe to run garbage collection. With regards the challenge you've mentioned above about not deleting objects that haven't been touched in |
The first step would be to decide what parts of Reading what the manpage says, git-gc seems to depend on git-repack (no support), which uses git-pack-objects (some support). As part of the ODB backend API, we have In short, designing the |
The main part I think I'd need is a common approach/algorithm for identifying all the unreachable objects and deleting them. The backend in my case is another ODB (JADE), which fits in quite nicely with the pluggable backend approach provided by libgit2. While it would be needed for cleaning up standard git file based ODB's (which can just be cleaned up using git), I don't believe there's any need to provide support for repacking objects into compressed pack files for other custom ODB backends. In my case, the JADE database already packs the objects away into it's own database structure. I suspect the same can be said for other kinds of databases that'd be used as a backend, as each would store all the objects in a way that's most appropriate to them. I don't think it'd make sense to pull objects out of other kinds of databases in order to archive them into a pack file stored outside of the database. |
Hmm, I see the issue: you don't have a loose/packed system with a custom ODB, and I'm too focused on that 😉 . So there must be a way to ask for each backend's idea of which objects are unreachable (that could be handled as public helper), and a way to ask for a list of objects to be deleted (I think that's where the transaction-y issues alluded to enter the picture). So we'd provide |
That sounds like the sort of thing I'd need, but wouldn't it need to be at the repository level, as the garbage collection would need to use its associated refdb to determine what can and can't be reached? |
Definitely. It could always be the case that some objects A is part of ODB X and only reachable via an object B in another ODB Y, only. So asking X whether A is unreachable would return |
So what would be required is callbacks that enumerate all objects in an ODB. We'd then have to recreate the complete tree from all objects of each ODB and mark all the other objects as unreachable. We could then notify ODBs that this set of objects is unreachable. Each ODB can then decide for itself whether it wants to delete those objects. This still ignores the issue of expiry times, though, and I'm not convinced that this can be handled at ODB level, only. |
There is already a 'foreach' callback on git_odb_backend which looks like it's intended to enumerate all objects in an ODB.
While I wouldn't want to update an object when it's being read to track when it was last accessed, there's another approach that may make the overall gc operation more efficient as it'd avoid the overhead of building up & trimming a potentially very large list of objects that can/can't be reached to work out what can be deleted. If the ODB backend provided functions to update/check a timestamp (which would be initially set on creation) - This could be updated while recursively visiting all objects that can be reached. All objects in the ODB could then be enumerated to delete those where the timestamp hasn't been updated and has expired as a result. |
Oops, yes. I've accidentally looked at the wrong struct and had been really confused that I didn't see any such callback.
Yeah, that'd work. The downside though is that we'd batch-update all the objects whenever a GC is being performed. Sounds like a lot of disk writes to me, and for example if you have a loose file backend then it would be quite inefficient. Might be preferable in some situations though compared to building a complete graph |
In my case, the disk writes wouldn't be a problem, but I can understand why the library can't make assumptions like that. Now that we've identified a strategy that'd work in my case - Perhaps all I really need is a way of visiting every reachable object. I'll be able to take care of the rest by setting an indicator elsewhere so I know to update the timestamp when an object is visited. Is there anything in the library that could be used to just visit every reachable object? |
On Thu, Jun 20, 2019 at 04:10:53AM -0700, Kevin Saul wrote:
In my case, the disk writes wouldn't be a problem, but I can
understand why the library can't make assumptions like that.
Now that we've identified a strategy that'd work in my case -
Perhaps all I really need is a way of visiting every reachable
object. I'll be able to take care of the rest by setting an
indicator elsewhere so I know to update the timestamp when an
object is visited.
Is there anything in the library that could be used to just
visit every reachable object?
No, right now there isn't, but it shouldn't be too hard to do if
using the revwalk API.
|
@pks-t @ethomson @kcsaul @tiennou any progress on such API? If not, can we re-open this issue? I have a stand-alone application which does not assume that git binary for git-repack is present, and I ran into the issue that after running non-stop fine for several days, git_revwalk_next() with prior git_revwalk_push_head() out of a sudden stopped working to my surprise. Turns out I ran into the default resource limit on open files deep down inside git_revwalk_next() when it preps the list due to having too many pack files. Doing a plain I worked around the issue by bumping the ulimit ... ... but it's not really /actually/ solving it just mitigating, hence proper API for repacking is needed for libgit2. Are there any other options I'd have available to workaround, which are already in upstream libgit2 but which I've been missing? |
I'm okay with reopening this issue though, but there are no promises on work actually taking place 😉. The goal at the moment is to try to make as much stabilization happen as possible in preparation of v1.0, so a "new feature" is pretty low priority, especially one with (IMHO) not enough preparatory design work. Adding more ways to have write/deletion problems, especially with objects, doesn't feel 1.0-ish to me 😜. That said, "a repacker" sure is something that would be nice, but it's unclear how it's supposed to work against custom refdb implementations, and those are already in-use… So I think the summary, to me, is: Should packing be implemented as a new backend callback ? Or be backend-agnostic, which makes it effectively a repository-level "system" ? Should we invest time in improving our own refdb code, or should we concentrate into trying to port "what git already is", relibrarizing their refdb code ? It should be theorically possible to yank our implementation for It should be possible to write a "dumb" version of the repacker with the code now, loading and "locking" objects touched by either being pointed to by a reference, or related to one such locked object. And then issue a final "purge/compact" action to the odb with the "untouched objects" list. This looks simple, but you have to make sure to keep tabs on the refdb while doing that, you have worktrees and alternates to handle consistenly, and someone might be commiting from another process entirely. I'm not sure how this is doable given all there constraints… |
I'm fine with reopening as it's still an issue. And as @tiennou said, an open isuse is no promise that it's going to be implemented anytime soon ;) But yeah, having it closed may seem like libgit2 isn't willing to consider any GC/repacking implementation at all.
Agreed, though I think it would be a nice to have in the long term.
Tough question. I feel like having it backend-specific is going to scale a lot better than a backend-agnostic way, especially so as for some backends repacking would essentially be needless/impossible. If you've got a database backend there is no need for you to repack as the database handles that for you, already. So having a generic layer above the backends that still does the repacking is going to perform a lot of work with zero benefit at all. So I think the answer is backend agnostic, but that again throws up the question of how to handle cross-backend repackings. The most obvious case here loose/packed objects, which are two backends where repacking of loose objects results in a packed object. But how to handle that generically where there's e.g. a loose and database backend is a tough question. So I think any judgement here is premature. There definitely is a rather big discussion to be had on this topic to nail down pros/cons of either of the approaches and get the design right. |
Yeah, to be frank, the main issue I see with the current design is that our loose/packed reference backend carries too much "knowledge" about how to do things — case in point, any custom backend has to implement reflog handling, as both an "internal" and "external" API (resp. itself, and the OTOH, it's the only working backend I know of, and it already has some conformance issues with git because of its "customizability". A big discussion indeed 😉. |
Before this commit, running Git's GC in a Git repo backing a Jujube repo would risk deleting the conflict data we store as blobs in the Git repo. This commit fixes that by adding a Git note pointing to the conflict blob. I wasn't able to add a test case for this because libgit2 doesn't support gc [1]. Just testing that the ref is there doesn't seem very useful. [1] libgit2/libgit2#3247
I'm wondering if I could implement some functionalities of git-gc and git-reflog by using the current public APIs. Do we have this kind of APIs? I'm basically looking for
If we don't have these APIs do we have a plan on it?
The text was updated successfully, but these errors were encountered: