Leverage libgit2 instead of the git CLI tool

[ysbaddaden]

Maybe it would be better to use libgit2 (API) instead of relying on the presence of the external CLI tool. It could eventually lead to a better design of GitResolver, a simpler way to detect and report errors (e.g. refs not found) and portability (i.e. windows).

[straight-shoota]

There is a shard crystal-git/libgit2 with bindings, but it seems not to be actively maintained and stuck in the middle of a restructuring (crystal-git/libgit2#2 (comment)).

[ysbaddaden]

I created this issue 2 years ago, and I'm not sure it's such a good idea no. Anyway it won't happen anytime soon (or even later). Shards doesn't need external dependencies to bootstrap itself (apart from Crystal) and I want to keep it that way.

Closing for the time being. If someone is willing to try to integrate libgit2, please try it and report successes and issues :-)

[skinnyjames]

Hi all, I'm not sure if this path is a good one, but I've been working on library that wraps libgit2, and I plugged it into the git_resolver for this project. (Just cloning and fetching so far).

It's more of a proof a concept than a suggestion, but the tests are passing, which is nice.

If anyone was interested in checking it out, I PR'd against my fork of shards.
https://github.com/skinnyjames-forks/shards/pull/1/files

[skinnyjames]

Following up on this, I accidentally made a PR against shards, but as Grits is moving forward I think it's worth revisiting this issue.

From the PR:

Benchmarks

Overall true git clones appear faster than libgit2, but libgit2 clones consume significantly less memory. If not employed already, concurrency may help.

A benchmark on a repo with many dependencies using hyperfine (gshards is the release binary using libgit2).

hyperfine 'rm -Rf lib && shards install --without-development' 'rm -Rf lib && gshards install --without-development'

Benchmark 1: rm -Rf lib && shards install --without-development
  Time (mean ± σ):      4.076 s ±  0.263 s    [User: 1.844 s, System: 1.789 s]
  Range (min … max):    3.847 s …  4.669 s    10 runs
 
Benchmark 2: rm -Rf lib && gshards install --without-development
  Time (mean ± σ):      5.929 s ±  0.827 s    [User: 0.721 s, System: 1.340 s]
  Range (min … max):    5.484 s …  8.264 s    10 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Summary
  'rm -Rf lib && shards install --without-development' ran
    1.45 ± 0.22 times faster than 'rm -Rf lib && gshards install --without-development'

Notes

• Libgit2 is highly configurable, and so is Grits so far. Supporting more complex features programmatically may be easier.
• Statically linking libgit2 would remove a dependency (known)
• Historically libgit2 hasn't had support for shallow clones. This is no longer the case as of 2 weeks ago. Clones can do shallow fetches and it appears a fetch can unshallow a shallow clone. This could significantly improve performance. Grits is currently locked at v1.3 so it would need to be updated.

[straight-shoota]

Statically linking libgit2 would remove a dependency (known)

Git require SSL and linking that statically is trouble. So even if libgit2 is linked statically, it will probably still need libssl as a dynamically loaded dependency.

That said, there are use cases where an additional dependency doesn't matter. For example, when shards is installed via a package manager, that manager can usually take care of having lib dependencies available.

So maybe a libgit2 based resolver can just be an optional alternative resolver to the CLI based resolver which can be used in situations where you want a fully self-contained executable that doesn't have any binary depdencies.

[skinnyjames]

Git require SSL and linking that statically is trouble. So even if libgit2 is linked statically, it will probably still need libssl as a dynamically loaded dependency.

Ah, makes sense. Thanks for speaking to it.

So maybe a libgit2 based resolver can just be an optional alternative resolver to the CLI based resolver which can be used in situations where you want a fully self-contained executable that doesn't have any binary depdencies.

That could be interesting. I think the biggest value add in using libgit2 might just be not having to shell out for everything - which could lead to a much easier time implementing and maintaining new features.

Clones are slower though, so I'm currently working on an experiment that sees if Crystal's concurrency mechanism can be applied to a custom transport. Will definitely keep this thread updated.

Another note that might be problematic is that Grits is linked against libgit2.so.1.3 specifically, because I noticed some of the APIs weren't compatible between versions. I'd love to remove this requirement by making it work with latest, but I haven't done so yet.

[hugopl]

I used libgit in one of my projects and what I could say is:

It breaks the API very often, so it's normal for me to upgrade my system then notice that I can't load my program anymore because it failed to load libgit.so... then I need to recompile it... OTOH git CLI is stable as a rock and works.

[skinnyjames]

It breaks the API very often, so it's normal for me to upgrade my system then notice that I can't load my program anymore because it failed to load libgit.so... then I need to recompile it... OTOH git CLI is stable as a rock and works.

Yeah. I've run into this too. Using libgit as a resolver would likely mean statically linking some version of libgit.
I haven't had much success in using a custom transport, and I also haven't spent much time in making sense of the resolvers.

Since I can't seem to make clones faster either, I'm not sure the benefits are currently there. I'm happy to speak to any of the work I've done if somebody is interested in picking it up, but I probably won't be following up on this anymore, sorry.

[straight-shoota]

Yeah, linking libgit2 statically isn't easy either. I think mostly because it depends on libssl. That's a nightmare to link statically.