[September 26] - SHA-1 is fast again! #3021
Sebastian Thiel (Byron)
announced in
Progress Update
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I know, the report is late, but hopefully, as always, I do have a very good reason for it: the error migration away from
thiserrortogix-errorhas finally concluded. It was a massive PR which held me hostage cognitively as it felt like a blocker for everything else, so I'd hope for long-waiting features to trickle in (i.e. pass the refackiew) much quicker now.thiserror->gix-errorFinally, it's done. And with the migration complete,
gitoxidefinally takes ownership over its error handling. This comes with a few advantages:gixcan now use plumbing without having to define error types for each methodOver time, the error classification will improve as demand will shape the way errors are created and annotated, making it easier than ever for the downstream applications to be even more robust in the face of fixable errors.
SHA-1 hashing is fast again!
This should be a community story, but it's so hugely important that I can't wait this long. Thanks to Sam Reis and his incredible performance work, we see a roughly 2.4x speedup, yes, 140% faster (!!), in SHA-1 hashing, an operation hugely influential to the overall runtime when cloning repositories.
All that, of course, still happens while checking for collision attacks, which is the original reason that the hashing performance decreased to around 1/3 of what it was before when that was enabled. I still remember how painful this was, so I am super happy we can now hash without the handbrakes pulled once again!
It's notable that SHA-256 hashes even faster than SHA-1 ever did, and it's definitely something I am also looking forward to experiencing one day.
gix-notesjust got fasterGit notes are the mechanism to attach metadata to Git objects.
As I keep hearing that they are slow (the last time was at the GitMerge in Lisbon), I put it to a test: create a note for every commit in the Linux kernel commit graph, which means creating shy of 1.4m notes.
How long does that take? A mere 2.4s on my machine. Of course, that deserves a footnote stating that this is only for the in-memory creation of these notes. Writing them directly into a pack is as fast as 210k/s when uncompressed, so that's by far the slowest operation, and an operation that can probably be faster anyway if one really wanted to optimise for that.
Reading this many notes is also incredibly fast, at about 700k/s, so it really does seem that performance won't be the limiting factor here.
You can try the benchmark yourself by running
cargo run --release -p gix --example gix-notes-bench.tixfeels like it's settlingWhile I didn't manage to merge
tixback in yet, it feels very robust already with only a few changes per week, none of which are features. Its biggest weaknesses still are pretty bad UX when transplanting/selecting commits or trees (compared to what's possible), but it wasn't important enough for me to try to fix it.A nice side effect of the continued work on it is that it motivates me to add features to
gix-*crates that benefit all Git client applications. The recent addition, refackiew pending, is a generalised filesystem notification framework which should work on all platforms, and ultimately allowgitoxideto ship with a filesystem daemon for greatly acceleratedgit statusqueries.I am definitely looking forward to using that myself; it's still early.
The next big step will be to merge part of the progress back in, but that can only happen after worktree support.
Oh, and before I forget, it's worth mentioning that
tixuses Git notes for per-worktree metadata already, and they are such an enabler for these features (all notes are encoded as simple Git config files):All these fulfil a specific purpose, and all are needed and used regularly, while surviving rewrites and rebases, usually. And if these markers don't survive, that's a great indicator that QA needs to be redone, or worse, an entire review. To prevent the latter, agents are instructed to not amend commits with sparkles and instead create fixup commits, so these can be reviewed separately and squashed later.
Git3 support: reftable
No work was done in this regard, but from what I can tell, there are maybe 4 - 6 months left on the clock until Git3 truly lands. This isn't much, especially since I am still not ready to start this endeavour, but I at least wanted to mention here that I am definitely keeping it in my mind firmly.
Before actual work can be done, I'd like to prepare
gix-refwith a better abstraction for different backends, and hope to accomplish this by adding an in-memory proxy which would allow us to perform edits in memory first, then flush them to disk later or discard them entirely. This is great for in-memory previews of complex operations while being transparent to the application.Community
A step closer to diff parity
Thanks to Tyler Smith, the diff slider correction in
gix-imara-diffis now 33% better, down to 0.45% disparity compared to Git in a specific sample of more than 500k diff lines. This makesgix-diffmore conforming to Git thanlibgit2on the same sample.And I have a feeling, and hopes, that they aren't quite done yet :D.
Welcoming Johannes Schindelin
In case the name doesn't ring a bell, let me introduce him as the core maintainer of Git for Windows. We have been in touch for some time now, and I'd be tempted to say that it's now going to happen: Johannes will contribute more to Gitoxide and start with no less than supporting me with a reftable implementation.
Given the state of the world, this is less of a matter of skill and more of a matter of working together effectively on a big feature, and I am looking forward to giving it my best. Now that we have it in writing, I suppose there is no way around it either 😁.
Let's do this!
Everything else!
~/~userexpansion by TrietCN, and continuation-line indentation fixes by jaideeppyne.Gix in Cargo
There is nothing new here, but let's keep the horizon active:
Cheers,
Sebastian
PS: The latest timesheets can be found here (2026).
All reactions