Support for shallow repositories #3058
Comments
|
Maybe this is super naive, but couldn't support for this be added by simply automatically hiding all commits that are in this list when creating a new |
|
Something like that might indeed be enough for the git-log use-case, but we still need to figure out how to make that work with merge-base. The network layer also needs to know this because we need to make the server aware that we're on a shallow repo so it doesn't try to re-use objects we don't have, and push needs to make sure we don't attempt to perform an impossible upload. |
|
+1 FWIW |
|
Just ran into #1430, which led me here. What’s the current status on this? Has anyone managed to get shallow repositories to work with fetches? Thanks! Re: use case: Using git for differential updates to data in iOS app. A shallow clone results in a ~50MB app; a full clone in ~150MB app (and, of course, growing). |
|
@aral No, there has been no update here - this is not currently supported, nor is it a high priority feature for us to implement, I'm afraid. |
|
@ethomson Understood. Thanks. |
since libgit2 does not expose cloning with --depth, I can't use it for larger repos. see libgit2/libgit2#3058 I would love to switch to git2go once this is solved.
|
🤷♂️ bump and +1 here. |
|
My use case: a huge github repo (more huge in terms of history than in terms of file sizes), but I only need to append to it with a single commit. To avoid the big (and growing) working directory, I just do a bare clone, and then use the API (via nodegit, actually) to create and commit the file, and push. But the repository also has a long history (and growing, by a few thousand commits per day). So over time, all those commits I'm doing just chew up space with all that local git history. Every 20,000 commits or so, I'd like to just clear out the repository and re-clone it shallow (depth=1), because I only need the very last commit to add a new one. Of course, I can do the clean/re-clone via a cron job, but the problem is that coordinating between a cron job and any running node process (to ensure I don't clobber in progress commit work) is much uglier/hacky than I'd like it to be. :( |
|
Just to clarify: there really is no need to provide usecases for this. It is definitly a thing we want included in libgit2, the only problem is that we lack the time to do so. Unfortunately, we are already fully loaded maintaining the current state of affairs with libgit2 with the little spare time we have for it. Because of this, it's quite hard to find time to even review merge requests implementing bigger features such as this one here. I'm really feeling sorry for it, but that's unfortunately how it is right now. |
Fetch GitHub commits by long hash more efficiently Closes #10078. **Tested with the following Cargo.toml:** ```toml [package] name = "repro" version = "0.0.0" edition = "2021" publish = false [dependencies] cargo = { git = "https://github.com/rust-lang/cargo", rev = "b30694b4d9b29141298870b7993e9aee10940524" } ``` ```console $ rm -rf ~/.cargo/git/db/cargo-* ~/.cargo/git/checkouts/cargo-* $ time $CARGO generate-lockfile $ du -shc ~/.cargo/git/db/cargo-* ~/.cargo/git/checkouts/cargo-* ``` Using current cargo from the most recent nightly, the `generate-lockfile` command downloads 69704 git objects in 7.0 seconds, consuming 41 MB on disk. Using cargo built from this PR by `cargo build --release`, the same command downloads 21481 objects in 2.2 seconds, consuming 17 MB on disk. Once libgit2 is able to do shallow clones (libgit2/libgit2#3058) this can be even more of a speedup. Using command-line git (which does not use libgit2) and `time git fetch --depth=1 https://github.com/rust-lang/cargo b30694b` indicates that it downloads just 262 objects in 1.1 seconds.
|
adding myself to the list of ppl. interested in this feature here. I see a lot of depending work merged already. does anyone track what's left to do until this can be implemented? |
|
This pull request (which itself builds on others) has an implementation that allows shallow cloning as well as unshallowing: #6396 |
|
Is there news about this work somewhere else? I'm watching this issue and the PR (#6396) closely, but seems like this has been paused for a while. Is there something blocking this work to move forward? I don't want to sound pushy or anything, I'm just very interested in this feature and unfortunately I don't have the knowledge to help.. |
|
Shallow cloning was just landed into |
git can write a file
.git/shallowto indicate that the history is cut off at particular commits. We currently do not read this at all and thus libgit2 will regularly fail to work on these repositories with error messages about failing to find objects.The typicall way such a repository is created is by the use of
git clone --depth N. We do not support this option to clone either, as we do not support the depth negotiation in the protocol.In order to provide support we would have to make sure to check against the list in
.git/shallowwhether we're at the end of the history we should expect to be at, and then not try to walk further back or ignore errors when looking up parents.There are a couple of issues which ask questions about this, I will close them and redirect them here to have a cleaner position from which to start.
The text was updated successfully, but these errors were encountered: