Verify and update upstream changes - #1
Open
edmwagner wants to merge 298 commits into
Open
Conversation
driusan
force-pushed
the
master
branch
3 times, most recently
from
January 2, 2018 18:39
d61922f to
c7ffbb6
Compare
Most options are unimplemented, but the basic "git grep" subcommand should work for the current git directory.
The basic "git apply" command should now work and atomically apply a patch, though it's missing tests and all options.
Added checkout -p to selectively discard chunks from the working tree. This necessitated adding the --reverse option to git apply, since it works by parsing the output of "git diff", and diff shows the patch in the opposite direction of what checkout needs.
Added the ability to apply patches to the indexes instead of the work tree with the --cached option. This required inverting the logic for how apply works to copy the files to a temporary directory and run the external patch program there, rather than apply it in the work directory then restore from the backup if there's an error (which is probably what it should have done all along for safety reasons anyways.)
Add the ability to interactively stage hunks to the index with the -p option.
If a file is modified according to stat information, they still need to be hashed in order to catch the case where they were modified, and then modified back to the original so the stat information is off. The stat info does, however, allow us to short-circuit out of files that are definitely not modified.
This implements the happy path for git revert.
GetAuthor and GetCommitter paniced if opening .gitconfig returned an error, this resulted in "git clone" (and git init, and various other commands) failing when trying to maintain the reflog. It now returns (and handles) an error instead, and git commit prints a warning and instructions if the config file doesn't exist. Fixes #9
SymbolicRefGet wasn't trimming the trailing newline that might be added by some git clients, confusing git log (and others) Also fixed compilation error with tests.
…f things) to fail
There was a lot of excessive looking up objects for information which was then discarded in git ls-tree. This fixes some of it to make the performance more reasonable for large trees. Fixes #15.
This more accurately matches the real git client Fixes #13
-C changes the directory that git executes in before doing anything. Fixes #21
If a file is a symlink, then git commands are operating on that symlink, not on the content being pointed to, so we need to ensure that checking if a file exists is checking the symlink, not the content of the symlink in most cases.
This fixes the case where git update-index is added on a broken symlink. (However, update-index is still not properly adding non-broken symlinks.)
Added support for adding symlinks to the index, and included a simple test case which ensures that both valid and broken symlinks are added with the same hash as the real git client. Fixes #14
Solves issue where write-tree could get into an infinite loop if an explicit prefix was passed and the final directory only contains one file. Fixes #24.
Add the git fsck test suite that are passing to be automatically run by Travis. The first tree test from fsck test is also fixed.
This gets the git tag tests working for fsck. It includes a lot of hacks, because the git tests regarding tags are incredibly inconsistent about whether things go to stdout, stderr, and whether they cause a non-zero exit status or not, but this seems to get them all to pass, at least for now.
This fixes up to t1450.49 in the fsck tests. Only 4 known failures remain.
…a1 (#249) A refactoring done in df1a505 resulted in recursively calling GetAllObjectsExcept on a variable that was not yet initialized. This resulted in trying to get objects on the empty Sha1, incorrectly. Fix variable reference to the proper variable name. This also removes the default-acl test from the official test suite, which was only passing incidentally and is now failing with this bug fix.
…es (#251) A blob object could be included in the results of git rev-list multiple times if it was in different tree objects. This improves the validation for GetAllObjectsExcept to verify blob objects as well, and ensures that RevList adds objects it encounters to the exclude list.
When running git push, the ".git/refs/remotes/remotename/branch" ref needs to be populated. Failing to do this on an initial push (with --set-upstream set) means that future pushes (both with dgit and git) fail, because it can't resolve the rev for the the refspec "remotename/branch" that it has locally.
Attempting to `go get github.com/golang/text` instead of the golang.org module name was causing go to fail. Use the vanity path instead.
This does just enough work to ensure that the go-get-modules-tests.sh passes. No attempt has been done to ensure that it can read the tags after they've been indexed or that deltas work against it.
When the branch is already up to date, the program exit code should be success (regardless of if the "Already up to date" error came from git.Fetch or git.Merge)
When an object doesn't exist in a packfile, checking for an object that starts with 0 can cause a panic, because the i >= 0 check is after the access in the loop condition. Switch the order to that the loop terminates before accessing the index. Also cast from a uint to an int so that i>=0 isn't always true.
This fixes some hacks to significantly improve the performance of index-pack (cloning the dgit repo on 9front went from ~160s to ~20s). Since we've forked the compress/zlib repository anyways, the signature is changed to always require a flate.Reader instead of an io.Reader. This means that reading a zlib stream doesn't overshoot the reader, but only works on things that have a ReadByte method. Since files don't implement ReadByte, zlib also grew the ability to tell you how many compressed bytes it read, meaning files can be wrapped in a bufio and then Seeked back to the proper place. In the refactoring the ability to simultaneously index as the pack is read from the network was lost, but this can be re-added in the future, for now the performance gains of not reading 1 byte at a time or seeking back looking for the digest more than makes up the difference.
The indexing of packs now happen as it's coming over the wire, rather than being performed in 2 distinct phases.
This provides some fixes for the go get tests on the master branch. - GOPROXY=direct is set, so that dgit is invoked instead of downloading packages through the Go proxy. - Options are parsed for git remote add (they aren't implemented, just parsed.) - A hack is added so that -- is thrown away in git remote add. Go arbitrarily adds it between the remote name and the remote URL, which results in dgit thinking the the remote URL is "--" - A hack is added so that "--" is thrown away at the end of git checkout. Go arbitrarily adds it, which causes dgit to think that we're trying to checkout files, not a branch. - rev-parse no longer thinks commit names are ambiguous if the commit id is arrived to multiple ways (ie. through a pack file and a loose object.) - The repository used for go get tests is changed. Nearly every repo I could find can't be retrieved with GOPROXY=direct, even with the real git client because Google's git servers require authentication even when doing a clone. - Ran go mod vendor to update the out of sync vendor directory. - 1.13, 1.14, and master are added to allowed failures. They invoke git cat-file --batch which we don't yet support, but should be implemented in a separate PR, because we should ensure the cat-file tests from the official git client pass when we implement it.
This updates index-pack to resolve deltas in files concurrently. It also fixes the CRC32 in the index generated, which is now byte-for-byte identical to that which comes from the official git client.
This fixes all the versions of Go tested by Travis with dgit. - cat-file --batch is implemented - cat-file tests from the official test client are added (up until the point that they deal with packs because while we read packs, we don't write them.) - a bug where :path in rev-parse is supposed to refer to the index, not HEAD is fixed - a new temporary $GOPATH is used by the go get tests so that the tests aren't added to dgit's go.mod in module mode.
This is causing "no procs" errors on 9front, but isn't giving much performance gains, so for now call the callback on each index in sequence.This is causing "no procs" errors on 9front, but isn't giving much performance gains, so for now call the callback on each index in sequence.
git9 incorrectly uses 040755 instead of 040000 for the mode of subdirectories in trees. Since even if it's fixed upsteam the trees will still be out in the wild, this updates dgit to be a little more lenient of what it considers a tree.
This adds support for the one tree version of git diff-tree (where it diffs against the parent), and includes the --root option. This is just enough to get the setup for t5300-pack-object to pass.
This makes a variety of improvements to improve the performance of git index-pack under dgit.
Deltas are now io.Readers so that they can be directly streamed into anything that takes an io.Reader without first having to calculate the entire delta content and store the result on the heap.
Various places that were using make([]byte, 1) now take a flate.Reader instead of an io.Reader to read a byte. This generates much less garbage for the GC to collect. We're also more careful about passing flate.Readers around where needed instead of io.Readers
Progress information is now sent to stderr at most once per 3 seconds.
Indexing is now split into 2 phases. The first hashes normal commit/tree/blob/tag entries and notes how many deltas are used against them, the second resolves deltas.
Getting objects while indexing no longer uses GetObject to resolve deltas. GetObject requires entries in the index to be sorted, and we now only sort once at the end.
An LRU cache is used to cache objects while indexing.
My first attempt at re-adding caching cached tried to mark everything with deltas in the first phase, and then purge them after all deltas against them were resolved, but that could only get to about 80% of indexing github.com/Perl/perl5 (before getting to the delta resolution) on my 2Gb vultr node before running out of memory, so I wrote an LRU cache implementation with a cap on the amount of memory used until it starts evicting content (rather than the number of entries), but that also kept running out of memory without a very tiny cap. I assumed my code was buggy and leaking memory and switched to package github.com/hashicorp/golang-lru and.. also kept running out of memory. I think Go's garbage collector just isn't very amenable to an LRU cache. For now it's still using the hashicorp/golang-lru implementation with a very small (250 entry) cap on the number of entries (larger for small repositories), which is small enough that I can index the perl pack, although the memory consumption is larger than it should be.
The result of all this is that indexing the pack from github.com/Perl/perl5 takes ~8.5 minutes (down from ~20 minutes) on my poor little vultr node and there should be a proportional improvement in cloning time. (There's still a lot of room for improvement compared to git which takes about 1.5 minutes and doesn't bloat up the memory like this does, but this is probably a good enough improvement to commit.)
unpack-objects is also updated to use the same infrastructure as index-pack so that the older, less efficient delta resolution code can be removed.
This also fixes #262 as a side-effect of point 4.
This adds support for the first few tests of t5300, which tests pack/unpack objects in the official git client. It passes until the point where it compares the delta flavours. This mostly involves slightly better support for the GIT_OBJECT_DIRECTORY environment variable, naming the packs correctly (with the trailer as a suffix) and parsing some command line options.
Move delta Reader into new package and create delta writing infrastructure This creates a new package "delta" which is concerned with reading and writing git deltas. The old git.delta type is now a delta.Reader and indexpack is updated accordingly. The new "delta.Calculate" function takes 2 byte slices and calculates a delta that goes from src to dst, writing the resulting delta to an io.Writer. PackObjects doesn't yet use it, but this puts the plumbing in place to calculate ref and offset deltas.
Fix push which was broken while fixing pack-object tests. cmd.PackObject's behaviour had changed to fix the official git tests and push was incorrectly using that instead of git.PackObjects.
This adds support for calculating either ref of ofs deltas in git pack-objects. It fixes all of the tests of t5300-pack-object.sh of the official git test suite up until it starts testing with git verify-pack (which we don't currently have any implementation of.) Other than deltas, this mostly involved improving support for GIT_OBJECTS_DIRECTORY throughout dgit and implementing index-pack --keep.
This implements the git send-pack command and refactors "git push" to use the send-pack plumbing rather than the hack which was previously in place to allow pushing in the early days of dgit. The result should be that it will work with all transport protocols that are supported, and not just HTTP. It also splits the push command into the same git/cmd package distinction for parsing command line and implementing the logic as other commands use instead of all being done in cmd. (Another leftover from the fact that push was one of the earliest things implemented in dgit.) In doing so, it brings the behavior of push closer in line with that of the official git client. (ie now the remote name is required, not the branch name, it can be called with no arguments, and --force is no longer implied.) Also fixes a bug where Linux wasn't loading SSH keys because of incorrect build tags.
This fixes some regressions in git push caused by porting the code to use git-send-pack instead of the old PackNegotiator https only hack. Our delta calculation isn't robust enough for the real world, so don't use it in send-pack, and we weren't taking the "new branch" case into consideration when checking if a push needs to be forced.
Update import path of no longer existant bitbucket hg repo to github mirror.
The incorrect n was being checked when an insert instruction was found, and the incorrect value was being used in the error message because of shadowing. Resolves #277
Switch from Travis CI (which wasn't running anyways) to GitHub actions. Tests are currently failing for a variety of reasons. Go test ./... is failing because the status tests are failing and need to be fixed and a couple tests are skipped. go-get-module-tests.sh is failing for an known reason and currently skipped.
Windows will not allow a file to be renamed while it's still open. Add an explicit close before os.Rename, so that it doesn't fail because the file is already in use. Resolves #281
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verify and update upstream changes