[DO NOT MERGE] prepare for vfs 2.23.0 #165
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dscho
changed the title
|
@dscho I think you forgot the 32 commits from ds/commit-graph-incremental that we merged into vfs-2.22.0 but were not in v2.22.0.windows.1. ;) They just were the identical commits from upstream, so they did not show up in your rebase. |
The command-line I used was: |
dscho
force-pushed
the
prepare-for-vfs-v2.23.0
branch
from
0d3d18c
to
b63209f
Compare
dscho
requested review from
derrickstolee,
jamill,
jeffhostetler,
kewillford and
wilbaker
dscho
force-pushed
the
prepare-for-vfs-v2.23.0
branch
from
b63209f
to
bf142bd
Compare
While using the reset --stdin feature on windows path added may have a \r at the end of the path that wasn't getting removed so didn't match the path in the index and wasn't reset. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Signed-off-by: Saeed Noursalehi <sanoursa@microsoft.com>
Signed-off-by: Johannes Schindelin <johasc@microsoft.com>
This header file will accumulate GVFS-specific definitions. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
This does not do anything yet. The next patches will add various values for that config setting that correspond to the various features offered/required by GVFS. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
The two existing members of the run_hook*() family, run_hook_ve() and run_hook_le(), are good for callers that know the precise number of parameters already. Let's introduce a new sibling that takes an argv array for callers that want to pass a variable number of parameters. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This takes a substantial amount of time, and if the user is reasonably sure that the files' integrity is not compromised, that time can be saved. Git no longer verifies the SHA-1 by default, anyway. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
The idea is to allow blob objects to be missing from the local repository, and to load them lazily on demand. After discussing this idea on the mailing list, we will rename the feature to "lazy clone" and work more on this. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Hydrate missing loose objects in check_and_freshen() when running virtualized. Add test cases to verify read-object hook works when running virtualized. This hook is called in check_and_freshen() rather than check_and_freshen_local() to make the hook work also with alternates. Helped-by: Kevin Willford <kewillf@microsoft.com> Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
Prevent the sparse checkout to delete files that were marked with skip-worktree bit and are not in the sparse-checkout file. This is because everything with the skip-worktree bit turned on is being virtualized and will be removed with the change of HEAD. There was only one failing test when running with these changes that was checking to make sure the worktree narrows on checkout which was expected since we would no longer be narrowing the worktree. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
…ng objects This commit converts the existing read_object hook proc model for downloading missing blobs to use a background process that is started the first time git encounters a missing blob and stays running until git exits. Git and the read-object process communicate via stdin/stdout and a versioned, capability negotiated interface as documented in Documentation/technical/read-object-protocol.txt. The advantage of this over the previous hook proc is that it saves the overhead of spawning a new hook process for every missing blob. The model for the background process was refactored from the recent git LFS work. I refactored that code into a shared module (sub-process.c/h) and then updated convert.c to consume the new library. I then used the same sub-process module when implementing the read-object background process. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
While performing a fetch with a virtual file system we know that there will be missing objects and we don't want to download them just because of the reachability of the commits. We also don't want to download a pack file with commits, trees, and blobs since these will be downloaded on demand. This flag will skip the first connectivity check and by returning zero will skip the upload pack. It will also skip the second connectivity check but continue to update the branches to the latest commit ids. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
If we are going to write an object there is no use in calling the read object hook to get an object from a potentially remote source. We would rather just write out the object and avoid the potential round trip for an object that doesn't exist. This change adds a flag to the check_and_freshen() and freshen_loose_object() functions' signatures so that the hook is bypassed when the functions are called before writing loose objects. The check for a local object is still performed so we don't overwrite something that has already been written to one of the objects directories. Based on a patch by Kevin Willford. Signed-off-by: Johannes Schindelin <johasc@microsoft.com>
Ensure all filters and EOL conversions are blocked when running under GVFS so that our projected file sizes will match the actual file size when it is hydrated on the local machine. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
This adds hard-coded call to GVFS.hooks.exe before and after each Git command runs. To make sure that this is only called on repositories cloned with GVFS, we test for the tell-tale .gvfs. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
The use case here is to allow usage statistics to be gathered by running hooks before and after every hook, and to make that configurable via hooks. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
GVFS Git introduced pre-command and post-command hooks, to gather usage statistics and to be able to adjust the worktree if necessary. As run_hooks() implicitly calls setup_git_directory(), and that function does surprising things to the global state (sometimes even changing the current working directory), it cannot be used here. This commit introduces the pre-command/post-command hooks, based on the previous patches that culminate in support for running hooks early, i.e. before setup_git_directory() was called. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
gvfs: block unsupported commands when running in a GVFS repo
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
BRANCHES.md: Add explanation of branches and using forks
Add trace2_thread_start() and trace2_thread_exit() events to the worker threads used to read the index. This gives per-thread perf data. These workers were introduced in: abb4bb8 read-cache: load cache extensions on a worker thread 77ff112 read-cache: load cache entries on worker threads Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
…ension Add regions around code to read and write the cache-tree extension when the index is read or written. This is an experiment and may be dropped in future releases if we don't need it anymore. This experiment demonstrates that it takes more time to parse and deserialize the cache-tree extension than it does to read the cache-entries. Commits [1] and [2] spreads cache-entry reading across N-1 cores and dedicates a single core to simultaneously read the index extensions. Local testing (on my machine) shows that reading the cache-tree extension takes ~0.28 seconds. The 11 cache-entry threads take ~0.08 seconds. The main thread is blocked for 0.15 to 0.20 seconds waiting for the extension thread to finish. Let's use this commit to gather some telemetry and confirm this. My point is that improvements, such as index V5 which makes the cache entries smaller, may improve performance, but the gains may be limited because of this extension. And that we may need to look inside the cache-tree extension to truly improve do_read_index() performance. [1] abb4bb8 read-cache: load cache extensions on a worker thread [2] 77ff112 read-cache: load cache entries on worker threads Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
…and report_tracking() Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
It took this developer more than a moment to verify that was_dirty() really returns 0 (i.e. "false") if the file was not even tracked. In other words, the `dirty` variable that was initialized to 1 (i.e. "true") and then negated to be returned was not helping readability. The same holds for the final return: rather than assigning the value to return to `dirty` and then *immediately* returning that, we can simplify it to a single statement.
Add data for the number of files created/overwritten and deleted during the checkout. Give proper category name to all events in unpack-trees.c and eliminate "exp". Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
It took this developer quite a good while to understand why the current code cannot get a `NULL` returned by `index_file_exists()`. To un-confuse readers (and future-proof the code), let's just be safe and check before we dereference the returned pointer.
Update tracing around report_tracking() to use 'tracking' category rather than 'exp' category. Add ahead/behind results from stat_tracking_info(). Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The idea of the virtual file system really is to tell Git to avoid accessing certain paths. This fixes the case where a given path is not yet included in the virtual file system and we are about to write a conflicted version of it.
Includes gvfs-specific commits from PR#115
With the "--untracked-files=complete" option status computes a superset of the untracked files. We use this when writing the status cache. If subsequent deserialize commands ask for either the complete set or one of the "no", "normal", or "all" subsets, it can still use the cache file because of filtering in the deserialize parser. When running status with the "-uno" option, the long format status would print a "(use -u to show untracked files)" hint. When deserializing with the "-uno" option and using a cache computed with "-ucomplete", the "nothing to commit, working tree clean" message would be printed instead of the hint. It was easy to miss because the correct hint message was printed if the cache was rejected for any reason (and status did the full fallback). The "struct wt_status des" structure was initialized with the content of the status cache (and thus defaulted to "complete"). This change sets "des.show_untracked_files" to the requested subset from the command-line or config. This allows the long format to print the hint. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The major work on merge-recursive.c that went into v2.21.0 necessitates an adjustment to work well with VFSforGit's virtualfilesystem hook.
…cked-hint status: deserialize with -uno does not print correct hint
When our patches to support that hook were upstreamed, the hook's name was eliciting some reviewer suggestions, and it was renamed to `post-index-change`. These patches (with the new name) made it into v2.22.0. However, VFSforGit users may very well have checkouts with that hook installed under the original name. To support this, let's just introduce a hack where we look a bit more closely when we just failed to find the `post-index-change` hook, and allow any `post-indexchanged` hook to run instead (if it exists).
dscho
force-pushed
the
prepare-for-vfs-v2.23.0
branch
from
bf142bd
to
50a7847
Compare
|
(Note, this got "merged" because I updated the |
derrickstolee
added a commit
to microsoft/scalar
that referenced
this pull request
Aug 20, 2019
See microsoft/git#165 for the details of the latest rebase. This PR is an adaptation of microsoft/VFSForGit#1389.
derrickstolee
added a commit
to microsoft/VFSForGit
that referenced
this pull request
Aug 22, 2019
See microsoft/git#165 for the rebase onto v2.23.0.windows.1. This PR has a few reactions to that update: A newline was dropped in a message. Remove it from our expectation in a test. Rebase now has unpredictable extra line of whitespace sometimes. However, -q doesn't work in tandem with --merge, but using GIT_QUIET=true makes the output quiet on Windows. Still doesn't work on OSX for some reason, so also ignore the error output and focus on the end-result. Use git rev-parse HEAD^{tree} to ensure we got the same resulting working directory. FileContentsShouldMatch() used to fail if both of the files didn't exist. Something changed in v2.23.0 such that a cherry-pick loses a file in both vanilla Git and VFS for Git, so the test started failing on that path. Relax the condition, since the files do match. KnownGitCommands needs to include restore and switch, resolving #1463. Includes the performance fix for git checkout -b from microsoft/git#183.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is set up to prepare for v2.23.0: its target branch tracks git-for-windows#2276, rebasing VFS for Git's patches on top of it as frequently as possible.
The topic branch
ahead-behind-and-serialized-statuswas partially applied upstream, asjh/status-aheadbehind:The topic branches
show-forced-updatesandforced-updates-after-branch-listwere applied upstream, asds/fetch-disable-force-notice:The topic branches adding trace2 regions were all squashed into a single topic branch:
vfs-trace2.The topic branch
derrickstolee/midx-expire/msmade it into upstream asds/midx-expire-repack:Also, the fix to compile with GCC v9.x made it upstream: 8fc2610 was applied as 9dae4fe.
Finally, 2023a0c was dropped (reverted) via 050b8ff.
The full range-diff relative to vfs-2.22.0:
(As you can see, there are currently 84 patches, i.e. non-merge commits, on top of Git for Windows in
vfs-2.22.0, and only 70 in the candidate forvfs-2.23.0, meaning that we managed to upstream 10 patches (two patches were about trace2'ingprime_cache_tree()and then reverting it, so they don't count 😃.)