Making 404s on gvfs/config silent #503
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When two `git maintenance` processes try to write the `.plist` file, we need to help them with serializing their efforts. The 150ms time-out value was determined from thin air. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
On macOS, we use launchctl to manage the background maintenance schedule. This uses a set of .plist files to describe the schedule, but these files are also registered with 'launchctl bootstrap'. If multiple 'git maintenance start' commands run concurrently, then they can collide replacing these schedule files and registering them with launchctl. To avoid extra launchctl commands, do a check for the .plist files on disk and check if they are registered using 'launchctl list <name>'. This command will return with exit code 0 if it exists, or exit code 113 if it does not. We can test this behavior using the GIT_TEST_MAINT_SCHEDULER environment variable. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
While using the reset --stdin feature on windows path added may have a \r at the end of the path that wasn't getting removed so didn't match the path in the index and wasn't reset. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Signed-off-by: Saeed Noursalehi <sanoursa@microsoft.com>
Since we really want to be based on a `.vfs.*` tag, let's make sure that there was a new-enough one, i.e. one that agrees with the first three version numbers of the recorded default version. This prevents e.g. v2.22.0.vfs.0.<some-huge-number>.<commit> from being used when the current release train was not yet tagged. It is important to get the first three numbers of the version right because e.g. Scalar makes decisions depending on those (such as assuming that the `git maintenance` built-in is not available, even though it actually _is_ available). Signed-off-by: Johannes Schindelin <johasc@microsoft.com>
This adds hard-coded call to GVFS.hooks.exe before and after each Git command runs. To make sure that this is only called on repositories cloned with GVFS, we test for the tell-tale .gvfs. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
Suggested by Ben Peart. Signed-off-by: Johannes Schindelin <johasc@microsoft.com>
This header file will accumulate GVFS-specific definitions. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Signed-off-by: Alejandro Pauly <alpauly@microsoft.com>
This does not do anything yet. The next patches will add various values for that config setting that correspond to the various features offered/required by GVFS. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
We need to respect that config setting even if we already know that we have a repository, but have not yet read the config. The regression test was written by Alejandro Pauly. Signed-off-by: Johannes Schindelin <johasc@microsoft.com>
This takes a substantial amount of time, and if the user is reasonably sure that the files' integrity is not compromised, that time can be saved. Git no longer verifies the SHA-1 by default, anyway. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
When using the sparse-checkout feature, the file might not be on disk because the skip-worktree bit is on. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Signed-off-by: Kevin Willford <kewillf@microsoft.com>
When using the sparse-checkout feature git should not write to the working directory for files with the skip-worktree bit on. With the skip-worktree bit on the file may or may not be in the working directory and if it is not we don't want or need to create it by calling checkout_entry. There are two callers of checkout_target. Both of which check that the file does not exist before calling checkout_target. load_current which make a call to lstat right before calling checkout_target and check_preimage which will only run checkout_taret it stat_ret is less than zero. It sets stat_ret to zero and only if !stat->cached will it lstat the file and set stat_ret to something other than zero. This patch checks if skip-worktree bit is on in checkout_target and just returns so that the entry doesn't not end up in the working directory. This is so that apply will not create a file in the working directory, then update the index but not keep the working directory up to date with the changes that happened in the index. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Prevent the sparse checkout to delete files that were marked with skip-worktree bit and are not in the sparse-checkout file. This is because everything with the skip-worktree bit turned on is being virtualized and will be removed with the change of HEAD. There was only one failing test when running with these changes that was checking to make sure the worktree narrows on checkout which was expected since we would no longer be narrowing the worktree. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
When using the sparse checkout feature the git reset command will add entries to the index that will have the skip-worktree bit off but will leave the working directory empty. File data is lost because the index version of the files has been changed but there is nothing that is in the working directory. This will cause the next status call to show either deleted for files modified or deleting or nothing for files added. The added files should be shown as untracked and modified files should be shown as modified. To fix this when the reset is running if there is not a file in the working directory and if it will be missing with the new index entry or was not missing in the previous version, we create the previous index version of the file in the working directory so that status will report correctly and the files will be availble for the user to deal with. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
While performing a fetch with a virtual file system we know that there will be missing objects and we don't want to download them just because of the reachability of the commits. We also don't want to download a pack file with commits, trees, and blobs since these will be downloaded on demand. This flag will skip the first connectivity check and by returning zero will skip the upload pack. It will also skip the second connectivity check but continue to update the branches to the latest commit ids. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Ensure all filters and EOL conversions are blocked when running under GVFS so that our projected file sizes will match the actual file size when it is hydrated on the local machine. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
This code change makes sure that the config value for core_gvfs is always loaded before checking it. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
The two existing members of the run_hook*() family, run_hook_ve() and run_hook_le(), are good for callers that know the precise number of parameters already. Let's introduce a new sibling that takes a strvec for callers that want to pass a variable number of parameters. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The idea is to allow blob objects to be missing from the local repository, and to load them lazily on demand. After discussing this idea on the mailing list, we will rename the feature to "lazy clone" and work more on this. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
String formatting can be a performance issue when there are hundreds of thousands of trees. Change to stop using the strbuf_addf and just add the strings or characters individually. There are a limited number of modes so added a switch for the known ones and a default case if something comes through that are not a known one for git. In one scenario regarding a huge worktree, this reduces the time required for a `git checkout <branch>` from 44 seconds to 38 seconds, i.e. it is a non-negligible performance improvement. Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Hydrate missing loose objects in check_and_freshen() when running virtualized. Add test cases to verify read-object hook works when running virtualized. This hook is called in check_and_freshen() rather than check_and_freshen_local() to make the hook work also with alternates. Helped-by: Kevin Willford <kewillf@microsoft.com> Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
The following commands and options are not currently supported when working in a GVFS repo. Add code to detect and block these commands from executing. 1) fsck 2) gc 4) prune 5) repack 6) submodule 8) update-index --split-index 9) update-index --index-version (other than 4) 10) update-index --[no-]skip-worktree 11) worktree Signed-off-by: Ben Peart <benpeart@microsoft.com>
…ng objects This commit converts the existing read_object hook proc model for downloading missing blobs to use a background process that is started the first time git encounters a missing blob and stays running until git exits. Git and the read-object process communicate via stdin/stdout and a versioned, capability negotiated interface as documented in Documentation/technical/read-object-protocol.txt. The advantage of this over the previous hook proc is that it saves the overhead of spawning a new hook process for every missing blob. The model for the background process was refactored from the recent git LFS work. I refactored that code into a shared module (sub-process.c/h) and then updated convert.c to consume the new library. I then used the same sub-process module when implementing the read-object background process. The read-object hook feature was designed before the SHA-256 support was even close to be started. As a consequence, its protocol hard-codes the key `sha1`, even if we now also support SHA-256 object IDs. Technically, this is wrong, and probably the best way forward would be to rename the key to `oid` (or `sha256`, but that is less future-proof). However, there are existing setups out there, with existing read-object hooks that most likely have no idea what to do with `oid` requests. So let's leave the key as `sha1` for the time being, even if it will be technically incorrect in SHA-256 repositories. Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
We found a user who had set "core.gvfs = false" in their global config. This should not have been necessary, but it also should not have caused a problem. However, it did. The reason is that gvfs_load_config_value() is called from config.c when reading config key/value pairs from all the config files. The local config should override the global config, and this is done by config.c reading the global config first then reading the local config. However, our logic only allowed writing the core_gvfs variable once. Put the guards against multiple assignments of core_gvfs into gvfs_config_is_set() instead, because that will fix the problem _and_ keep multiple calls to gvfs_config_is_set() from slowing down. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
If we are going to write an object there is no use in calling the read object hook to get an object from a potentially remote source. We would rather just write out the object and avoid the potential round trip for an object that doesn't exist. This change adds a flag to the check_and_freshen() and freshen_loose_object() functions' signatures so that the hook is bypassed when the functions are called before writing loose objects. The check for a local object is still performed so we don't overwrite something that has already been written to one of the objects directories. Based on a patch by Kevin Willford. Signed-off-by: Johannes Schindelin <johasc@microsoft.com>
This allows setting the GVFS-enabled cache server, or listing the one(s) associated with the remote repository. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In Scalar's functional tests, we do not do anything with authentication. Therefore, we do want to avoid accessing the `vsts/info` endpoint because it requires authentication even on otherwise public repositories. Let's introduce the environment variable `SCALAR_TEST_SKIP_VSTS_INFO` which can be set to `true` to simply skip that step (and force the `url_*` style repository IDs instead of `id_*` whenever possible). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
… when plist is registered
This adds the bare minimum to compile the `scalar` executable. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This implements the subcommands `register`, `unregister` and `list`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This implements `clone`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This implements `scalar run`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This allows fixing settings after a Scalar upgrade, or after botching the enlistments configuration. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This implements the `diagnose` subcommand. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This is a convenient shortcut for `scalar unregister <enlistment> && rm -rf <enlistment>`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This implements the `version` command for backwards-compatibility with the .NET version of Scalar. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
For convenience, this ports the `git -c <key>=<value> -C <dir> <command>` functionality to `scalar`, allowing config settings and workig directories to be set for the duration of the Scalar invocation. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Make `contrib/scalar/` work nicely with the built-in FSMonitor. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Document the whole thing. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Allow concurrent `scalar register` and `scalar unregister` calls to be more collaborative when trying to lock the global Git config at the very same time. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch offers to include `scalar` in a regular Git build, simply by setting `INCLUDE_SCALAR=YesPlease`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
For ease of development, build and test `scalar`, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Prepare `scalar` to use the GVFS protocol instead of partial clone (required to support Azure Repos). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Adjust the Scalar Functional Tests CI run to adjust to Scalar/C. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Apparently, there _might_ be left-over Git processes even outside of using the built-in FSMonitor... (for example, see https://github.com/microsoft/git/pull/366/checks?check_run_id=2786318460#step:19:20) Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
As Scalar/C ignores `feature.scalar`, let's not run essentially the same thing twice. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Integrate Scalar (ported to C) into vfs-2.32.0
Let's only use the `ours` and `theirs` variables if they are initialized, i.e. when `stat_tracking_info()` returned without error. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
… PRs against features/* I'm running PRs pointing at `features/sparse-index` but I also want the Scalar functional tests to run for those PRs.
…-vars Avoid using uninitialized variables in `format_tracking_info()`
Due to how we are calling 'git index-pack' and piping stdout, I found an instance where our 'git index-pack' subcommand is timing out. Since we are using pipe_command() to get the stdout and use that hash to create a new filename, the 'ip.stderr = -1' line breaks the expected behavior, especially when providing '-v'. Let's be more careful about stderr when cloning. The case I saw was for using the prefetch endpoint on Azure Repos on a repo that doesn't have configured cache servers. The pack-file was over 1GB so the progress indicator reliably hit a problem. Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
… silent Due to how we are calling 'git index-pack' and piping stdout, I found an instance where our 'git index-pack' subcommand is timing out. Since we are using pipe_command() to get the stdout and use that hash to create a new filename, the 'ip.stderr = -1' line breaks the expected behavior, especially when providing '-v'. Let's be more careful about stderr when cloning. The case I saw was for using the prefetch endpoint on Azure Repos on a repo that doesn't have configured cache servers. The pack-file was over 1GB so the progress indicator reliably hit a problem.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is supposed to be a single commit 92c8e11 to address #384. No idea why github thinks it's 205 commits.
I made it before finding https://github.com/microsoft/git/blob/vfs-2.36.1/CONTRIBUTING.md, so I can abandon this and follow that process instead, but this was just a quick way for me to test out codespaces.
Not sure if we want all 404s suppressed, or only a subset, so let me know if something slightly different would be better.
Thanks for taking the time to contribute to Git!
This fork contains changes specific to monorepo scenarios. If you are an
external contributor, then please detail your reason for submitting to
this fork:
GVFS Protocol.