Rebase to v2.38.0-rc1 #4032

This simplifies the workflow dramatically, as well as accelerating it because the Action recently learned to use the partial clone/parallel checkout features of Git. Note that we have to reinstate that `/usr/bin/git` hack (a shell script that simply redirects to `/mingw64/bin/git.exe`) in the `pkg` job manually, since we no longer cache the `build-installers` artifact _after_ installing that hack in `bundle-artifacts`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Dennis Ameling <dennis@dennisameling.com>

Correct some wording and inform users regarding the Visual Studio changes (from V16.6) to the default generator. Subsequent commits ensure that Git for Windows can be directly opened in modern Visual Studio without needing special configuration of the CMakeLists settings. It appeares that internally Visual Studio creates it's own version of the .sln file (etc.) for extension tools that expect them. The large number of references below document the shifting of Visual Studio default and CMake setting options. refs: https://docs.microsoft.com/en-us/search/?scope=C%2B%2B&view=msvc-150&terms=Ninja 1. https://docs.microsoft.com/en-us/cpp/linux/cmake-linux-configure?view=msvc-160 (note the linux bit) "In Visual Studio 2019 version 16.6 or later ***, Ninja is the default generator for configurations targeting a remote system or WSL. For more information, see this post on the C++ Team Blog [https://devblogs.microsoft.com/cppblog/linux-development-with-visual-studio-first-class-support-for-gdbserver-improved-build-times-with-ninja-and-updates-to-the-connection-manager/]. For more information about these settings, see CMakeSettings.json reference [https://docs.microsoft.com/en-us/cpp/build/cmakesettings-reference?view=msvc-160]." 2. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160 "CMake supports two files that allow users to specify common configure, build, and test options and share them with others: CMakePresets.json and CMakeUserPresets.json." " Both files are supported in Visual Studio 2019 version 16.10 or later. ***" 3. https://devblogs.microsoft.com/cppblog/linux-development-with-visual-studio-first-class-support-for-gdbserver-improved-build-times-with-ninja-and-updates-to-the-connection-manager/ " Ninja has been the default generator (underlying build system) for CMake configurations targeting Windows for some time***, but in Visual Studio 2019 version 16.6 Preview 3*** we added support for Ninja on Linux." 4. https://docs.microsoft.com/en-us/cpp/build/cmakesettings-reference?view=msvc-160 " `generator`: specifies CMake generator to use for this configuration. May be one of: Visual Studio 2019 only: Visual Studio 16 2019 Visual Studio 16 2019 Win64 Visual Studio 16 2019 ARM Visual Studio 2017 and later: Visual Studio 15 2017 Visual Studio 15 2017 Win64 Visual Studio 15 2017 ARM Visual Studio 14 2015 Visual Studio 14 2015 Win64 Visual Studio 14 2015 ARM Unix Makefiles Ninja Because Ninja is designed for fast build speeds instead of flexibility and function, it is set as the default. However, some CMake projects may be unable to correctly build using Ninja. If this occurs, you can instruct CMake to generate Visual Studio projects instead. To specify a Visual Studio generator in Visual Studio 2017, open the settings editor from the main menu by choosing CMake | Change CMake Settings. Delete "Ninja" and type "V". This activates IntelliSense, which enables you to choose the generator you want." "To specify a Visual Studio generator in Visual Studio 2019, right-click on the CMakeLists.txt file in Solution Explorer and choose CMake Settings for project > Show Advanced Settings > CMake Generator. When the active configuration specifies a Visual Studio generator, by default MSBuild.exe is invoked with` -m -v:minimal` arguments." 5. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#enable-cmakepresetsjson-integration-in-visual-studio-2019 "Enable CMakePresets.json integration in Visual Studio 2019 CMakePresets.json integration isn't enabled by default in Visual Studio 2019. You can enable it for all CMake projects in Tools > Options > CMake > General: (tick a box)" ... see more. 6. https://docs.microsoft.com/en-us/cpp/build/cmakesettings-reference?view=msvc-140 (whichever v140 is..) "CMake projects are supported in Visual Studio 2017 and later." 7. https://docs.microsoft.com/en-us/cpp/overview/what-s-new-for-cpp-2017?view=msvc-150 "Support added for the CMake Ninja generator." 8. https://docs.microsoft.com/en-us/cpp/overview/what-s-new-for-cpp-2017?view=msvc-150#cmake-support-via-open-folder "CMake support via Open Folder Visual Studio 2017 introduces support for using CMake projects without converting to MSBuild project files (.vcxproj). For more information, see CMake projects in Visual Studio[https://docs.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-150]. Opening CMake projects with Open Folder automatically configures the environment for C++ editing, building, and debugging." ... +more! 9. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#supported-cmake-and-cmakepresetsjson-versions "Visual Studio reads and evaluates CMakePresets.json and CMakeUserPresets.json itself and doesn't invoke CMake directly with the --preset option. So, CMake version 3.20 or later isn't strictly required when you're building with CMakePresets.json inside Visual Studio. We recommend using CMake version 3.14 or later." 10. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#enable-cmakepresetsjson-integration-in-visual-studio-2019 "If you don't want to enable CMakePresets.json integration for all CMake projects, you can enable CMakePresets.json integration for a single CMake project by adding a CMakePresets.json file to the root of the open folder. You must close and reopen the folder in Visual Studio to activate the integration. 11. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#default-configure-presets ***(doesn't actually say which version..) "Default Configure Presets If no CMakePresets.json or CMakeUserPresets.json file exists, or if CMakePresets.json or CMakeUserPresets.json is invalid, Visual Studio will fall back*** on the following default Configure Presets: Windows example JSON { "name": "windows-default", "displayName": "Windows x64 Debug", "description": "Sets Ninja generator, compilers, x64 architecture, build and install directory, debug build type", "generator": "Ninja", "binaryDir": "${sourceDir}/out/build/${presetName}", "architecture": { "value": "x64", "strategy": "external" }, "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug", "CMAKE_INSTALL_PREFIX": "${sourceDir}/out/install/${presetName}" }, "vendor": { "microsoft.com/VisualStudioSettings/CMake/1.0": { "hostOS": [ "Windows" ] } } }, " Signed-off-by: Philip Oakley <philipoakley@iee.email>

This is needed for the next commit, where we add HOST_CPU to the CMake invocation. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The CMakeSettings.json file is tool generated. Developers may track it should they provide additional settings. Signed-off-by: Philip Oakley <philipoakley@iee.email>

This is needed for the next change, where we add HOST_CPU support to the CMake definition. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The intention of this change is to align with how the top-level git `Makefile` defines its own test target (which also internally calls `$(MAKE) -C t/ all`). This change also ensures the consistency of `make -C contrib/subtree test` with other testing in CI executions (which rely on `$DEFAULT_TEST_TARGET` being defined as `prove`). Signed-off-by: Victoria Dye <vdye@github.com>

In Git-for-Windows, work on using ARM64 has progressed. The commit 2d94b77 (cmake: allow building for Windows/ARM64, 2020-12-04) failed to notice that /compat/vcbuild/vcpkg_install.bat will default to using the "x64-windows" architecture for the vcpkg installation if not set, but CMake is not told of this default. Commit 635b6d9 (vcbuild: install ARM64 dependencies when building ARM64 binaries, 2020-01-31) later updated vcpkg_install.bat to accept an arch (%1) parameter, but retained the default. This default is neccessary for the use case where the project directory is opened directly in Visual Studio, which will find and build a CMakeLists.txt file without any parameters, thus expecting use of the default setting. Also Visual studio will generate internal .sln solution and .vcxproj project files needed for some extension tools. Inform users of the additional .sln/.vcxproj generation. ** How to test: rm -rf '.vs' # remove old visual studio settings rm -rf 'compat/vcbuild/vcpkg' # remove any vcpkg downloads rm -rf 'contrib/buildsystems/out' # remove builds & CMake artifacts with a fresh Visual Studio Community Edition, File>>Open>>(git *folder*) to load the project (which will take some time!). check for successful compilation. The implicit .sln (etc.) are in the hidden .vs directory created by Visual Studio. Signed-off-by: Philip Oakley <philipoakley@iee.email>

Reintroduce the 'core.useBuiltinFSMonitor' config setting (originally added in 0a756b2 (fsmonitor: config settings are repository-specific, 2021-03-05)) after its removal from the upstream version of FSMonitor. Upstream, the 'core.useBuiltinFSMonitor' setting was rendered obsolete by "overloading" the 'core.fsmonitor' setting to take a boolean value. However, several applications (e.g., 'scalar') utilize the original config setting, so it should be preserved for a deprecation period before complete removal: * if 'core.fsmonitor' is a boolean, the user is correctly using the new config syntax; do not use 'core.useBuiltinFSMonitor'. * if 'core.fsmonitor' is unspecified, use 'core.useBuiltinFSMonitor'. * if 'core.fsmonitor' is a path, override and use the builtin FSMonitor if 'core.useBuiltinFSMonitor' is 'true'; otherwise, use the FSMonitor hook indicated by the path. Additionally, for this deprecation period, advise users to switch to using 'core.fsmonitor' to specify their use of the builtin FSMonitor. Signed-off-by: Victoria Dye <vdye@github.com>

As mentioned in the Makefile and CMakeLists.txt: "When cross-compiling, define HOST_CPU as the canonical name of the CPU on which the built Git will run (for instance "x86_64")" This commit sets the HOST_CPU variable since Git for Windows arm64 is cross-compiled from an amd64 host. Signed-off-by: Dennis Ameling <dennis@dennisameling.com>

We now have a GitHub Action to download and cache Azure Pipelines artifacts (such as the `vcpkg` artifacts), hiding gnarly internals, and also providing some robustness against network glitches. Let's use it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

As reported in newren/git-filter-repo#225, it looks like 99 bytes is not really sufficient to represent e.g. the full path to Python when installed via Windows Store (and this path is used in the hasb bang line when installing scripts via `pip`). Let's increase it to what is probably the maximum sensible path size: MAX_PATH. This makes `parse_interpreter()` in line with what `lookup_prog()` handles. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Vilius Šumskas <vilius@sumskas.eu>

We used to have that `make vcxproj` hack, but a hack it is. In the meantime, we have a much cleaner solution: using CMake, either explicitly, or even more conveniently via Visual Studio's built-in CMake support (simply open Git's top-level directory via File>Open>Folder...). Let's let the `README` reflect this. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This adds support for a new http.sslAutoClientCert config value. In cURL 7.77 or later the schannel backend does not automatically send client certificates from the Windows Certificate Store anymore. This config value is only used if http.sslBackend is set to "schannel", and can be used to opt in to the old behavior and force cURL to send client certificates. This fixes git-for-windows#3292 Signed-off-by: Pascal Muller <pascalmuller@gmail.com>

Because `git subtree` (unlike most other `contrib` modules) is included as part of the standard release of Git for Windows, its stability should be verified as consistently as it is for the rest of git. By including the `git subtree` tests in the CI workflow, these tests are as much of a gate to merging and indicator of stability as the standard test suite. Signed-off-by: Victoria Dye <vdye@github.com>

Ensure key CMake option values are part of the CMake output to facilitate user support when tool updates impact the wider CMake actions, particularly ongoing 'improvements' in Visual Studio. These CMake displays perform the same function as the build-options.txt provided in the main Git for Windows. CMake is already chatty. The setting of CMAKE_EXPORT_COMPILE_COMMANDS is also reported. Include the environment's CMAKE_EXPORT_COMPILE_COMMANDS value which may have been propogated to CMake's internal value. Testing the CMAKE_EXPORT_COMPILE_COMMANDS processing can be difficult in the Visual Studio environment, as it may be cached in many places. The 'environment' may include the OS, the user shell, CMake's own environment, along with the Visual Studio presets and caches. See previous commit for arefacts that need removing for a clean test. Signed-off-by: Philip Oakley <philipoakley@iee.email>

In Git for Windows, `has_symlinks` is set to 0 by default. Therefore, we need to parse the config setting `core.symlinks` to know if it has been set to `true`. In `git init`, we must do that before copying the templates because they might contain symbolic links. Even if the support for symbolic links on Windows has not made it to upstream Git yet, we really should make sure that all the `core.*` settings are parsed before proceeding, as they might very well change the behavior of `git init` in a way the user intended. This fixes git-for-windows#3414 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

On LLP64 systems, such as Windows, the size of `long`, `int`, etc. is only 32 bits (for backward compatibility). Git's use of `unsigned long` for file memory sizes in many places, rather than size_t, limits the handling of large files on LLP64 systems (commonly given as `>4GB`). Provide a minimum test for handling a >4GB file. The `hash-object` command, with the `--literally` and without `-w` option avoids writing the object, either loose or packed. This avoids the code paths hitting the `bigFileThreshold` config test code, the zlib code, and the pack code. Subsequent patches will walk the test's call chain, converting types to `size_t` (which is larger in LLP64 data models) where appropriate. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The previous commit adds a test that demonstrates a problem in the `hash-object --literally` command, manifesting in an unnecessary file size limit on systems using the LLP64 data model (which includes Windows). Walking the affected code path is `cmd_hash_object()` >> `hash_fd()` >> `hash_literally()` >> `hash_object_file_literally()`. The function `hash_object_file_literally()` is the first with a file length parameter (via a mem buffer). This commit changes the type of that parameter to the LLP64 compatible `size_t` type. There are no other uses of the function. The `strbuf` type is already `size_t` compatible. Note: The hash-object test does not yet pass. Subsequent commits will continue to walk the call tree's lower level functions to identify further fixes. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This is another fall-out of the recent refactoring flurry. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Continue walking the code path for the >4GB `hash-object --literally` test. The `hash_object_file_literally()` function internally uses both `hash_object_file()` and `write_object_file_prepare()`. Both function signatures use `unsigned long` rather than `size_t` for the mem buffer sizes. Use `size_t` instead, for LLP64 compatibility. While at it, convert those function's object's header buffer length to `size_t` for consistency. The value is already upcast to `uintmax_t` for print format compatibility. Note: The hash-object test still does not pass. A subsequent commit continues to walk the call tree's lower level hash functions to identify further fixes. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This fixes the build after 7bc341e (git-compat-util: add a test balloon for C99 support, 2021-12-01). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Continue walking the code path for the >4GB `hash-object --literally` test to the hash algorithm step for LLP64 systems. This patch lets the SHA1DC code use `size_t`, making it compatible with LLP64 data models (as used e.g. by Windows). The interested reader of this patch will note that we adjust the signature of the `git_SHA1DCUpdate()` function without updating _any_ call site. This certainly puzzled at least one reviewer already, so here is an explanation: This function is never called directly, but always via the macro `platform_SHA1_Update`, which is usually called via the macro `git_SHA1_Update`. However, we never call `git_SHA1_Update()` directly in `struct git_hash_algo`. Instead, we call `git_hash_sha1_update()`, which is defined thusly: static void git_hash_sha1_update(git_hash_ctx *ctx, const void *data, size_t len) { git_SHA1_Update(&ctx->sha1, data, len); } i.e. it contains an implicit downcast from `size_t` to `unsigned long` (before this here patch). With this patch, there is no downcast anymore. With this patch, finally, the t1007-hash-object.sh "files over 4GB hash literally" test case is fixed. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This is now passed by default, ever since 6a8cbc4 (developer: enable pedantic by default, 2021-09-03). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Just like the `hash-object --literally` code path, the `--stdin` code path also needs to use `size_t` instead of `unsigned long` to represent memory sizes, otherwise it would cause problems on platforms using the LLP64 data model (such as Windows). To limit the scope of the test case, the object is explicitly not written to the object store, nor are any filters applied. The `big` file from the previous test case is reused to save setup time; To avoid relying on that side effect, it is generated if it does not exist (e.g. when running via `sh t1007-*.sh --long --run=1,41`). Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

To complement the `--stdin` and `--literally` test cases that verify that we can hash files larger than 4GB on 64-bit platforms using the LLP64 data model, here is a test case that exercises `hash-object` _without_ any options. Just as before, we use the `big` file from the previous test case if it exists to save on setup time, otherwise generate it. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Derrick Stolee <derrickstolee@github.com>

Since ef8a6c6 (reftable: utility functions, 2021-10-07) we not only have a libreftable, but also a libreftable_test. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

If `feature.experimental` and `feature.manyFiles` are set and the user has not explicitly turned off the builtin FSMonitor, we now start the built-in FSMonitor by default. Only forcing it when UNSET matches the behavior of UPDATE_DEFAULT_BOOL() used for other repo settings. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

To verify that the `clean` side of the `clean`/`smudge` filter code is correct with regards to LLP64 (read: to ensure that `size_t` is used instead of `unsigned long`), here is a test case using a trivial filter, specifically _not_ writing anything to the object store to limit the scope of the test case. As in previous commits, the `big` file from previous test cases is reused if available, to save setup time, otherwise re-generated. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

In the case of Git for Windows (say, in a Git Bash window) running in a Windows Subsystem for Linux (WSL) directory, the GetNamedSecurityInfoW() call in is_path_owned_By_current_side() returns an error code other than ERROR_SUCCESS. This is consistent behavior across this boundary. In these cases, the owner would always be different because the WSL owner is a different entity than the Windows user. The change here is to suppress the error message that looks like this: error: failed to get owner for '//wsl.localhost/...' (1) Before this change, this warning happens for every Git command, regardless of whether the directory is marked with safe.directory. Signed-off-by: Derrick Stolee <derrickstolee@github.com>

For Windows builds >= 15063 set $env:TERM to "xterm-256color" instead of "cygwin" because they have a more capable console system that supports this. Also set $env:COLORTERM="truecolor" if unset. $env:TERM is initialized so that ANSI colors in color.c work, see 29a3963 (Win32: patch Windows environment on startup, 2012-01-15). See git-for-windows#3629 regarding problems caused by always setting $env:TERM="cygwin". This is the same heuristic used by the Cygwin runtime. Signed-off-by: Rafael Kitover <rkitover@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

NtQueryObject under Wine can return a success but fill out no name. In those situations, Wine will set Buffer to NULL, and set result to the sizeof(OBJECT_NAME_INFORMATION). Running a command such as echo "$(git.exe --version 2>/dev/null)" will crash due to a NULL pointer dereference when the code attempts to null terminate the buffer, although, weirdly, removing the subshell or redirecting stdout to a file will not trigger the crash. Code has been added to also check Buffer and Length to ensure the check is as robust as possible due to the current behavior being fragile at best, and could potentially change in the future This code is based on the behavior of NtQueryObject under wine and reactos. Signed-off-by: Christopher Degawa <ccom@randomderp.com>

Atomic append on windows is only supported on local disk files, and it may cause errors in other situations, e.g. network file system. If that is the case, this config option should be used to turn atomic append off. Co-Authored-By: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: 孙卓识 <sunzhuoshi@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Visual Studio 2022 does not like that at all. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The mingw-w64 GCC seems to link implicitly to libwinpthread, which does implement a pthread emulation (that is more complete than Git's). Let's keep preferring Git's. To avoid linker errors where it thinks that the `pthread_self` and the `pthread_create` symbols are defined twice, let's give our version a `win32_` prefix, just like we already do for `pthread_join()`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

These refactorings are really gifts that keep on giving. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We are about to vendor in `mimalloc`'s source code which we will want to include `git-compat-util.h` after defining that constant. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

It is not useful because we do not have any persisted directory anymore, not since dropping our Travis CI support. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This commit imports mimalloc's source code as per v2.0.6, fetched from the tag at https://github.com/microsoft/mimalloc. The .c files are from the src/ subdirectory, and the .h files from the include/ subdirectory. We will subsequently modify the source code to accommodate building within Git's context. Since we plan on using the `mi_*()` family of functions, we skip the C++-specific source code, some POSIX compliant functions to interact with mimalloc, and the code that wants to support auto-magic overriding of the `malloc()` function (mimalloc-new-delete.h, alloc-posix.c, mimalloc-override.h, alloc-override.c, alloc-override-osx.c, alloc-override-win.c and static.c). To appease the `check-whitespace` job of Git's Continuous Integration, this commit was washed one time via `git rebase --whitespace=fix`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

... so that we can test a MinGit backport in a private repository (with GitHub Actions, minutes and parallel jobs are limited way more than with Azure Pipelines in private repositories). In this commit, we reinstate the exact version of `azure-pipelines.yml` as 6081d38 (ci: retire the Azure Pipelines definition, 2020-04-11) deleted. Naturally, many adjustments are required to make it work again. Some of the changes are actually outside of that file (such as the `runs_on_pool` changes that are needed in the Azure Pipelines part of `ci/lib.sh`) and they were made in the commits leading up to this here commit. However, other adjustments are required in the `azure-pipelines.yml` file itself, and for ease of review (read: to build confidence in those changes) they will be made in subsequent, individual commits that explain the intent, context, implementation and justification like every good commit message should do. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We want to compile mimalloc's source code as part of Git, rather than requiring the code to be built as an external library: mimalloc uses a CMake-based build, which is not necessarily easy to integrate into the flavors of Git for Windows (which will be the main benefitting port). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This is a follow-up to 6c280b4 (ci: remove GETTEXT_POISON jobs, 2021-01-20) after reinstating the Azure Pipeline. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This format is not supported by MSVC runtimes targeted by the MINGW toolchain. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We have `ci/install-dependencies.sh` for that. Incidentally, this avoids the following error in the linux-* jobs: The following packages have unmet dependencies: git-email : Depends: git (< 1:2.25.1-.) but 1:2.35.1-0ppa1~ubuntu20.04.1 is to be installed Recommends: libemail-valid-perl but it is not going to be installed Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Instead, load the `GetProcessMemoryInfo()` function dynamically. When needed. If needed. This is necessary because the start-up cost of Git processes spent on loading dynamic libraries is non-negligible. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We haven't used this feature in ages, we don't actually need to. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Instead, load the `BCryptGenRandom()` function dynamically. When needed. If needed. This is necessary because the start-up cost of Git processes spent on loading dynamic libraries is non-negligible. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…imal-sdk The Azure Pipeline `git-sdk-64-minimal` was retired... Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

By defining `USE_MIMALLOC`, Git can now be compiled with that nicely-fast and small allocator. Note that we have to disable a couple `DEVELOPER` options to build mimalloc's source code, as it makes heavy use of declarations after statements, among other things that disagree with Git's conventions. For example, the `-Wno-array-bounds` flag is needed because in `-O2` builds, trying to call `NtCurrentTeb()` (which `_mi_thread_id()` does on Windows) causes the bogus warning about a system header, likely related to https://sourceforge.net/p/mingw-w64/mailman/message/37674519/ and to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99578: C:/git-sdk-64-minimal/mingw64/include/psdk_inc/intrin-impl.h:838:1: error: array subscript 0 is outside array bounds of 'long long unsigned int[0]' [-Werror=array-bounds] 838 | __buildreadseg(__readgsqword, unsigned __int64, "gs", "q") | ^~~~~~~~~~~~~~ Also: The `mimalloc` library uses C11-style atomics, therefore we must require that standard when compiling with GCC if we want to use `mimalloc` (instead of requiring "only" C99). This is what we do in the CMake definition already, therefore this commit does not need to touch `contrib/buildsystems/`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

These many refactorings in Git sure are gifts that keep on giving. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Setting `MIMALLOC_SHOW_STATS` to ask mimalloc to print out something after the process is done is the easiest way to verify that a mimalloc-enabled Git is running. So it better work and not try to write to a Win32 Console when it got a regular file handle instead or, as is the case in Git for Windows' regular Git Bash window, an emulated pseudo terminal. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This is a backport of d051ed7 (.github/workflows/main.yml: run static-analysis on bionic, 2021-02-08) to the Azure Pipeline. When Azure Pipelines' build agents transitioned 'ubuntu-latest' from 18.04 to 20.04, it broke our `static-analysis` job, since Coccinelle isn't available on Ubuntu focal (it is only available in the universe suite). Until Coccinelle can be installed from 20.04's main suite, pin the static-analysis build to run on 18.04, where it can be installed by default. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Whith Windows 2000, Microsoft introduced a flag to the PE header to mark executables as "terminal server aware". Windows terminal servers provide a redirected Windows directory and redirected registry hives when launching legacy applications without this flag set. Since we do not use any INI files in the Windows directory and don't write to the registry, we don't need this additional preparation. Telling the OS that we don't need this should provide slightly improved startup times in terminal server environments. When building for supported Windows Versions with MSVC the /TSAWARE linker flag is automatically set, but MinGW requires us to set the --tsaware flag manually. This partially addresses git-for-windows#3935. Signed-off-by: Matthias Aßhauer <mha1993@live.de>

Thorough benchmarking with repacking a subset of linux.git (the commit history reachable from 93a6fef ([PATCH] fix the SYSCTL=n compilation, 2007-02-28), to be precise) suggest that this allocator is on par, in multi-threaded situations maybe even better than nedmalloc: `git repack -adfq` with mimalloc, 8 threads: 31.166991900 27.576763800 28.712311000 27.373859000 27.163141900 `git repack -adfq` with nedmalloc, 8 threads: 31.915032900 27.149883100 28.244933700 27.240188800 28.580849500 In a different test using GitHub Actions build agents (probably single-threaded, a core-strength of nedmalloc)): `git repack -q -d -l -A --unpack-unreachable=2.weeks.ago` with mimalloc: 943.426 978.500 939.709 959.811 954.605 `git repack -q -d -l -A --unpack-unreachable=2.weeks.ago` with nedmalloc: 995.383 952.179 943.253 963.043 980.468 While these measurements were not executed with complete scientific rigor, as no hardware was set aside specifically for these benchmarks, it shows that mimalloc and nedmalloc perform almost the same, nedmalloc with a bit higher variance and also slightly higher average (further testing suggests that nedmalloc performs worse in multi-threaded situations than in single-threaded ones). In short: mimalloc seems to be slightly better suited for our purposes than nedmalloc. Seeing that mimalloc is developed actively, while nedmalloc ceased to see any updates in eight years, let's use mimalloc on Windows instead. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

These fixes were necessary for Sverre Rabbelier's remote-hg to work, but for some magic reason they are not necessary for the current remote-hg. Makes you wonder how that one gets away with it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This topic branch allows us to specify absolute paths without the drive prefix e.g. when cloning. Example: C:\Users\me> git clone https://github.com/git/git \upstream-git This will clone into a new directory C:\upstream-git, in line with how Windows interprets absolute paths. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Fix gitk (long cmdline)

This topic branch teaches `git clean` to respect NTFS junctions and Unix bind mounts: it will now stop at those boundaries. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

In MSYS2, we have two Python interpreters at our disposal, so we can include the Python stuff in the build. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…eband-config Config option to disable side-band-64k for transport

Make sure `errno` is set when socket operations fail

…ymlinks Do resolve symlinks in `getcwd()`

mingw: fix fatal error working on mapped network drives on Windows

…g-curl clink.pl: fix MSVC compile script to handle libcurl-d.lib

…nction Handle `git add <file>` where <file> traverses an NTFS junction

Allow running Git directly from `C:\Program Files\Git\mingw64\bin\git.exe`

…st-effort Introduce and use the new "best effort" strategy for Secure Channel revoke checking

…-in-vs/master ci: avoid d/f conflict in vs/master

Make use of the new environment variable GIT_ASK_YESNO to support the recently implemented fallback in case unlink, rename or rmdir fail for files in use on Windows. The added dialog will present a yes/no question to the the user which will currently be used by the windows compat layer to let the user retry a failed file operation. Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>

The text wrapping seems to be aligned to the right side of the Yes button, leaving an awful lot of empty space. Let's try to counter this by using pixel units. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Since v2.9.0, Git knows about the config variable core.hookspath that allows overriding the path to the directory containing the Git hooks. Since v2.10.0, the `--git-path` option respects that config variable, too, so we may just as well use that command. For Git versions older than v2.5.0 (which was the first version to support the `--git-path` option for the `rev-parse` command), we simply fall back to the previous code. This fixes git-for-windows#1755 Initial-patch-by: Philipp Gortan <philipp@gortan.org> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Assumes file names in git tree objects are UTF-8 encoded. On most unix systems, the system encoding (and thus the TCL system encoding) will be UTF-8, so file names will be displayed correctly. On Windows, it is impossible to set the system encoding to UTF-8. Changing the TCL system encoding (via 'encoding system ...', e.g. in the startup code) is explicitly discouraged by the TCL docs. Change gitk functions dealing with file names to always convert from and to UTF-8. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Git for Windows now ships with the new Git icon from git-scm.com. Use that icon file if it exists instead of the old procedurally drawn one. This patch was sent upstream but so far no decision on its inclusion was made, so commit it to our fork. Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>

"Question?" is maybe not the most informative thing to ask. In the absence of better information, it is the best we can do, of course. However, Git for Windows' auto updater just learned the trick to use git-gui--askyesno to ask the user whether to update now or not. And in this scripted scenario, we can easily pass a command-line option to change the window title. So let's support that with the new `--title <title>` option. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Tcl/Tk 8.6 introduced new events for the cursor left/right keys and apparently changed the behavior of the previous event. Let's work around that by using the new events when we are running with Tcl/Tk 8.6 or later. This fixes git-for-windows#495 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

For additional GUI goodness. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Let's try to address git-for-windows#1755 this way. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

When using remotes (with git-flow especially), the remote reference names are almost always wordwrapped in the "list references" window because it's somewhat narrow by default. It's possible to resize it with a mouse, but it's annoying to have to do this every time, especially on Windows 10, where the window border seems to be only one (1) pixel wide, thus making the grabbing of the window border tricky. Signed-off-by: James J. Raden <james.raden@gmail.com>

These changes are necessary to support better Git for Windows' new auto-update feature. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The Git project followed suite and added their Code of Conduct, based on the Contributors' Covenant v1.4. We edit it slightly to reflect Git for Windows' particulars. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Getting started contributing to Git can be difficult on a Windows machine. CONTRIBUTING.md contains a guide to getting started, including detailed steps for setting up build tools, running tests, and submitting patches to upstream. [includes an example by Pratik Karki how to submit v2, v3, v4, etc.] Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Includes touch-ups by 마누엘, Philip Oakley and 孙卓识. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

With improvements by Clive Chan, Adric Norris, Ben Bodenmiller and Philip Oakley. Helped-by: Clive Chan <cc@clive.io> Helped-by: Adric Norris <landstander668@gmail.com> Helped-by: Ben Bodenmiller <bbodenmiller@hotmail.com> Helped-by: Philip Oakley <philipoakley@iee.org> Signed-off-by: Brendan Forster <brendan@github.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Git for Windows accepts pull requests; Core Git does not. Therefore we need to adjust the template (because it only matches core Git's project management style, not ours). Also: direct Git for Windows enhancements to their contributions page, space out the text for easy reading, and clarify that the mailing list is plain text, not HTML. Signed-off-by: Philip Oakley <philipoakley@iee.org> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The sentiment bot will help detect when things get too heated. Hopefully. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Git documentation refers to $HOME and $XDG_CONFIG_HOME often, but does not specify how or where these values come from on Windows where neither is set by default. The new documentation reflects the behavior of setup_windows_environment() in compat/mingw.c. Signed-off-by: Alejandro Barreto <alejandro.barreto@ni.com>

The `--stdin` option was a well-established paradigm in other commands, therefore we implemented it in `git reset` for use by Visual Studio. Unfortunately, upstream Git decided that it is time to introduce `--pathspec-from-file` instead. To keep backwards-compatibility for some grace period, we therefore reinstate the `--stdin` option on top of the `--pathspec-from-file` option, but mark it firmly as deprecated. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

On Windows, an absolute POSIX path needs to be turned into a Windows one. We used to interpret paths starting with a single `/` as relative to the runtime-prefix, but now these need to be prefixed with `%(prefix)/`. Let's warn for now, but still handle it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Rather than using private IFTTT Applets that send mails to this maintainer whenever a new version of a Git for Windows component was released, let's use the power of GitHub workflows to make this process publicly visible. This workflow monitors the Atom/RSS feeds, and opens a ticket whenever a new version was released. Note: Bash sometimes releases multiple patched versions within a few minutes of each other (i.e. 5.1p1 through 5.1p4, 5.0p15 and 5.0p16). The MSYS2 runtime also has a similar system. We can address those patches as a group, so we shouldn't get multiple issues about them. Note further: We're not acting on newlib releases, OpenSSL alphas, Perl release candidates or non-stable Perl releases. There's no need to open issues about them. Co-authored-by: Matthias Aßhauer <mha1993@live.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This is the recommended way on GitHub to describe policies revolving around security issues and about supported versions. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This topic branch allows `add -p` and `add -i` with a large number of files. It is kind of a hack that was never really meant to be upstreamed. Let's see if we can do better in the built-in `add -p`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

git add -i: handle CR/LF line endings in the interactive input

Rationalize line endings for scissors-cleanup

…xtra_info t/t0014: fix: eliminate additional lines from trace

This merges the current version of the patch that tries to address Git GUI's problems with intent-to-add files. This patch will likely be improved substantially before it is merged into Git GUI's main branch, but we want to have _something_ resembling a fix already in Git for Windows v2.29.0. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…e-and-headless Include Windows-specific maintenance and headless-git

Vcpkg Install: detect lack of working Git, and note possible vcpkg time outs

…m64-support Windows arm64 support

cmake(): allow setting HOST_CPU for cross-compilation

…ifacts-g4w ci: cache vcpkg artifacts

…ength-of-interpreter-path mingw: allow for longer paths in `parse_interpreter()`

…ter-anymore Let the documentation reflect that there is no vs/master anymore

…t-automatically-sending-client-certificate http: Add support for enabling automatic sending of SSL client certificate

…tests Add `contrib/subtree` test execution to CI builds

Make Git for Windows start builds in modern Visual Studio

…mlinks-earlier init: respect core.symlinks before copying the templates

Begin `unsigned long`->`size_t` conversion to support large files on Windows

…irectory` The first three commits are rebased versions of those in gitgitgadget#1215. These allow the following: 1. Fix `git config --global foo.bar <path>` from allowing the `<path>`. As a bonus, users with a config value starting with `/` will not get a warning about "old-style" paths needing a "`%(prefix)/`". 2. When in WSL, the path starts with `/` so it needs to be interpolated properly. Update the warning to include `%(prefix)/` to get the right value for WSL users. (This is specifically for using Git for Windows from Git Bash, but in a WSL directory.) 3. When using WSL, the ownership check fails and reports an error message. This is noisy, and happens even if the user has marked the path with `safe.directory`. Remove that error message.

mingw: set $env:TERM=xterm-256color for newer OSes

winansi: check result before using Name for pty

Fix append failure issue under remote directories git-for-windows#2753 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We will use them in the upcoming "FSCache" patches (to accelerate sequential lstat() calls). Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Backport a couple fixes to make the CI build run again (so much for reproducible builds...). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Move opendir down in preparation for the next patch. Signed-off-by: Karsten Blees <blees@dcon.de>

MinGW: link as terminal server aware

Emulating the POSIX `dirent` API on Windows via `FindFirstFile()`/`FindNextFile()` is pretty staightforward, however, most of the information provided in the `WIN32_FIND_DATA` structure is thrown away in the process. A more sophisticated implementation may cache this data, e.g. for later reuse in calls to `lstat()`. Make the `dirent` implementation pluggable so that it can be switched at runtime, e.g. based on a config option. Define a base DIR structure with pointers to `readdir()`/`closedir()` that match the `opendir()` implementation (similar to vtable pointers in Object-Oriented Programming). Define `readdir()`/`closedir()` so that they call the function pointers in the `DIR` structure. This allows to choose the `opendir()` implementation on a call-by-call basis. Make the fixed-size `dirent.d_name` buffer a flex array, as `d_name` may be implementation specific (e.g. a caching implementation may allocate a `struct dirent` with _just_ the size needed to hold the `d_name` in question). Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Switch Git for Windows to using mimalloc instead of nedmalloc

Emulating the POSIX lstat API on Windows via GetFileAttributes[Ex] is quite slow. Windows operating system APIs seem to be much better at scanning the status of entire directories than checking single files. A caching implementation may improve performance by bulk-reading entire directories or reusing data obtained via opendir / readdir. Make the lstat implementation pluggable so that it can be switched at runtime, e.g. based on a config option. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This is the branch thicket of patches in Git for Windows that are considered ready for upstream. To keep them in a ready-to-submit shape, they are kept as close to the beginning of the branch thicket as possible.

Add a macro to mark code sections that only read from the file system, along with a config option and documentation. This facilitates implementation of relatively simple file system level caches without the need to synchronize with the file system. Enable read-only sections for 'git status' and preload_index. Signed-off-by: Karsten Blees <blees@dcon.de>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Teach read_directory_recursive() and add_excludes() to be aware of optional fscache and avoid trying to open() and fstat() non-existant ".gitignore" files in every directory in the worktree. The current code in add_excludes() calls open() and then fstat() for a ".gitignore" file in each directory present in the worktree. Change that when fscache is enabled to call lstat() first and if present, call open(). This seems backwards because both lstat needs to do more work than fstat. But when fscache is enabled, fscache will already know if the .gitignore file exists and can completely avoid the IO calls. This works because of the lstat diversion to mingw_lstat when fscache is enabled. This reduced status times on a 350K file enlistment of the Windows repo on a NVMe SSD by 0.25 seconds. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

Checking the work tree status is quite slow on Windows, due to slow `lstat()` emulation (git calls `lstat()` once for each file in the index). Windows operating system APIs seem to be much better at scanning the status of entire directories than checking single files. Add an `lstat()` implementation that uses a cache for lstat data. Cache misses read the entire parent directory and add it to the cache. Subsequent `lstat()` calls for the same directory are served directly from the cache. Also implement `opendir()`/`readdir()`/`closedir()` so that they create and use directory listings in the cache. The cache doesn't track file system changes and doesn't plug into any modifying file APIs, so it has to be explicitly enabled for git functions that don't modify the working copy. Note: in an earlier version of this patch, the cache was always active and tracked file system changes via ReadDirectoryChangesW. However, this was much more complex and had negative impact on the performance of modifying git commands such as 'git checkout'. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Teach FSCACHE to remember "not found" directories. This is a performance optimization. FSCACHE is a performance optimization available for Windows. It intercepts Posix-style lstat() calls into an in-memory directory using FindFirst/FindNext. It improves performance on Windows by catching the first lstat() call in a directory, using FindFirst/ FindNext to read the list of files (and attribute data) for the entire directory into the cache, and short-cut subsequent lstat() calls in the same directory. This gives a major performance boost on Windows. However, it does not remember "not found" directories. When STATUS runs and there are missing directories, the lstat() interception fails to find the parent directory and simply return ENOENT for the file -- it does not remember that the FindFirst on the directory failed. Thus subsequent lstat() calls in the same directory, each re-attempt the FindFirst. This completely defeats any performance gains. This can be seen by doing a sparse-checkout on a large repo and then doing a read-tree to reset the skip-worktree bits and then running status. This change reduced status times for my very large repo by 60%. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Make fscache_enabled() function public rather than static. Remove unneeded fscache_is_enabled() function. Change is_fscache_enabled() macro to call fscache_enabled(). is_fscache_enabled() now takes a pathname so that the answer is more precise and mean "is fscache enabled for this pathname", since fscache only stores repo-relative paths and not absolute paths, we can avoid attempting lookups for absolute paths. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

If multiple threads access a directory that is not yet in the cache, the directory will be loaded by each thread. Only one of the results is added to the cache, all others are leaked. This wastes performance and memory. On cache miss, add a future object to the cache to indicate that the directory is currently being loaded. Subsequent threads register themselves with the future object and wait. When the first thread has loaded the directory, it replaces the future object with the result and notifies waiting threads. Signed-off-by: Karsten Blees <blees@dcon.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Teach "add" to use preload-index and fscache features to improve performance on very large repositories. During an "add", a call is made to run_diff_files() which calls check_remove() for each index-entry. This calls lstat(). On Windows, the fscache code intercepts the lstat() calls and builds a private cache using the FindFirst/FindNext routines, which are much faster. Somewhat independent of this, is the preload-index code which distributes some of the start-up costs across multiple threads. We need to keep the call to read_cache() before parsing the pathspecs (and hence cannot use the pathspecs to limit any preload) because parse_pathspec() is using the index to determine whether a pathspec is, in fact, in a submodule. If we would not read the index first, parse_pathspec() would not error out on a path that is inside a submodule, and t7400-submodule-basic.sh would fail with not ok 47 - do not add files from a submodule We still want the nice preload performance boost, though, so we simply call read_cache_preload(&pathspecs) after parsing the pathspecs. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Fix regression described in: git-for-windows#1392 which was introduced in: git-for-windows@b235337 Problem Symptoms ================ When the user has a .gitignore file that is a symlink, the fscache optimization introduced above caused the stat-data from the symlink, rather that of the target file, to be returned. Later when the ignore file was read, the buffer length did not match the stat.st_size field and we called die("cannot use <path> as an exclude file") Optimization Rationale ====================== The above optimization calls lstat() before open() primarily to ask fscache if the file exists. It gets the current stat-data as a side effect essentially for free (since we already have it in memory). If the file does not exist, it does not need to call open(). And since very few directories have .gitignore files, we can greatly reduce time spent in the filesystem. Discussion of Fix ================= The above optimization calls lstat() rather than stat() because the fscache only intercepts lstat() calls. Calls to stat() stay directed to the mingw_stat() completly bypassing fscache. Furthermore, calls to mingw_stat() always call {open, fstat, close} so that symlinks are properly dereferenced, which adds *additional* open/close calls on top of what the original code in dir.c is doing. Since the problem only manifests for symlinks, we add code to overwrite the stat-data when the path is a symlink. This preserves the effect of the performance gains provided by the fscache in the normal case. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

When I do git fetch, git call file stats under .git/objects for each refs. This takes time when there are many refs. By enabling fscache, git takes file stats by directory traversing and that improved the speed of fetch-pack for repository having large number of refs. In my windows workstation, this improves the time of `git fetch` for chromium repository like below. I took stats 3 times. * With this patch TotalSeconds: 9.9825165 TotalSeconds: 9.1862075 TotalSeconds: 10.1956256 Avg: 9.78811653333333 * Without this patch TotalSeconds: 15.8406702 TotalSeconds: 15.6248053 TotalSeconds: 15.2085938 Avg: 15.5580231 Signed-off-by: Takuto Ikuta <tikuta@chromium.org> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This is retry of git-for-windows#1419. I added flush_fscache macro to flush cached stats after disk writing with tests for regression reported in git-for-windows#1438 and git-for-windows#1442. git checkout checks each file path in sorted order, so cache flushing does not make performance worse unless we have large number of modified files in a directory containing many files. Using chromium repository, I tested `git checkout .` performance when I delete 10 files in different directories. With this patch: TotalSeconds: 4.307272 TotalSeconds: 4.4863595 TotalSeconds: 4.2975562 Avg: 4.36372923333333 Without this patch: TotalSeconds: 20.9705431 TotalSeconds: 22.4867685 TotalSeconds: 18.8968292 Avg: 20.7847136 I confirmed this patch passed all tests in t/ with core_fscache=1. Signed-off-by: Takuto Ikuta <tikuta@chromium.org>

On file systems that support it, this can dramatically speed up operations like add, commit, describe, rebase, reset, rm that would otherwise have to lstat() every file to "re-match" the stat information in the index to that of the file system. On a synthetic repo with 1M files, "git reset" dropped from 52.02 seconds to 14.42 seconds for a savings of 72%. Signed-off-by: Ben Peart <benpeart@microsoft.com>

Use FindFirstFileExW with FindExInfoBasic to avoid forcing NTFS to look up the short name. Also switch to a larger (64K vs 4K) buffer using FIND_FIRST_EX_LARGE_FETCH to minimize round trips to the kernel. In a repo with ~200K files, this drops warm cache status times from 3.19 seconds to 2.67 seconds for a 16% savings. Signed-off-by: Ben Peart <benpeart@microsoft.com>

At the end of the status command, disable and free the fscache so that we don't leak the memory and so that we can dump the fscache statistics. Signed-off-by: Ben Peart <benpeart@microsoft.com>

Add support to fscache to enable running the entire test suite with the fscache enabled. Signed-off-by: Ben Peart <benpeart@microsoft.com>

Add tracing around initializing and discarding mempools. In discard report on the amount of memory unused in the current block to help tune setting the initial_size. Signed-off-by: Ben Peart <benpeart@microsoft.com>

Update enable_fscache() to take an optional initial size parameter which is used to initialize the hashmap so that it can avoid having to rehash as additional entries are added. Add a separate disable_fscache() macro to make the code clearer and easier to read. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We will use this in the next commit to implement an FSCache-aware version of is_mount_point(). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The threading model for fscache has been to have a single, global cache. This puts requirements on it to be thread safe so that callers like preload-index can call it from multiple threads. This was implemented with a single mutex and completion events which introduces contention between the calling threads. Simplify the threading model by making fscache thread specific. This allows us to remove the global mutex and synchronization events entirely and instead associate a fscache with every thread that requests one. This works well with the current multi-threading which divides the cache entries into blocks with a separate thread processing each block. At the end of each worker thread, if there is a fscache on the primary thread, merge the cached results from the worker into the primary thread cache. This enables us to reuse the cache later especially when scanning for untracked files. In testing, this reduced the time spent in preload_index() by about 25% and also reduced the CPU utilization significantly. On a repo with ~200K files, it reduced overall status times by ~12%. Signed-off-by: Ben Peart <benpeart@microsoft.com>

When FSCache is active, we can cache the reparse tag and use it directly to determine whether a path refers to an NTFS junction, without any additional, costly I/O. Note: this change only makes a difference with the next commit, which will make use of the FSCache in `git clean` (contingent on `core.fscache` set, of course). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Track fscache hits and misses for lstat and opendir requests. Reporting of statistics is done when the cache is disabled for the last time and freed and is only reported if GIT_TRACE_FSCACHE is set. Sample output is: 11:33:11.836428 compat/win32/fscache.c:433 fscache: lstat 3775, opendir 263, total requests/misses 4052/269 Signed-off-by: Ben Peart <benpeart@microsoft.com>

Now that the fscache is single threaded, take advantage of the mem_pool as the allocator to significantly reduce the cost of allocations and frees. With the reduced cost of free, in future patches, we can start freeing the fscache at the end of commands instead of just leaking it. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The recent change to make fscache thread specific relied on fscache_enable() being called first from the primary thread before being called in parallel from worker threads. Make that more robust and protect it with a critical section to avoid any issues. Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Ben Peart <benpeart@microsoft.com>

Using FindFirstFileExW() requires the OS to allocate a 64K buffer for each directory and then free it when we call FindClose(). Update fscache to call the underlying kernel API NtQueryDirectoryFile so that we can do the buffer management ourselves. That allows us to allocate a single buffer for the lifetime of the cache and reuse it for each directory. This change improves performance of 'git status' by 18% in a repo with ~200K files and 30k folders. Documentation for NtQueryDirectoryFile can be found at: https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/content/ntifs/nf-ntifs-ntquerydirectoryfile https://docs.microsoft.com/en-us/windows/desktop/FileIO/file-attribute-constants https://docs.microsoft.com/en-us/windows/desktop/fileio/reparse-point-tags To determine if the specified directory is a symbolic link, inspect the FileAttributes member to see if the FILE_ATTRIBUTE_REPARSE_POINT flag is set. If so, EaSize will contain the reparse tag (this is a so far undocumented feature, but confirmed by the NTFS developers). To determine if the reparse point is a symbolic link (and not some other form of reparse point), test whether the tag value equals the value IO_REPARSE_TAG_SYMLINK. The NtQueryDirectoryFile() call works best (and on Windows 8.1 and earlier, it works *only*) with buffer sizes up to 64kB. Which is 32k wide characters, so let's use that as our buffer size. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

When updating the skip-worktree bits in the index to align with new values in a sparse-checkout file, Git scans the entire working directory with lstat() calls. In a sparse-checkout, many of these lstat() calls are for paths that do not exist. Enable the fscache feature during this scan. Since enable_fscache() calls nest, the disable_fscache() method decrements a counter and would only clear the cache if that counter reaches zero. In a local test of a repo with ~2.2 million paths, updating the index with git read-tree -m -u HEAD with a sparse-checkout file containing only /.gitattributes improved from 2-3 minutes to ~6 seconds. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The `git clean` command needs to enumerate plenty of files and directories, and can therefore benefit from the FSCache. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…er/fscache_nfd fscache: add not-found directory cache to fscache

…er/add_preload_fscache add: use preload-index and fscache for performance

…xcludes_with_fscache dir.c: make add_excludes aware of fscache during status

fetch-pack.c: enable fscache for stats under .git/objects

…t_flush checkout.c: enable fscache for checkout again Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…_index Enable the filesystem cache (fscache) in refresh_index().

…-gfw fscache: use FindFirstFileExW to avoid retrieving the short name

…ter-status-gfw status: disable and free fscache at the end of the status command

…e-gfw fscache: add GIT_TEST_FSCACHE support

…ics-gfw fscache: add fscache hit statistics

This brings substantial wins in performance because the FSCache is now per-thread, being merged to the primary thread only at the end, so we do not have to lock (except while merging). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…safe-enable-gfw fscache: make fscache_enable() thread safe

…DirectoryFile-gfw fscache: teach fscache to use NtQueryDirectoryFile

…size strbuf_readlink() calls readlink() twice if the hint argument specifies the exact size of the link target (e.g. by passing stat.st_size as returned by lstat()). This is necessary because 'readlink(..., hint) == hint' could mean that the buffer was too small. Use hint + 1 as buffer size to prevent this. Signed-off-by: Karsten Blees <blees@dcon.de>

There is a problem in the way 9ac3f0e (pack-objects: fix performance issues on packing large deltas, 2018-07-22) initializes that mutex in the `packing_data` struct. The problem manifests in a segmentation fault on Windows, when a mutex (AKA critical section) is accessed without being initialized. (With pthreads, you apparently do not really have to initialize them?) This was reported in git-for-windows#1839. Signed-off-by: Doug Kelly <dougk.ff7@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The `git_terminal_prompt()` function expects the terminal window to be attached to a Win32 Console. However, this is not the case with terminal windows other than `cmd.exe`'s, e.g. with MSys2's own `mintty`. Non-cmd terminals such as `mintty` still have to have a Win32 Console to be proper console programs, but have to hide the Win32 Console to be able to provide more flexibility (such as being resizeable not only vertically but also horizontally). By writing to that Win32 Console, `git_terminal_prompt()` manages only to send the prompt to nowhere and to wait for input from a Console to which the user has no access. This commit introduces a function specifically to support `mintty` -- or other terminals that are compatible with MSys2's `/dev/tty` emulation. We use the `TERM` environment variable as an indicator for that: if the value starts with "xterm" (such as `mintty`'s "xterm_256color"), we prefer to let `xterm_prompt()` handle the user interaction. The most prominent user of `git_terminal_prompt()` is certainly `git-remote-https.exe`. It is an interesting use case because both `stdin` and `stdout` are redirected when Git calls said executable, yet it still wants to access the terminal. When running inside a `mintty`, the terminal is not accessible to the `git-remote-https.exe` program, though, because it is a MinGW program and the `mintty` terminal is not backed by a Win32 console. To solve that problem, we simply call out to the shell -- which is an *MSys2* program and can therefore access `/dev/tty`. Helped-by: nalla <nalla@hamal.uberspace.de> Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

strbuf_readlink() refuses to read link targets that exceed PATH_MAX (even if a sufficient size was specified by the caller). As some platforms support longer paths, remove this restriction (similar to strbuf_getcwd()). Signed-off-by: Karsten Blees <blees@dcon.de>

Windows paths are typically limited to MAX_PATH = 260 characters, even though the underlying NTFS file system supports paths up to 32,767 chars. This limitation is also evident in Windows Explorer, cmd.exe and many other applications (including IDEs). Particularly annoying is that most Windows APIs return bogus error codes if a relative path only barely exceeds MAX_PATH in conjunction with the current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG. Many Windows wide char APIs support longer than MAX_PATH paths through the file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path. Notable exceptions include functions dealing with executables and the current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...). Introduce a handle_long_path function to check the length of a specified path properly (and fail with ENAMETOOLONG), and to optionally expand long paths using the '\\?\' file namespace prefix. Short paths will not be modified, so we don't need to worry about device names (NUL, CON, AUX). Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be limited to MAX_PATH (at least not on Win7), so we can use it to do the heavy lifting of the conversion (translate '/' to '\', eliminate '.' and '..', and make an absolute path). Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH limit. Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs that support long paths. While improved error checking is always active, long paths support must be explicitly enabled via 'core.longpaths' option. This is to prevent end users to shoot themselves in the foot by checking out files that Windows Explorer, cmd/bash or their favorite IDE cannot handle. Test suite: Test the case is when the full pathname length of a dir is close to 260 (MAX_PATH). Bug report and an original reproducer by Andrey Rogozhnikov: msysgit#122 (comment) [jes: adjusted test number to avoid conflicts, added support for chdir(), etc] Thanks-to: Martin W. Kirst <maki@bitkings.de> Thanks-to: Doug Kelly <dougk.ff7@gmail.com> Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com> Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Stepan Kasal <kasal@ucw.cz> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Accessing the Windows console through the special CONIN$ / CONOUT$ devices doesn't work properly for non-ASCII usernames an passwords. It also doesn't work for terminal emulators that hide the native console window (such as mintty), and 'TERM=xterm*' is not necessarily a reliable indicator for such terminals. The new shell_prompt() function, on the other hand, works fine for both MSys1 and MSys2, in native console windows as well as mintty, and properly supports Unicode. It just needs bash on the path (for 'read -s', which is bash-specific). On Windows, try to use the shell to read from the terminal. If that fails with ENOENT (i.e. bash was not found), use CONIN/OUT as fallback. Note: To test this, create a UTF-8 credential file with non-ASCII chars, e.g. in git-bash: 'echo url=http://täst.com > cred.txt'. Then in git-cmd, 'git credential fill <cred.txt' works (shell version), while calling git without the git-wrapper (i.e. 'mingw64\bin\git credential fill <cred.txt') mangles non-ASCII chars in both console output and input. Signed-off-by: Karsten Blees <blees@dcon.de>

Signed-off-by: Karsten Blees <blees@dcon.de>

When updating the skip-worktree bits in the index to align with new values in a sparse-checkout file, Git scans the entire working directory with lstat() calls. In a sparse-checkout, many of these lstat() calls are for paths that do not exist. Enable the fscache feature during this scan. In a local test of a repo with ~2.2 million paths, updating the index with `git read-tree -m -u HEAD` with a sparse-checkout file containing only `/.gitattributes` improved from 2-3 minutes to 15-20 seconds. More work could be done to stop running lstat() calls when recursing into directories that are known to not exist.

Use a suffciently large buffer to strip the trailing slash. Signed-off-by: Karsten Blees <blees@dcon.de>

To support Git Bash running in a MinTTY, we use a dirty trick to access the MSYS2 pseudo terminal: we execute a Bash snippet that accesses /dev/tty. The idea was to fall back to writing to/reading from CONOUT$/CONIN$ if that Bash call failed because Bash was not found. However, we should fall back even in other error conditions, because we have not successfully read the user input. Let's make it so. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

GetFileAttributes cannot handle paths with trailing dir separator. The current [l]stat implementation calls GetFileAttributes twice if the path has trailing slashes (first with the original path passed to [l]stat, and and a second time with a path copy with trailing '/' removed). With Unicode conversion, we get the length of the path for free and also have a (wide char) buffer that can be modified. Remove trailing directory separators before calling the Win32 API. Signed-off-by: Karsten Blees <blees@dcon.de>

We already avoid traversing NTFS junction points in `git clean -dfx`. With this topic branch, we do that when the FSCache is enabled, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

A ton of Git commands simply do not read (or at least parse) the core.* settings. This is not good, as Git for Windows relies on the core.longPaths setting to be read quite early on. So let's just make sure that all commands read the config and give platform_core_config() a chance. This patch teaches tons of Git commands to respect the config setting `core.longPaths = true`, including `pack-refs`, thereby fixing git-for-windows#1218 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Update wchar_t buffers to use MAX_LONG_PATH instead of MAX_PATH and call xutftowcs_long_path() in the Win32 backend source files. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

On Windows, git repositories may have extra files which need cleaned (e.g., a build directory) that may be arbitrarily deep. Suggest using `core.longPaths` if such situations are encountered. Fixes: git-for-windows#2715 Signed-off-by: Ben Boeckel <mathstuf@gmail.com>

With the recent update in efee955 (gpg-interface: check gpg signature creation status, 2016-06-17), we ask GPG to send all status updates to stderr, and then catch the stderr in an strbuf. But GPG might fail, and send error messages to stderr. And we simply do not show them to the user. Even worse: this swallows any interactive prompt for a passphrase. And detaches stderr from the tty so that the passphrase cannot be read. So while the first problem could be fixed (by printing the captured stderr upon error), the second problem cannot be easily fixed, and presents a major regression. So let's just revert commit efee955. This fixes git-for-windows#871 Cc: Michael J Gruber <git@drmicha.warpmail.net> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

With respect to symlinks, the current stat() implementation is almost the same as lstat(): except for the file type (st_mode & S_IFMT), it returns information about the link rather than the target. Implement stat by opening the file with as little permissions as possible and calling GetFileInformationByHandle on it. This way, all link resoltion is handled by the Windows file system layer. If symlinks are disabled, use lstat() as before, but fail with ELOOP if a symlink would have to be resolved. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

With the new mingw_stat() implementation, do_lstat() is only called from mingw_lstat() (with follow == 0). Remove the extra function and the old mingw_stat()-specific (follow == 1) logic. Signed-off-by: Karsten Blees <blees@dcon.de>

When obtaining lstat information for reparse points, we need to call FindFirstFile() in addition to GetFileInformationEx() to obtain the type of the reparse point (symlink, mount point etc.). However, currently there is no error handling whatsoever if FindFirstFile() fails. Call FindFirstFile() before modifying the stat *buf output parameter and error out if the call fails. Note: The FindFirstFile() return value includes all the data that we get from GetFileAttributesEx(), so we could replace GetFileAttributesEx() with FindFirstFile(). We don't do that because GetFileAttributesEx() is about twice as fast for single files. I.e. we only pay the extra cost of calling FindFirstFile() in the rare case that we encounter a reparse point. Note: The indentation of the remaining reparse point code will be fixed in the next patch. Signed-off-by: Karsten Blees <blees@dcon.de>

Move S_IFLNK detection to file_attr_to_st_mode() and reuse it in fscache. Implement DT_LNK detection in dirent.c and the fscache readdir version. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Git typically doesn't trust the stat.st_size member of symlinks (e.g. see strbuf_readlink()). However, some functions take shortcuts if st_size is 0 (e.g. diff_populate_filespec()). In mingw_lstat() and fscache_lstat(), make sure to return an adequate size. The extra overhead of opening and reading the reparse point to calculate the exact size is not necessary, as git doesn't rely on the value anyway. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The retry pattern is duplicated in three places. It also seems to be too hard to use: mingw_unlink() and mingw_rmdir() duplicate the code to retry, and both of them do so incompletely. They also do not restore errno if the user answers 'no'. Introduce a retry_ask_yes_no() helper function that handles retry with small delay, asking the user, and restoring errno. mingw_unlink: include _wchmod in the retry loop (which may fail if the file is locked exclusively). mingw_rmdir: include special error handling in the retry loop. Signed-off-by: Karsten Blees <blees@dcon.de>

Symlinks on Windows don't work the same way as on Unix systems. E.g. there are different types of symlinks for directories and files, creating symlinks requires administrative privileges etc. By default, disable symlink support on Windows. I.e. users explicitly have to enable it with 'git config [--system|--global] core.symlinks true'. The test suite ignores system / global config files. Allow testing *with* symlink support by checking if native symlinks are enabled in MSys2 (via 'MSYS=winsymlinks:nativestrict'). Reminder: This would need to be changed if / when we find a way to run the test suite in a non-MSys-based shell (e.g. dash). Signed-off-by: Karsten Blees <blees@dcon.de>

Signed-off-by: Karsten Blees <blees@dcon.de>

_wunlink() / DeleteFileW() refuses to delete symlinks to directories. If _wunlink() fails with ERROR_ACCESS_DENIED, try _wrmdir() as well. Signed-off-by: Karsten Blees <blees@dcon.de>

MSVCRT's _wrename() cannot rename symlinks over existing files: it returns success without doing anything. Newer MSVCR*.dll versions probably do not have this problem: according to CRT sources, they just call MoveFileEx() with the MOVEFILE_COPY_ALLOWED flag. Get rid of _wrename() and call MoveFileEx() with proper error handling. Signed-off-by: Karsten Blees <blees@dcon.de>

If symlinks are enabled, resolve all symlinks when changing directories, as required by POSIX. Note: Git's real_path() function bases its link resolution algorithm on this property of chdir(). Unfortunately, the current directory on Windows is limited to only MAX_PATH (260) characters. Therefore using symlinks and long paths in combination may be problematic. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Implement readlink() by reading NTFS reparse points. Works for symlinks and directory junctions. If symlinks are disabled, fail with ENOSYS. Signed-off-by: Karsten Blees <blees@dcon.de>

This commit fixes mingw_lstat by computing the proper size for symlinks according to POSIX. POSIX specifies that upon successful return from lstat: "the value of the st_size member shall be set to the length of the pathname contained in the symbolic link not including any terminating null byte". Prior to this commit the mingw_lstat function returned a fixed size of 4096. This caused problems in git repositories that were accessed by git for Cygwin or git for WSL. For example, doing `git reset --hard` using git for Windows would update the size of symlinks in the index to be 4096; at a later time git for Cygwin or git for WSL would find that symlinks have changed size during `git status`. Vice versa doing `git reset --hard` in git for Cygwin or git for WSL would update the size of symlinks in the index with the correct value, only for git for Windows to find incorrectly at a later time that the size had changed. Signed-off-by: Bill Zissimopoulos <billziss@navimatics.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Implement symlink() that always creates file symlinks. Fails with ENOSYS if symlinks are disabled or unsupported. Note: CreateSymbolicLinkW() was introduced with symlink support in Windows Vista. For compatibility with Windows XP, we need to load it dynamically and fail gracefully if it isnt's available. Signed-off-by: Karsten Blees <blees@dcon.de>

Symlinks on Windows have a flag that indicates whether the target is a file or a directory. Symlinks of wrong type simply don't work. This even affects core Win32 APIs (e.g. DeleteFile() refuses to delete directory symlinks). However, CreateFile() with FILE_FLAG_BACKUP_SEMANTICS doesn't seem to care. Check the target type by first creating a tentative file symlink, opening it, and checking the type of the resulting handle. If it is a directory, recreate the symlink with the directory flag set. It is possible to create symlinks before the target exists (or in case of symlinks to symlinks: before the target type is known). If this happens, create a tentative file symlink and postpone the directory decision: keep a list of phantom symlinks to be processed whenever a new directory is created in mingw_mkdir(). Limitations: This algorithm may fail if a link target changes from file to directory or vice versa, or if the target directory is created in another process. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This will come in handy in the next commit. Signed-off-by: JiSeop Moon <zcube@zcube.kr> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Bert Belder <bertbelder@gmail.com>

With Windows 10 Build 14972 in Developer Mode, a new flag is supported by CreateSymbolicLink() to create symbolic links even when running outside of an elevated session (which was previously required). This new flag is called SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE and has the numeric value 0x02. Previous Windows 10 versions will not understand that flag and return an ERROR_INVALID_PARAMETER, therefore we have to be careful to try passing that flag only when the build number indicates that it is supported. For more information about the new flag, see this blog post: https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/ This patch is loosely based on the patch submitted by Samuel D. Leslie as git-for-windows#1184. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

It is a known issue that a rename() can fail with an "Access denied" error at times, when copying followed by deleting the original file works. Let's just fall back to that behavior. Signed-off-by: JiSeop Moon <zcube@zcube.kr> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

On Windows, symbolic links actually have a type depending on the target: it can be a file or a directory. In certain circumstances, this poses problems, e.g. when a symbolic link is supposed to point into a submodule that is not checked out, so there is no way for Git to auto-detect the type. To help with that, we will add support over the course of the next commits to specify that symlink type via the Git attributes. This requires an index_state, though, something that Git for Windows' `symlink()` replacement cannot know about because the function signature is defined by the POSIX standard and not ours to change. So let's introduce a helper function to create symbolic links that *does* know about the index_state. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

When creating directories via `safe_create_leading_directories()`, we might encounter an already-existing directory which is not readable by the current user. To handle that situation, Git's code calls `stat()` to determine whether we're looking at a directory. In such a case, `CreateFile()` will fail, though, no matter what, and consequently `mingw_stat()` will fail, too. But POSIX semantics seem to still allow `stat()` to go forward. So let's call `mingw_lstat()` for the rescue if we fail to get a file handle due to denied permission in `mingw_stat()`, and fill the stat info that way. We need to be careful to not allow this to go forward in case that we're looking at a symbolic link: to resolve the link, we would still have to create a file handle, and we just found out that we cannot. Therefore, `stat()` still needs to fail with `EACCES` in that case. This fixes git-for-windows#2531. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

In preparation for making this function a bit more complicated (to allow for special-casing the `ContainerMappedDirectories` in Windows containers, which look like a symbolic link, but are not), let's move it out of the header. Signed-off-by: JiSeop Moon <zcube@zcube.kr> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

On Windows, symbolic links have a type: a "file symlink" must point at a file, and a "directory symlink" must point at a directory. If the type of symlink does not match its target, it doesn't work. Git does not record the type of symlink in the index or in a tree. On checkout it'll guess the type, which only works if the target exists at the time the symlink is created. This may often not be the case, for example when the link points at a directory inside a submodule. By specifying `symlink=file` or `symlink=dir` the user can specify what type of symlink Git should create, so Git doesn't have to rely on unreliable heuristics. Signed-off-by: Bert Belder <bertbelder@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

In git-for-windows#2637, we fixed a bug where symbolic links' target path sizes were recorded incorrectly in the index. The downside of this fix was that every user with tracked symbolic links in their checkouts would see them as modified in `git status`, but not in `git diff`, and only a `git add <path>` (or `git add -u`) would "fix" this. Let's do better than that: we can detect that situation and simply pretend that a symbolic link with a known bad size (or a size that just happens to be that bad size, a _very_ unlikely scenario because it would overflow our buffers due to the trailing NUL byte) means that it needs to be re-checked as if we had just checked it out. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

... even if they may look like them. As looking up the target of the "symbolic link" (just to see whether it starts with `/ContainerMappedDirectories/`) is pretty expensive, we do it when we can be *really* sure that there is a possibility that this might be the case. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: JiSeop Moon <zcube@zcube.kr>

The main idea of this patch is that even if we have to look up the absolute path of the script, if only the basename was specified as argv[0], then we should use that basename on the command line, too, not the absolute path. This patch will also help with the upcoming patch where we automatically substitute "sh ..." by "busybox sh ..." if "sh" is not in the PATH but "busybox" is: we will do that by substituting the actual executable, but still keep prepending "sh" to the command line. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

BusyBox comes with a ton of applets ("applet" being the identical concept to Git's "builtins"). And similar to Git's builtins, the applets can be called via `busybox <command>`, or the BusyBox executable can be copied/hard-linked to the command name. The similarities do not end here. Just as with Git's builtins, it is problematic that BusyBox' hard-linked applets cannot easily be put into a .zip file: .zip archives have no concept of hard-links and therefore would store identical copies (and also extract identical copies, "inflating" the archive unnecessarily). To counteract that issue, MinGit already ships without hard-linked copies of the builtins, and the plan is to do the same with BusyBox' applets: simply ship busybox.exe as single executable, without hard-linked applets. To accommodate that, Git is being taught by this commit a very special trick, exploiting the fact that it is possible to call an executable with a command-line whose argv[0] is different from the executable's name: when `sh` is to be spawned, and no `sh` is found in the PATH, but busybox.exe is, use that executable (with unchanged argv). Likewise, if any executable to be spawned is not on the PATH, but busybox.exe is found, parse the output of `busybox.exe --help` to find out what applets are included, and if the command matches an included applet name, use busybox.exe to execute it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

It is a bit strange, and even undesirable, to require Perl just to run the test suite even when NO_PERL was set. This patch does not fix this problem by any stretch of imagination. However, it fixes *the* Perl invocation that *every single* test script has to run. While at it, it makes the source code also more grep'able, as the code that unsets some, but not all, GIT_* environment variables just became a *lot* more explicit. And all that while still reducing the total number of lines. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This helper is slightly more performant than the script with MSYS2's Bash. And a lot more readable. To accommodate t1050, which wants to compare files weighing in with 3MB (falling outside of t1050's malloc limit of 1.5MB), we simply lift the allocation limit by setting the environment variable GIT_ALLOC_LIMIT to zero when calling the helper. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

It is convenient to assume that everybody who wants to build & test Git has access to a working `iconv` executable (after all, we already pretty much require libiconv). However, that limits esoteric test scenarios such as Git for Windows', where an end user installation has to ship with `iconv` for the sole purpose of being testable. That payload serves no other purpose. So let's just have a test helper (to be able to test Git, the test helpers have to be available, after all) to act as `iconv` replacement. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We already have a directory where we store files intended for use by multiple test scripts. The same directory is a better home for the test-binary-*.png files than t/. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The idea is to allow running the test suite on MinGit with BusyBox installed in /mingw64/bin/sh.exe. In that case, we will want to exclude sort & find (and other Unix utilities) from being bundled. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

BusyBox-w32 is a true Win32 application, i.e. it does not come with a POSIX emulation layer. That also means that it does *not* use the Unix convention of separating the entries in the PATH variable using colons, but semicolons. However, there are also BusyBox ports to Windows which use a POSIX emulation layer such as Cygwin's or MSYS2's runtime, i.e. using colons as PATH separators. As a tell-tale, let's use the presence of semicolons in the PATH variable: on Unix, it is highly unlikely that it contains semicolons, and on Windows (without POSIX emulation), it is virtually guaranteed, as everybody should have both $SYSTEMROOT and $SYSTEMROOT/system32 in their PATH. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Traditionally, Git for Windows' SDK uses Bash as its default shell. However, other Unix shells are available, too. Most notably, the Win32 port of BusyBox comes with `ash` whose `pwd` command already prints Windows paths as Git for Windows wants them, while there is not even a `builtin` command. Therefore, let's be careful not to override `pwd` unless we know that the `builtin` command is available. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The -W option is only understood by MSYS2 Bash's pwd command. We already make sure to override `pwd` by `builtin pwd -W` for MINGW, so let's not double the effort here. This will also help when switching the shell to another one (such as BusyBox' ash) whose pwd does *not* understand the -W option. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

When running with BusyBox, we will want to avoid calling executables on the PATH that are implemented in BusyBox itself. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Since c6b0831 (docs: warn about possible '=' in clean/smudge filter process values, 2016-12-03), t0021 writes out a file with quotes in its name, and MSYS2's path conversion heuristics mistakes that to mean that we are not talking about a path here. Therefore, we need to use Windows paths, as the test-helper is a Win32 program that would otherwise have no idea where to look for the file. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

At some stage, t5003-archive-zip wants to add a file that is not ASCII. To that end, it uses /bin/sh. But that file may actually not exist (it is too easy to forget that not all the world is Unix/Linux...)! Besides, we already have perfectly fine binary files intended for use solely by the tests. So let's use one of them instead. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

While it may seem super convenient to some old Unix hands to simpy require Perl to be available when running the test suite, this is a major hassle on Windows, where we want to verify that Perl is not, actually, required in a NO_PERL build. As a super ugly workaround, we "install" a script into /usr/bin/perl reading like this: #!/bin/sh # We'd much rather avoid requiring Perl altogether when testing # an installed Git. Oh well, that's why we cannot have nice # things. exec c:/git-sdk-64/usr/bin/perl.exe "$@" The problem with that is that BusyBox assumes that the #! line in a script refers to an executable, not to a script. So when it encounters the line #!/usr/bin/perl in t5532's proxy-get-cmd, it barfs. Let's help this situation by simply executing the Perl script with the "interpreter" specified explicitly. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

When t5605 tries to verify that files are hardlinked (or that they are not), it uses the `-links` option of the `find` utility. BusyBox' implementation does not support that option, and BusyBox-w32's lstat() does not even report the number of hard links correctly (for performance reasons). So let's just switch to a different method that actually works on Windows. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Git for Windows uses MSYS2's Bash to run the test suite, which comes with benefits but also at a heavy price: on the plus side, MSYS2's POSIX emulation layer allows us to continue pretending that we are on a Unix system, e.g. use Unix paths instead of Windows ones, yet this is bought at a rather noticeable performance penalty. There *are* some more native ports of Unix shells out there, though, most notably BusyBox-w32's ash. These native ports do not use any POSIX emulation layer (or at most a *very* thin one, choosing to avoid features such as fork() that are expensive to emulate on Windows), and they use native Windows paths (usually with forward slashes instead of backslashes, which is perfectly legal in almost all use cases). And here comes the problem: with a $PWD looking like, say, C:/git-sdk-64/usr/src/git/t/trash directory.t5813-proto-disable-ssh Git's test scripts get quite a bit confused, as their assumptions have been shattered. Not only does this path contain a colon (oh no!), it also does not start with a slash. This is a problem e.g. when constructing a URL as t5813 does it: ssh://remote$PWD. Not only is it impossible to separate the "host" from the path with a $PWD as above, even prefixing $PWD by a slash won't work, as /C:/git-sdk-64/... is not a valid path. As a workaround, detect when $PWD does not start with a slash on Windows, and simply strip the drive prefix, using an obscure feature of Windows paths: if an absolute Windows path starts with a slash, it is implicitly prefixed by the drive prefix of the current directory. As we are talking about the current directory here, anyway, that strategy works. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

On Windows, the current working directory is pretty much guaranteed to contain a colon. If we feed that path to CVS, it mistakes it for a separator between host and port, though. This has not been a problem so far because Git for Windows uses MSYS2's Bash using a POSIX emulation layer that also pretends that the current directory is a Unix path (at least as long as we're in a shell script). However, that is rather limiting, as Git for Windows also explores other ports of other Unix shells. One of those is BusyBox-w32's ash, which is a native port (i.e. *not* using any POSIX emulation layer, and certainly not emulating Unix paths). So let's just detect if there is a colon in $PWD and punt in that case. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

To verify that the symlink is resolved correctly, we use the fact that `git.exe` is a native Win32 program, and that `git.exe config -f <path>` therefore uses the native symlink resolution. Signed-off-by: Bert Belder <bertbelder@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The TerminateProcess() function does not actually leave the child processes any chance to perform any cleanup operations. This is bad insofar as Git itself expects its signal handlers to run. A symptom is e.g. a left-behind .lock file that would not be left behind if the same operation was run, say, on Linux. To remedy this situation, we use an obscure trick: we inject a thread into the process that needs to be killed and to let that thread run the ExitProcess() function with the desired exit status. Thanks J Wyman for describing this trick. The advantage is that the ExitProcess() function lets the atexit handlers run. While this is still different from what Git expects (i.e. running a signal handler), in practice Git sets up signal handlers and atexit handlers that call the same code to clean up after itself. In case that the gentle method to terminate the process failed, we still fall back to calling TerminateProcess(), but in that case we now also make sure that processes spawned by the spawned process are terminated; TerminateProcess() does not give the spawned process a chance to do so itself. Please note that this change only affects how Git for Windows tries to terminate processes spawned by Git's own executables. Third-party software that *calls* Git and wants to terminate it *still* need to make sure to imitate this gentle method, otherwise this patch will not have any effect. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The Makefile target `install-mingit-test-artifacts` simply copies stuff and things directly into a MinGit directory, including an init.bat script to set everything up so that the tests can be run in a cmd window. Sadly, Git's test suite still relies on a Perl interpreter even if compiled with NO_PERL=YesPlease. We punt for now, installing a small script into /usr/bin/perl that hands off to an existing Perl of a Git for Windows SDK. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Previously, we did not install any handler for Ctrl+C, but now we really want to because the MSYS2 runtime learned the trick to call the ConsoleCtrlHandler when Ctrl+C was pressed. With this, hitting Ctrl+C while `git log` is running will only terminate the Git process, but not the pager. This finally matches the behavior on Linux and on macOS. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

The `xutftowcs_path` function canonicalizes absolute paths using GetFullPathNameW. This canonicalization may change the length of the string (e.g. getting rid of \.\), which breaks callers that pass the template string in a strbuf and expect the length of the string to remain the same. In my particular case, the tmp-objdir code is passing a strbuf to mkdtemp and is breaking since the strbuf.len is no longer synchronized with strlen(strbuf.buf). Signed-off-by: Neeraj K. Singh <neerajsi@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…ITOR" In e3f7e01 (Revert "editor: save and reset terminal after calling EDITOR", 2021-11-22), we reverted the commit wholesale where the terminal state would be saved and restored before/after calling an editor. The reverted commit was intended to fix a problem with Windows Terminal where simply calling `vi` would cause problems afterwards. To fix the problem addressed by the revert, but _still_ keep the problem with Windows Terminal fixed, let's revert the revert, with a twist: we restrict the save/restore _specifically_ to the case where `vi` (or `vim`) is called, and do not do the same for any other editor. This should still catch the majority of the cases, and will bridge the time until the original patch is re-done in a way that addresses all concerns. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

These are Git for Windows' Git GUI and gitk patches. We will have to decide at some point what to do about them, but that's a little lower priority (as Git GUI seems to be unmaintained for the time being, and the gitk maintainer keeps a very low profile on the Git mailing list, too). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…dvice clean: suggest using `core.longPaths` if paths are too long to remove

This was pull request git-for-windows#1645 from ZCube/master Support windows container. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Specify symlink type in .gitattributes

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Handle Ctrl+C in Git Bash nicely Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Switch to batched fsync by default

A fix for calling `vim` in Windows Terminal caused a regression and was reverted. We partially un-revert this, to get the fix again. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This topic branch re-adds the deprecated --stdin/-z options to `git reset`. Those patches were overridden by a different set of options in the upstream Git project before we could propose `--stdin`. We offered this in MinGit to applications that wanted a safer way to pass lots of pathspecs to Git, and these applications will need to be adjusted. Instead of `--stdin`, `--pathspec-from-file=-` should be used, and instead of `-z`, `--pathspec-file-nul`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Previously, we interpolated paths in config variables that start with a forward-slash as relative to the runtime prefix. This was not portable and has been replaced with `%(prefix)/`. Let's warn users when they use the now-deprecated form. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

…updates Start monitoring updates of Git for Windows' component in the open

Add a README.md for GitHub goodness. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rebase to v2.38.0-rc1 #4032

Rebase to v2.38.0-rc1 #4032

Commits on Sep 22, 2022