Repository navigation
Invalid path error during sparse checkout in versions newer than 2.24.0 #2777
Description
Setup
- Which version of Git for Windows are you using? Is it 32-bit or 64-bit?
$ git --version --build-options
git version 2.28.0.windows.1
cpu: x86_64
built from commit: 77982caf269b7ee713a76da2bcf260c34d3bf7a7
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
- Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?
$ cmd.exe /c ver
Microsoft Windows [Versión 10.0.17763.1339]
- What options did you set as part of the installation? Or did you choose the
defaults?
# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt
Editor Option: Notepad++
Custom Editor Path:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled
- Any other interesting things about your environment that might be related
to the issue you're seeing?
Nothing special about the enviroment. Can reproduce on a fresh install.
However, the repository I'm working with contains a folder with filenames considered invalid on Windows (e.g :)
Details
- Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other
CMD
- What commands did you run to trigger this issue? If you can provide a
Minimal, Complete, and Verifiable example
this will help us understand the issue.
git config core.sparseCheckout true
(echo /* & echo !other.7z) >.git\info\sparse-checkout
git checkout
- What did you expect to occur after running these commands?
The repository would be checked out except for the folder other.7z, which contains filenames considered invalid on Windows.
Works as expected if I downgrade to a version older than 2.25.0, such as 2.24.0 or 2.23.0.
- What actually happened instead?
git reports invalid path errors and doesn't check out the repository, despite the path being excluded by the sparse checkout
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_04/NEWS_04.tar/home/sugiyama/fly/CON.CGX'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_04/NEWS_04.tar/home/sugiyama/fly/CON.CGX.BAK'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_04/NEWS_04.tar/home/sugiyama/fly/CON.COL'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_04/NEWS_04.tar/home/sugiyama/fly/CON.COL.BAK'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/5000vi.lzh/srd|/current_dir'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/5000vi.lzh/srd|/var/current_dir'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/5000vi.lzh/srd|/var/srd/current_dir'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/exp/:'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/fzero.lzh/fzero/RP5A22:'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/fzero.lzh/fzero/RP5C77:'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/kart/mak.lzh/mak/:'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/msdos/prn.c'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/util/...'
error: invalid path 'other.7z/NEWS.7z/NEWS/テープリストア/NEWS_05/NEWS_05.tar/home/kimura/util/prn.c'
- If the problem was occurring with a specific repository, can you provide the
URL to that repository to help us with testing?
https://github.com/XXLuigiMario/gigaleak
EDIT: Found the origin of this issue, this is caused by the core.protectNTFS setting. Disabling it lets the checkout go through without issues. However, this is a bug because no problematic paths are involved in this checkout!
Activity
rimrul commentedon Aug 3, 2020
This is not a bug.
CONandPRNare reserved names,...ends in a period, the others contain a pipe or a colon.PhilipOakley commentedon Aug 3, 2020
Maybe we need a note somewhere to clarify that sparse checkout will still check the validity of all filenames, and the excluding the reserved names from the checkout won't stop the error report of an invalid path.
Side thought: While
CONandPRNare reserved, so are any files with any extensions to those base names (CON.*,PRN.*) - I haven't looked at the sparse definitions given by @XXLuigiMario for that aspect (and don't for get the case insensitivity, socon.*andprn.*).puigru commentedon Aug 3, 2020
@rimrul I'm aware, but that's precisely why I chose to do a sparse checkout. To prevent git from trying to create these files. It shouldn't stop me in my tracks for something I'm not trying to do.
In addition, the error message doesn't make it clear what's going on. It sounds as if it is still trying to create those files despite being excluded, and doesn't provide any guidance in case you know what you're doing.
rimrul commentedon Aug 3, 2020
Honestly, I didn't read the issue carefully enough. It does indeed seem counter-intuitive to get these errors for files that shouldn't be part of the sparse checkout.
PhilipOakley commentedon Aug 3, 2020
@XXLuigiMario did you confirm that your
.git\info\sparse-checkoutwas actually effective in stopping the checkout of the directory at that level.Test with a non zip directory and file of appropriate names. (the
.7zsuggests that you have a zipped file containing those inappropriate paths - I could be wrong)Also, it maybe that Windows is "confusing" the zip file as being a directory for the checkout purposes. (I have not attempted to copy, clone, or test the repo..)
puigru commentedon Aug 3, 2020
@PhilipOakley Yes,
other.7zoriginated from an archive but is actually the name of a folder in this instance. Thesparse-checkoutrule was effective because once settingcore.protectNTFSto false it checks out the repository without issue with the exception of the problematic folder.I have created a test repo on a Linux machine as follows:
Cloned it on Windows and repeated the commands from the issue, replacing
other.7zwithproblem.With
core.protectNTFSenabled I get:Once disabled:
And the
READMEfile is created, as expected.PhilipOakley commentedon Aug 3, 2020
So, the
core.protectNTFSis being over zealous, or over early, in its 'protection' of the NTFS system from potential bad file names. (an opportunity to review the code @XXLuigiMario ;-)Maybe the sparse checkout needs to be more cautious about where the protection happens. Maybe @derrickstolee has some insight on the sparse approach.
derrickstolee commentedon Aug 3, 2020
I don't think this is a sparse-checkout issue, but instead it is just a
core.protectNTFSissue. That config option doesn't allow bad paths into the index at all, even if those paths are not being populated by the sparse-checkout.Changing this behavior has the potential to re-introduce security problems, so it is not something I will recommend. Perhaps someone who really knows what is going on could look into it, but I doubt this is a good idea.
PhilipOakley commentedon Aug 3, 2020
Thanks for that clarification.
Worth a clarification to make sure everyone is on the same page about how the issues are handled (separate discussion)
Looks like the docs need a tweak to clarify the index vs file system aspect:
Any change would also affect
core.protectHFSpuigru commentedon Aug 3, 2020
@PhilipOakley I would also suggest tweaking how this situation is presented to the user within git, currently it makes it seem like it is actually trying to create these paths (and failing) instead of a security feature.
rimrul commentedon Aug 3, 2020
Maybe we could add a hint to sparse checkout if one of those 2 options is enabled, that there might be error messages about those files, even if they're excluded from checkout?
iamlgh commentedon Aug 20, 2020
I think this something should be done about it, even if it's just documentation. If you don't set "git config core.protectNTFS false", then you can't clone, change to a branch, or pull files from a branch that has these errors, with sparse checkout. It's fatal, it's not that there are just error messages, as @rimrul stated.
wdtj commentedon Sep 4, 2020
This was indeed the case with my issue (above). I've spent about 2 days trying to resolve the issue. I whole hardheartedly suggest that something be changed to prevent this in the future. Either a warning to set core.protectNTFS when sparse-checkout is specified or similar an enhancement to remap bad filepaths.
With the frequency of hit's I have seen suggesting core.sparse-checkout to fix the error, you would be well advised to re-open this issue or accept #2803.
derrickstolee commentedon Dec 30, 2020
As @dscho went to great lengths to detail what would need to happen here without re-introducing a security issue, I would say that this is currently not meeting the bar for our effort at the moment.
The best workaround is to not have these paths in your repo if you can avoid it. (I know we would have a difficult time making the case that torvalds/linux should rename their
aux.cfile, but usually it is best to use paths that work on all platforms whenever possible.)I'm investigating an area that may make this possible to work around in some cases but not 100% of the time. It just happens to be a side-effect of the much-more-important case that I'm handling. When that is ready I'll update this issue if that works to help this situation.
Update git submodules
Remove a file which causes cloning on Windows platform to fail
mikekaganski commentedon Dec 20, 2022
I don't know if it's the same issue, but the same error happens when the paths contain a backslash (which is normal on Windows), like this one for https://bibisect.libreoffice.org/win32-5.1
PhilipOakley commentedon Dec 20, 2022
Unfortunately, that's the one 'special' character on the Linux/Posix side of Git. While some care is applied to try and correctly detect the 'isDir' character, it's best to avoid mixed usage if possible.
dscho commentedon Dec 20, 2022
I think there is a misunderstanding going on here. This path contains a directory whose name is
lib\__pycache__. It's not a__pycachedirectory inside alibdirectory. The directory's name is literallylib\__pycache__.This is important because on Linux, you can have such a directory, and you can have a
__pycache__directory inside alibdirectory next to it.But on Windows, directory names cannot contain backslash characters. Git is doing the right thing here to error out.
mikekaganski commentedon Dec 20, 2022
I am very much sure that the repository was created on Windows. The files (and directories) for it were created on Windows, this is absolutely guaranteed (LibreOffice for Windows can only be built on Windows still). As such, I doubt very much that this repo contains some files/paths that are impossible on Windows.
But since I'm not a git expert, I can't check what is in the metadata of this. Just sharing my thoughts :)
PhilipOakley commentedon Dec 20, 2022
But.... Git assumes Linux/Posix paths by default (it is the SCM for Linux, by design and intent), so Git for Windows is between a rock and a hard place on these issues. Sometimes you have to bite the bullet and focus on the common path 🥲 .
mikekaganski commentedon Dec 20, 2022
@PhilipOakley :) Please note that issues are to identify and resolve problems, and I posted here to inform about a possible problem, to provide a way to check it (with the respective URL), and possibly to fix. I had workarounded it for myself using
git config core.protectNTFS false; and I am not the author of the repo. So I find any comments telling how to avoid it at creation to be really off-topic, unless they mean official "NOTABUG" (which I'd be OK as a resolution, and then simply considered that some bug happened at creation time long ago, using some unknown old git version - then it would be useful to know what specifically happened, and if there was a user error that could be avoided).dscho commentedon Dec 31, 2022
@mikekaganski is it possible that the invalid path (I am talking about
lib\__pycache__, which should belib/__pycache__instead) is committed at the tip of the branch you are checking out? If so, that needs to be fixed. You can work around the problem by settingcore.protectNTFS=falsebut you should immediately rename the path usinggit mvafterwards and commit, and somehow get that integrated into the upstream repository.dscho commentedon Dec 31, 2022
@Dheeraj-kumar1 this is indeed an invalid path. The
:Zone.Identifiersuffix is a so-called "Alternate Data Stream" (see e.g. this blog post for more information) that is attached to the file in question, and it is not supposed to be committed, and it is certainly not supposed to be checked out. You need to remove this before checking out (e.g. by cloning with--no-checkoutand then playing games with the Git index all while operating undercore.protectNTFS=false).fixed ADS identifier files stuffing up git checkouts - updated gitign…