Fresh install unable to install anything

[sirideain]

Here is what I see with verbose. I get this trying to install clippy or rustfmt. I assume I would get it with other crates.

Updating registry https://github.com/rust-lang/crates.io-index
error: [20/-1] Cannot checkout to invalid path '3/n/nul'

[ncoish]

I get the same failure when using cargo build or cargo update --verbose:

cargo update --verbose
Updating registry https://github.com/rust-lang/crates.io-index
error: failed to load source for a dependency on num

Caused by:
Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
[20/-1] Cannot checkout to invalid path '3/n/nul'

Platform: Windows 10
cargo 0.19.0-nightly (fa7584c 2017-04-26)

[azriel91]

Yay. NUL is a reserved name.

Other names to avoid / handle:

Do not use the following reserved names for the name of a file:
CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9:

source: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

[withoutboats]

ping @alexcrichton @carols10cents not sure who else

We probably need to delete this crate and add all of these windows reserved names to the list of reserved names in crates.io.

[withoutboats]

Once nul is deleted I have a PR to reserve the names listed in @azriel91's post rust-lang/crates.io#695

[albel727]

@withoutboats Why wait? Make your PR sooner, for it better get some review before someone else has an idea to get famous quickly.

[withoutboats]

@albel727 The PR is open (I linked to it), but I'm not sure what will happen if its deployed before nul is deleted.

[carols10cents]

What will happen is if there are any updates to the nul crate after the deploy, those updates won't work. Which I think is fine. I'd rather prevent more things that cause problems first.

[albel727]

@withoutboats Ah, I thought it would be a PR against cargo itself. Nevermind me.

[carols10cents]

Deploying to production of the crates.io pr starting now.

[Manishearth]

This issue cannot be fixed by blacklisting crate names on crates.

Cargo uses a trie, so you could always trigger the trie to break via things like uploading packages nula, nulb,... so that a nul folder is created in the filesystem trie. We need a crates trie blacklist that makes sure these folder names don't get created in the trie (falling back to storing them flat one level above), as well as blacklists for the existing name.

Unless I'm misunderstanding how the registry trie works.

[carols10cents]

@Manishearth The trie is 2 letters, so a crate named nula would go in nu/la. Idk if that still leaves problematic files

[withoutboats]

All of the Windows names are 3 or 4 characters so it should be fine.

[shepmaster]

Cargo uses a trie, so you could always trigger the trie to break via things like uploading packages nula, nulb,... so that a nul folder is created in the filesystem trie.

If the directory scheme were to be changed, I'd recommend switching to using a hash of the name for the prefix directories; this leads to better distribution of files per directory. Too many things in a directory can have poor performance implications.

[retep998]

If the directory scheme were to be changed, I'd recommend switching to using a hash of the name for the prefix directories; this leads to better distribution of files per directory. Too many things in a directory can have poor performance implications.

With the current system you could trivially create a few million crates with a certain prefix and easily DOS anyone with a filesystem that doesn't handle that many files in a folder.

[carols10cents]

Production deploy of crates.io done, and crate removed from the index. Please open new issues for any remaining problems.

[jckarter]

A more robust way to address these kinds of problems on Windows might be to do your own path normalization before calling any Win32 or C filesystem APIs. A fully qualified absolute path like \\?\C:\nul\aux\prn\whatever gets minimal Win32-level processing before getting handed off to the NT layer where these legacy device names aren't a problem. Another benefit is that \\?\ paths can be up to 32768 TCHARs long instead of only MAX_PATH.

[shepmaster]

@jckarter the error message seems to indicate that its during the git checkout that the issue occurs; not from within Cargo's own code. Perhaps there's an underlying issue with git / libgit?

[ticki]

Ideally, the crates with reserved names should be banned. This should be trivial to add in the crates.io server.

[ghost]

I don't think any names that some OS does not allow, should be banned. This is a Windows bug and not a problem with the crate name.

[withoutboats]

@ticki That's exactly what we did. rust-lang/crates.io#695

[Manishearth]

This is a Windows bug and not a problem with the crate name.

We can't pass the buck on to windows like that. Yes, it is a windows bug. Windows won't change it and certainly not in a timeframe that solves the current breakage. It is our responsibility to fix it, regardless of the entity that can be blamed for the bug. Fixing a bug does not necessitate self-assigning fault for it.

[nightpool]

I think mangling could also be a solution here that preserves backwards compatibility. What if we mangled only names with a reserved words with a prefix? So "nul" becomes "--nul", or something similar? Then current and future versions of cargo can add these crates, and old versions of cargo miss out on a vanishingly small number of crates but also don't get borked by checking out the repository.

[carols10cents]

Please open new issues for any new problems or discussions on further improvements

[the-Arioch]

...but GIT implemented Windows long paths @jckarter mentioned as far ago as 2014, pity no one tested it.
And today even GUI packages like TortoiseGit turn long paths on.
Maybe they were not considered tested enough in 2017.

https://www.brycevandyk.com/of-too-long-file-names-in-windows-and-git/
https://stackoverflow.com/questions/22575662/filename-too-long-in-git-for-windows

Wonder if [ab]using Git for package database was correct idea. It means that quirks of every single platform would be escalated into world-global Crates ban-list. Instead of fixing the platform "driver". In a sense, that was repeating that Microsoft "special CP/M names" escalation quirk....