Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fresh install unable to install anything #3982

Closed
sirideain opened this issue Apr 30, 2017 · 23 comments
Closed

Fresh install unable to install anything #3982

sirideain opened this issue Apr 30, 2017 · 23 comments

Comments

@sirideain
Copy link

@sirideain sirideain commented Apr 30, 2017

Here is what I see with verbose. I get this trying to install clippy or rustfmt. I assume I would get it with other crates.

Updating registry https://github.com/rust-lang/crates.io-index
error: [20/-1] Cannot checkout to invalid path '3/n/nul'

@ncoish
Copy link

@ncoish ncoish commented Apr 30, 2017

I get the same failure when using cargo build or cargo update --verbose:

cargo update --verbose
Updating registry https://github.com/rust-lang/crates.io-index
error: failed to load source for a dependency on num

Caused by:
Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
[20/-1] Cannot checkout to invalid path '3/n/nul'

Platform: Windows 10
cargo 0.19.0-nightly (fa7584c 2017-04-26)

@azriel91
Copy link

@azriel91 azriel91 commented Apr 30, 2017

Yay. NUL is a reserved name.

Other names to avoid / handle:

Do not use the following reserved names for the name of a file:
CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9:

source: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

@withoutboats
Copy link
Contributor

@withoutboats withoutboats commented Apr 30, 2017

ping @alexcrichton @carols10cents not sure who else

We probably need to delete this crate and add all of these windows reserved names to the list of reserved names in crates.io.

@withoutboats
Copy link
Contributor

@withoutboats withoutboats commented Apr 30, 2017

Once nul is deleted I have a PR to reserve the names listed in @azriel91's post rust-lang/crates.io#695

@albel727
Copy link

@albel727 albel727 commented Apr 30, 2017

@withoutboats Why wait? Make your PR sooner, for it better get some review before someone else has an idea to get famous quickly.

@withoutboats
Copy link
Contributor

@withoutboats withoutboats commented Apr 30, 2017

@albel727 The PR is open (I linked to it), but I'm not sure what will happen if its deployed before nul is deleted.

@carols10cents
Copy link
Member

@carols10cents carols10cents commented Apr 30, 2017

What will happen is if there are any updates to the nul crate after the deploy, those updates won't work. Which I think is fine. I'd rather prevent more things that cause problems first.

@albel727
Copy link

@albel727 albel727 commented Apr 30, 2017

@withoutboats Ah, I thought it would be a PR against cargo itself. Nevermind me.

@carols10cents
Copy link
Member

@carols10cents carols10cents commented Apr 30, 2017

Deploying to production of the crates.io pr starting now.

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Apr 30, 2017

This issue cannot be fixed by blacklisting crate names on crates.

Cargo uses a trie, so you could always trigger the trie to break via things like uploading packages nula, nulb,... so that a nul folder is created in the filesystem trie. We need a crates trie blacklist that makes sure these folder names don't get created in the trie (falling back to storing them flat one level above), as well as blacklists for the existing name.

Unless I'm misunderstanding how the registry trie works.

@carols10cents
Copy link
Member

@carols10cents carols10cents commented Apr 30, 2017

@Manishearth The trie is 2 letters, so a crate named nula would go in nu/la. Idk if that still leaves problematic files

@withoutboats
Copy link
Contributor

@withoutboats withoutboats commented Apr 30, 2017

All of the Windows names are 3 or 4 characters so it should be fine.

@shepmaster
Copy link
Member

@shepmaster shepmaster commented Apr 30, 2017

Cargo uses a trie, so you could always trigger the trie to break via things like uploading packages nula, nulb,... so that a nul folder is created in the filesystem trie.

If the directory scheme were to be changed, I'd recommend switching to using a hash of the name for the prefix directories; this leads to better distribution of files per directory. Too many things in a directory can have poor performance implications.

@retep998
Copy link
Member

@retep998 retep998 commented Apr 30, 2017

If the directory scheme were to be changed, I'd recommend switching to using a hash of the name for the prefix directories; this leads to better distribution of files per directory. Too many things in a directory can have poor performance implications.

With the current system you could trivially create a few million crates with a certain prefix and easily DOS anyone with a filesystem that doesn't handle that many files in a folder.

@carols10cents
Copy link
Member

@carols10cents carols10cents commented Apr 30, 2017

Production deploy of crates.io done, and crate removed from the index. Please open new issues for any remaining problems.

@jckarter
Copy link

@jckarter jckarter commented May 1, 2017

A more robust way to address these kinds of problems on Windows might be to do your own path normalization before calling any Win32 or C filesystem APIs. A fully qualified absolute path like \\?\C:\nul\aux\prn\whatever gets minimal Win32-level processing before getting handed off to the NT layer where these legacy device names aren't a problem. Another benefit is that \\?\ paths can be up to 32768 TCHARs long instead of only MAX_PATH.

@shepmaster
Copy link
Member

@shepmaster shepmaster commented May 1, 2017

@jckarter the error message seems to indicate that its during the git checkout that the issue occurs; not from within Cargo's own code. Perhaps there's an underlying issue with git / libgit?

@ticki
Copy link

@ticki ticki commented May 1, 2017

Ideally, the crates with reserved names should be banned. This should be trivial to add in the crates.io server.

@ghost
Copy link

@ghost ghost commented May 1, 2017

I don't think any names that some OS does not allow, should be banned. This is a Windows bug and not a problem with the crate name.

@withoutboats
Copy link
Contributor

@withoutboats withoutboats commented May 1, 2017

@ticki That's exactly what we did. rust-lang/crates.io#695

@Manishearth
Copy link
Member

@Manishearth Manishearth commented May 1, 2017

This is a Windows bug and not a problem with the crate name.

We can't pass the buck on to windows like that. Yes, it is a windows bug. Windows won't change it and certainly not in a timeframe that solves the current breakage. It is our responsibility to fix it, regardless of the entity that can be blamed for the bug. Fixing a bug does not necessitate self-assigning fault for it.

@nightpool
Copy link

@nightpool nightpool commented May 1, 2017

I think mangling could also be a solution here that preserves backwards compatibility. What if we mangled only names with a reserved words with a prefix? So "nul" becomes "--nul", or something similar? Then current and future versions of cargo can add these crates, and old versions of cargo miss out on a vanishingly small number of crates but also don't get borked by checking out the repository.

@carols10cents
Copy link
Member

@carols10cents carols10cents commented May 1, 2017

Please open new issues for any new problems or discussions on further improvements

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.