New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fresh install unable to install anything #3982

Closed
sirideain opened this Issue Apr 30, 2017 · 23 comments

Comments

Projects
None yet
@sirideain

sirideain commented Apr 30, 2017

Here is what I see with verbose. I get this trying to install clippy or rustfmt. I assume I would get it with other crates.

Updating registry https://github.com/rust-lang/crates.io-index
error: [20/-1] Cannot checkout to invalid path '3/n/nul'

@ncoish

This comment has been minimized.

Show comment
Hide comment
@ncoish

ncoish Apr 30, 2017

I get the same failure when using cargo build or cargo update --verbose:

cargo update --verbose
Updating registry https://github.com/rust-lang/crates.io-index
error: failed to load source for a dependency on num

Caused by:
Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
[20/-1] Cannot checkout to invalid path '3/n/nul'

Platform: Windows 10
cargo 0.19.0-nightly (fa7584c 2017-04-26)

ncoish commented Apr 30, 2017

I get the same failure when using cargo build or cargo update --verbose:

cargo update --verbose
Updating registry https://github.com/rust-lang/crates.io-index
error: failed to load source for a dependency on num

Caused by:
Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
[20/-1] Cannot checkout to invalid path '3/n/nul'

Platform: Windows 10
cargo 0.19.0-nightly (fa7584c 2017-04-26)

@azriel91

This comment has been minimized.

Show comment
Hide comment
@azriel91

azriel91 Apr 30, 2017

Yay. NUL is a reserved name.

Other names to avoid / handle:

Do not use the following reserved names for the name of a file:
CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9:

source: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

azriel91 commented Apr 30, 2017

Yay. NUL is a reserved name.

Other names to avoid / handle:

Do not use the following reserved names for the name of a file:
CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9:

source: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

@withoutboats

This comment has been minimized.

Show comment
Hide comment
@withoutboats

withoutboats Apr 30, 2017

Contributor

ping @alexcrichton @carols10cents not sure who else

We probably need to delete this crate and add all of these windows reserved names to the list of reserved names in crates.io.

Contributor

withoutboats commented Apr 30, 2017

ping @alexcrichton @carols10cents not sure who else

We probably need to delete this crate and add all of these windows reserved names to the list of reserved names in crates.io.

@withoutboats

This comment has been minimized.

Show comment
Hide comment
@withoutboats

withoutboats Apr 30, 2017

Contributor

Once nul is deleted I have a PR to reserve the names listed in @azriel91's post rust-lang/crates.io#695

Contributor

withoutboats commented Apr 30, 2017

Once nul is deleted I have a PR to reserve the names listed in @azriel91's post rust-lang/crates.io#695

@albel727

This comment has been minimized.

Show comment
Hide comment
@albel727

albel727 Apr 30, 2017

@withoutboats Why wait? Make your PR sooner, for it better get some review before someone else has an idea to get famous quickly.

albel727 commented Apr 30, 2017

@withoutboats Why wait? Make your PR sooner, for it better get some review before someone else has an idea to get famous quickly.

@withoutboats

This comment has been minimized.

Show comment
Hide comment
@withoutboats

withoutboats Apr 30, 2017

Contributor

@albel727 The PR is open (I linked to it), but I'm not sure what will happen if its deployed before nul is deleted.

Contributor

withoutboats commented Apr 30, 2017

@albel727 The PR is open (I linked to it), but I'm not sure what will happen if its deployed before nul is deleted.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Apr 30, 2017

Member

What will happen is if there are any updates to the nul crate after the deploy, those updates won't work. Which I think is fine. I'd rather prevent more things that cause problems first.

Member

carols10cents commented Apr 30, 2017

What will happen is if there are any updates to the nul crate after the deploy, those updates won't work. Which I think is fine. I'd rather prevent more things that cause problems first.

@albel727

This comment has been minimized.

Show comment
Hide comment
@albel727

albel727 Apr 30, 2017

@withoutboats Ah, I thought it would be a PR against cargo itself. Nevermind me.

albel727 commented Apr 30, 2017

@withoutboats Ah, I thought it would be a PR against cargo itself. Nevermind me.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Apr 30, 2017

Member

Deploying to production of the crates.io pr starting now.

Member

carols10cents commented Apr 30, 2017

Deploying to production of the crates.io pr starting now.

@Manishearth

This comment has been minimized.

Show comment
Hide comment
@Manishearth

Manishearth Apr 30, 2017

Member

This issue cannot be fixed by blacklisting crate names on crates.

Cargo uses a trie, so you could always trigger the trie to break via things like uploading packages nula, nulb,... so that a nul folder is created in the filesystem trie. We need a crates trie blacklist that makes sure these folder names don't get created in the trie (falling back to storing them flat one level above), as well as blacklists for the existing name.

Unless I'm misunderstanding how the registry trie works.

Member

Manishearth commented Apr 30, 2017

This issue cannot be fixed by blacklisting crate names on crates.

Cargo uses a trie, so you could always trigger the trie to break via things like uploading packages nula, nulb,... so that a nul folder is created in the filesystem trie. We need a crates trie blacklist that makes sure these folder names don't get created in the trie (falling back to storing them flat one level above), as well as blacklists for the existing name.

Unless I'm misunderstanding how the registry trie works.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Apr 30, 2017

Member

@Manishearth The trie is 2 letters, so a crate named nula would go in nu/la. Idk if that still leaves problematic files

Member

carols10cents commented Apr 30, 2017

@Manishearth The trie is 2 letters, so a crate named nula would go in nu/la. Idk if that still leaves problematic files

@withoutboats

This comment has been minimized.

Show comment
Hide comment
@withoutboats

withoutboats Apr 30, 2017

Contributor

All of the Windows names are 3 or 4 characters so it should be fine.

Contributor

withoutboats commented Apr 30, 2017

All of the Windows names are 3 or 4 characters so it should be fine.

@shepmaster

This comment has been minimized.

Show comment
Hide comment
@shepmaster

shepmaster Apr 30, 2017

Member

Cargo uses a trie, so you could always trigger the trie to break via things like uploading packages nula, nulb,... so that a nul folder is created in the filesystem trie.

If the directory scheme were to be changed, I'd recommend switching to using a hash of the name for the prefix directories; this leads to better distribution of files per directory. Too many things in a directory can have poor performance implications.

Member

shepmaster commented Apr 30, 2017

Cargo uses a trie, so you could always trigger the trie to break via things like uploading packages nula, nulb,... so that a nul folder is created in the filesystem trie.

If the directory scheme were to be changed, I'd recommend switching to using a hash of the name for the prefix directories; this leads to better distribution of files per directory. Too many things in a directory can have poor performance implications.

@retep998

This comment has been minimized.

Show comment
Hide comment
@retep998

retep998 Apr 30, 2017

Member

If the directory scheme were to be changed, I'd recommend switching to using a hash of the name for the prefix directories; this leads to better distribution of files per directory. Too many things in a directory can have poor performance implications.

With the current system you could trivially create a few million crates with a certain prefix and easily DOS anyone with a filesystem that doesn't handle that many files in a folder.

Member

retep998 commented Apr 30, 2017

If the directory scheme were to be changed, I'd recommend switching to using a hash of the name for the prefix directories; this leads to better distribution of files per directory. Too many things in a directory can have poor performance implications.

With the current system you could trivially create a few million crates with a certain prefix and easily DOS anyone with a filesystem that doesn't handle that many files in a folder.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Apr 30, 2017

Member

Production deploy of crates.io done, and crate removed from the index. Please open new issues for any remaining problems.

Member

carols10cents commented Apr 30, 2017

Production deploy of crates.io done, and crate removed from the index. Please open new issues for any remaining problems.

@jckarter

This comment has been minimized.

Show comment
Hide comment
@jckarter

jckarter May 1, 2017

A more robust way to address these kinds of problems on Windows might be to do your own path normalization before calling any Win32 or C filesystem APIs. A fully qualified absolute path like \\?\C:\nul\aux\prn\whatever gets minimal Win32-level processing before getting handed off to the NT layer where these legacy device names aren't a problem. Another benefit is that \\?\ paths can be up to 32768 TCHARs long instead of only MAX_PATH.

jckarter commented May 1, 2017

A more robust way to address these kinds of problems on Windows might be to do your own path normalization before calling any Win32 or C filesystem APIs. A fully qualified absolute path like \\?\C:\nul\aux\prn\whatever gets minimal Win32-level processing before getting handed off to the NT layer where these legacy device names aren't a problem. Another benefit is that \\?\ paths can be up to 32768 TCHARs long instead of only MAX_PATH.

@shepmaster

This comment has been minimized.

Show comment
Hide comment
@shepmaster

shepmaster May 1, 2017

Member

@jckarter the error message seems to indicate that its during the git checkout that the issue occurs; not from within Cargo's own code. Perhaps there's an underlying issue with git / libgit?

Member

shepmaster commented May 1, 2017

@jckarter the error message seems to indicate that its during the git checkout that the issue occurs; not from within Cargo's own code. Perhaps there's an underlying issue with git / libgit?

@ticki

This comment has been minimized.

Show comment
Hide comment
@ticki

ticki May 1, 2017

Ideally, the crates with reserved names should be banned. This should be trivial to add in the crates.io server.

ticki commented May 1, 2017

Ideally, the crates with reserved names should be banned. This should be trivial to add in the crates.io server.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 1, 2017

I don't think any names that some OS does not allow, should be banned. This is a Windows bug and not a problem with the crate name.

ghost commented May 1, 2017

I don't think any names that some OS does not allow, should be banned. This is a Windows bug and not a problem with the crate name.

@withoutboats

This comment has been minimized.

Show comment
Hide comment
@withoutboats

withoutboats May 1, 2017

Contributor

@ticki That's exactly what we did. rust-lang/crates.io#695

Contributor

withoutboats commented May 1, 2017

@ticki That's exactly what we did. rust-lang/crates.io#695

@Manishearth

This comment has been minimized.

Show comment
Hide comment
@Manishearth

Manishearth May 1, 2017

Member

This is a Windows bug and not a problem with the crate name.

We can't pass the buck on to windows like that. Yes, it is a windows bug. Windows won't change it and certainly not in a timeframe that solves the current breakage. It is our responsibility to fix it, regardless of the entity that can be blamed for the bug. Fixing a bug does not necessitate self-assigning fault for it.

Member

Manishearth commented May 1, 2017

This is a Windows bug and not a problem with the crate name.

We can't pass the buck on to windows like that. Yes, it is a windows bug. Windows won't change it and certainly not in a timeframe that solves the current breakage. It is our responsibility to fix it, regardless of the entity that can be blamed for the bug. Fixing a bug does not necessitate self-assigning fault for it.

@nightpool

This comment has been minimized.

Show comment
Hide comment
@nightpool

nightpool May 1, 2017

I think mangling could also be a solution here that preserves backwards compatibility. What if we mangled only names with a reserved words with a prefix? So "nul" becomes "--nul", or something similar? Then current and future versions of cargo can add these crates, and old versions of cargo miss out on a vanishingly small number of crates but also don't get borked by checking out the repository.

nightpool commented May 1, 2017

I think mangling could also be a solution here that preserves backwards compatibility. What if we mangled only names with a reserved words with a prefix? So "nul" becomes "--nul", or something similar? Then current and future versions of cargo can add these crates, and old versions of cargo miss out on a vanishingly small number of crates but also don't get borked by checking out the repository.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents May 1, 2017

Member

Please open new issues for any new problems or discussions on further improvements

Member

carols10cents commented May 1, 2017

Please open new issues for any new problems or discussions on further improvements

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment