Some recently built crates fail to unpack

[causal-agent]

Error observed so far with selectors = "0.4.0" and glutin = "0.4.6" dependencies.

Output of cargo build --verbose:

    Updating registry `https://github.com/rust-lang/crates.io-index`
unable to get packages from source

Caused by:
  failed to unpack package `selectors v0.4.0`

Caused by:
  No such file or directory (os error 2)

    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading xml-rs v0.2.2
 Downloading gleam v0.2.3
 Downloading objc v0.1.8
 Downloading dlib v0.2.0
 Downloading dylib v0.0.1
 Downloading core-foundation v0.2.0
 Downloading core-graphics v0.2.1
 Downloading wayland-kbd v0.3.4
 Downloading crossbeam v0.1.6
 Downloading osmesa-sys v0.0.5
 Downloading tempfile v1.1.3
 Downloading byteorder v0.4.2
 Downloading wayland-window v0.2.2
 Downloading shared_library v0.1.2
 Downloading wayland-scanner v0.5.8
 Downloading x11-dl v2.3.0
 Downloading glutin v0.4.6
unable to get packages from source

Caused by:
  failed to unpack package `glutin v0.4.6`

Caused by:
  No such file or directory (os error 2)

[thepowersgang]

Running strace cargo install selectors on my machine yields this snippet just before it fails:

utimes("/home/tpg/.multirust/toolchains/stable/cargo/registry/src/github.com-88ac128001ac3a9a/12652521203/selectors-0.4.0/src/tree.rs", {{1454023299, 0}, {1454023299, 0}}) = 0
chmod("/home/tpg/.multirust/toolchains/stable/cargo/registry/src/github.com-88ac128001ac3a9a/12652521203/selectors-0.4.0/src/tree.rs", 0644) = 0
open("/home/tpg/.multirust/toolchains/stable/cargo/registry/src/github.com-88ac128001ac3a9a/selectors-0.4.0/.cargo-ok", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = -1 ENOENT (No such file or directory)

It appears that there's a timestamp inserted in the path which isn't being added everywhere.

[wolfiestyle]

I discovered this while trying glium. Setting the glutin version to 0.4.5 in Cargo.lock seems to work as a workaround.

[thepowersgang]

Latest nightly does not show this error, but current beta does.

[tpickett66]

I'm seeing this behavior with the 0.7.0 release of cargo when fetching glutin 0.4.6.

[alexcrichton]

I'm working on a fix, thanks for the report!

cc @SimonSapin, this is a bug where a newer Cargo published an archive that is triggering a bug in older Cargos. You may want to republish selectors with an older Cargo to ensure the tarball is compatible with older Cargos.

[alexcrichton]

Ok, I've posted #2327 which should fix this error in the future.

If this crops up, the fix is unfortunately just that the original crate author needs to be contacted to republish with a version of Cargo with the fix or an older Cargo. Sorry for the inconvenience!

[SimonSapin]

You may want to republish selectors with an older Cargo to ensure the tarball is compatible with older Cargos.

That’s… ugh. This is gonna be annoying, the "bad" tarballs are immutable on crates.io, so maintainers need to pick new version numbers.

• What range of Cargo versions (nightly dates?) create "bad" tarballs? (How old "an older Cargo"?) The current Nightly according to multirust is 6ffd134 2016-01-26, and it looks like it it doesn’t have the fix yet.
• What versions fail to install them? @programble @thepowersgang @Darkstalker Is it practical for you to upgrade Cargo? Contrary to what multirust believes this can be done independently from upgrading rustc.
• Is there a way to tell that a given version of a crate.io package is "bad", other than trying to install it?

[alexcrichton]

Yes unfortunately maintainers will need to:

1. Yank these versions
2. Publish new versions
3. Recommend newer versions of Cargo

This bug was merged on 44c0d7b which was commited on 1/25 but didn't make it out until the 1/27 nightly. It looks like the fix didn't make it outlast night, sot he affected Cargo versions are the 1/27 nightly to what I presume is today's, 1/29 nightly.

Any version of Cargo prior to 1/27 will fail to install tarballs published by the affected Cargo versions.

There's not really anyway to tell crates.io whether a package is bad, all the tarballs being upload are indeed valid tarballs. Older cargos just interpreted a valid GNU archive in a buggy manner.

[nox]

There should be a test which tries to unpack a package with an older Cargo version.

[kamalmarhubi]

Is there any issue tracking remediation here? I agree with @nox that there should be a check, ideally on the server side at upload time.

[nox]

I meant that Cargo itself should include tests that unpack newly-created packages with older versions of Cargo, not a safety check at upload-time of third-party crates.

[kamalmarhubi]

Ah, that makes sense. I think that there should be a check at upload time given the append only nature of crates.io. I guess this should be an issue against that project instead.

[SimonSapin]

This downloads 389 crates uploaded between January 25 and Feb 4 16:04:06 2016 +0000 (according to git commit dates in the index)

git -C ~/.multirust/toolchains/nightly/cargo/registry/index/github.com-* log --format='%ai %s'|
grep -E '^2016-0(2|1-2[5-9])'|
grep Updating|
cut -d'`' -f2|
tr '#' ' '|
awk '{print "https://crates-io.s3-us-west-1.amazonaws.com/crates/" $1 "/" $1 "-" $2 ".crate"}'|
xargs wget

This shows the 27 of them whose tarball is in GNU format:

for i in *.crate; (gzcat $i|file -|grep -q GNU) && echo $i

blake2-rfc-0.2.14.crate
blas-sys-0.5.1.crate
blas-sys-0.5.2.crate
cargo-edit-0.1.1.crate
ears-0.3.3.crate
egc-0.1.1.crate
egc-0.1.2.crate
glutin-0.4.6.crate
indexing-0.1.0-alpha3.crate
inner-0.1.0.crate
lapack-sys-0.6.0.crate
lapack-sys-0.6.1.crate
libedgegrid-0.1.1.crate
libnotify-0.3.0.crate
libnotify-sys-0.3.0.crate
linked-hash-map-0.0.8.crate
lru-cache-0.0.6.crate
murmurhash64-0.3.0.crate
ndarray-0.4.0-alpha.1.crate
pandoc-0.3.0.crate
pandoc_ast-0.2.0.crate
rx-0.0.0.crate
selectors-0.4.0.crate
serde_xml-0.6.2.crate
skiplist-0.2.10.crate
splitmut-0.1.0.crate
xdrgen-0.1.9.crate

This shows other versions of the same crate for each of them:

for i in *.crate; (gzcat $i|file -|grep -q GNU) && 
(echo; echo '## ' $i; ls $(echo $i|rev|cut -d- -f2-|rev)-*|grep -v $i)

And with some manual filtering, these are the crates that likely need to be fixed (the crates in GNU format who do not have a later semver-compatible version in non-GNU format)

blake2-rfc-0.2.14.crate
cargo-edit-0.1.1.crate
egc-0.1.2.crate
indexing-0.1.0-alpha3.crate
inner-0.1.0.crate
lapack-sys-0.6.1.crate (has later, semver *incompatible* versions)
libedgegrid-0.1.1.crate
libnotify-0.3.0.crate
libnotify-sys-0.3.0.crate
linked-hash-map-0.0.8.crate
lru-cache-0.0.6.crate
murmurhash64-0.3.0.crate
pandoc-0.3.0.crate
pandoc_ast-0.2.0.crate
rx-0.0.0.crate
serde_xml-0.6.2.crate
skiplist-0.2.10.crate
splitmut-0.1.0.crate
xdrgen-0.1.9.crate