rustup nightly install failing #1092

davidroeca · 2017-05-05T03:34:01Z

Just getting back into rust and installed rustup. Tried to install the nightly build and encountered the following error:

$ rustup install nightly
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: downloading component 'rustc'
info: downloading component 'rust-std'
info: downloading component 'cargo'
info: downloading component 'rust-docs'
info: installing component 'rustc'
info: rolling back changes
error: failed to extract package
info: caused by: failed to unpack `rustc-nightly-x86_64-unknown-linux-gnu/rustc/share/man/man1/rustdoc.1` into `/home/david/.rustup/tmp/wnj79prt3m8vg6f2_dir/rustc/share/man/man1/rustdoc.1`
info: caused by: No such file or directory (os error 2)

Not sure if this is a nightly issue, a rustup issue, or my own lack of experience with rustup. Currently running Linux Mint 18

messense · 2017-05-05T03:58:53Z

Same issue on macOS 11.12

est31 · 2017-05-05T04:03:05Z

I can confirm on Ubuntu 17.04. rustup update is broken as well. @davidroeca this is a very recent issue.

gyscos · 2017-05-05T04:10:22Z

Here is what I get after installing the current rustup from git (to get debug info and backtraces):

% ~/.cargo/bin/rustup update nightly
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: downloading component 'rustc'
info: downloading component 'rust-std'
info: downloading component 'cargo'
info: downloading component 'rust-docs'
info: downloading component 'rust-analysis'
info: downloading component 'rls'
info: downloading component 'rust-src'
info: downloading component 'rust-std' for 'asmjs-unknown-emscripten'
info: downloading component 'rust-std' for 'wasm32-unknown-emscripten'
info: installing component 'rustc'
info: rolling back changes
error: failed to extract package
info: caused by: failed to unpack `rustc-nightly-x86_64-unknown-linux-gnu/rustc/share/man/man1/rustdoc.1` into `/home/gyscos/.rustup/tmp/_mffp8148_7wf8w9_dir/rustc/share/man/man1/rustdoc.1`
info: caused by: No such file or directory (os error 2)
info: backtrace:

stack backtrace:
   0:     0x55f2010a2894 - backtrace::backtrace::libunwind::trace
                        at /home/gyscos/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.0/src/backtrace/libunwind.rs:53
                         - backtrace::backtrace::trace<closure>
                        at /home/gyscos/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.0/src/backtrace/mod.rs:42
   1:     0x55f2010a372f - backtrace::capture::{{impl}}::new
                        at /home/gyscos/fetched/rustup.rs/target/debug/build/backtrace-6ddf4a4263d5dc15/out/capture.rs:79
   2:     0x55f20109ae3c - error_chain::make_backtrace
                        at /home/gyscos/.cargo/registry/src/github.com-1ecc6299db9ec823/error-chain-0.7.1/src/lib.rs:383
   3:     0x55f200b8032c - core::ops::FnOnce::call_once<fn() -> core::option::Option<alloc::arc::Arc<backtrace::capture::Backtrace>>,()>
                        at /checkout/src/libcore/ops.rs:2626
   4:     0x55f200b5926c - core::option::{{impl}}::or_else<alloc::arc::Arc<backtrace::capture::Backtrace>,fn() -> core::option::Option<alloc::arc::Arc<backtrace::capture::Backtrace>>>
                        at /checkout/src/libcore/option.rs:640
   5:     0x55f200b44c6a - error_chain::{{impl}}::new<rustup_dist::errors::Error>
                        at /home/gyscos/.cargo/registry/src/github.com-1ecc6299db9ec823/error-chain-0.7.1/src/lib.rs:437
   6:     0x55f200bf8a5b - rustup_dist::errors::{{impl}}::chain_err::{{closure}}<(),std::io::error::Error,closure,rustup_dist::errors::ErrorKind>
                        at /home/gyscos/fetched/rustup.rs/<error_chain_processed macros>:117
   7:     0x55f200b74c46 - core::result::{{impl}}::map_err<(),std::io::error::Error,rustup_dist::errors::Error,closure>
                        at /checkout/src/libcore/result.rs:486
   8:     0x55f200bf79d1 - rustup_dist::errors::{{impl}}::chain_err<(),std::io::error::Error,closure,rustup_dist::errors::ErrorKind>
                        at /home/gyscos/fetched/rustup.rs/<error_chain_processed macros>:115
   9:     0x55f200bdacde - rustup_dist::component::package::unpack_without_first_dir<flate2::gz::DecoderReader<std::fs::File>>
                        at src/rustup-dist/src/component/package.rs:202
  10:     0x55f200bd9f7b - rustup_dist::component::package::{{impl}}::new<flate2::gz::DecoderReader<std::fs::File>>
                        at src/rustup-dist/src/component/package.rs:183
  11:     0x55f200bdb4f9 - rustup_dist::component::package::{{impl}}::new<std::fs::File>
                        at src/rustup-dist/src/component/package.rs:232
  12:     0x55f200bdba4c - rustup_dist::component::package::{{impl}}::new_file
                        at src/rustup-dist/src/component/package.rs:236
  13:     0x55f200be427b - rustup_dist::manifestation::{{impl}}::update
                        at src/rustup-dist/src/manifestation.rs:170
  14:     0x55f200bcc817 - rustup_dist::dist::update_from_dist_
                        at src/rustup-dist/src/dist.rs:656
  15:     0x55f200bcbcce - rustup_dist::dist::update_from_dist
                        at src/rustup-dist/src/dist.rs:619
  16:     0x55f200b1e26e - rustup::install::{{impl}}::run
                        at src/rustup/install.rs:50
  17:     0x55f200b09851 - rustup::toolchain::{{impl}}::install
                        at src/rustup/toolchain.rs:113
  18:     0x55f200b0aa6f - rustup::toolchain::{{impl}}::install_from_dist_inner
                        at src/rustup/toolchain.rs:175
  19:     0x55f200b0a5b5 - rustup::toolchain::{{impl}}::install_from_dist
                        at src/rustup/toolchain.rs:170
  20:     0x55f2009f7a2d - rustup_init::rustup_mode::update
                        at src/rustup-cli/rustup_mode.rs:488
  21:     0x55f2009ec639 - rustup_init::rustup_mode::main
                        at src/rustup-cli/rustup_mode.rs:33
  22:     0x55f200a15dd8 - rustup_init::run_multirust
                        at src/rustup-cli/main.rs:82
  23:     0x55f200a15b86 - rustup_init::main
                        at src/rustup-cli/main.rs:57
  24:     0x55f2010fc15a - panic_unwind::__rust_maybe_catch_panic
                        at /checkout/src/libpanic_unwind/lib.rs:98
  25:     0x55f2010f59b1 - std::panicking::try<(),fn()>
                        at /checkout/src/libstd/panicking.rs:433
                         - std::panic::catch_unwind<fn(),()>
                        at /checkout/src/libstd/panic.rs:361
                         - std::rt::lang_start
                        at /checkout/src/libstd/rt.rs:57
  26:     0x55f200a1a8a2 - main
  27:     0x7f5759081510 - __libc_start_main
  28:     0x55f200971539 - _start
  29:                0x0 - <unknown>

TimNN · 2017-05-05T04:16:40Z

strace output:

info: installing component 'rustc'
open("/home/logic/.rustup/tmp/hmzwj4cotkanpzvj_file", O_RDONLY|O_CLOEXEC) = 3
mkdir("/home/logic/.rustup/tmp/3v47lhv9_ptrqrvj_dir", 0777) = 0
open("/home/logic/.rustup/tmp/3v47lhv9_ptrqrvj_dir/rustc/share/man/man1/rustdoc.1", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = -1 ENOENT (No such file or directory)
open("/home/logic/.rustup/tmp/3v47lhv9_ptrqrvj_dir", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3

TimNN · 2017-05-05T04:22:48Z

Ok, the problem seems to be that the tar.gz generation changed and no longer includes directory entries, so the tar.gz extractor used by rustup no longer creates any directories.

jonhoo · 2017-05-05T04:24:30Z

Can confirm on Arch Linux as well.

alexcrichton · 2017-05-05T04:25:16Z

@ranma42 this is likely related to rust-lang/rust#41600, notably this range of changes, mind taking a look?

brson · 2017-05-05T04:25:20Z

Thanks for investigating @TimNN.

I might guess it was this PR updating rust-installer that caused the change on Rust's side.

brson · 2017-05-05T04:25:31Z

Damn you @alexcrichton!

TimNN · 2017-05-05T04:28:33Z

This causes the problem, I think rust-lang/rust-installer@4f99485...4cf7397#diff-534b12d64247754d4938410a9ba07bd3R274

find "$CFG_INPUT" \( -type d -empty \) -or \( -not -type d \) \
    | rev | sort | rev | tar -cf "$CFG_OUTPUT.tar" -T -
need_ok "failed to tar"

Not a find expert here, but I think that means that only empty directories are included in the output .tar (and files, of course).

TimNN · 2017-05-05T04:33:10Z

Rustup does the unpacking of tar files here: https://github.com/rust-lang-nursery/rustup.rs/blob/master/src/rustup-dist/src/component/package.rs#L202, however unpack requires all intermediary directories to have been created: https://docs.rs/tar/0.4.11/tar/struct.Entry.html#method.unpack

brson · 2017-05-05T04:42:50Z

Is it obvious to anyone how to change the tar command to include the directories again? I think we could just remove all of "( -type d -empty ) -or ( -not -type d ) " but I'm not sure.

Safe thing would be to just revert rust-installer.

@alexcrichton What time does the next nightly start building?

alexcrichton · 2017-05-05T04:44:14Z

@brson needs to be past @bors by 0:00 UTC which gives us ~19 hours from this comment

jonhoo · 2017-05-05T04:44:46Z

@brson I believe the find options can be removed, and in fact that point we could probably just use tar directly instead of using find at all..

jonhoo · 2017-05-05T04:45:30Z

Haha, yes, exactly like rust-lang/rust-installer#61.

ranma42 · 2017-05-05T06:21:38Z

@brson just removing the \( -type d -empty \) -or \( -not -type d \) condition would cause files to be included multiple times, because by default tar recurses into subdirectories.
This would list all folders (both empty and non-empty) in the archive, but I expect it to still cause issues because the folder entry might appear after the files it contains.

find "$CFG_INPUT" | rev | sort | rev | tar --no-recursion -cf "$CFG_OUTPUT.tar" -T -
need_ok "failed to tar"

@jonhoo yes, that restores the order in which tar lists the archive entries, but it also worsen the compression.

A fix on the rustup side would be to always create the full path (just like tar and the other tools that manage tar files do).
EDIT: a fix on the rust-installer side that does not regress the compression might be something along the lines of

(find "$CFG_INPUT" -type d ; \ # depth-first listing of directory tree
 find "$CFG_INPUT" -not -type d | rev | sort | rev) \ # regrouped listing
  | tar --no-recursion -cf "$CFG_OUTPUT.tar" -T -
need_ok "failed to tar"

What is the best way to test if it works?

brson · 2017-05-05T07:21:24Z

I submitted a PR to update rust-installer with @TimNN's patch to use the simple tar invocation. It can be reoptimized later.

A fix on the rustup side would be to always create the full path (just like tar and the other tools that manage tar files do).

That does seem reasonable. After it was in place for some length of time we could modify the tarballs again to remove these directories.

What is the best way to test if it works?

Hm, it's actually not simple to feed rustup an arbitrary tarball that I can think of. The easiest way to test might be to make the update and send it back through nightly and see if it breaks again.

leonardo-m · 2017-05-05T07:38:37Z

I hate tar. I prefer to use a single program like 7-zip that fetches by itself the files, compresses them, and later rebuilds the original directory tree and decompresses the files.

fschutt · 2017-05-05T08:19:17Z

Is there a way to fix this yet for travis, like a configuration setting to fetch the latest rustup? Tried to setup Travis CI, but it currently fails to install rustc: https://travis-ci.org/sharazam/printpdf/jobs/229028283#L131

frewsxcv · 2017-05-05T12:44:56Z

@Sharazam If I were you (and to others who get this email), I'd enable the allow_failures setting for nightly Rust on Travis CI. Even their docs recommend this:

https://docs.travis-ci.com/user/languages/rust/#Choosing-the-Rust-version

language: rust
rust:
  - stable
  - beta
  - nightly
matrix:
  allow_failures:
    - rust: nightly

EDIT: forgot to mention 'nightly Rust'

cuviper · 2017-05-05T14:35:52Z

@ranma42 FWIW in my Rust rewrite (rust-lang/rust#41569) I can easily tar just directories and then still have your rev-sorted file list. I've noted this issue, so I'll add something to that effect today.

These changes configure the trust template to use the correct credentials and project name. The rest was left as is. There seems to be an issue with rustup at the moment rust-lang/rustup#1092. As suggested in rust-lang/rustup#1092 (comment) I've enabled the `allow_failures` feature for nightly builds.

ranma42 · 2017-05-05T15:08:09Z

@cuviper You're right that is probably easier and less error prone in your rewrite (I see that you also skip the temporary tar file... nice! :D ).
I still think rustup should be able to untar archives like those generated last night, but leaving rust-installer as-is until the Rust rewrite is merged should be fine (well, I hope so; the next nightly build will tell us).

See rust-lang/rustup#1092 Solution: pin nightly to a specific release

brson · 2017-05-05T16:12:05Z

@Sharazam One way is to temporarily 'pin' your CI to a specific nightly from the archives with e.g. --toolchain=nightly-2017-05-03.

@TimNN

Update rust-installer to fix rust-lang/rustup#1092 r? @TimNN cc @alexcrichton @ranma42

brson · 2017-05-05T23:15:53Z

Not actually fixed yet @bors!

brson · 2017-05-05T23:16:32Z

The fix is upstream and tomorrow's nightly should work.

* Move CI to trust These changes configure the trust template to use the correct credentials and project name. The rest was left as is. There seems to be an issue with rustup at the moment rust-lang/rustup#1092. As suggested in rust-lang/rustup#1092 (comment) I've enabled the `allow_failures` feature for nightly builds. * Allow failures for beta channel * Add beta channels for travis and appveyor

fschutt · 2017-05-06T09:48:00Z

Nightly works again (Travis). Thanks.

Update tar and use new safe unpacking function The `unpack_in` function automatically handles the creation of the directories and validates the path name, but it is only available since version 0.4.11 of the `tar` crate. This is the rustup side of fixing #1092.

The `unpack` function assumes that the directory in which the file is being extracted exists, while most `tar` tools will automatically create the intermediate directories if they are missing. This would have avoided rust-lang#1092.

Ensure that intermediate directories exist when unpacking an entry The `unpack` function assumes that the directory in which the file is being extracted exists, while most `tar` tools will automatically create the intermediate directories if they are missing. This would have avoided #1092.

TimNN mentioned this issue May 5, 2017

Remove intelligent tar sort logic rust-lang/rust-installer#61

Merged

brson mentioned this issue May 5, 2017

Update rust-installer to fix rust-lang-nursery/rustup.rs#1092 rust-lang/rust#41765

Merged

sztosz mentioned this issue May 5, 2017

Clippy 0.0.131 fails to compile with nightly 2017-05-04 rust-lang/rust-clippy#1727

Closed

Diggsey mentioned this issue May 5, 2017

Test generated manifests against rustup rust-lang/rust#41767

Closed

alexeyzab mentioned this issue May 5, 2017

Move CI to trust XAMPPRocky/tokei#122

Merged

mixedCase mentioned this issue May 5, 2017

RLS installation assumes rustup can self-update editor-rs/vscode-rust#225

Closed

This was referenced May 5, 2017

Problem: make comparison doctests coverage better PumpkinDB/PumpkinDB#271

Merged

Problem: Make error messages more helpful PumpkinDB/PumpkinDB#270

Merged

yrashk added a commit to PumpkinDB/PumpkinDB that referenced this issue May 5, 2017

Problem: current nightly builds are broken

85c13a3

See rust-lang/rustup#1092 Solution: pin nightly to a specific release

bors added a commit to rust-lang/rust that referenced this issue May 5, 2017

Auto merge of #41765 - brson:installer, r=alexcrichton

302dfd6

Update rust-installer to fix rust-lang/rustup#1092 r? @TimNN cc @alexcrichton @ranma42

bors closed this as completed in rust-lang/rust@f59931d May 5, 2017

pgerber mentioned this issue May 5, 2017

zero exit code on failing rustup update #1095

Closed

FaultyRAM mentioned this issue May 5, 2017

rustup default nightly fails on windows #1096

Closed

brson reopened this May 5, 2017

Diggsey closed this as completed May 6, 2017

ranma42 mentioned this issue May 8, 2017

Ensure that intermediate directories exist when unpacking an entry #1098

Merged

cuviper mentioned this issue May 8, 2017

Update to the Rusty rust-installer rust-lang/rust#41843

Merged

Mark-Simulacrum mentioned this issue Jun 22, 2017

Test generated manifests against rustup rust-lang/rust-central-station#17

Closed

reading123 mentioned this issue Dec 20, 2017

Failed to install because of compile error osa1/tiny#61

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustup nightly install failing #1092

rustup nightly install failing #1092

davidroeca commented May 5, 2017

messense commented May 5, 2017

est31 commented May 5, 2017

gyscos commented May 5, 2017

TimNN commented May 5, 2017

TimNN commented May 5, 2017

jonhoo commented May 5, 2017

alexcrichton commented May 5, 2017 •

edited

Loading

brson commented May 5, 2017

brson commented May 5, 2017

TimNN commented May 5, 2017 •

edited

Loading

TimNN commented May 5, 2017

brson commented May 5, 2017

alexcrichton commented May 5, 2017

jonhoo commented May 5, 2017

jonhoo commented May 5, 2017

ranma42 commented May 5, 2017 •

edited

Loading

brson commented May 5, 2017

leonardo-m commented May 5, 2017

fschutt commented May 5, 2017

frewsxcv commented May 5, 2017 •

edited

Loading

cuviper commented May 5, 2017 •

edited

Loading

ranma42 commented May 5, 2017

brson commented May 5, 2017

brson commented May 5, 2017

brson commented May 5, 2017

fschutt commented May 6, 2017

rustup nightly install failing #1092

rustup nightly install failing #1092

Comments

davidroeca commented May 5, 2017

messense commented May 5, 2017

est31 commented May 5, 2017

gyscos commented May 5, 2017

TimNN commented May 5, 2017

TimNN commented May 5, 2017

jonhoo commented May 5, 2017

alexcrichton commented May 5, 2017 • edited Loading

brson commented May 5, 2017

brson commented May 5, 2017

TimNN commented May 5, 2017 • edited Loading

TimNN commented May 5, 2017

brson commented May 5, 2017

alexcrichton commented May 5, 2017

jonhoo commented May 5, 2017

jonhoo commented May 5, 2017

ranma42 commented May 5, 2017 • edited Loading

brson commented May 5, 2017

leonardo-m commented May 5, 2017

fschutt commented May 5, 2017

frewsxcv commented May 5, 2017 • edited Loading

cuviper commented May 5, 2017 • edited Loading

ranma42 commented May 5, 2017

brson commented May 5, 2017

brson commented May 5, 2017

brson commented May 5, 2017

fschutt commented May 6, 2017

alexcrichton commented May 5, 2017 •

edited

Loading

TimNN commented May 5, 2017 •

edited

Loading

ranma42 commented May 5, 2017 •

edited

Loading

frewsxcv commented May 5, 2017 •

edited

Loading

cuviper commented May 5, 2017 •

edited

Loading