sccache v0.4.0 failed due to invalid `-E` options #1674

NobodyXu · 2023-03-21T00:30:16Z

This bug is discovered in rust-secure-code/cargo-auditable#87

It appears that sccache calls the Rust workspace wrapper with the arguments intended for the C compiler instead of the Rust compiler. It invokes /home/runner/.cargo/bin/cargo-auditable-v0.6.1 -E /tmp/sccachemNgxBK/testfile.c, which is not a valid call to rustc. The Rust compiler does not accept the -E option (gcc does), and it's being passed a C file, not a Rust file.

There's nothing I can do to fix this in cargo auditable. You'll have to file a bug against sccache. I'll research workarounds so you could get the caching going without waiting on a fix.

Shnatsel · 2023-03-21T00:39:17Z

A bit of context: cargo-auditable sets itself as the RUSTC_WORKSPACE_WRAPPER, proper support for which has been added to sccache in #1280

There's detailed technical info in the linked issue, starting here.

NobodyXu · 2023-03-21T12:23:04Z

Tried sccache v0.4.0, still same error.

NobodyXu · 2023-03-21T12:32:02Z

P.S. The bug was found in cargo-bins/cargo-quickinstall#206

NobodyXu · 2023-03-22T05:39:16Z

The same error now also occurs in cargo-binstall cargo-bins/cargo-binstall#934 even without cargo-auditable:

cargo test  --target x86_64-unknown-linux-gnu --no-default-features --features rustls,fancy-with-backtrace,zstd-thin,log_release_max_level_debug,pkg-config
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `sccache rustc - --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 2)
  --- stderr
  sccache: error: failed to execute compile
  sccache: caused by: Failed to send data to or receive data from server
  sccache: caused by: Failed to read response header
  sccache: caused by: failed to fill whole buffer
error: Recipe `unit-tests` failed on line 209 with exit code 101
Error: Process completed with exit code 101.

And also from running sccache --show-stats:

/home/runner/.cargo/bin/sccache --show-stats
sccache: error: failed to execute compile
sccache: caused by: Compiler not supported: "error: Found argument \'-E\' which wasn\'t expected, or isn\'t valid in this context\n\n  If you tried to supply \'-E\' as a value rather than a flag, use \'-- -E\'\n\nUsage: sccache [OPTIONS] <--dist-auth|--dist-status|--show-stats|--start-server|--stop-server|--zero-stats|--package-toolchain <EXE> <OUT>|CMD>\n\nFor more information try \'--help\'\n"
/home/runner/work/_actions/webiny/action-post-run/3.0.0/dist/post/index.js:1702
                error = new Error(`The process '${this.toolPath}' failed with exit code ${this.processExitCode}`);
                        ^

Error: The process '/home/runner/.cargo/bin/sccache' failed with exit code 2
    at ExecState._setResult (/home/runner/work/_actions/webiny/action-post-run/3.0.0/dist/post/index.js:1702:25)
    at ExecState.CheckComplete (/home/runner/work/_actions/webiny/action-post-run/3.0.0/dist/post/index.js:1685:18)
    at ChildProcess.<anonymous> (/home/runner/work/_actions/webiny/action-post-run/3.0.0/dist/post/index.js:1579:27)
    at ChildProcess.emit (node:events:527:28)
    at maybeClose (node:internal/child_process:1092:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

From CI " test (x86_64-unknown-linux-gnu, ubuntu-latest)"

NobodyXu · 2023-03-22T05:53:50Z

This error only occurs on Linux and does not occur on MacOS or Windows.

cc @Xuanwo @sylvestre

NobodyXu · 2023-03-22T05:57:27Z

Also, cargo-binstall seems to work fine when using sccache v0.4.0-pre.10 for caching, so this might also be a regression.

Xuanwo · 2023-03-22T06:00:21Z

Yes, I think it's a sccache bug. sccache --version is broken too:

:) sccache --version
sccache: error: failed to execute compile
sccache: caused by: Compiler not supported: "error: Found argument \'-E\' which wasn\'t expected, or isn\'t valid in this context\n\n  If you tried to supply \'-E\' as a value rather than a flag, use \'-- -E\'\n\nUsage: sccache [OPTIONS] <--dist-auth|--dist-status|--show-stats|--start-server|--stop-server|--zero-stats|--package-toolchain <EXE> <OUT>|CMD>\n\nFor more information try \'--help\'\n"

But I am unable to reproduce it by

building from the source of main.
buidling from the source of tag v0.4
download the binary from github releases.

But I can reproduce it via install via cargo binstall sccache.

NobodyXu · 2023-03-22T06:07:01Z

But I am unable to reproduce it by building from the source.

@Shnatsel was able to reproduce this using the command from rust-secure-code/cargo-auditable#87 (comment)

For cargo-binstall CI failure, you can try reproducing using:

RUSTC_WRAPPER=sccache CARGO_INCREMENTAL=0 cargo test  --target x86_64-unknown-linux-gnu --no-default-features --features rustls,fancy-with-backtrace,zstd-thin,log_release_max_level_debug,pkg-config

which is almost identical to the command we run on the CI except the use of GHA for caching.

Xuanwo · 2023-03-22T06:12:10Z

But I am unable to reproduce it by

building from the source of main.

buidling from the source of tag v0.4

download the binary from github releases.

But I can reproduce it via install via cargo binstall sccache.

The minimal repro looks like the following:

> rm ~/.cargo/bin/sccache
> cargo install sccache --force
> sccache --version <------- works
sccache 0.4.0

> rm ~/.cargo/bin/sccache
> cargo binstall sccache --force
> sccache --version <------- broken
sccache: error: failed to execute compile
sccache: caused by: Compiler not supported: "error: Found argument \'-E\' which wasn\'t expected, or isn\'t valid in this context\n\n  If you tried to supply \'-E\' as a value rather than a flag, use \'-- -E\'\n\nUsage: sccache [OPTIONS] <--dist-auth|--dist-status|--show-stats|--start-server|--stop-server|--zero-stats|--package-toolchain <EXE> <OUT>|CMD>\n\nFor more information try \'--help\'\n"

Xuanwo · 2023-03-22T06:13:17Z

Hi, @NobodyXu, is anything special to install sccache from cargo-binstall?

:( paru -Q cargo-binstall
cargo-binstall 0.21.3-1

NobodyXu · 2023-03-22T06:18:04Z

@Xuanwo No, it simply downloads and unpacks pre-built sccache from mozilla/sccache github release.

Xuanwo · 2023-03-22T06:18:33Z

Hi, @NobodyXu, is anything special to install sccache from cargo-binstall?

Oh, I got it. cargo-binstall will create a symlink for sccache:

lrwxrwxrwx    14 xuanwo 22 Mar 14:11  sccache -> sccache-v0.4.0
.rwxr-xr-x   14M xuanwo 21 Mar 20:50  sccache-v0.4.0

Sadly, it doesn't work on sccache. sccache requires to be called as sccache.

Xuanwo · 2023-03-22T06:19:05Z

After remove the symlink and rename sccache-v0.4.0 to sccache, it works again.

There is a short doc for this: https://github.com/mozilla/sccache#symbolic-links

NobodyXu · 2023-03-22T06:20:03Z

Thanks!

Also, the same error occurs on cargo-quickinstall where we use mozilla-actions/sccache-action@v0.0.2

rust-secure-code/cargo-auditable#87 (comment).

Xuanwo · 2023-03-22T06:21:16Z

Also, the same error occurs on cargo-quickinstall where we use mozilla-actions/sccache-action@v0.0.2

I think they are facing the same issues. Perhaps sccache can provide a clearer error message for this situation.

NobodyXu · 2023-03-22T06:25:04Z

@Xuanwo But mozilla-actions/sccache-action@v0.0.2 does not install a symlink, though cargo-auditable is installed with a symlink but TBF rustc itself is also a symlink.

Xuanwo · 2023-03-22T06:32:22Z

@Xuanwo But mozilla-actions/sccache-action@v0.0.2 does not install a symlink, though cargo-auditable is installed with a symlink but TBF rustc itself is also a symlink.

Yep, you are correct. cargo-bins/cargo-quickinstall#206 is another issue. I'll take a look now.

Xuanwo · 2023-03-22T06:36:41Z

Yep, you are correct. cargo-bins/cargo-quickinstall#206 is another issue. I'll take a look now.

I'm not sure about this:

 INFO This will install the following binaries:
 INFO   - cargo-auditable (cargo-auditable -> /home/runner/.cargo/bin/cargo-auditable-v0.6.1)
 INFO And create (or update) the following symlinks:
 INFO   - cargo-auditable (/home/runner/.cargo/bin/cargo-auditable -> cargo-auditable-v0.6.1)

Can you try install cargo-auditable directly? rustc is Ok to be a symlink.

Xuanwo · 2023-03-22T06:38:50Z

The code related to symlinks was written six years ago. I need some time to revisit it and see if we can improve it.

Xuanwo · 2023-03-22T06:44:32Z

Oh, I got it. cargo-binstall will create a symlink for sccache:
lrwxrwxrwx    14 xuanwo 22 Mar 14:11  sccache -> sccache-v0.4.0
.rwxr-xr-x   14M xuanwo 21 Mar 20:50  sccache-v0.4.0
Sadly, it doesn't work on sccache. sccache requires to be called as sccache.

Hi, @NobodyXu, is it a good idea for you to change symlink into hardlink on linux platform? This may be the quickest solution so far. This change can help users who use cargo-binstall to install sccache. Sorry for asking those changes because of the sccache behavior.

NobodyXu · 2023-03-22T06:45:22Z

Hi, @NobodyXu, is it a good idea for you to change symlink into hardlink on linux platform? This may be the quickest solution so far. This change can help users who use cargo-binstall to install sccache.

We actually have a tracking issue to enable --no-symlinks by default cargo-bins/cargo-binstall#731

NobodyXu · 2023-03-22T06:52:45Z

We actually have a tracking issue to enable --no-symlinks by default cargo-bins/cargo-binstall#731

Opened cargo-bins/cargo-binstall#936 for this

Xuanwo · 2023-03-22T06:55:47Z

Opened cargo-bins/cargo-binstall#936 for this

Thank you very much for considering my advice!

NobodyXu · 2023-03-22T06:58:27Z

@Xuanwo I tried install cargo-auditable with --no-symlinks but it still failed with the same error in CI:

error: failed to compile `cargo-deny v0.13.7`, intermediate artifacts can be found at `/tmp/cargo-install4jaa7D`

Caused by:
  failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `sccache /home/runner/.cargo/bin/cargo-auditable rustc - --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 2)
  --- stderr
  sccache: error: failed to execute compile
  sccache: caused by: Compiler not supported: "thread \'main\' panicked at \'Failed to invoke rustc! Make sure it\'s in your $PATH: Os { code: 2, kind: NotFound, message: \"No such file or directory\" }\', /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/cargo-auditable-0.6.1/src/rustc_wrapper.rs:90:10\nstack backtrace:\n   0: rust_begin_unwind\n             at ./rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:575:5\n   1: core::panicking::panic_fmt\n             at ./rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/panicking.rs:64:14\n   2: core::result::unwrap_failed\n             at ./rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/result.rs:1791:5\n   3: cargo_auditable::main\nnote: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.\n"

NOTE that it fails on Linux, MacOS and Windows all with the same error.

NobodyXu · 2023-03-22T06:59:02Z

Thank you very much for considering my advice!

No worries, it's actually long overdue.

Xuanwo · 2023-03-22T06:59:47Z

NOTE that it fails on Linux, MacOS and Windows all with the same error.

Yes, it's another issue and I am still investigating it.

drahnr · 2023-03-22T07:38:37Z

We actually have a tracking issue to enable --no-symlinks by default cargo-bins/cargo-binstall#731

Opened cargo-bins/cargo-binstall#936 for this

The short term solution is to mention in the README.md install with binstall is only supported with --no-symlinks. And then investigate further why the symlinks are such a headache.

NobodyXu · 2023-03-22T09:14:02Z

The short term solution is to mention in the README.md install with binstall is only supported with --no-symlinks. And then investigate further why the symlinks are such a headache.

I plan to publish a new release v0.22.0 once this PR and few others trivial PRs get merged, likely in 1 day or two.

NobodyXu · 2023-03-23T01:43:28Z

@drahnr cargo-binstall v0.22.0 has released.

NobodyXu · 2023-04-11T04:09:32Z

NOTE that it fails on Linux, MacOS and Windows all with the same error.

Yes, it's another issue and I am still investigating it.

@Xuanwo Friendly ping on this issue.
Is there any progress on this one?

Xuanwo · 2023-04-11T04:13:51Z

Is there any progress on this one?

Sorry for the delay. I haven't been working on this issue in recent days, so there has been no progress thus far.

NobodyXu · 2023-06-16T14:30:06Z

@Xuanwo I tried sccache v0.5.3 and this bug is still reproducible.

Xuanwo · 2023-06-16T14:35:04Z

@Xuanwo I tried sccache v0.5.3 and this bug is still reproducible.

Sorry, but I'm not working on this issue these days.

NobodyXu · 2023-06-16T14:42:32Z

@Xuanwo It's ok, I just want to try if latest sccache version has fixed this bug or not.

Xuanwo · 2023-06-16T14:44:00Z

@Xuanwo It's ok, I just want to try if latest sccache version has fixed this bug or not.

We will ping this issue if it got fixed.

NobodyXu · 2023-06-16T14:47:19Z

Thank you @Xuanwo

Seems to cause CI errors due to mozilla/sccache#1674

NobodyXu changed the title ~~[bug] Using sccache v0.4.0.pre-11 with cargo-auditable failed due to invalid -E options~~ sccache v0.4.0 failed due to invalid -E options Mar 22, 2023

Marwes added a commit to Marwes/gluon that referenced this issue Sep 9, 2023

chore: Disable scccache on CI

37b7ab9

Seems to cause CI errors due to mozilla/sccache#1674

Marwes mentioned this issue Sep 9, 2023

chore: Fix build on newer versions of rust gluon-lang/gluon#959

Merged

sccache v0.4.0 failed due to invalid -E options #1674

sccache v0.4.0 failed due to invalid -E options #1674

Comments

NobodyXu commented Mar 21, 2023

Shnatsel commented Mar 21, 2023

NobodyXu commented Mar 21, 2023

NobodyXu commented Mar 21, 2023 • edited

NobodyXu commented Mar 22, 2023 • edited

NobodyXu commented Mar 22, 2023

NobodyXu commented Mar 22, 2023

Xuanwo commented Mar 22, 2023 • edited

NobodyXu commented Mar 22, 2023

Xuanwo commented Mar 22, 2023

Xuanwo commented Mar 22, 2023 • edited

NobodyXu commented Mar 22, 2023 • edited

Xuanwo commented Mar 22, 2023

Xuanwo commented Mar 22, 2023 • edited

NobodyXu commented Mar 22, 2023 • edited

Xuanwo commented Mar 22, 2023

NobodyXu commented Mar 22, 2023

Xuanwo commented Mar 22, 2023

Xuanwo commented Mar 22, 2023

Xuanwo commented Mar 22, 2023

Xuanwo commented Mar 22, 2023 • edited

NobodyXu commented Mar 22, 2023

NobodyXu commented Mar 22, 2023

Xuanwo commented Mar 22, 2023

NobodyXu commented Mar 22, 2023

NobodyXu commented Mar 22, 2023

Xuanwo commented Mar 22, 2023

drahnr commented Mar 22, 2023

NobodyXu commented Mar 22, 2023

NobodyXu commented Mar 23, 2023

NobodyXu commented Apr 11, 2023

Xuanwo commented Apr 11, 2023

NobodyXu commented Jun 16, 2023

Xuanwo commented Jun 16, 2023

NobodyXu commented Jun 16, 2023

Xuanwo commented Jun 16, 2023

NobodyXu commented Jun 16, 2023

sccache v0.4.0 failed due to invalid `-E` options #1674

sccache v0.4.0 failed due to invalid `-E` options #1674

NobodyXu commented Mar 21, 2023 •

edited

NobodyXu commented Mar 22, 2023 •

edited

Xuanwo commented Mar 22, 2023 •

edited

Xuanwo commented Mar 22, 2023 •

edited

NobodyXu commented Mar 22, 2023 •

edited

Xuanwo commented Mar 22, 2023 •

edited

NobodyXu commented Mar 22, 2023 •

edited

Xuanwo commented Mar 22, 2023 •

edited