Skip to content

/ servo

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sccache consistently errors on CI since Rust nightly-2019-11-07 #24714

Closed
SimonSapin opened this issue Nov 12, 2019 · 8 comments
Closed

sccache consistently errors on CI since Rust nightly-2019-11-07 #24714

SimonSapin opened this issue Nov 12, 2019 · 8 comments
Labels
I-bustage

Comments

@SimonSapin
Copy link
Member

@SimonSapin SimonSapin commented Nov 12, 2019

https://community-tc.services.mozilla.com/tasks/NvN6U22bQ-C626hsVGu4fA

error: failed to execute compile
caused by: error reading compile response from server
caused by: Failed to read response header
caused by: failed to fill whole buffer
error: could not compile `script`.

The error message is the same as in #24510, but setting SCCACHE_IDLE_TIMEOUT to zero does not help.

Merges in the regression range of nightly-2019-11-06 to nightly-2019-11-07 rust-lang/rust@1423bec...3804876:
@jdm jdm added the I-bustage label Nov 12, 2019
@jdm
Copy link
Member

@jdm jdm commented Nov 12, 2019

What's the best way to tackle this? Try to reproduce this locally?
@SimonSapin
Copy link
Member Author

@SimonSapin SimonSapin commented Nov 12, 2019

I haven’t managed to reproduce when building on my machine with the same version of sccache.

The output shows that this errors occurs when read_exact fails to read 4 bytes from a BufReader<TcpStream>.

https://github.com/mozilla/sccache/blob/f1322dca19577310cfcfb40059ff954d2140598b/src/client.rs#L56-L59

This happens when libc::recv returns 0, which indicates that the TCP connection has been closed.
@jdm
Copy link
Member

@jdm jdm commented Nov 12, 2019

Is it worth seeing if there's any interesting compiler output on CI if we don't use sccache?
@SimonSapin
Copy link
Member Author

@SimonSapin SimonSapin commented Nov 12, 2019

Good idea, I’ve started https://community-tc.services.mozilla.com/tasks/Wxja6BOTQ6OYbp-79FhFtA to try that.
@SimonSapin
Copy link
Member Author

@SimonSapin SimonSapin commented Nov 12, 2019

By the way, my workflow for this kind of "try build" without a PR is:

  • Make a local branch
  • Make changes to try
  • In the try-taskcluster list in the by_branch_name dict in etc/taskcluster/decision_task.py, add tasks to be run
  • Commit
  • git push origin -f HEAD:try-taskcluster
  • Navigate to https://github.com/servo/servo/commits/try-taskcluster
  • Click the orange dot on the latest commit (which should be the one you just pushed)
  • Click "Community-TC (push)" to view the task group on Taskcluster

It’s mostly out of habit, I don’t know if it’s really better or worse than using try* Homu commands in a PR that is not destined to be merged.
@SimonSapin
Copy link
Member Author

@SimonSapin SimonSapin commented Nov 12, 2019

if we don't use sccache?

https://community-tc.services.mozilla.com/tasks/Wxja6BOTQ6OYbp-79FhFtA succeeded. The only thing of note is that it generated more than 300k stderr lines worth of warnings, most of them from #24684. And rust-lang/rust#65134 is in the regression range.

I’ve also started https://community-tc.services.mozilla.com/tasks/BgPRR43VTKWDVM8nY6A5Ew with RUST_LOG=trace into a log file for the sccache server.
@SimonSapin
Copy link
Member Author

@SimonSapin SimonSapin commented Nov 12, 2019

https://community-tc.services.mozilla.com/tasks/BgPRR43VTKWDVM8nY6A5Ew/runs/0/logs/https%3A%2F%2Fcommunity-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FBgPRR43VTKWDVM8nY6A5Ew%2Fruns%2F0%2Fartifacts%2Fpublic%2Fsccache.log#L321219

ERROR 2019-11-12T15:31:54Z: sccache::server: frame size too big

https://github.com/tokio-rs/tokio/blob/master/tokio-util/src/codec/length_delimited.rs

            // Default max frame length of 8MB
            max_frame_len: 8 * 1_024 * 1_024,

mozilla/sccache#329 / https://github.com/mozilla/sccache/blob/master/src/server.rs

        if let Ok(max_frame_length_str) = env::var("SCCACHE_MAX_FRAME_LENGTH") {
            if let Ok(max_frame_length) = max_frame_length_str.parse::<usize>() {
                builder.max_frame_length(max_frame_length);
SimonSapin added a commit that referenced this issue Nov 12, 2019
@SimonSapin
Fix sccache failure when rustc generates hundreds of thousands of war…
Loading status checks…
8517643 
…nings

Fixes #24714
@SimonSapin
Copy link
Member Author

@SimonSapin SimonSapin commented Nov 12, 2019

https://community-tc.services.mozilla.com/tasks/LpLx0VqfTaqiCXEmEo-Qfw is green with SCCACHE_MAX_FRAME_LENGTH set to 100 MB.
SimonSapin added a commit that referenced this issue Nov 12, 2019
@SimonSapin
Fix sccache failure when rustc generates hundreds of thousands of war…
969b666 
…nings

Fixes #24714
@SimonSapin SimonSapin mentioned this issue Nov 12, 2019
Merged
bors-servo added a commit that referenced this issue Nov 13, 2019
@bors-servo
Auto merge of #24717 - servo:fraaaaaaaaaaaaame, r=jdm
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
1523ff4 
Fix sccache failure when rustc generates thousands of warnings

Fixes #24714
bors-servo added a commit that referenced this issue Nov 13, 2019
@bors-servo
Auto merge of #24717 - servo:fraaaaaaaaaaaaame, r=jdm
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
7cdde60 
Fix sccache failure when rustc generates thousands of warnings

Fixes #24714
bors-servo added a commit that referenced this issue Nov 14, 2019
@bors-servo
Auto merge of #24717 - servo:fraaaaaaaaaaaaame, r=jdm
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
8fae8b4 
Fix sccache failure when rustc generates thousands of warnings

Fixes #24714
bors-servo added a commit that referenced this issue Nov 14, 2019
@bors-servo
Auto merge of #24717 - servo:fraaaaaaaaaaaaame, r=jdm
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
c10da72 
Fix sccache failure when rustc generates thousands of warnings

Fixes #24714
bors-servo added a commit that referenced this issue Nov 14, 2019
@bors-servo
Auto merge of #24717 - servo:fraaaaaaaaaaaaame, r=jdm
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
76685af 
Fix sccache failure when rustc generates thousands of warnings

Fixes #24714
bors-servo added a commit that referenced this issue Nov 14, 2019
@bors-servo
Auto merge of #24717 - servo:fraaaaaaaaaaaaame, r=jdm
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
36c2a13 
Fix sccache failure when rustc generates thousands of warnings

Fixes #24714
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
I-bustage
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

Fix sccache failure when rustc generates thousands of warnings
2 participants
@jdm @SimonSapin
You can’t perform that action at this time.