Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incremental compilation: could not create session directory lock file: No locks available (os error 37) #49773

Open
szignal opened this issue Apr 8, 2018 · 16 comments

Comments

@szignal
Copy link

@szignal szignal commented Apr 8, 2018

I was unable to incrementally compile using cargo build.

I had added the rand dependency to the toml file, per the advice of the Guessing Game exercise in Rust Programming Language (Second Edition).

Before this particular error, I had just finished setting up a synced folder with NFS on my vagrant box (ubuntu 16.04) where rust was installed. There is some issue using the default vagrant shared folder setup with rust.

I tried this code:

use std::io;

fn main() {
println!("Guess the number");

println!("Please input your guess.");

let mut guess = String::new();

io::stdin().read_line(&mut guess)
    .expect("Failed to read line");

print!("You guessed: {}", guess);

}

I expected to see this happen:

compile

Instead, this happened:

Updating registry `https://github.com/rust-lang/crates.io-index`

Downloading rand v0.3.22
Downloading libc v0.2.40
Downloading rand v0.4.2
Compiling libc v0.2.40
Compiling rand v0.4.2
Compiling rand v0.3.22
Compiling guessing_game v0.1.0 (file:///Rust/projects/guessing_game)
error: incremental compilation: could not create session directory lock file: No locks available (os error 37)

thread 'rustc' panicked at 'librustc/session/mod.rs:727: Trying to get session directory from IncrCompSession NotInitialized', librustc/session/mod.rs:1180:26
note: Run with RUST_BACKTRACE=1 for a backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.25.0 (84203ca 2018-03-25) running on x86_64-unknown-linux-gnu

error: Could not compile guessing_game.

rustc --version --verbose:

rustc 1.25.0 (84203ca 2018-03-25)
binary: rustc
commit-hash: 84203ca
commit-date: 2018-03-25
host: x86_64-unknown-linux-gnu
release: 1.25.0
LLVM version: 6.0

Backtrace:

thread 'rustc' panicked at 'librustc/session/mod.rs:727: Trying to get session directory from IncrCompSession NotInitialized', librustc/session/mod.rs:1180:26
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
stack backtrace:
0: std::sys::unix::backtrace::tracing:👿:unwind_backtrace
at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
1: std::sys_common::backtrace::_print
at libstd/sys_common/backtrace.rs:71
2: std::panicking::default_hook::{{closure}}
at libstd/sys_common/backtrace.rs:59
at libstd/panicking.rs:380
3: std::panicking::default_hook
at libstd/panicking.rs:396
4: std::panicking::rust_panic_with_hook
at libstd/panicking.rs:576
5: std::panicking::begin_panic
6: rustc::session::opt_span_bug_fmt::{{closure}}
7: rustc::ty::context::tls::with_opt
8: rustc::session::opt_span_bug_fmt
9: rustc::session::bug_fmt
10: rustc::session::Session::incr_comp_session_dir
11: rustc_incremental::persist::fs::garbage_collect_session_directories
12: rustc_driver::driver::phase_2_configure_and_expand_inner::{{closure}}
13: rustc_driver::driver::phase_2_configure_and_expand_inner
14: rustc_driver::driver::compile_input
15: rustc_driver::run_compiler

@glaubitz

This comment has been minimized.

Copy link
Contributor

@glaubitz glaubitz commented Apr 9, 2018

This is an issue we are currently seeing on linux-sparc64 as well [1]:

error: incremental compilation: could not create session directory lock file: No such process (os error 3)

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.24.1 running on sparc64-unknown-linux-gnu

thread 'rustc' panicked at 'src/librustc/session/mod.rs:665: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/session/mod.rs:1141:26
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: Could not compile `build_helper`.
warning: build failed, waiting for other jobs to finish...

I have not yet figured out where this comes from. The same rust version worked fine in the past.

[1] https://buildd.debian.org/status/fetch.php?pkg=rustc&arch=sparc64&ver=1.25.0%2Bdfsg1-1%7Eexp2&stamp=1523052949&raw=0

CC @jrtc27 @psumbera @mkarcher

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

@michaelwoerister michaelwoerister commented May 3, 2018

NFS does not support locking files very well, I guess. It would be good if there was some way to make Cargo put the incremental folder somewhere else (into /tmp for example).

cc @rust-lang/cargo

@jrtc27

This comment has been minimized.

Copy link

@jrtc27 jrtc27 commented May 3, 2018

Although the original report mentions NFS, we're not using NFS for the sparc64 buildds; the two issues have similar but different error messages ("No locks available" vs "No such process") so they could well be different problems.

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

@michaelwoerister michaelwoerister commented May 3, 2018

@jrtc27, yes, sounds like a different root cause.

@alexcrichton

This comment has been minimized.

Copy link
Member

@alexcrichton alexcrichton commented May 3, 2018

In Cargo we disable file locking whenever NFS is detected, although that may not work well for rustc as it's semi-reasonable to expect Cargo to not be run in parallel much on one system but rustc probably is being executed in parallel by Cargo at least.

An alternative to moving the incremental directory is moving the locks for the incremental directory. I've considered this for Cargo where we actually want to enable file locking on NFS directories. That'd require more intrusive support in rustc, though.

@glaubitz

This comment has been minimized.

Copy link
Contributor

@glaubitz glaubitz commented Sep 22, 2018

If anyone wants to reproduce this problem, there are sparc64 porterboxes available in the gcc compile farm, both running Solaris and Linux. Any open source developer can apply for an account:

https://cfarm.tetaneutral.net/machines/list/

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

@michaelwoerister michaelwoerister commented Sep 24, 2018

Is this on a file system that should support locks in general?

An alternative to moving the incremental directory is moving the locks for the incremental directory. I've considered this for Cargo where we actually want to enable file locking on NFS directories. That'd require more intrusive support in rustc, though.

As long as we don't want to support two different machines working on the same NFS share simultaneously (which we don't really do anyway), we could indeed just allocate that lock files in /tmp. The only real restriction for lock files is that they mustn't be contained in the session directory that they protect.

@glaubitz

This comment has been minimized.

Copy link
Contributor

@glaubitz glaubitz commented Sep 24, 2018

Is this on a file system that should support locks in general?

It's a standard off-the-shelf installation of Debian on ext4. Nothing special. There is no NFS or anything involved and it affects sparc64 machines with the same configuration as comparable x86_64 machines.

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

@michaelwoerister michaelwoerister commented Sep 24, 2018

Interesting. That's unexpected.

@glaubitz

This comment has been minimized.

Copy link
Contributor

@glaubitz glaubitz commented Sep 24, 2018

You can test it yourself on the Linux/sparc64 porterbox available in the gcc compile farm. It's a rather fast machine.

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

@michaelwoerister michaelwoerister commented Sep 24, 2018

I've requested an account. Thanks for the tip!

@glaubitz

This comment has been minimized.

Copy link
Contributor

@glaubitz glaubitz commented Sep 24, 2018

I have root privileges on the sparc64 machine (but not the others), so I can install additional packages if required.

@glaubitz

This comment has been minimized.

Copy link
Contributor

@glaubitz glaubitz commented Oct 14, 2018

@michaelwoerister Did you have any success getting an account in the gcc compile farm?

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

@michaelwoerister michaelwoerister commented Oct 23, 2018

@glaubitz Yes, I have an account. I haven't gotten around to actually using it though.

@glaubitz

This comment has been minimized.

Copy link
Contributor

@glaubitz glaubitz commented Dec 20, 2018

@michaelwoerister @alexcrichton

@karcherm has had a look at the Rust sources and he found the problem rather quickly. Rust is hard-wiring the constants for the fnctls from glibc assuming they are constant across which they are not.

Compare:

https://github.molgen.mpg.de/git-mirror/glibc/blob/master/sysdeps/unix/sysv/linux/sparc/bits/fcntl.h#L53

and:

https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_data_structures/flock.rs.html#45

Rust is trying to call "Set Lock" and uses function control number 6. But that's "F_SETOWN" on sparc64 which expects a PID as a parameter which is why the error message is "No such process".

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

@michaelwoerister michaelwoerister commented Dec 20, 2018

Awesome find, @karcherm & @glaubitz!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.