Replace `std::task` with `std::thread`, remove librustrt, make final "runtime" features lazily initialize #19654

Merged
merged 24 commits into from Dec 19, 2014

Projects

None yet
@aturon
Contributor
aturon commented Dec 9, 2014

This PR substantially narrows the notion of a "runtime" in Rust, and allows calling into Rust code directly without any setup or teardown.

After this PR, the basic "runtime support" in Rust will consist of:

  • Unwinding and backtrace support
  • Stack guards

Other support, such as helper threads for timers or the notion of a "current thread" are initialized automatically upon first use.

When using Rust in an embedded context, it should now be possible to call a Rust function directly as a C function with absolutely no setup, though in that case panics will cause the process to abort. In this regard, the C/Rust interface will look much like the C/C++ interface.

In more detail, this PR:

  • Merges librustrt back into std::rt, undoing the facade. While doing so, it removes a substantial amount of redundant functionality (such as mutexes defined in the rt module). Code using librustrt can now call into std::rt to e.g. start executing Rust code with unwinding support.
  • Allows all runtime data to be initialized lazily, including the "current thread", the "at_exit" infrastructure, and the "args" storage.
  • Deprecates and largely removes std::task along with the widespread requirement that there be a "current task" for many APIs in std. The entire task infrastructure is replaced with std::thread, which provides a more standard API for manipulating and creating native OS threads. In particular, it's possible to join on a created thread, and to get a handle to the currently-running thread. In addition, threads are equipped with some basic blocking support in the form of park/unpark operations (following a tradition in some OSes as well as the JVM). See the std::thread documentation for more details.
  • Channels are refactored to use a new internal blocking infrastructure that itself sits on top of park/unpark.

One important change here is that a Rust program ends when its main thread does, following most threading models. On the other hand, threads will often be created with an RAII-style join handle that will re-institute blocking semantics naturally (and with finer control).

This is very much a:

[breaking-change]

Closes #18000
r? @alexcrichton

@aturon
Contributor
aturon commented Dec 9, 2014

Note: this PR does not attempt to exhaustively replace the task terminology with thread, nor does it clear out all use of the now-deprecated APIs. Both are left to follow-up work.

@aturon
Contributor
aturon commented Dec 9, 2014

Note also: the implementation of park/unpark is currently pretty silly: it just uses a lock and condvar. In the future these can be implemented in more clever ways, e.g. following the JVM.

@eddyb
Member
eddyb commented Dec 9, 2014
@alexcrichton alexcrichton commented on an outdated diff Dec 9, 2014
src/libcore/borrow.rs
@@ -92,7 +92,7 @@ impl<'a, T, Sized? B> BorrowFrom<Cow<'a, T, B>> for B where B: ToOwned<T> {
/// Trait for moving into a `Cow`
pub trait IntoCow<'a, T, Sized? B> {
- /// Moves `self` into `Cow`
+ /// Moves `serlf` into `Cow`
@alexcrichton alexcrichton and 1 other commented on an outdated diff Dec 9, 2014
src/librustc_driver/lib.rs
@@ -473,18 +473,22 @@ pub fn monitor(f: proc():Send) {
static STACK_SIZE: uint = 32000000; // 32MB
let (tx, rx) = channel();
- let w = io::ChanWriter::new(tx);
+ let mut w = Some(io::ChanWriter::new(tx)); // option dance
@alexcrichton
alexcrichton Dec 9, 2014 Member

How come this dance is necessary? I would have thought that it could move into the proc below

@engstad
engstad Dec 9, 2014

If I am not mistaken, this is due to a limitation of closures not being able to capture w directly. I've seen this trick several places in the code-base, hoping it would disappear at some point - perhaps with unboxed closures?

@alexcrichton alexcrichton commented on an outdated diff Dec 9, 2014
src/librustc_trans/back/write.rs
@@ -868,7 +868,11 @@ fn run_work_multithreaded(sess: &Session,
let diag_emitter = diag_emitter.clone();
let remark = sess.opts.cg.remark.clone();
- let future = TaskBuilder::new().named(format!("codegen-{}", i)).try_future(proc() {
+ let (tx, rx) = channel();
+ let mut tx = Some(tx);
@alexcrichton
alexcrichton Dec 9, 2014 Member

Similar to above, how come this needs an option?

@alexcrichton alexcrichton and 1 other commented on an outdated diff Dec 9, 2014
src/libstd/comm/blocking.rs
+use clone::Clone;
+
+struct Inner {
+ thread: Thread,
+ woken: AtomicBool,
+}
+
+#[deriving(Clone)]
+pub struct SignalToken {
+ inner: Arc<Inner>,
+}
+
+pub struct WaitToken {
+ inner: Arc<Inner>,
+ no_send: NoSend,
+}
@alexcrichton
alexcrichton Dec 9, 2014 Member

Should this also require NoSync to prevent from sharing amongst fork-join parallelism? I'm thinking of a function like this, but it may only be safe with Send + Sync anyway, so this is probably enough.

@reem
reem Dec 9, 2014 Contributor

Right now the only safe bound for that function is Send + Sync, since Sync is overly permissive (allows stuff like thread-local references or other non-transferrable types).

@aturon
Contributor
aturon commented Dec 9, 2014

Once piece of feedback I'd like: is spawn and with_join the right setup? That makes detached threads the "default". We could alternatively do spawn and spawn_detached, but I worry this will be surprising to people.

@alexcrichton alexcrichton and 1 other commented on an outdated diff Dec 9, 2014
src/libstd/comm/blocking.rs
+struct Inner {
+ thread: Thread,
+ woken: AtomicBool,
+}
+
+#[deriving(Clone)]
+pub struct SignalToken {
+ inner: Arc<Inner>,
+}
+
+pub struct WaitToken {
+ inner: Arc<Inner>,
+ no_send: NoSend,
+}
+
+pub fn tokens() -> (WaitToken, SignalToken) {
@alexcrichton
alexcrichton Dec 9, 2014 Member

Out of curiosity, doesn't this abstraction break if the thread is blocked on anything other than WaitToken while some SignalTokens are signaled? I'm thinking that you call tokens, signal the token, manually call Thread::park(), and then call wait, you'd sleep forever, right? That seems pretty silly though, and I imagine we could document this example when it's a public-facing API.

@aturon
aturon Dec 9, 2014 Contributor

It's correct that you won't get unparked by that signal, but you could be unparked by a different signal. But in any case, you're probably doing something wrong.

(Note that the intent of structures like this and park/unpark is for building blocking concurrency abstractions like channels; they aren't a good fit for application-level concurrent programming. See http://gee.cs.oswego.edu/dl/papers/aqs.pdf)

@alexcrichton alexcrichton commented on the diff Dec 9, 2014
src/libstd/comm/sync.rs
pub struct Packet<T> {
/// Only field outside of the mutex. Just done for kicks, but mainly because
/// the other shared channel already had the code implemented
channels: atomic::AtomicUint,
- /// The state field is protected by this mutex
- lock: NativeMutex,
- state: UnsafeCell<State<T>>,
+ lock: Mutex<State<T>>,
@aturon
aturon Dec 9, 2014 Contributor

You know, I probably could've saved myself a lot of pain by also replacing the queue and buf data structures with something from std. Mutexes around Copy data ... bad news.

@alexcrichton alexcrichton commented on the diff Dec 9, 2014
src/libstd/comm/oneshot.rs
- let n = unsafe { task.cast_to_uint() };
- match self.state.compare_and_swap(EMPTY, n, atomic::SeqCst) {
- // Nothing on the channel, we legitimately block
- EMPTY => Ok(()),
-
- // If there's data or it's a disconnected channel, then we
- // failed the cmpxchg, so we just wake ourselves back up
- DATA | DISCONNECTED => {
- unsafe { Err(BlockedTask::cast_from_uint(n)) }
- }
-
- // Only one thread is allowed to sleep on this port
- _ => unreachable!()
- }
- });
+ let (wait_token, signal_token) = blocking::tokens();
@alexcrichton
alexcrichton Dec 9, 2014 Member

I somewhat fear the perf impact of creating a new set of tokens each time a task blocks, but I'd personally rather land the runtime removal and optimize later.

@aturon
aturon Dec 9, 2014 Contributor

Same here. FWIW, it would be quite easy to re-instate the BlockedTask-style enum.

@aturon
aturon Dec 9, 2014 Contributor

Although note, this is basically just one allocation per context switch. Probably a drop in the bucket.

@alexcrichton
alexcrichton Dec 9, 2014 Member

agreed, I'd want to measure before taking any action

@alexcrichton alexcrichton and 1 other commented on an outdated diff Dec 9, 2014
src/libstd/io/stdio.rs
@@ -531,7 +522,7 @@ mod tests {
assert_eq!(r.read_to_string().unwrap(), "hello!\n");
}
- #[test]
+ //#[test]
@aturon
aturon Dec 9, 2014 Contributor

Nope: it's the reason this PR is titled [WIP].

@alexcrichton
alexcrichton Dec 9, 2014 Member

aha! I see :)

@alexcrichton alexcrichton commented on an outdated diff Dec 9, 2014
src/libstd/lib.rs
pub mod task;
+#[allow(missing_docs)]
@alexcrichton
alexcrichton Dec 9, 2014 Member

Could this be removed now?

@alexcrichton alexcrichton and 1 other commented on an outdated diff Dec 9, 2014
src/libstd/rt/args.rs
static mut GLOBAL_ARGS_PTR: uint = 0;
- static LOCK: StaticNativeMutex = NATIVE_MUTEX_INIT;
+ static LOCK: NativeMutex = MUTEX_INIT;
@alexcrichton
alexcrichton Dec 9, 2014 Member

NativeMutex is gone, right?

@aturon
aturon Dec 9, 2014 Contributor

As you can see, I develop on macos.

@alexcrichton alexcrichton commented on an outdated diff Dec 9, 2014
src/libstd/rt/at_exit_imp.rs
}
}
pub fn push(f: proc():Send) {
+ INIT.doit(init);
@alexcrichton alexcrichton commented on an outdated diff Dec 9, 2014
src/libstd/rt/at_exit_imp.rs
unsafe {
// Note that the check against 0 for the queue pointer is not atomic at
// all with respect to `run`, meaning that this could theoretically be a
// use-after-free. There's not much we can do to protect against that,
// however. Let's just assume a well-behaved runtime and go from there!
- rtassert!(!RUNNING.load(atomic::SeqCst));
@alexcrichton
alexcrichton Dec 9, 2014 Member

Isn't this an important assertion to keep? If you re-enqueue some cleanup while we're cleaning up we'll just leak it?

@alexcrichton
alexcrichton Dec 9, 2014 Member

Or... you'll hit the assertion below, nevermind!

@alexcrichton alexcrichton and 1 other commented on an outdated diff Dec 9, 2014
src/libstd/rt/mod.rs
#[allow(experimental)]
pub fn init(argc: int, argv: *const *const u8) {
- rustrt::init(argc, argv);
- unsafe { rustrt::unwind::register(failure::on_fail); }
+ unsafe {
+ args::init(argc, argv);
+ sys::thread::guard::init();
+ sys::stack_overflow::init();
+ unwind::register(failure::on_fail);
@alexcrichton
alexcrichton Dec 9, 2014 Member

Do you think this could be moved to a Once?

@aturon
aturon Dec 9, 2014 Contributor

The whole init function? It's an interesting idea...

@alexcrichton
alexcrichton Dec 9, 2014 Member

Maybe not the whole function, but at least unwind::register. The others may be more difficult to move around.

@alexcrichton
alexcrichton Dec 9, 2014 Member

I'm also fine developing our story here over time and not hammering it out all immediately.

@alexcrichton alexcrichton and 1 other commented on an outdated diff Dec 9, 2014
src/libstd/rt/mod.rs
@@ -99,13 +118,10 @@ fn lang_start(main: *const u8, argc: int, argv: *const *const u8) -> int {
/// This procedure is guaranteed to run on the thread calling this function, but
/// the stack bounds for this rust task will *not* be set. Care must be taken
/// for this function to not overflow its stack.
-///
-/// This function will only return once *all* native threads in the system have
-/// exited.
+// FIXME: this should be unsafe
@alexcrichton
alexcrichton Dec 9, 2014 Member

Feel free to make it so :)

@alexcrichton
alexcrichton Dec 9, 2014 Member

Also feel free to merge this upwards into lang_start

@aturon
aturon Dec 9, 2014 Contributor

I tried to mark it unsafe and ran into a weird issue so left it alone. I can try it again.

@alexcrichton alexcrichton commented on the diff Dec 9, 2014
src/libstd/rt/mod.rs
@@ -162,5 +195,7 @@ pub fn start(argc: int, argv: *const *const u8, main: proc()) -> int {
/// Invoking cleanup while portions of the runtime are still in use may cause
/// undefined behavior.
pub unsafe fn cleanup() {
- rustrt::cleanup();
+ args::cleanup();
+ sys::stack_overflow::cleanup();
@alexcrichton
alexcrichton Dec 9, 2014 Member

Perhaps both args and stack_overflow could use at_exit to schedule cleanups? That way we only have to get one thing working with atexit to get it all to work.

@sfackler sfackler commented on an outdated diff Dec 9, 2014
src/libstd/sys/windows/thread.rs
+ // that's because pthreads enforces that stacks are at least
+ // PTHREAD_STACK_MIN bytes big. Windows has no such lower limit, it's
+ // just that below a certain threshold you can't do anything useful.
+ // That threshold is application and architecture-specific, however.
+ // For now, the only requirement is that it's big enough to hold the
+ // red zone. Round up to the next 64 kB because that's what the NT
+ // kernel does, might as well make it explicit. With the current
+ // 20 kB red zone, that makes for a 64 kB minimum stack.
+ let stack_size = (cmp::max(stack, RED_ZONE) + 0xfffe) & (-0xfffe - 1);
+ let ret = CreateThread(ptr::null_mut(), stack_size as libc::size_t,
+ thread_start, arg, 0, ptr::null_mut());
+
+ if ret as uint == 0 {
+ // be sure to not leak the closure
+ let _p: Box<proc():Send> = mem::transmute(arg);
+ panic!("failed to spawn native thread: {}", ret);
@sfackler
sfackler Dec 9, 2014 Member

Should there be an option to handle this error rather than panic?

@alexcrichton alexcrichton commented on the diff Dec 9, 2014
src/libstd/rt/util.rs
@@ -73,3 +81,130 @@ pub fn default_sched_threads() -> uint {
}
}
}
+
+// Indicates whether we should perform expensive sanity checks, including rtassert!
+//
+// FIXME: Once the runtime matures remove the `true` below to turn off rtassert,
+// etc.
@alexcrichton
alexcrichton Dec 9, 2014 Member

This comment makes me chuckle

@sfackler sfackler commented on the diff Dec 9, 2014
src/libstd/sys/windows/stack_overflow.rs
+use mem;
+use libc;
+use libc::types::os::arch::extra::{LPVOID, DWORD, LONG, BOOL};
+use sys_common::{stack, thread_info};
+
+pub struct Handler {
+ _data: *mut libc::c_void
+}
+
+impl Handler {
+ pub unsafe fn new() -> Handler {
+ make_handler()
+ }
+}
+
+impl Drop for Handler {
@sfackler
sfackler Dec 9, 2014 Member

Does this need to impl Drop explicitly or can it just contain NoCopy?

@alexcrichton alexcrichton and 1 other commented on an outdated diff Dec 9, 2014
src/libstd/sys/common/thread_info.rs
+
+use thread::Thread;
+use cell::RefCell;
+use string::String;
+
+struct ThreadInfo {
+ // This field holds the known bounds of the stack in (lo, hi)
+ // form. Not all threads necessarily know their precise bounds,
+ // hence this is optional.
+ stack_bounds: (uint, uint),
+ stack_guard: uint,
+ unwinding: bool,
+ thread: Thread,
+}
+
+thread_local!(static THREAD_INFO: RefCell<Option<ThreadInfo>> = RefCell::new(None))
@alexcrichton
alexcrichton Dec 9, 2014 Member

Instead of an Option, perhaps this could have the lazy initialization expression here?

@aturon
aturon Dec 14, 2014 Contributor

The lazy init expression is only used if you access the thread info from a thread not launched via Rust APIs; otherwise it is explicitly initialized with an already-created Thread.

@alexcrichton
alexcrichton Dec 14, 2014 Member

Ah yeah after looking below I see that this is necessary, carry on!

@alexcrichton alexcrichton and 1 other commented on an outdated diff Dec 9, 2014
src/libstd/sys/common/thread_info.rs
+
+struct ThreadInfo {
+ // This field holds the known bounds of the stack in (lo, hi)
+ // form. Not all threads necessarily know their precise bounds,
+ // hence this is optional.
+ stack_bounds: (uint, uint),
+ stack_guard: uint,
+ unwinding: bool,
+ thread: Thread,
+}
+
+thread_local!(static THREAD_INFO: RefCell<Option<ThreadInfo>> = RefCell::new(None))
+
+impl ThreadInfo {
+ fn with<R>(f: |&mut ThreadInfo| -> R) -> R {
+ THREAD_INFO.with(|c| {
@alexcrichton
alexcrichton Dec 9, 2014 Member

This borrow worries me a little in that once the THREAD_INFO has its destructor run, all blocking services will abort the process because they won't be able to access the current thread.

I'm not really sure of a great way to get around that, but perhaps this could yield a nicer panic message?

@aturon
aturon Dec 14, 2014 Contributor

I can imagine a few ways to do better, but they involve bubbling up this possibility in APIs, e.g. by making Thread::current() yield an Option. I'm not sure how much to worry about this edge case.

I'm also not sure how to generate a better panic message here -- isn't with going to cause the panic? Looking at the docs, it doesn't seem like .destroyed covers all the cases that might lead to this panic.

@alexcrichton
alexcrichton Dec 14, 2014 Member

For generating a better panic message you'd want to check .destroyed() beforehand yeah. I think that may be good enough for now, we could possibly add more support later for something like repeated re-construction if necessary.

@sfackler sfackler commented on the diff Dec 9, 2014
src/libstd/rt/util.rs
+ Ok(())
+ }
+ }
+
+ // Convert the arguments into a stack-allocated string
+ let mut msg = [0u8, ..512];
+ let mut w = BufWriter { buf: &mut msg, pos: 0 };
+ let _ = write!(&mut w, "{}", args);
+ let msg = str::from_utf8(w.buf[mut ..w.pos]).unwrap_or("aborted");
+ let msg = if msg.is_empty() {"aborted"} else {msg};
+
+ // Give some context to the message
+ let hash = msg.bytes().fold(0, |accum, val| accum + (val as uint) );
+ let quote = match hash % 10 {
+ 0 => "
+It was from the artists and poets that the pertinent answers came, and I
@sfackler
sfackler Dec 9, 2014 Member

Glad to see this is surviving the transition :)

@alexcrichton alexcrichton and 1 other commented on an outdated diff Dec 9, 2014
src/libstd/thread.rs
+//! Native threads
+//!
+//! ## The threading model
+//!
+//! An executing Rust program consists of a collection of native OS threads,
+//! each with their own stack and local state.
+//!
+//! Threads generally have their memory *isolated* from each other by virtue of
+//! Rust's owned types (which of course may only be owned by a single thread at
+//! a time). Communication between threads can be done through
+//! [channels](../../std/comm/index.html), Rust's message-passing types, along
+//! with [other forms of thread synchronization](../../std/sync/index.html) and
+//! shared-memory data structures. In particular, types that are guaranteed to
+//! be threadsafe are easily shared between threads using the
+//! atomically-reference-counted container,
+//! [`Arc`](../../std/sync/struct.Arc.html).
@alexcrichton
alexcrichton Dec 9, 2014 Member

This is somewhat oddly worded to me where it starts out by saying memory is isolated, but then it explains that I can indeed share memory. Maybe the emphasis at the top could be toned down a bit?

@aturon
aturon Dec 9, 2014 Contributor

Sure. (This was brought over from the old std::task docs, so I was trying not to change it too much, but I agree with you.)

@alexcrichton alexcrichton commented on the diff Dec 9, 2014
src/libstd/thread.rs
+
+use sys::thread as imp;
+use sys_common::{stack, thread_info};
+
+/// Thread configuation. Provides detailed control over the properties
+/// and behavior of new threads.
+pub struct Cfg {
+ // A name for the thread-to-be, for identification in panic messages
+ name: Option<String>,
+ // The size of the stack for the spawned thread
+ stack_size: Option<uint>,
+ // Thread-local stdout
+ stdout: Option<Box<Writer + Send>>,
+ // Thread-local stderr
+ stderr: Option<Box<Writer + Send>>,
+}
@alexcrichton
alexcrichton Dec 9, 2014 Member

The stdout and stderr options here seem pretty sketchy to me, I'd be fine removing them. They're implemented with a trivial wrapper around the provide proc regardless.

@alexcrichton alexcrichton commented on an outdated diff Dec 9, 2014
src/libstd/thread.rs
+
+use any::Any;
+use borrow::IntoCow;
+use boxed::Box;
+use mem;
+use sync::{Mutex, Condvar, Arc};
+use string::String;
+use rt::{mod, unwind};
+use io::{Writer, stdio};
+
+use sys::thread as imp;
+use sys_common::{stack, thread_info};
+
+/// Thread configuation. Provides detailed control over the properties
+/// and behavior of new threads.
+pub struct Cfg {
@alexcrichton
alexcrichton Dec 9, 2014 Member

Cfg seems like a somewhat odd name for this, convention-wise it seems like it may want to be ThreadBuilder, but I do agree that's pretty long. We may be out of a noun like Command is to Process, but perhaps this could just be called Builder to avoid stuttering?

@alexcrichton alexcrichton commented on an outdated diff Dec 9, 2014
src/libstd/thread.rs
+ unsafe {
+ stack::record_os_managed_stack_bounds(my_stack_bottom, my_stack_top);
+ }
+ thread_info::set(
+ (my_stack_bottom, my_stack_top),
+ unsafe { imp::guard::current() },
+ their_thread
+ );
+
+ // There are two primary reasons that general try/catch is
+ // unsafe. The first is that we do not support nested try/catch. The
+ // fact that this is happening in a newly-spawned thread
+ // suffices. The second is that unwinding while unwinding is not
+ // defined. We take care of that by having an 'unwinding' flag in
+ // the thread itself. For these reasons, this unsafety should be ok.
+ unsafe {
@alexcrichton
alexcrichton Dec 9, 2014 Member

Can this unsafety be contained to just unwind::try?

@alexcrichton alexcrichton and 1 other commented on an outdated diff Dec 9, 2014
src/libstd/thread.rs
+ name: name,
+ lock: Mutex::new(false),
+ cvar: Condvar::new(),
+ })
+ }
+ }
+
+ /// Spawn a detached thread, and return a handle to it.
+ ///
+ /// The new child thread may outlive its parent.
+ pub fn spawn(f: proc():Send) -> Thread {
+ Cfg::new().spawn(f)
+ }
+
+ /// Spawn a joinable thread, and return an RAII guard for it.
+ pub fn with_join<T: Send>(f: proc():Send -> T) -> JoinGuard<T> {
@alexcrichton
alexcrichton Dec 9, 2014 Member

This is a bit of an interesting choice of words here! Perhaps we could move spawn to the constructor new (aka Thread::new) and free up the word spawn for this to give us some more room, but that's still not exactly the best state of affairs.

@alexcrichton
alexcrichton Dec 9, 2014 Member

Could you also expand the documentation to explain where the type parameter T comes into play and how you can use a JoinGuard to get it out? It may also be nice to warn about ignoring the JoinGuard and how it may block.

@alexcrichton alexcrichton commented on an outdated diff Dec 9, 2014
src/libstd/thread.rs
+pub type Result<T> = ::result::Result<T, Box<Any + Send>>;
+
+#[must_use]
+/// An RAII guard that will block until thread termination when dropped.
+pub struct JoinGuard<T> {
+ native: imp::rust_thread,
+ thread: Thread,
+ joined: bool,
+ packet: Option<Box<Result<T>>>,
+}
+
+impl<T: Send> JoinGuard<T> {
+ /// Extract a handle to the thread this guard will join on.
+ pub fn thread(&self) -> Thread {
+ self.thread.clone()
+ }
@alexcrichton
alexcrichton Dec 9, 2014 Member

Could this return &Thread and push the clone to the caller?

@alexcrichton alexcrichton commented on an outdated diff Dec 9, 2014
src/libstd/thread.rs
+}
+
+impl Thread {
+ fn new(name: Option<String>) -> Thread {
+ Thread {
+ inner: Arc::new(Inner {
+ name: name,
+ lock: Mutex::new(false),
+ cvar: Condvar::new(),
+ })
+ }
+ }
+
+ /// Spawn a detached thread, and return a handle to it.
+ ///
+ /// The new child thread may outlive its parent.
@alexcrichton
alexcrichton Dec 9, 2014 Member

Could you expand the documentation here to explain that all threads spawned via this method will not block the main process from exiting?

@alexcrichton alexcrichton commented on an outdated diff Dec 9, 2014
src/libstd/thread.rs
+ res
+ }
+}
+
+#[unsafe_destructor]
+impl<T: Send> Drop for JoinGuard<T> {
+ fn drop(&mut self) {
+ // This is required for correctness. If this is not done then the thread
+ // would fill in a return box which no longer exists.
+ if !self.joined {
+ unsafe { imp::join(self.native) };
+ }
+ }
+}
+
+// TODO: fix tests
@alexcrichton
alexcrichton Dec 9, 2014 Member

still relevant?

@sfackler
Member
sfackler commented Dec 9, 2014

There's also the C++ route of Thread::new or Thread::spawn or whatever spawning attached, and having a fn detach(self) method on JoinGuard.

@alexcrichton
Member

I'm a little worried about the ergonomics of spawning a thread here, having to import std::thread::Thread and then calling Thread::spawn is a lot of "thread"s going on there. The precise API may rely on the names we settle on.

@sfackler's suggestion is interesting to only have Thread::new and then you call detach(self). As another data point, it looks like C++ requires join or detach to be called or otherwise it will abort the process. We may not want to get that drastic, however. I'd personally be a little sad if spawning a thread required two function calls at all times, but it is somewhat consistent in that nothing is favored over another.

All that being said, I don't think I actually spawn threads that frequently, so it may be somewhat of a moot point.

@alexcrichton
Member

Oh and amazing work here @aturon, I'm super happy to see the runtime removal enter its final stage!

@thestinger
Contributor

it looks like C++ requires join or detach to be called or otherwise it will abort the process.

It won't abort the process. It detaches in the destructor. It is considered a bad design, because it needs to be wrapped to correctly use joined threads.

@petrochenkov
Contributor

It won't abort the process. It detaches in the destructor.

?
boost::thread used to do this, not std

@brson
Contributor
brson commented Dec 9, 2014

\o/

@sfackler
Member
sfackler commented Dec 9, 2014

If *this has an associated thread (joinable() == true), std::terminate() is called.

http://en.cppreference.com/w/cpp/thread/thread/~thread

@wycats
Contributor
wycats commented Dec 10, 2014

HELL'S YES!

@aturon
Contributor
aturon commented Dec 15, 2014

@alexcrichton Update: I've rebased (over proc removal in particular) and updated the API according to our earlier discussion. That is, with_join is gone, and instead spawn creates a joinable thread (returning a JoinGuard). Then you can call detach on the guard to get a detached thread.

Regarding the question of what to do when dropping a JoinGuard that has not been joined: I believe that the unused result lint should suffice (JoinGuard has always been must_use).

The Cfg type has been renamed to Builder.

I declined to add a top-level spawn function for the time being; I don't think that

use std::thread::Thread;
let t = Thread::spawn(f);

is any worse than

use std::vec::Vec;
let v = Vec::new();

I've address most (but not quite all) nits -- please check over the docs for std::thread in particular. The most thorough documentation is at the module level, but I've added some repeat docs on the methods as well.

I'll be working on getting this passing the test suite again, but I think it's ready for re-r?

@alexcrichton alexcrichton commented on an outdated diff Dec 15, 2014
src/libstd/thread.rs
+ (*self.packet.get()).take().unwrap()
+ }
+ }
+
+ /// Detaches the child thread, allowing it to outlive its parent.
+ pub fn detach(mut self) {
+ unsafe { imp::detach(self.native) };
+ self.joined = true; // avoid joining in the destructor
+ }
+}
+
+#[unsafe_destructor]
+impl<T: Send> Drop for JoinGuard<T> {
+ fn drop(&mut self) {
+ // This is required for correctness. If this is not done then the thread
+ // would fill in a return box which no longer exists.
@alexcrichton
alexcrichton Dec 15, 2014 Member

This comment threw me for a bit of a spin, I think that we can drop the "for correctness" clause here now that it's an Arc instead of a Box.

@alexcrichton alexcrichton commented on the diff Dec 15, 2014
src/libstd/sys/unix/thread.rs
+ (-(page_size as int - 1) as uint - 1);
+ assert_eq!(pthread_attr_setstacksize(&mut attr, stack_size as libc::size_t), 0);
+ },
+ errno => {
+ // This cannot really happen.
+ panic!("pthread_attr_setstacksize() error: {}", errno);
+ },
+ };
+
+ let arg: *mut libc::c_void = mem::transmute(box p); // must box since sizeof(p)=2*uint
+ let ret = pthread_create(&mut native, &attr, thread_start, arg);
+ assert_eq!(pthread_attr_destroy(&mut attr), 0);
+
+ if ret != 0 {
+ // be sure to not leak the closure
+ let _p: Box<Box<FnOnce()+Send>> = mem::transmute(arg);
@alexcrichton
alexcrichton Dec 15, 2014 Member

This type needs to be Box<Thunk> I believe.

@aturon
aturon Dec 15, 2014 Contributor

Hm. So I brought in this code from https://github.com/rust-lang/rust/blob/master/src/librustrt/thread.rs#L452-L461 after @nikomatsakis killed proc. I will take a closer look...

@alexcrichton alexcrichton commented on an outdated diff Dec 15, 2014
src/libstd/sys/common/thread_info.rs
+struct ThreadInfo {
+ // This field holds the known bounds of the stack in (lo, hi)
+ // form. Not all threads necessarily know their precise bounds,
+ // hence this is optional.
+ stack_bounds: (uint, uint),
+ stack_guard: uint,
+ unwinding: bool,
+ thread: Thread,
+}
+
+thread_local!(static THREAD_INFO: RefCell<Option<ThreadInfo>> = RefCell::new(None))
+
+impl ThreadInfo {
+ fn with<R>(f: |&mut ThreadInfo| -> R) -> R {
+ if THREAD_INFO.destroyed() {
+ panic!("Use of std::thread::Thread::current() is not possible after\
@alexcrichton
alexcrichton Dec 15, 2014 Member

space after the word after

@alexcrichton
Member

This looks awesome @aturon! We can always add a convenience spawn function if necessary. Looks like there are some tidy errors and I left a few more nits, but other than that r=me

@aturon aturon changed the title from [WIP] Replace `std::task` with `std::thread`, remove librustrt, make final "runtime" features lazily initialize to Replace `std::task` with `std::thread`, remove librustrt, make final "runtime" features lazily initialize Dec 16, 2014
@sfackler
Member

:D

@bors bors added a commit that referenced this pull request Dec 17, 2014
@bors bors auto merge of #19654 : aturon/rust/merge-rt, r=alexcrichton
This PR substantially narrows the notion of a "runtime" in Rust, and allows calling into Rust code directly without any setup or teardown. 

After this PR, the basic "runtime support" in Rust will consist of:

* Unwinding and backtrace support
* Stack guards

Other support, such as helper threads for timers or the notion of a "current thread" are initialized automatically upon first use.

When using Rust in an embedded context, it should now be possible to call a Rust function directly as a C function with absolutely no setup, though in that case panics will cause the process to abort. In this regard, the C/Rust interface will look much like the C/C++ interface.

In more detail, this PR:

* Merges `librustrt` back into `std::rt`, undoing the facade. While doing so, it removes a substantial amount of redundant functionality (such as mutexes defined in the `rt` module). Code using `librustrt` can now call into `std::rt` to e.g. start executing Rust code with unwinding support.

* Allows all runtime data to be initialized lazily, including the "current thread", the "at_exit" infrastructure, and the "args" storage.

* Deprecates and largely removes `std::task` along with the widespread requirement that there be a "current task" for many APIs in `std`. The entire task infrastructure is replaced with `std::thread`, which provides a more standard API for manipulating and creating native OS threads. In particular, it's possible to join on a created thread, and to get a handle to the currently-running thread. In addition, threads are equipped with some basic blocking support in the form of `park`/`unpark` operations (following a tradition in some OSes as well as the JVM). See the `std::thread` documentation for more details.

* Channels are refactored to use a new internal blocking infrastructure that itself sits on top of `park`/`unpark`.

One important change here is that a Rust program ends when its main thread does, following most threading models. On the other hand, threads will often be created with an RAII-style join handle that will re-institute blocking semantics naturally (and with finer control).

This is very much a:

[breaking-change]

Closes #18000
r? @alexcrichton
7a33188
@bors bors added a commit that referenced this pull request Dec 17, 2014
@bors bors auto merge of #19654 : aturon/rust/merge-rt, r=alexcrichton
This PR substantially narrows the notion of a "runtime" in Rust, and allows calling into Rust code directly without any setup or teardown. 

After this PR, the basic "runtime support" in Rust will consist of:

* Unwinding and backtrace support
* Stack guards

Other support, such as helper threads for timers or the notion of a "current thread" are initialized automatically upon first use.

When using Rust in an embedded context, it should now be possible to call a Rust function directly as a C function with absolutely no setup, though in that case panics will cause the process to abort. In this regard, the C/Rust interface will look much like the C/C++ interface.

In more detail, this PR:

* Merges `librustrt` back into `std::rt`, undoing the facade. While doing so, it removes a substantial amount of redundant functionality (such as mutexes defined in the `rt` module). Code using `librustrt` can now call into `std::rt` to e.g. start executing Rust code with unwinding support.

* Allows all runtime data to be initialized lazily, including the "current thread", the "at_exit" infrastructure, and the "args" storage.

* Deprecates and largely removes `std::task` along with the widespread requirement that there be a "current task" for many APIs in `std`. The entire task infrastructure is replaced with `std::thread`, which provides a more standard API for manipulating and creating native OS threads. In particular, it's possible to join on a created thread, and to get a handle to the currently-running thread. In addition, threads are equipped with some basic blocking support in the form of `park`/`unpark` operations (following a tradition in some OSes as well as the JVM). See the `std::thread` documentation for more details.

* Channels are refactored to use a new internal blocking infrastructure that itself sits on top of `park`/`unpark`.

One important change here is that a Rust program ends when its main thread does, following most threading models. On the other hand, threads will often be created with an RAII-style join handle that will re-institute blocking semantics naturally (and with finer control).

This is very much a:

[breaking-change]

Closes #18000
r? @alexcrichton
e45347f
@bors bors added a commit that referenced this pull request Dec 18, 2014
@bors bors auto merge of #19654 : aturon/rust/merge-rt, r=alexcrichton
This PR substantially narrows the notion of a "runtime" in Rust, and allows calling into Rust code directly without any setup or teardown. 

After this PR, the basic "runtime support" in Rust will consist of:

* Unwinding and backtrace support
* Stack guards

Other support, such as helper threads for timers or the notion of a "current thread" are initialized automatically upon first use.

When using Rust in an embedded context, it should now be possible to call a Rust function directly as a C function with absolutely no setup, though in that case panics will cause the process to abort. In this regard, the C/Rust interface will look much like the C/C++ interface.

In more detail, this PR:

* Merges `librustrt` back into `std::rt`, undoing the facade. While doing so, it removes a substantial amount of redundant functionality (such as mutexes defined in the `rt` module). Code using `librustrt` can now call into `std::rt` to e.g. start executing Rust code with unwinding support.

* Allows all runtime data to be initialized lazily, including the "current thread", the "at_exit" infrastructure, and the "args" storage.

* Deprecates and largely removes `std::task` along with the widespread requirement that there be a "current task" for many APIs in `std`. The entire task infrastructure is replaced with `std::thread`, which provides a more standard API for manipulating and creating native OS threads. In particular, it's possible to join on a created thread, and to get a handle to the currently-running thread. In addition, threads are equipped with some basic blocking support in the form of `park`/`unpark` operations (following a tradition in some OSes as well as the JVM). See the `std::thread` documentation for more details.

* Channels are refactored to use a new internal blocking infrastructure that itself sits on top of `park`/`unpark`.

One important change here is that a Rust program ends when its main thread does, following most threading models. On the other hand, threads will often be created with an RAII-style join handle that will re-institute blocking semantics naturally (and with finer control).

This is very much a:

[breaking-change]

Closes #18000
r? @alexcrichton
acd0294
@bors bors added a commit that referenced this pull request Dec 19, 2014
@bors bors auto merge of #19654 : aturon/rust/merge-rt, r=alexcrichton
This PR substantially narrows the notion of a "runtime" in Rust, and allows calling into Rust code directly without any setup or teardown. 

After this PR, the basic "runtime support" in Rust will consist of:

* Unwinding and backtrace support
* Stack guards

Other support, such as helper threads for timers or the notion of a "current thread" are initialized automatically upon first use.

When using Rust in an embedded context, it should now be possible to call a Rust function directly as a C function with absolutely no setup, though in that case panics will cause the process to abort. In this regard, the C/Rust interface will look much like the C/C++ interface.

In more detail, this PR:

* Merges `librustrt` back into `std::rt`, undoing the facade. While doing so, it removes a substantial amount of redundant functionality (such as mutexes defined in the `rt` module). Code using `librustrt` can now call into `std::rt` to e.g. start executing Rust code with unwinding support.

* Allows all runtime data to be initialized lazily, including the "current thread", the "at_exit" infrastructure, and the "args" storage.

* Deprecates and largely removes `std::task` along with the widespread requirement that there be a "current task" for many APIs in `std`. The entire task infrastructure is replaced with `std::thread`, which provides a more standard API for manipulating and creating native OS threads. In particular, it's possible to join on a created thread, and to get a handle to the currently-running thread. In addition, threads are equipped with some basic blocking support in the form of `park`/`unpark` operations (following a tradition in some OSes as well as the JVM). See the `std::thread` documentation for more details.

* Channels are refactored to use a new internal blocking infrastructure that itself sits on top of `park`/`unpark`.

One important change here is that a Rust program ends when its main thread does, following most threading models. On the other hand, threads will often be created with an RAII-style join handle that will re-institute blocking semantics naturally (and with finer control).

This is very much a:

[breaking-change]

Closes #18000
r? @alexcrichton
b8f840c
aturon and others added some commits Nov 24, 2014
@aturon aturon libs: merge librustrt into libstd
This commit merges the `rustrt` crate into `std`, undoing part of the
facade. This merger continues the paring down of the runtime system.

Code relying on the public API of `rustrt` will break; some of this API
is now available through `std::rt`, but is likely to change and/or be
removed very soon.

[breaking-change]
2b3477d
@aturon aturon Refactor std::os to use sys::os 74d0769
@aturon aturon Allow args to work without rt initialization b66681c
@aturon aturon Make at_exit initialize lazily c009bfd
@aturon aturon Remove rt::bookkeeping
This commit removes the runtime bookkeeping previously used to ensure
that all Rust tasks were joined before the runtime was shut down.

This functionality will be replaced by an RAII style `Thread` API, that
will also offer a detached mode.

Since this changes the semantics of shutdown, it is a:

[breaking-change]
9b03b72
@aturon aturon Introduce std::thread
Also removes:

* `std::task`
* `std::rt::task`
* `std::rt::thread`

Notes for the new API are in a follow-up commit.

Closes #18000
cac133c
@aturon aturon Remove rt::{local, local_data, thread_local_storage} 84cb6cd
@aturon aturon Add blocking support module for channels 7fd7ce6
@aturon aturon Remove rt::{mutex, exclusive} d8e4780
@aturon aturon Fallout from new thread API 43ae4b3
@aturon aturon Revise rt::unwind 14c1a10
@aturon aturon Disable capture_stderr test for now ced2239
@alexcrichton @aturon alexcrichton Fix the capture_stderr test
There's always a fun time having two sets of standard libraries when testing!
9644d60
@alexcrichton @aturon alexcrichton Fix compilation on linux 7a6c54c
@alexcrichton @aturon alexcrichton Tweak the startup routine to pass on linux
We need to be sure to init thread_info before we init args for example because
args is grabbing locks which may entail looking at the local thread eventually.
a7061d0
@alexcrichton @aturon alexcrichton Avoid .take().unwrap() with FnOnce closures 4ffd9f4
@aturon aturon Revise std::thread API to join by default
This commit is part of a series that introduces a `std::thread` API to
replace `std::task`.

In the new API, `spawn` returns a `JoinGuard`, which by default will
join the spawned thread when dropped. It can also be used to join
explicitly at any time, returning the thread's result. Alternatively,
the spawned thread can be explicitly detached (so no join takes place).

As part of this change, Rust processes now terminate when the main
thread exits, even if other detached threads are still running, moving
Rust closer to standard threading models. This new behavior may break code
that was relying on the previously implicit join-all.

In addition to the above, the new thread API also offers some built-in
support for building blocking abstractions in user space; see the module
doc for details.

Closes #18000

[breaking-change]
a27fbac
@aturon aturon Update doc comment for std::rt 13f302d
@alexcrichton @aturon alexcrichton std: Lower abstractions for thread_local/at_exit
The current implementations use `std::sync` primitives, but these primitives
currently end up relying on `thread_info` and a local `Thread` being available
(mainly for checking the panicking flag).

To get around this, this commit lowers the abstractions used by the windows
thread_local implementation as well as the at_exit_imp module. Both of these
modules now use a `sys::Mutex` and a `static mut` and manage the
allocation/locking manually.
5759cff
@alexcrichton @aturon alexcrichton std: Move the panic flag to its own thread local
This flag is somewhat tied to the `unwind` module rather than the `thread_info`
module, so this commit moves it into that module as well as allowing the same OS
thread to call `unwind::try` multiple times. Previously once a thread panicked
its panic flag was never reset, even after exiting the panic handler.
d08600b
@aturon aturon Delete rest of rustrt
... and address other rebasing fallout.
0ce5faa
@aturon aturon Disable stack overflow test on android, which seems to be failing spu…
…riously.

cc rust-lang#20004
f4c0c0f
@aturon aturon Rebasing fixes. a9e7669
@aturon aturon Disable at_exit handlers
The [final step](rust-lang#19654) of
runtime removal changes the threading/process model so that the process
shuts down when the main thread exits. But several shared resources,
like the helper thread for timeouts, are shut down when the main thread
exits (but before the process ends), and they are not prepared to be
used after shut down, but other threads may try to access them during
the shutdown sequence of the main thread.

As an interim solution, the `at_exit` cleanup routine is simply skipped.

Ultimately, these resources should be made to safely handle asynchronous
shutdown, usually by panicking if called from a detached thread when the
main thread is ending.

See issue for details rust-lang#20012

This is a [breaking-change] for anyone relying on `at_exit`.
903c5a8
@alexcrichton

p=10

Contributor
bors replied Dec 19, 2014

saw approval from alexcrichton
at aturon@903c5a8

Contributor
bors replied Dec 19, 2014

merging aturon/rust/merge-rt = 903c5a8 into auto

Contributor
bors replied Dec 19, 2014

status: {"merge_sha": "0efafac398ff7f28c5f0fe756c15b9008b3e0534"}

Contributor
bors replied Dec 19, 2014

aturon/rust/merge-rt = 903c5a8 merged ok, testing candidate = 0efafac

Contributor
bors replied Dec 19, 2014

fast-forwarding master to auto = 0efafac

Contributor
bors replied Dec 19, 2014

fast-forwarding master to auto = 0efafac

@bors bors added a commit that referenced this pull request Dec 19, 2014
@bors bors auto merge of #19654 : aturon/rust/merge-rt, r=alexcrichton
This PR substantially narrows the notion of a "runtime" in Rust, and allows calling into Rust code directly without any setup or teardown. 

After this PR, the basic "runtime support" in Rust will consist of:

* Unwinding and backtrace support
* Stack guards

Other support, such as helper threads for timers or the notion of a "current thread" are initialized automatically upon first use.

When using Rust in an embedded context, it should now be possible to call a Rust function directly as a C function with absolutely no setup, though in that case panics will cause the process to abort. In this regard, the C/Rust interface will look much like the C/C++ interface.

In more detail, this PR:

* Merges `librustrt` back into `std::rt`, undoing the facade. While doing so, it removes a substantial amount of redundant functionality (such as mutexes defined in the `rt` module). Code using `librustrt` can now call into `std::rt` to e.g. start executing Rust code with unwinding support.

* Allows all runtime data to be initialized lazily, including the "current thread", the "at_exit" infrastructure, and the "args" storage.

* Deprecates and largely removes `std::task` along with the widespread requirement that there be a "current task" for many APIs in `std`. The entire task infrastructure is replaced with `std::thread`, which provides a more standard API for manipulating and creating native OS threads. In particular, it's possible to join on a created thread, and to get a handle to the currently-running thread. In addition, threads are equipped with some basic blocking support in the form of `park`/`unpark` operations (following a tradition in some OSes as well as the JVM). See the `std::thread` documentation for more details.

* Channels are refactored to use a new internal blocking infrastructure that itself sits on top of `park`/`unpark`.

One important change here is that a Rust program ends when its main thread does, following most threading models. On the other hand, threads will often be created with an RAII-style join handle that will re-institute blocking semantics naturally (and with finer control).

This is very much a:

[breaking-change]

Closes #18000
r? @alexcrichton
0efafac
@bors bors merged commit 903c5a8 into rust-lang:master Dec 19, 2014

2 checks passed

continuous-integration/travis-ci The Travis CI build passed
Details
default all tests passed
@tomaka tomaka referenced this pull request in tomaka/hlua Dec 19, 2014
Closed

rust-hl-modules fails with file operations? #15

@bluss
Contributor
bluss commented Dec 19, 2014

spawn() and other adjacent features like channel() were in the prelude, saying something to the new Rust user about Rust the language in the process. I think it would be nice to preserve that, the feel of being setup up for concurrency by default.

@reem
Contributor
reem commented Dec 19, 2014

It also sort of suggested that they were builtin constructs, which I think their removal from the prelude is supposed to prevent.

@aturon
Contributor
aturon commented Dec 19, 2014

@bluss You might want to leave a comment at rust-lang/rfcs#503, which talks about the future of the prelude.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment