Introduce the Tokio runtime: Reactor + Threadpool #141

carllerche · 2018-02-15T18:19:39Z

This patch is an initial implementation of the Tokio runtime. The Tokio runtime provides an out of the box configuration for running I/O heavy asynchronous applications.

tl;dr

This patch includes a work-stealing thread pool and an easy way to get a reactor + thread pool:

extern crate tokio;
extern crate futures;
use futures::{Future, Stream};
use tokio::net::TcpListener;

let addr = "127.0.0.1:8080".parse().unwrap();
let listener = TcpListener::bind(&addr).unwrap();

let server = listener.incoming()
    .map_err(|e| println!("error = {:?}", e))
    .for_each(|socket| {
        tokio::spawn(process(socket))
    });

tokio::run(server);

This will result in sockets being processed across all available cores.

Background

Most I/O heavy application require the same general setup: a reactor that drives I/O resources and an executor to run application logic. Historically, tokio-core provided both of these in one structure. A reactor and a single threaded executor for application logic that ran on the same thread as the reactor.

The Tokio reform RFC proposed splitting those two constructs. The Tokio reactor now can be run standalone. To replace the single threaded executor, current_thread was introduced. However, this leaves figuring out concurrency up to the user.

Idea

Modern computers provide many cores. Applications really need to be able to take advantage of this available concurrency opportunity. It would be nice if Tokio provided an out of the box concurrency solution that is solid for most use cases.

Thread Pool

First, to support scheduling futures efficiently across all cores, a thread pool is needed. Currently, the consensus regarding the best strategy is to use a work-stealing thread pool. So, Tokio now has a new crate: tokio-threadpool.

Now, there are many ways to implement a work-stealing thread pool. If you are familiar with Rayon, you might already know that Rayon also uses a work-stealing thread pool. However, Rayon's pool is optimized for a different use case than tokio-threadpool. Rayon's goal is to provide concurrency when attempting to compute a single end result. Tokio's goal is to multiplex many independent tasks with as much fairness as possible while still having good performance. So, the implementation of tokio-threadpool and Rayon's pool are going to be fairly different.

`tokio-executor`

Another new crate is included, tokio-executor. This is mostly pulling in the design work happening in the futures executor RFC. Since the goal between now and Tokio 0.2 is to get as much experimentation done as possible, I wanted to give the traits / types discussed in that RFC a spin. Ideally, the tokio-executor crate will be short lived and gone by 0.2.

Putting it together

The tokio crate includes a new Runtime type, which is a handle to both the reactor and the thread pool. This allows starting and stopping both in a coordinated way and will provide configuration options and more over time.

Path to 0.2

Again, the goal for the time period between now and 0.2 is to get as much experimentation done as possible. The Runtime construct is an initial proposal with an implementation, giving users a change to try it out and provide feedback.

Before 0.2 is released, an RFC will be written up detailing what was learned, and what is being proposed for inclusion in the 0.2 release. This will be a forum for all to provide feedback based on usage.

Remaining for this PR

Update documentation
Audit APIs being introduced
Check for any changes in the futures executor RFC.
Should park return a Result?
Actually add tokio::spawn fn.
Rename run_seeded.

This patch is an intial implementation of the Tokio runtime. The Tokio runtime provides an out of the box configuration for running I/O heavy asynchronous applications. As of now, the Tokio runtime is a combination of a work-stealing thread pool as well as a background reactor to drive I/O resources. This patch also includes tokio-executor, a hopefully short lived crate that is based on the futures 0.2 executor RFC.

carllerche · 2018-02-15T18:40:55Z

The Runtime type is here. This file introduces the bulk of the new public API. The rest is the implementation.

kpp · 2018-02-15T19:15:51Z

Cargo.toml

 ]

 [badges]
 travis-ci = { repository = "tokio-rs/tokio" }
 appveyor = { repository = "carllerche/tokio" }

 [dependencies]
+tokio-io = "0.1"


Why do you set path in tokio-executor like path = "tokio-executor" but you get tokio-io from crates.io?

The tokio-io dependency probably should be updated as well 👍 That can be done on master.

This enables the reactor to be used as the thread parker for executors. This also adds an `Error` component to `Park`.

olix0r · 2018-02-15T22:09:34Z

tokio-threadpool/src/deque.rs

+
+    #[inline]
+    pub fn poll(&self) -> Poll<T> {
+        unsafe {


Is this whole function actually unsafe? ISTM that the scope could be narrowed.

Ah, thanks. This entire file is actually dead code right now :) The deque is provided by a lib.

olix0r · 2018-02-15T22:18:56Z

src/runtime.rs

+        // Get a handle to the reactor.
+        let handle = reactor.handle().clone();
+
+        let pool = threadpool::Builder::new()


we probably want a way to configure the upper bounds of number of threads this threadpool may run?

alkis · 2018-02-15T23:18:37Z

src/reactor/mod.rs

+///
+/// A `Handle` is used for associating I/O objects with an event loop
+/// explicitly. Typically though you won't end up using a `Handle` that often
+/// and will instead use and implicitly configured handle for your thread.


alkis · 2018-02-15T23:23:15Z

src/reactor/mod.rs

+
+        // If the fallback hasn't been previously initialized then let's spin
+        // up a helper thread and try to initialize with that. If we can't
+        // actually create a helper thread then we'll just return a "defunkt"


s/defunkt/defunct/

djg · 2018-02-15T23:38:43Z

tokio-threadpool/README.md

+
+## License
+
+`futures-pool` is primarily distributed under the terms of both the MIT license


Should this be tokio-threadpool instead of futures-pool?

ian-p-cooke · 2018-02-16T10:50:13Z

src/runtime.rs

+}
+
+/// Start the Tokio runtime and spawn the provided future.
+pub fn run_seeded<F>(f: F)


this is trivial but the 'seeded' terminology seems very unfamiliar for what's probably a common entrypoint to the API. The only 'seed' that comes to mind is that of random number generation. What about just 'run_future' as the name here?

I agree that the name isn't great.

The issue with run_future is that the function does more than this. It spawns the future onto the runtime, runs it AND any other futures that get spawned.

Any other thoughts for names?

go ?
How about run_async ?

run for closure, poll for future?

I'm thinking of just renaming run_seeded -> run and getting rid of the other fn (in favor of using the Runtime instance directly).

This patch updates `CurrentThread` to take a `&mut Enter` reference as part of all functions that require an executor context. This allows the callers to configure the executor context before hand. For example, the default Tokio reactor can be set for the `CurrentThread` executor.

alkis · 2018-02-17T07:46:39Z

tokio-threadpool/src/lib.rs

+
+        let mut found_work = false;
+
+        worker_loop!(idx = len; len; {


Shouldn't this be stealing from a random, active worker? This seems to be going through the workers in order.

Also this is the only use of worker_loop macro. Perhaps inline it?

Hmm... you are correct. It was supposed to start at a random index but that seems to have gotten lost / forgotten! Thanks.

On mobile ATM and didn't inspect the loop mentioned, but are we confident that starting at a random index is sufficiently random? Something tells me we actually need to randomly index into a proper set of available workers, otherwise there may be statistical bias towards picking higher numbered workers

It works as a ring, so there isn't really a "higher" index worker.

IIRC, this bit was inspired from Java's ForkJoin pool.

alkis · 2018-02-17T07:50:47Z

tokio-threadpool/src/task.rs

+        mem::transmute(unpark_id)
+    }
+
+    /// Execute the task returning `true` if the task needs to be scheduled again.


s/true/Run::Schedule/

alkis · 2018-02-17T19:56:13Z

@carllerche is it possible to add more toplevel docs explaining some the role of each abstraction? It would help a lot when doing a deeper review.

alkis · 2018-02-18T21:03:20Z

src/executor/current_thread/mod.rs

+/// In more detail, this function will block until:
+/// - All executing futures are complete, or
+/// - `cancel_all_spawned` is invoked.
+pub fn run<F, R>(f: F) -> R


Instead of run and run_seeded, how about: run_with_context (closure) and poll_with_context (future)?

@alkis

Thanks @alkis

alkis · 2018-02-18T21:50:02Z

src/executor/current_thread/scheduler.rs

+        self.inner.clone().into()
+    }
+
+    pub fn schedule(&mut self, item: Box<Future<Item = (), Error = ()>>) {


Any way to avoid one of the two allocations here? One from Arc and one from the Box'ed future.

There might be a way, but it is actually pretty tricky due to the need for trait objects. I opted to not worry about it for now until it is proven to be an issue.

alkis · 2018-02-18T22:18:38Z

tokio-threadpool/src/lib.rs

+
+    /// Generates a random number
+    ///
+    /// Uses a thread-local seeded XorShift.


Can't this be part of WorkerEntry instead of thread-local?

carllerche · 2018-02-19T03:12:48Z

@alkis re: randomizing work stealing, the logic was taken from Java's implementation. The pool is intended to work well with a partial worker set, i.e. with any number of worker threads shutdown.

There probably are ways to improve the heuristics, but I will punt any such tuning until later and will want a solid set of benchmarks before making such changes.

alkis · 2018-02-19T08:53:57Z

@carllerche Have you looked into the design of the go scheduler? It was designed by Vyukov:

Notable differences to current implementation:

when stealing, thief steals half the queue of the victim
workers only look at global queue only once per 61 turns
stealing goes through the active workers in a random permutation
thieves spin 4 times trying to find victims, before giving up and parking
workers do not steal at all if there is enough thieves already

There is a comment in the source code which starts with "Worker thread parking/unparking" which explains the current design and why other approaches work badly.

carllerche · 2018-02-19T17:32:12Z

@alkis All good thoughts, especially stealing more than one task at a time. I was going to include this initially, but the deque impl lacked the ability. The impl has since moved to crossbeam, so threadpool should be updated.

Re: all the other items. It's definitely worth experimenting with, but backed with numbers :) This PR represents an initial commit bringing in the feature, enabling iteration over time. The Go scheduler is intended for a slightly different use case than Tokio's. Go has one scheduler for the entire process and everything uses it. All threads are constantly active in some capacity (threads exist, but might be sleeping). Tokio's thread pool is (currently) intended to handle:

More than one instance of a thread pool in a process
Threads can spin up and shutdown.
There might only be a single thread running in the pool.
TBD additional features that don't make sense in Go's runtime.

If you are interested, iterating the scheduler is something that you could champion, starting with setting up a suite of benchmarks. Then, each change can be made and we can check the impact in various scenarios.

Not to mention, there currently exist some bigger known issues (there is a case of false sharing... a TODO in the code mentions this). Also, the code base is generally immature and needs people to use it, so getting it out sooner than later is good!

alkis · 2018-02-20T10:12:01Z

src/executor/current_thread/mod.rs

+
+    /// Run the executor to completion, blocking the thread until all
+    /// spawned futures have completed **or** `duration` time has elapsed.
+    pub fn run_timeout(&mut self, duration: Duration)


General question: why does Rust prefer timeouts vs deadlines? Timeouts don't compose well:

// I want to finish something in 10 secs, with timeouts. let mut start = Instant::now(); let elapsed = || { let now = Instant::now(); let res = start - res; start = now; res }; // Not even entirely correct because elapsed() can be greater than 10 secs and subtraction will panic. let c = socket.open_with_timeout(Duration::from_secs(10))?; let data = socket.read_with_timeout(Duration::from_secs(10) - elapsed())?; let processed = process(data)?; socket.write_with_timeout(processed, Duration::from_secs(10) - elapsed())?;

// I want to finish something in 10 secs, with deadline. let deadline = Instant::now() + Duration::from_secs(10); let c = socket.open_with_deadline(deadlline)?; let data = socket.read_with_deadline(deadlline)?; let processed = process(data)?; socket.write_with_deadline(processed, deadlline)?;

IIRC, Rust itself picked timeouts as that is what gets passed to the syscalls.

One can add a deadline shim on top of a timeout based API, but if the APIs themselves are deadline based, you would have to make additional syscalls within each one to get Instant::now().

I personally have never hit the composability issue as working w/ futures it is a non-issue (you define the computation w/o any timeout or deadline and set a timeout on the entire computation). However, if it did become an issue, I'd probably add a deadline shim on top :)

I think adding a timeout shim on top of deadline API is easier (no need to check for negative timeouts). Also deadline is the right thing for the implementation anyway.

I don't think there is additional Instant::now() call unless the timeout based code has bugs (see below).

The composability issue is a real problem. Also timeouts usually lead to bugs whenever there are retries or partial reads. Go switched from SetTimeout to SetDeadline before 1.0 mainly for the latter reason.

FWIW I just check stdlib and it suffers from this as well. If we set a timeout on a socket and then use read_to_end or read_exact the timeout won't be obeyed if the loops inside said functions execute more than once. Even if we try to describe the current semantics they are super weird: the effective timeout is K * the set timeout where K is the number of successful partial reads of read_exact or read_to_end respectively.

@alkis For me, it comes down to this. How do you change this function to take a deadline instead of a timeout without increasing the number of syscalls?

@carllerche the function you linked will gain one Instant::now and this function will lose this syscall.

@alkis for example in automotive, having multiple internal clocks, such as the dashboard-clock, power-control-unit-clock, system-clock, and steady-clock. The latter being a monotonic clock; the time points of this clock cannot decrease as physical time moves forward. In contrast a system clock may be adjusted by user or via GPS. If you define a deadline-timer relative to the system clock, and in the meanwhile the clock is adjusted to a timepoint in the past, the timer will fire after a much longer interval than intended; the corresponding functionality of the app may seem frozen and malfunctioning. Using a timeout is not associated to a specific clock, is much more robust against adjustments of clocks

That's sounds like a bug. There is access to monotonic clock yet one is defined on a non monotonic clock.

I am looking for an example where the system does not have a monotonic clock and the program specifies a timeout and that is handled correctly.

FYI I wrote this on the topic.

@alkis you probably saw, but tokio-timer is now using deadlines.

No, I didn't. Thanks for the pointer @carllerche!

carllerche · 2018-02-21T07:04:27Z

Ok, I think that this is ready to merge (assuming CI passes).

kpp · 2018-02-22T08:38:34Z

src/runtime.rs

+//!     .shutdown_on_idle()
+//!     .wait().unwrap();
+//!
+//! tokio::run(server);


Do we have to call tokio::run after we created Runtime?

Bathtor · 2018-03-01T08:53:48Z

I'm slightly wondering why are we duplicating work on executors? I did exactly the same thing (i.e. writing a workstealing threadpool scheduler on crossbream) months ago, because I noticed it was missing, and I published it to crate.io so people could use it.
Wouldn't it be better to work towards one really good implementation of this, than a bunch of mediocre ones? (Or at least one per load type. As pointed out before there are differences between computing a single result as fast as possible and scheduling reactive tasks well.)

jeehoonkang · 2018-03-06T14:38:50Z

@carllerche you mentioned in #141 (comment) that stealing multiple elements will be helpful. I have an implementation of many-stealing deque, and I wonder if you are interested in it.

While I "guess" this algorithm is correct, without proofs I cannot be confident. I'm currently proving it, and I hope it'll be finished within a month. In the meantime, I'd like to see if stealing multiple elements will actually improve the performance of Tokio. I'll be back within days with benchmark results.

FYI, Rayon developers are also interested in stealing multiple elements (relevant issue here), but they are not sure whether it's a performance win. I'll bench Rayon with the new deque, too.

jeehoonkang · 2018-03-06T17:31:18Z

I changed tokio-threadpool to use the many-stealing deque, and got the following result on a benchmark. Here, control is the PR #185 (replacing coco with crossbeam), and variable is this branch. In variable, Worker::try_steal_task() steals multiple elements, and if there are more than one, it executes one and pushes the others into its own deque. I believe there's a better strategy, but it was the easiest to implement. The benchmark was performed in a Xeon E5-2620 v3 @ 2.40GHz w/ 12 cores (24 hw threads).

 name                                control ns/iter  variable ns/iter  diff ns/iter   diff %  speedup
 connect_churn::multi_threads        7,376,326        7,385,806                9,480    0.13%   x 1.00
 connect_churn::one_thread           9,325,834        7,109,259           -2,216,575  -23.77%   x 1.31
 connect_churn::two_threads          15,843,802       15,970,003             126,201    0.80%   x 0.99
 futures_channel_latency             8,223            5,680                   -2,543  -30.93%   x 1.45
 mio_poll                            125              168                         43   34.40%   x 0.74
 mio_register_deregister             439              588                        149   33.94%   x 0.75
 mio_reregister                      127              129                          2    1.57%   x 0.98
 transfer::big_chunks::one_thread    3,343,033        3,330,092              -12,941   -0.39%   x 1.00
 transfer::small_chunks::one_thread  60,798,726       60,740,570             -58,156   -0.10%   x 1.00
 udp_echo_latency                    19,158           17,840                  -1,318   -6.88%   x 1.07

Overall, it shows that many-stealing is very helpful for some use cases, but maybe not for all. In particular, mio benchmarks are significantly worsened. I'll investigate why.

carllerche · 2018-03-06T17:43:46Z

The Mio benchmarks are unrelated to anything in Tokio. They are setting the baseline. Any change to tokio-threadpool will not impact those. So, it is odd that there is such a huge diff.

alkis · 2018-03-06T20:45:33Z

How many does steal_many() steal? IIRC Go steals half the queue.

jeehoonkang · 2018-03-07T01:15:59Z

@alkis currently it steals half the elements. I named it steal_many() because it can be configurable in the future, but maybe is it better to name it steal_half()?

ghost · 2018-06-16T10:38:44Z

examples/hello_world.rs

-        // In our example, we have not defined a shutdown strategy, so
-        // this will block until `ctrl-c` is pressed at the terminal.
-    });
+    // current thread. This means that spawned tasks must not implement `Send`.


Wasn't the old wording correct - "spawned tasks are not required to implement Send"? :)

You are correct!

Well, the good news is it has been fixed on master it looks like.

kpp reviewed Feb 15, 2018

View reviewed changes

Implement Park for Reactor

815eee5

This enables the reactor to be used as the thread parker for executors. This also adds an `Error` component to `Park`.

olix0r reviewed Feb 15, 2018

View reviewed changes

Remove deque.rs as it is not used anymore.

4121146

olix0r reviewed Feb 15, 2018

View reviewed changes

alkis reviewed Feb 15, 2018

View reviewed changes

djg reviewed Feb 15, 2018

View reviewed changes

ian-p-cooke reviewed Feb 16, 2018

View reviewed changes

carllerche added 3 commits February 16, 2018 13:23

Add Executor::status impl for current_thread.

eb823f8

Add CurrentThread::turn

4b55003

alkis reviewed Feb 17, 2018

View reviewed changes

carllerche added 2 commits February 17, 2018 10:00

Minor tweaks to the various APIs

d664b21

Fix typos

e4cfc5e

alkis reviewed Feb 18, 2018

View reviewed changes

carllerche added 2 commits February 18, 2018 13:26

Fix tests

3b051bc

Fix threadpool steal routine

a366f7c

Thanks @alkis

alkis reviewed Feb 18, 2018

View reviewed changes

carllerche added 3 commits February 18, 2018 21:01

Misc CurrentThread tweaks

b2953f0

Some API tweaks to CurrentThread

658afb8

Work on executor facade and docs

76ce551

More documentation

234558c

alkis reviewed Feb 20, 2018

View reviewed changes

carllerche added 14 commits February 20, 2018 13:29

Add docs for park

f4f4168

Simplify ParkThread::park implementation

a4c6225

More docs

8480f11

Start adding threadpool documentation

607958f

Threadpool API docs

b60bfb6

Add Sender API docs

d9e6806

Some more ThreadPool documentation

626a49e

Write some Runtime documentation

53ad363

Documentation + fixes

beaf747

EnterError is unused

9c6cd94

Add a basic Runtime test

8f53991

Check in the runtime test

753bce0

Don't expose reactor::with_default / set_default yet

8188c4b

Audit API + docs

adfcf10

carllerche merged commit fe14e7b into master Feb 21, 2018

carllerche mentioned this pull request Feb 21, 2018

Explore improvements to the thread pool scheduling logic #145

Closed

kpp reviewed Feb 22, 2018

View reviewed changes

carllerche deleted the tokio-runtime branch February 26, 2018 21:54

smangelsdorf mentioned this pull request Feb 28, 2018

Track Tokio reform and consider what refactoring is required for Gotham to work with Tokio 0.2 gotham-rs/gotham#165

Closed

ghost reviewed Jun 16, 2018

View reviewed changes

Nic0w mentioned this pull request Sep 4, 2022

Calling fill_buf on a BufReader<OwnedReadHalf> does not always bring fresh data when it should #4974

Closed


		## License

		`futures-pool` is primarily distributed under the terms of both the MIT license

Introduce the Tokio runtime: Reactor + Threadpool #141

Introduce the Tokio runtime: Reactor + Threadpool #141

Conversation

carllerche commented Feb 15, 2018 • edited Loading

tl;dr

Background

Idea

Thread Pool

tokio-executor

Putting it together

Path to 0.2

Remaining for this PR

carllerche commented Feb 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alkis Feb 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alkis commented Feb 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carllerche commented Feb 19, 2018

alkis commented Feb 19, 2018 • edited Loading

carllerche commented Feb 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frehberg Mar 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carllerche commented Feb 21, 2018

Choose a reason for hiding this comment

Bathtor commented Mar 1, 2018

jeehoonkang commented Mar 6, 2018

jeehoonkang commented Mar 6, 2018 • edited Loading

carllerche commented Mar 6, 2018

alkis commented Mar 6, 2018

jeehoonkang commented Mar 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carllerche commented Feb 15, 2018 •

edited

Loading

`tokio-executor`

alkis Feb 17, 2018 •

edited

Loading

alkis commented Feb 19, 2018 •

edited

Loading

frehberg Mar 26, 2018 •

edited

Loading

jeehoonkang commented Mar 6, 2018 •

edited

Loading