RFC 0002: MT Execution Contexts #2

ysbaddaden · 2024-02-05T18:31:51Z

No description provided.

beta-ziliani · 2024-02-05T18:35:40Z

Preview: https://github.com/crystal-lang/rfcs/blob/rfc-0002-mt-execution-contexts/text/0002-execution-contexts.md

0002-execution-contexts.md

straight-shoota · 2024-02-06T20:08:46Z

The distinction between execution context and scheduler could need a bit refinement. There is definitely some overlap in functionality, just by comparing the API. I guess execution contexts might take over some features of the current scheduler?

RX14

Absolutely lovely, well-written proposal. I completely agree with the design intent here, and only have a few—mostly overlapping—comments about event loops and the default context. The vast majority of this design is exactly what I would like to see in crystal.

0002-execution-contexts.md

RX14 · 2024-02-06T20:48:28Z

0002-execution-contexts.md

+- a scheduler to run the fibers (or many schedulers for a MT context);
+- an event loop (IO & timers):
+
+  => this might be complex: I don’t think we can share a libevent across event bases? we already need to have a “thread local” libevent object for IO objects as well as for PCRE2 (though this is an optimization).


Are multiple event bases required? It doesn't seem obvious to me that they are.

Yeah, I'm wondering about that. We currently have one libevent base per thread, and that overcomplexifies IO::Evented with lots of objects to allocate in the HEAP for each IO and thread.

I'm probably being naive (though Go seems to do that) but maybe a global event loop wouldn't behave so badly? Even with the potential contention on adding an event to the libevent base, especially on machines with too many cores to count (e.g. ARM Neoverse).

On the other hand not having to synchronize everything and their mother make other things a lot less complex. It is quite liberating to not give a shit about what other threads are doing.

FWIW, having one dedicated ring per thread is also how the makers of io_uring recommend multi thread usage.

Hum, implementing our own wrapper on top of epoll/kqueue becomes more and more compelling.

Since we reschedule the fiber, we could put events on the stack —this is possible with libevent but not recommended: the struct size may change, and not have to keep them somewhere to try and avoid reallocating events all the time.

Then we could have one or many event loops and not care about thread locals (especially in IO); we could keep Fiber#resume_event and merely take care that it can only be in one event loop queue at a time.

Yes, storing it on the stack is a very useful technique for io_uring as well (to keep it alive during submission). I've made a lot of use of that and it helped a lot. There the completion event need to be handled somehow though - I stored it in the fiber itself until it was awakened and could process it, but there may be better techniques.

One good thing about the current implementation though is that it handles the thundering herd problem decently, as far as I've been able to see. That is, if multiple fibers are waiting for something only one of them will wake when something happens. With many listeners that may become something to keep track of.

We shall move the "refactor the event loop" into a proper issue on https://github.com/crystal-lang/crystal/issues

It doesn't need a RFC as it's mostly internal implementation detail.

It might be interesting to consider one event loop (EL) per execution context (EC) 🤔

With a single EL per EC, a starving scheduler will run the EL and enqueue every resumable fibers for the EC.

With one EL per scheduler, a starving scheduler will run its own EL and push the fraction of fibers it happened to have, possibly flooding the EC queues and delaying other schedulers from running their own EL, and delaying resumable fibers from being resumed.

That may be a reason why Go has a single EL: it could be unfair otherwise.

0002-execution-contexts.md

RX14 · 2024-02-07T09:05:29Z

0002-execution-contexts.md

+
+## Default context configuration
+
+This proposal doesn’t solve the inherent problem of: how can applications configure the default context at runtime (e.g. number of MT schedulers) since we create the context before the application’s main can start. 


This proposal supports creating multiple execution contexts, let the application configure it's own EC and start fibers in it if required. That allows all the complexity the app needs when configuring the context the application actually runs in, because it's initialized by the application. The root context does not have to be well-used by the application.

ysbaddaden · 2024-02-13T10:13:06Z

@RX14 Tell me if I'm wrong, the plan would be:

Crystal 1

introduce EC with ST and MT;
deprecate same_thread argument;
same_thread: false is NOOP;
ST accepts same_thread: true (always true anyway);
MT raises on same_thread: true (new API, no breaking change);
default EC is ST (no breaking change);
consider a -Dmt flag to force default EC to be MT (?);

Crystal 2

remove deprecated same_thread (breaking change);
default EC becomes MT (breaking change).

Isolated context

I think I see that context for UI loops only, and want to prevent blocking behaviors, but there's nothing wrong with doing blocking calls in other use cases, and using the event-loop normally is fine. Still, spawning a fiber without an explicit context should either raise or the default context should be configured (as you suggest):

abstract class ExecutionContext
  class Isolated < ExecutionContext
    def initialize(name : String, @spawn_context : ExecutionContext? = nil, &)
      @thread = Thread.new(name) { yield }
    end

    def spawn(**args, &) : Fiber
      if ctx = @spawn_context
        ctx.spawn(**args) { yield }
      else
        raise RuntimeError.new("Can't spawn in isolated context (need a spawn context)")
      end
    end
  end
end

mt = ExecutionContext::MultiThreaded.new
ui = ExecutionContext::Isolated.new("GTK", spawn_context: mt) { Gtk.main }

Instead of raising, the spawn context could be the default EC.

ysbaddaden · 2024-02-13T10:50:43Z

@RX14 I applied your suggestions to the RFC.

There's no such method

RX14 · 2024-02-14T10:14:56Z

@ysbaddaden I think -Dmt is probably not necessary, and the exact implementation plan for crystal 2 is best left deferred until there's operational experience, but I agree on everything else.

I envision the root execution context being MT or ST a moot point, because every well-architected app has a single App.run line at the top-level and converting that top-level code to be spawning a MT context and waiting for that fiber should be a one-liner if we have the right helper methods in place.

If we all agree, maybe we can start on the other 90% of the RFC: bikeshedding naming. I like ExecutionContext::Parallel, because I don't like the idea of implementation details (threads) leaking into the name.

ysbaddaden · 2024-02-19T09:54:09Z

@RX14 The mt flag may not be necessary in Crystal v1, as the default context could be MT:1 and resized on demand (still no breaking change). I'll still push for MT:N to be the default in Crystal v2. Execution contexts are a mean to further control the parallelism in very specific cases, not the end solution. I believe developers shouldn't have to care about it until you have to.

I wouldn't bikeshed the namings just yet. As I'm experimenting with the types, I feel that the difference is getting thinner and thinner. In fact, Kotlin only has a single scheduler implementation, and a couple constructors to start execution contexts with 1 (ST) or many threads (MT).

I'm also struggling with the inheritance: EC::MT < EC makes sense, but so does EC::MT::Scheduler < EC as we want EC.current to point to the current MT scheduler running on the thread, not the shared MT context (it's easier to reach the context from the scheduler).

crysbot · 2024-02-21T01:44:00Z

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/crystal-multithreading-support/6622/8

ysbaddaden · 2024-02-23T10:30:13Z

I forgot again but MT:1 would break spawn(same_thread: true) in Crystal 1. It's a NOOP without the preview_mt flag but the parameter was still exposed to the public API 😭

straight-shoota · 2024-02-23T10:50:45Z

I think we can accept breakage with same_thread: true. It only works with preview_mt which is explicitly a preview feature. There should be no expectations on compatibility in a setting outside of preview_mt.

yxhuvud

I like this, I like this a lot.

yxhuvud · 2024-02-23T15:40:31Z

0002-execution-contexts.md

+
+Such a group of fibers will never run in parallel. This can vastly simplify the synchronization logic since you don’t have to deal with parallelism anymore, only concurrency, which is much easier & faster to deal with. For example no need for costly atomic operations, you can simply access a value directly. Parallelism issues and their impact on the application performance is limited to the global communication.
+
+## Issues


I think data and especially execution locality could show up on the negative side as well, as the round robin take away a lot of programmatic control of data locality as well. It is .. possible.. to manually schedule fibers to dedicated threads but it really is not the way it currently is meant to be used.

0002-execution-contexts.md

yxhuvud · 2024-02-23T15:57:31Z

0002-execution-contexts.md

+- a scheduler to run the fibers (or many schedulers for a MT context);
+- an event loop (IO & timers):
+
+  => this might be complex: I don’t think we can share a libevent across event bases? we already need to have a “thread local” libevent object for IO objects as well as for PCRE2 (though this is an optimization).


I'm in the 'let the event loop decide if it want to be instantiated on a thread, execution level or global level' camp. How that would look API-wise I'm less sure - especially not if dynamic amount of threads in a context is to be supported.

This might be complex: I don’t think we can share a libevent across event bases

From what I have gathered from libevent docs, it is possible, but would necessitate a lot more synchronization when io happens (*), so it is probably slower.

But yes, it is complex. Windows, and its weird file handles says hi. Each open file handle is specific for each instance of whatever it uses, so it needs to be only one global one event instance there.

we already enable some structures for thread safety but then create separate bases for each thread anyhow, IIRC. It was quite a while since I looked at it. I think we can remove that enabling without danger - they should really only be used when actually reusing a libevent base between threads). We don't use the specialized mt safe functions libevent that make use of it.

yxhuvud · 2024-02-23T16:01:41Z

0002-execution-contexts.md

+- configuration (e.g. number of threads, …);
+- methods to spawn, enqueue, yield and reschedule fibers within its premises;
+- a scheduler to run the fibers (or many schedulers for a MT context);
+- an event loop (IO & timers):


It probably needs to be mentioned that it needs to continue to work with channels and mutexes. This is somewhat straightforward today with how fibers are bound to a thread once executed, so the simple version is to just schedule it using the normal interfaces. But interfaces for scheduling are not necessarily the same between execution contexts as they are within it!

For example, a basic work stealing scheduler can default to just enqueue a fiber in the executing scheduler/thread context, and let distribution between threads happen in other ways. But that doesn't work if the thing to wake up doesn't live in the same context - then it needs to be communicated somehow. And then the question is what to communicate it to.

Also somewhere it should probably be explicitly defined what happens if channel interaction happens in an isolated context (as defined above).

This is explicit in the Guide side:

Applications can create any number of execution contexts in parallel. These contexts are isolated but they shall still be capable to communicate together with the usual synchronization primitives (e.g. Channel, Mutex) that must be thread-safe.

Once spawned a fiber shouldn’t move to another execution context. For example on re-enqueue the fiber must be resumed into it’s execution context: context B enqueues waiting sender from context A. That being said, we could allow to send a fiber to another context.

It's not detailed in the Technical side, though.

0002-execution-contexts.md

yxhuvud · 2024-02-23T16:17:44Z

0002-execution-contexts.md

+    def initialize(@name : String, @minimum : Int32, @maximum : Int32)
+      # todo: start @minimum threads
+    end


Allowing a dynamic amount of threads requires more synchronization and complexity than having a static amount. While it sounds nice to be able to adjust, it probably warrants its own separate class. Making certain all threads are in a waiting state before starting to actually queue stuff allows a bunch of simplifications with less mutexes and risks a lot fewer possible race conditions too.

Enqueue doesn't have to need much sync. Go pushes to a bounded local queue (per scheduler) with overflow to a global queue; threads can be started at any time: when they reach the run loop they will grab a batch of fibers from the global queue or steal from another scheduler. Stopping ain't more complex, schedulers aren't tied to a specific thread, the thread detaches the scheduler and returns itself to the thread pool.

The complexity is more in when to start / stop a thread.

And that complexity pushes for it to become a "future evolution".

0002-execution-contexts.md

Blacksmoke16 · 2024-02-23T17:01:47Z

As someone who isn't familiar with this stuff at all, my random question is:

Do we need to do anything in relation to like how Intel has P and E cores now? Like as a way to signal to the OS's thread scheduler that a fiber should have a preference on where it runs? Or is that something the OS itself handles somehow?

0002-execution-contexts.md

Co-authored-by: Linus Sellberg <sellberg@gmail.com>

ysbaddaden · 2024-02-26T18:15:08Z

@Blacksmoke16 From what I read specifying a thread priority can hint the OS to schedule the thread on a big (efficient) or little (power) core. We can also set a thread affinity to a given core, but we must detect the core type beforehand.

Adds notes about wrapping an existing EC, and thread affinities (to pin a thread to a core) in addition to set priorities (still no API). Simplifies the EC API to remove `yield` and `sleep` that may not be needed (the `Fiber.yield` and `sleep` methods can create the resume events), but adds `spawn(same_thread)` to handle the transition.

0002-execution-contexts.md

Co-authored-by: Johannes Müller <straightshoota@gmail.com>

ysbaddaden · 2024-03-04T16:49:02Z

@RX14 I'm growing fond of Parallel.
I think Exclusive could be better and more explicit than Isolated?
I'm not sure about the single threaded context... or maybe Single?

crysbot · 2024-03-14T10:34:45Z

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/fiber-usage-in-high-io-application/6689/2

ysbaddaden · 2024-03-18T09:58:41Z

After spending 2 days chasing after a bug, I realized I mixed up two different things into a single ExecutionContext class: the execution context (cross context) and the fiber scheduler (thread local) which led to a nasty bug (cross scheduler local enqueues, when only cross context enqueues should be allowed).

I'm trying to separate these into 2 different types (e.g. ExecutionContext and ExecutionContext::Scheduler), and probably make them as modules, so we can have a single type implementing both modules (e.g. Isolated) or a couple types each implementing one module (e.g. MultiThreaded).

crysbot · 2024-04-11T12:41:18Z

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/glib-main-loop-and-fibers-integration/6156/11

1. the context itself (that may have many threads); 2. the fiber scheduling (on _one_ thread). The context itself is meant to create and maintain (or monitor) one or many threads (that each need a scheduler) and the object itself can be used for cross context communication, for example context A telling context B to spawn a fiber. Their usage is both external (public API) and internal (private API). The schedulers are meant to do the actual scheduling of fibers inside a single thread. Their usage is mostly internal (private API). The choice of modules is to be more versatile in the way we want to implement each execution context. We might want to implement both modules in a single type (e.g. single threaded context) or as distinct types.

text/0002-execution-contexts.md

Co-authored-by: Sijawusz Pur Rahnama <sija@sija.pl>

ysbaddaden · 2024-04-19T10:15:23Z

Thinking again about names:

ST: Mono? Single? Concurrent (to oppose on Parallel)?
MT: Parallel is still the best for MT 👍
I prefer Isolated over Exclusive after all (the fiber gets isolated), but no strong opinion;

I'm also thinking about simple constructors:

io_workers = ExecutionContext.concurrent
cpu_workers = ExecutionContext.parallel(size: 8)
ui = ExecutionContext.isolate { UI.main_loop }

Yet, I'm still struggling for a nice name to the single threaded context 😞

straight-shoota · 2024-04-19T12:24:40Z

SingleThreaded maybe?

I'm not sure we need such convenience constructors. This isn't essential anyway and we can figure it out later.

ysbaddaden · 2024-04-22T10:38:47Z

@straight-shoota yes, the convenience fonctions aren't needed, but while ExecutionContext::Parallel and ExecutionContext::Isolated feel nice, ExecutionContext::SingleThreaded doesn't have the same catchy feeling.

crysbot · 2024-05-03T20:15:21Z

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/how-to-build-for-release/6808/5

RFC 0002 - MT Execution Contexts

da62312

ysbaddaden self-assigned this Feb 5, 2024

straight-shoota reviewed Feb 5, 2024

View reviewed changes

0002-execution-contexts.md Outdated Show resolved Hide resolved

ysbaddaden added 2 commits February 6, 2024 10:06

Format notes, some rephrasing + example

c0496fe

fix: use crystal code for the reference guide

e17b582

RX14 reviewed Feb 7, 2024

View reviewed changes

straight-shoota changed the title ~~RFC 0002 - MT Execution Contexts~~ RFC 0002: MT Execution Contexts Feb 8, 2024

Apply suggestions from RX14

7213d8a

Remove refs to #yield_to

e75bb54

There's no such method

yxhuvud reviewed Feb 23, 2024

View reviewed changes

0002-execution-contexts.md Outdated Show resolved Hide resolved

Apply suggestions from code review

291828c

Co-authored-by: Linus Sellberg <sellberg@gmail.com>

ysbaddaden added 2 commits February 27, 2024 10:56

Fix: transition plan

7281cb6

straight-shoota reviewed Feb 27, 2024

View reviewed changes

0002-execution-contexts.md Outdated Show resolved Hide resolved

0002-execution-contexts.md Outdated Show resolved Hide resolved

ysbaddaden and others added 4 commits February 27, 2024 15:14

Apply suggestions from code review

53592f0

Co-authored-by: Johannes Müller <straightshoota@gmail.com>

fixup to previous typo commit

e8270e4

Deprecate Fiber#resume

2e4d505

Push dynamic number of threads to future evolution

775c2cd

ysbaddaden mentioned this pull request Feb 29, 2024

Fix: init schedulers before we spawn fibers crystal-lang/crystal#14339

Merged

straight-shoota mentioned this pull request Mar 15, 2024

RFC: Refactor Crystal::EventLoop to disconnect it from LibEvent crystal-lang/crystal#10766

Open

ysbaddaden mentioned this pull request Mar 22, 2024

Refactor and add comments to IOCP #run_once crystal-lang/crystal#14380

Merged

ysbaddaden added 2 commits April 15, 2024 15:06

Fix: move to text/ folder

2fcda75

Sija reviewed Apr 15, 2024

View reviewed changes

text/0002-execution-contexts.md Outdated Show resolved Hide resolved

text/0002-execution-contexts.md Outdated Show resolved Hide resolved

Apply suggestions from code review

9a451ec

Co-authored-by: Sijawusz Pur Rahnama <sija@sija.pl>

straight-shoota mentioned this pull request Apr 23, 2024

Async DNS resolution crystal-lang/crystal#13619

Open

straight-shoota mentioned this pull request May 2, 2024

Thread owns its current fiber (instead of Crystal::Scheduler) crystal-lang/crystal#14554

Merged

ysbaddaden mentioned this pull request May 6, 2024

Add EventLoop#run(blocking) and EventLoop#interrupt crystal-lang/crystal#14568

Draft


		## Default context configuration

		This proposal doesn’t solve the inherent problem of: how can applications configure the default context at runtime (e.g. number of MT schedulers) since we create the context before the application’s main can start.


		Such a group of fibers will never run in parallel. This can vastly simplify the synchronization logic since you don’t have to deal with parallelism anymore, only concurrency, which is much easier & faster to deal with. For example no need for costly atomic operations, you can simply access a value directly. Parallelism issues and their impact on the application performance is limited to the global communication.

		## Issues

RFC 0002: MT Execution Contexts #2

Are you sure you want to change the base?

RFC 0002: MT Execution Contexts #2

Conversation

ysbaddaden commented Feb 5, 2024

beta-ziliani commented Feb 5, 2024 • edited by straight-shoota

straight-shoota commented Feb 6, 2024

RX14 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ysbaddaden Feb 27, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ysbaddaden commented Feb 13, 2024 • edited

ysbaddaden commented Feb 13, 2024

RX14 commented Feb 14, 2024

ysbaddaden commented Feb 19, 2024

crysbot commented Feb 21, 2024

ysbaddaden commented Feb 23, 2024 • edited

straight-shoota commented Feb 23, 2024 • edited

yxhuvud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yxhuvud Feb 23, 2024 • edited

Choose a reason for hiding this comment

yxhuvud Feb 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Blacksmoke16 commented Feb 23, 2024

ysbaddaden commented Feb 26, 2024

ysbaddaden commented Mar 4, 2024 • edited

crysbot commented Mar 14, 2024

ysbaddaden commented Mar 18, 2024 • edited

crysbot commented Apr 11, 2024

ysbaddaden commented Apr 19, 2024 • edited

straight-shoota commented Apr 19, 2024

ysbaddaden commented Apr 22, 2024

crysbot commented May 3, 2024

beta-ziliani commented Feb 5, 2024 •

edited by straight-shoota

ysbaddaden Feb 27, 2024 •

edited

ysbaddaden commented Feb 13, 2024 •

edited

ysbaddaden commented Feb 23, 2024 •

edited

straight-shoota commented Feb 23, 2024 •

edited

yxhuvud Feb 23, 2024 •

edited

yxhuvud Feb 23, 2024 •

edited

ysbaddaden commented Mar 4, 2024 •

edited

ysbaddaden commented Mar 18, 2024 •

edited

ysbaddaden commented Apr 19, 2024 •

edited