Newbie's questions for migrating from oneTBB

[solbjorn]

Can I ask a few silly questions since I'm new to coroutines, but eager to switch my oneTBB-based engine to TMC?

1. Do I get it correctly that I can't place an aw_spawn_fork in a global struct and co_await it later from some other coroutine, not the one that spawned it?
   (my assumption comes from that aw_spawn_fork doesn't have a default constructor, so putting it in a struct is tricky. Moreover, you mentioned in the docs that they contain pointers to this, which I got at "you must always co_await the results of fork() within the coroutine that spawned them").

What would be the best way for the following scenario:

• from one function, I run two tasks and return
• in another function, which happens several code blocks later, I need to wait for them

Currently, I run two task_groups from the former and them wait for them from the latter. task_groups lay in a shared struct, I don't pass pointers for them around the code.
I'd like to not use post_awaitable() and them block on the futures later, since that's not what coroutines are about.

I thought of something like:

• in the first function, spawn 2 coroutines/tasks and detach() them immediately
• put either an async version of condvar or barrier in the shared struct and toggle them the end of those coroutines
• later, when I want to wait for them, co_await the condvar/barrier to suspend instead of blocking in case they are not ready yet.

But will this guarantee that the coroutines which I'm waiting for will execute for sure even if someone has 1 core and everything executes serially there?
And this doesn't look like an intended/obvious way...

2. Since aw_mutex and ex_braid do nearly the same stuff, which one is faster from your code PoV? Mutex seems to be a bit heavier since it builds an awaiters list and need to repost every awaiter on each unlock? But if the contention is really narrow and it's unlikely for this mutex to be blocked, ex_braid can incur more overhead?

3. aw_mutex has unlock() and co_unlock(). The latter can do sync transfer. Can sync transfer lead to that let's say 8 threads suspended on the same lock, but, if using co_unlock(), these 8 coroutines will continue execution on only one thread serially even after this mutex' scope?

4. If I have the following tree of function calls:

tmc::task f1() // coroutine
  void f2() // regular function called by the coroutine above
    void f3() // called by f2()

Does it mean that if I want to run a coroutine from f3(), I need to call post*() from it, even though it's still run on ex_cpu already, but doesn't have a suspension point? Or, the preferred alternative would be to convert each of those functions to a coroutine?

My main idea is to not block any of the coroutines I want to introduce with serial stuff like generic mutexes/futures/etc, so that only one thread (which runs main synchronous code) could be blocked at a time.

5. Unrelated: BTW, you can use std::hardware_destructive_interference_size instead of hardcoding to 64, it's a constexpr IIRC.
   But I've also seen that some developers started multiplying it by two since modern CPUs (at least x86_64) often tend to fetch 2 CLs at a time instead of one which could still provoke false-sharing.
   Anyway, only benchmarking could give a reliable answer here.

[tzcnt]

Can I ask a few silly questions since I'm new to coroutines, but eager to switch my oneTBB-based engine to TMC?

1. Do I get it correctly that I can't place an aw_spawn_fork in a global struct and co_await it later from some other coroutine, not the one that spawned it?
   (my assumption comes from that aw_spawn_fork doesn't have a default constructor, so putting it in a struct is tricky. Moreover, you mentioned in the docs that they contain pointers to this, which I got at "you must always co_await the results of fork() within the coroutine that spawned them").

What would be the best way for the following scenario:

• from one function, I run two tasks and return
• in another function, which happens several code blocks later, I need to wait for them

Currently, I run two task_groups from the former and them wait for them from the latter. task_groups lay in a shared struct, I don't pass pointers for them around the code. I'd like to not use post_awaitable() and them block on the futures later, since that's not what coroutines are about.

I thought of something like:

• in the first function, spawn 2 coroutines/tasks and detach() them immediately
• put either an async version of condvar or barrier in the shared struct and toggle them the end of those coroutines
• later, when I want to wait for them, co_await the condvar/barrier to suspend instead of blocking in case they are not ready yet.

But will this guarantee that the coroutines which I'm waiting for will execute for sure even if someone has 1 core and everything executes serially there? And this doesn't look like an intended/obvious way...

aw_spawn_fork contains 2 things which are non-movable:

• a control block which allows synchronization of the completion between the awaiter (parent), the forked task (child), and any other forked tasks that might be awaited as part of its group (siblings)
• storage for all results of the group (if non-void)

Since the child tasks begin executing immediately and capture a pointer to this control block (this happens during the call to fork()), its location must be pinned before they begin execution.

There are several possible workarounds using detached tasks:

• use a barrier or latch initialized to N+1 that is shared between the child tasks and eventual awaiter
• use a channel and push the results into it from each task - this is the most flexible approach since you can kick off new tasks at any time. However, like the barrier you do need to know exactly how many results to expect, at least until channel::try_pull() #69 is implemented.

The need for these workarounds is a weakness of the current API that will be rectified in the future by #62 , which will enable std::future-like behaviors for tasks. Another option would be #75.

As long as you don't use any blocking waits (tmc::post_waitable, std::future, std::mutex), all of the work will be executed eventually, even on a single core machine.

[tzcnt]

2. Since aw_mutex and ex_braid do nearly the same stuff, which one is faster from your code PoV? Mutex seems to be a bit heavier since it builds an awaiters list and need to repost every awaiter on each unlock? But if the contention is really narrow and it's unlikely for this mutex to be blocked, ex_braid can incur more overhead?

In my benchmarks the performance is similar. ex_braid should have higher throughput/bandwidth under high contention, but mutex is more lightweight and should have lower latency under low contention. I'd say the choice between them lies in how long-lived the object is, since ex_braid incurs several allocations on creation, but mutex is allocation-free. In the next release, ex_braid is being rewritten on top of channel which will reduce this difference.

One major difference is that ex_braid is tied to a particular parent executor, and only threads from that executor will participate in executing tasks on the braid. However, a mutex can be shared by tasks that run on different executors, and those tasks will each execute on their own original executor.

An additional reason to use ex_braid is the ability to spawn().run_on() a child task on the braid and have just that task run on the braid. Whereas with mutex you would need to have an explicit reference to the mutex in the child task function parameter list, and then do the lock/unlock inside the function body.

[tzcnt]

3. aw_mutex has unlock() and co_unlock(). The latter can do sync transfer. Can sync transfer lead to that let's say 8 threads suspended on the same lock, but, if using co_unlock(), these 8 coroutines will continue execution on only one thread serially even after this mutex' scope?

Threads don't suspend on locks in TMC. Only tasks. If you have an ex_cpu with 8 threads, they will all steal work from each other. I think you are asking about the scenario below. Thread 1 takes care of the problem you are asking about. As long as work is available in the ex_cpu queue, all threads will participate in work-stealing.

• thread 0 running task 0 acquires the mutex
• task 0 co_unlock()s the mutex
• thread 0 finds task 1 in the mutex queue
• thread 0 posts task 0 back to its own ex_cpu queue
  • this awakens thread 1 from the ex_cpu thread pool
• thread 0 transfers the mutex lock to task 1 and begins executing task 1
• thread 1 steals task 0 from thread 0's ex_cpu queue and resumes it

[tzcnt]

4. If I have the following tree of function calls:

tmc::task f1() // coroutine
  void f2() // regular function called by the coroutine above
    void f3() // called by f2()

Does it mean that if I want to run a coroutine from f3(), I need to call post*() from it, even though it's still run on ex_cpu already, but doesn't have a suspension point? Or, the preferred alternative would be to convert each of those functions to a coroutine?

My main idea is to not block any of the coroutines I want to introduce with serial stuff like generic mutexes/futures/etc, so that only one thread (which runs main synchronous code) could be blocked at a time.

This is one of the core issues with stackless coroutines - that they introduce the "function coloring problem". If you want to call tmc::task<void> f4() from f3() and await for a result, then f3() and by extension f2() all need to be coroutines.

You could call post_waitable() and then block the thread waiting for a result, which is known as the "sync over async" antipattern. This is a bad idea to do on the primary thread pool as it can cause resource exhaustion / deadlocks quite easily. I see that you are planning to do this only on a dedicated single thread / main thread, and if you need to interact with legacy external components that offer std::mutex/std::future-based interfaces then that may be a good way to handle it.

[tzcnt]

5. Unrelated: BTW, you can use std::hardware_destructive_interference_size instead of hardcoding to 64, it's a constexpr IIRC.
   But I've also seen that some developers started multiplying it by two since modern CPUs (at least x86_64) often tend to fetch 2 CLs at a time instead of one which could still provoke false-sharing.
   Anyway, only benchmarking could give a reliable answer here.

Yes... like you said, on some systems std::hardware_destructive_interference_size really should be 128, but the compiler still returns 64 everywhere... so I didn't bother to introduce a changeable constant. But it's the right thing to do to get rid of this magic number and make it easier to experiment with different values in the future.

[solbjorn]

OMG, thanks a lot for such detailed explanations! I'll keep the issue opened for a while in case I have more quesitons, ok?
From your explanations, it seems like I got everything right from your documentation, which is another + for you :>

[solbjorn]

6. I was designing some drafts yesterday (in my head) how I could convert the main rendering function to coroutines.
   It would go mostly smoothly as it has only a couple mutexes (to be replaced with an async mutex and/or braid), however, DirectX 11.2 has several stupid APIs which can be called only the following way (one particular example):

    while ((hr = GetData(used[ID].Q.Get(), &fragments, sizeof(fragments))) == S_FALSE) { }

It's not THAT bad in the engine I work on (this piece was not written by me)

    while ((hr = GetData(used[ID].Q.Get(), &fragments, sizeof(fragments))) == S_FALSE)
    {
        if (!SwitchToThread())
            Sleep(ps_r2_wait_sleep);

        if (T.GetElapsed_ms() > ps_r2_wait_timeout)
        {
            fragments = (occq_result)-1; // 0xffffffff;
            break;
        }
    }

but still is a synchronous while-yield-blah waste of resources.

I haven't found ANY DirectX API that would allow to do that asynchronously (post a request and get notified when the data is ready). Ideally we could suspend the rendering coroutine here and resume it once notified.

What would be the best approach for this? I don't want to do sync-over-async crap.

Something like, move such pieces to the main/legacy thread, suspend the rendering coro once it hits the point where we need to get the result, then wait for this loop in the main thread and post a coro resuming the rendering one?

7. Are the compilers able to inline coroutines when they're small? Like

inline tmc::task<int> fn() { co_return 2; }

tmc::task<void> fn2()
{
    ...
    int x = co_await tmc::spawn(fn);
    ...
}

Wouldn't this generate more code / yield worse optimization than a regular sync call to an inline which returns 2?

(I use Clang 21 with -O3 -flto -fwhole-program-vtables)

8. In case when I want to call an async coroutine from legacy code and immediately wait for the result, there are at least 2 options:

• use the future that tmc::post_awaitable() returns;
• use atomic wait-notify like you do in tmc::async_main().

I suspect it would be more efficient to use the second approach?

[tzcnt]

6. I was designing some drafts yesterday (in my head) how I could convert the main rendering function to coroutines.
   It would go mostly smoothly as it has only a couple mutexes (to be replaced with an async mutex and/or braid), however, DirectX 11.2 has several stupid APIs which can be called only the following way (one particular example):

    while ((hr = GetData(used[ID].Q.Get(), &fragments, sizeof(fragments))) == S_FALSE) { }

It's not THAT bad in the engine I work on (this piece was not written by me)

    while ((hr = GetData(used[ID].Q.Get(), &fragments, sizeof(fragments))) == S_FALSE)
    {
        if (!SwitchToThread())
            Sleep(ps_r2_wait_sleep);

        if (T.GetElapsed_ms() > ps_r2_wait_timeout)
        {
            fragments = (occq_result)-1; // 0xffffffff;
            break;
        }
    }

but still is a synchronous while-yield-blah waste of resources.

I haven't found ANY DirectX API that would allow to do that asynchronously (post a request and get notified when the data is ready). Ideally we could suspend the rendering coroutine here and resume it once notified.

What would be the best approach for this? I don't want to do sync-over-async crap.

Something like, move such pieces to the main/legacy thread, suspend the rendering coro once it hits the point where we need to get the result, then wait for this loop in the main thread and post a coro resuming the rendering one?

If the API requires you to poll periodically, you will need to use a timer. Some coroutine libraries offer async timer facilities, but TMC does not offer them directly - rather you can use them via the Asio integration in tmc-asio. There are a couple examples of using the asio timer facilities here:

https://github.com/tzcnt/tmc-examples/blob/main/examples/asio/timer_mem_bench.cpp
https://github.com/tzcnt/tmc-examples/blob/main/examples/asio/delay.cpp

However it's worth noting that there is no truly async timer. Under the hood, a thread is blocking on the OS timer syscall, and then posting the results back to the executor queue when ready. Your approach of doing this manually using the main thread is equivalent. I like the SwitchToThread() call - this may actually help performance in some cases as it's more lightweight than actually blocking on the timer.

[tzcnt]

7. Are the compilers able to inline coroutines when they're small? Like

inline tmc::task<int> fn() { co_return 2; }

tmc::task<void> fn2()
{
    ...
    int x = co_await tmc::spawn(fn);
    ...
}

Wouldn't this generate more code / yield worse optimization than a regular sync call to an inline which returns 2?

(I use Clang 21 with -O3 -flto -fwhole-program-vtables)

Compilers are supposed to be able to inline coroutines, but often fail to do so at this time. The Clang 20 attributes (#61) are supposed to help with this. This particular item is near the top of my priority list... so it will be coming soon(tm).

Notably you don't need to use tmc::spawn unless you want to customize the behavior of the coroutine. In the snippet you provided, you don't need it, and you're more likely to get a successful inlining if you just do a direct call:

inline tmc::task<int> fn() { co_return 2; }

tmc::task<void> fn2()
{
    ...
    int x = co_await fn();
    ...
}

[tzcnt]

8. In case when I want to call an async coroutine from legacy code and immediately wait for the result, there are at least 2 options:

• use the future that tmc::post_awaitable() returns;
• use atomic wait-notify like you do in tmc::async_main().

I suspect it would be more efficient to use the second approach?

I'm not sure about this. It depends on the implementation of std::future::wait vs. std::atomic::wait in your stdlib. Maybe using std::atomic::wait would be slightly faster as it doesn't introduce additional allocations, which std::promise does. However, I'd expect both of them to call the same syscall under the hood (WaitForSingleObject?? I'm mostly a Linux developer.) and for that syscall to dominate the runtime. You can try rolling your own synchronization primitive if you want - post_waitable() was provided only as an easy entry point for this kind of thing. Please let me know if you find a significant performance win from doing this.

[solbjorn]

I'm not sure about this. It depends on the implementation of std::future::wait vs. std::atomic::wait in your stdlib. Maybe using std::atomic::wait would be slightly faster as it doesn't introduce additional allocations, which std::promise does. However, I'd expect both of them to call the same syscall under the hood (WaitForSingleObject?? I'm mostly a Linux developer.) and for that syscall to dominate the runtime. You can try rolling your own synchronization primitive if you want - post_waitable() was provided only as an easy entry point for this kind of thing. Please let me know if you find a significant performance win from doing this.

In LLVM libc++, which I use, future::wait uses a condition_variable + mutex under the hood, std::atomic uses C11 atomics. I didn't dig deeper to the OS calls. Worth trying and comparing I guess.

I'm a Linux dev, too, but this game is Windows-exclusive unfortunately :D (and porting its render from Dx 11.2 to Vulkan would be hell to me)

[solbjorn]

9. Some classes like mutex provide await() and co_await(), with the difference that the latter allows synchronous transfer.
   Why is it optimal, are there any downsides or cases where synchronous transfer might be unwanted?

[tzcnt]

co_unlock will maximize the throughput of the mutex, but the current (unlocking) task is suspended and posted back to the executor queue for continuation.

unlock or the scope destructor will continue executing the current (unlocking) task afterward. The task that gets the lock next has to be posted to the executor queue first. Until it is taken off the queue and resumed, the mutex is "locked" but nothing running with it.

The advantage of the synchronous unlock are if the unlocking task is latency sensitive and you don't want it to suspend, or if you want to make use of the RAII lock scope object.

[tzcnt]

I'm going to convert this to a discussion; feel free to continue with any further questions there.

[solbjorn]

10. Is there any way to easily "perfect-forward" a coroutine?

Let's say we have the following:

tmc::task<int> do_some_work(int a)
{
    ...
    co_return ret;
}

...

// caller
int a = co_await do_some_work(b);

"Delegate"-like things (e.g. C++26's std::function_ref) wrap the target into an invoker:

https://github.com/solbjorn/callme/blob/master/Impl/CallMe.h#L413

(all the implementations I'm aware of: LLVM, Folly, Abseil etc., do the same)

I'm actually planning to add support for coroutines to my fork of CallMe and I suspect (haven't tried TBH, maybe it really is just that simple) that such invokers would not work as-is:

tmc::task<int> do_some_work(int a)
{
    ...
    co_return ret;
}

...

// caller
CallMe::Delegate<tmc::task(int)> del = CallMe::fromFunction<&do_some_work>();

int a = co_await del(b);

as invokers are just regular functions.

If invokers for coroutines must differ from regular ones, would something like this be enough?

// R is tmc::task<blah> -- the same type which the target returns
R co_invoke(Args&& args)
{
    co_return co_await target(std::forward<Args>(args)...);
}

Some side info:

1. libfork's for_each supports both regular functions and coroutines as the projection:

https://github.com/ConorWilliams/libfork/blob/main/include/libfork/algorithm/for_each.hpp#L60

(in his tests he passes both std::identity and a coroutine which does the same, but with co_return)

but it's wrapped with lf::just so I dunno what magic he does there.

2. Why I need this at all:

Lots of stuff in the engine is implemented as a vector/deque of such "delegates". When the main loop reaches the point where they should be executed, it iterates over the container and calls them.

Since the outer functions which handle these containers will be most likely converted to coroutines, it would be better to run each delegate as a coroutine as well to keep the function coloring etc etc.

(but maybe you have a better idea how to organize this, since this code processing queues of delegates originates from 2005 and is not mine obviously; I only switched it from some ancient implementation to C++20's CallMe)

[solbjorn]

Breh, I completely forgot about the utility awaitables and was wrapping oneliners in lambdas...

When I need to execute something on the standalone tmc::ex_cpu_st, I previously did it that way:

    co_await tmc::spawn([](auto& last) -> tmc::task<void> {
        PIX_EVENT(DEFER_FLUSH_OCCLUSION);

        for (auto light : last)
        {
            if (light == nullptr)
                continue;

            for (auto& svis : light->svis)
                svis.flushoccq();
        }

        last.clear();
        co_return;
    }(Lights_LastFrame)).run_on(xr::tmc_cpu_st_executor());

Now I'm thinking of just

    {
        auto scope = co_await tmc::enter(xr::tmc_cpu_st_executor());
        PIX_EVENT(DEFER_FLUSH_OCCLUSION);

        for (auto light : Lights_LastFrame)
        {
            if (light == nullptr)
                continue;

            for (auto& svis : light->svis)
                svis.flushoccq();
        }

        Lights_LastFrame.clear();
        co_await scope.exit();
    }

The second should be more optimal I guess?

IIRC .run_on() prevents Clang from doing HALO -- what if we just switch the executor in-place (the second snippet)? Does .run_on() make sense to use when I immediately co_await it or it's meant to be used with .fork()/.detach() only?

---

P. S. In case you haven't seen my comment in #175:

I'm planning to switch to a more efficient Chase-Lev queue in the future.

Just out of curiosity: how soon? Will it be wsq from TaskFlow or something new from scratch?

:3

It's funny nonetheless that TMC is 3x faster than TF even without Chase-Lev queues :D

[solbjorn]

I've used xoshiro in the past and had good results. For this I don't care about "cryptographically secure", so I'll just pick something that is really fast (for benchmarking) and has "good enough" entropy. Only if the benchmarks demonstrate that integrating randomness is worth it, then I would dig a little deeper into the quality of the generator output.

No PRNG (Pseudo RNG) is "cryptografically secure", so it's only a matter of performance and distribution. The second can have huge impact on results.
For example, in my mod, I have 8 weather groups ("clear", "cloudy" etc), each of them consists of 6-7 cycles (variations of the same weather), and a matrix of possible transitions. When several transitions are available, it can happen (or not happen) based on random. Before I switched both the C++ code from std::mt19937 and Lua from the built-in PRNG to Abseil's random (PCG-based), I had less variety in weather, which means I had poorer random distribution before the switch.
xoshiro is way better in both performance and distribution than std::* as well. plf_rand uses PCG for newer C++ standards and falls back to xoshiro for the older ones (and all that fits into 105 locs... Dunno why the "official" PCG for C++ is a couple thousand locs).

---

I've noticed you played a bit with snmalloc -- any interesting results comparing to tcmalloc? From their docs and articles, I got an impression that it performs better on clusters and 60+ thread Xeons rather than on consumer CPUs... But I could've misinterpreted it.

I'm limited to mimalloc as of now (but there's nothing bad in it, especially after the 3.2.7 release), as only this lib can work as a proxy for the entire process space -- sorta poor man's LD_PRELOAD (which I really miss on Windows). The rest usually only provide global new/delete replacement for the functions statically linked into the exe.

[tzcnt]

Snmalloc was close to tcmalloc but still slightly slower on my 64 core machine. The design seems theoretically good so it's one to keep an eye on.

It seems that tcmalloc at least has the patch functionality to override the global allocator even for other DLLs, although I didn't find clear documentation on how to do so. I suppose if you are able to achieve your goal of a fully statically linked exe then you could experiment with any allocator you like.

[solbjorn]

It seems that tcmalloc at least has the patch functionality to override the global allocator even for other DLLs, although I didn't find clear documentation on how to do so. I suppose if you are able to achieve your goal of a fully statically linked exe then you could experiment with any allocator you like.

tcmalloc from gperftools yes, but I believe you use the new tcmalloc which Google develops for Linux exclusively?

I currently have fully statically linked exe (now that I got rid of oneTBB), but with injection/redirection you also have 3rd party allocator taking care of any allocation that happens inside the process space. Let's say you can't link in Windows DLLs or DirectX DLLs or any proprietary DLLs which don't have source code statically. When you use mimalloc-redirect or any other similar hacks, you also have allocations from these DLLs managed by your allocator. If you link the allocator statically, you only have allocations that come from your binary managed by the allocator.

I'm fine with mimalloc anyway, Tracy didn't find any allocator-related bottlenecks. I've also found some completely new allocator which also injects entry points similarly to mimalloc-redirect, but it's on early WIP stage currently. I might play with it later when it's more mature.

[tzcnt]

I use the Debian and Arch packages which are based on the old gperfools version. Since installing them is dead simple I felt it was fair to recommend their usage with the library.

[solbjorn]

Hmmm, you should definitely give this a try:

https://github.com/google/tcmalloc

It is Linux-exclusive, so I can't test it with the engine, but IIUC this is a "new generation" of TCMalloc.

Also since you have slightly better results with the "classic" version than with mimalloc, maybe I'll also try the one from gperftools soon... It supports Windows and have DLL patching as you mentioned, although building it for Windows is a separate painful story.

[solbjorn]

I've found at least one such pattern in the code:

m_playing_sounds.erase(std::remove_if(m_playing_sounds.begin(), m_playing_sounds.end(), CInappropriateSoundPredicate(sound_mask)), m_playing_sounds.end());

where I need to change the operator() (of the predicate) to a coroutine in order to keep the function coloring (it calls a function which I converted to a coroutine recently).

For now it looks like I need to convert this to a manual loop in order to do that, there is no other way?

[solbjorn]

The reason why I'm converting a lot of functions to coroutines now is that xrSound loads next blocks of oggs asynchronously: when its state machine decides that we'll need the next block soon, it tosses a task to the executor. In order to correctly stop or destroy the sound, I need to make sure its task is done. That's how stop/destroy became coroutines (they co_await the task), but now I see that in many places I'm not able to use them directly (mostly due to Lua bindings, which can't be coroutines), so I created a queue of sounds to be stopped/destroyed to handle this a bit later from the coroutine context.

BTW for such async tasks (which don't fit the fork-join model) I use spawn().detach() + manual_reset_event to wait for the results later. Dunno if this is the most optimal way, let me know if there's anything better. The reset event struct is pretty tiny, I just reset it before detaching the task and set it at the end of the task.

(example: https://github.com/solbjorn/reaper-engine/blob/tmc/ogsr_engine/Layers/xrRender/DetailManager.cpp#L380)

[tzcnt]

If you only need to await the task in one place, then I'd say use a fork_group for that. It can be used in the same way as a member variable, and it also has a reset() function so you would call fg.reset(); fg.fork(calc_async()); The advantage of this is that you don't need to manually set the event at the end of calc_async(). The synchronization is controlled entirely in the wrapper. It also allows the task to symmetric transfer to an awaiter while destroying itself at the same time (doesn't have the issue we discussed here #137 (reply in thread) or the opposite issue with calling event.set() which must always post the awaiter)

(I should clarify the doc comment on fork_group to say it's fine to call reset() before the first use as well - all it does is set the internal counters to 0).

However if you might need multiple other tasks to wait for one event, then you should continue to use manual_reset_event, as fork_group can only be awaited by a single awaiter before it needs to be reset.

[solbjorn]

As for the example above (details calculation), then there's only one awaiter (details render task, by "details" the original devs meant grass).
You're saying that fork_group can be a member variable, but we await it like co_await std::move(fg). Doesn't it mean that the member is not usable after that?

Sounds can be awaited from multiple places, so yes, there can be multiple co_awaits.

[tzcnt]

Calling fork_group::reset() restores it to a usable state so you can dispatch and await another group of tasks. I'm trying to figure out how to better document this. I thought I might use the standard library as an example - e.g. a std::vector that's been moved-from can be clear()ed after which it becomes a usable empty vector. But I can't find comprehensive documentation on this for std types - just scattered references.

Actually I think the documentation does clearly state this: https://fleetcode.com/oss/tmc/docs/v1.3/awaitables/fork_group.html#_CPPv4N3tmc13aw_fork_group5resetEv . Should I also note that it's reusable in this manner at the top level of the docs? Since this resettable behavior is special to spawn_group and fork_group, I can see how it could get lost if it's buried on the function doc only.

[solbjorn]

I think it's just me not reading the docs deeply =\

So if I got it correctly,

• from start_async(), I call fg.reset() and then fg.fork_clang();
• from the rendering function I just do co_await std::move(fg)

It should behave similarly to detach + reset event, but more optimized.

I've just recalled that the details rendering function is called from several places, so it might not be a good candidate. But I have a couple others that for sure are (async rain calculation, async HOM calculation — each is awaited exactly once; maybe something else will come to my mind later).

Thanks!

[solbjorn]

Re "why I convert so many functions to coroutines" (note to myself mostly).

For sure, I could just leave everything as it is, just use tmc::post() + either std::future or atomic wait/notify to simply replace oneTBB, but what's the point then? I wanted a challenge, wanted to know whether it's possible / I'm able to make such huge engine coroutine-based and lock/sleep-free (without standard mutexes etc.). I also want to keep tmc::cpu_executor() fully non-blocking; if I can't make something non-blocking/async, I delegate it to xr::cpu_st_executor() (tmc::ex_cpu_st) which is the only thread I allow to block/sleep -- thankfully it's only a couple DirectX functions that require blocking/sleeping.
Mutexes and async tasks are scattered all over the engine, sometimes the call chain from the main loop to a task/mutex is ~30 functions and I need to keep the coloring. Sometimes I see that some function I converted to a coroutine is called from a constructor or a destructor. Then I make them private and create a friend coroutine co_create()/co_destroy() (it was your advice as well ;)).
Sometimes it's simply not possible to call a coroutine if the related function has a Lua binding (which can't be a coroutine) or is a callback called from a 3rd party library. For these, I have a queue of async tasks, I simply push a new task there and process this queue from the coroutine context at least once a frame (before the frame rendering, so that the delay between the call that pushed the task and the execution of this task is usually < 1 ms -- if this is not enough (testing will show), I'll process it more than once a frame. But for now, all such tasks are not time-critical -- let's say I queued a sound to stop, even if it's stopped 1 ms after the call, nobody will ever notice it).

Lots of folks kept telling me that it's impossible to replace Luabind with Sol in the engine. It took me half a year and several thousand locs, but I made it. So I'm pretty sure this challenge is doable as well, especially given that I receive so huge and helpful support from you.

(the same folks told me it's not possible to switch the engine from MSVC to Clang/clang-cl due to the legacy code that is too broken to be fixed -- lol that was EZ honestly, even though I wasn't good in C++ back then)

BTW I'm curious how Tracy will work after the engine is coroutine-based -- I haven't read its code deeply, but my impression was that Tracy expects every function to start and end on the same thread and doesn't expect that a function can suspend.

[tzcnt]

Yes, I think it is an admirable goal to make the entire thing non-blocking and should be achievable. I'm happy to help and appreciate your useful feedback on the library.

I just meant that only the outermost (awaiting) and innermost (first level that needs to be parallel or async) actually need to be coroutines. You can make everything in between a regular function if that simplifies development. The innermost regular function can construct or fork the coroutines without needing to await them, and then provide the awaitable back to the outermost coroutine which awaits it later. This can be done in a few ways:

• using spawn_group / fork_group passed by reference through the call stack, or stored in a member variable
• just pushing tasks to a vector/queue and then spawn_many or await them one-by-one afterward

Re: the sound example - after you get this working with the "queued to main thread" approach, a next step might be to decouple sound timing from the main thread entirely. If sound controls still need to be serialized, this might be a good place to use ex_braid - you can just post the work to the braid and let it run in the background. If you need both serialization and to eventually wait for some synchronization point in the main loop, then you could use a fork_group but also run it on the braid. Or you could use mutex if the part that needs to be synchronized is only a small portion of the code.

[tzcnt]

Re: Tracy I found this which sounds like a problem wolfpld/tracy#936 . Not 100% sure I understand but the suggestion appears to be to track the current coroutine in the worker thread, before every resume call. Not sure how I can support that.

I found this which is very cool https://github.com/tokio-rs/console but it implies that they built the entire thing from the ground up for Tokio. So we probably need a new tracing framework that is coroutine-aware for C++.

Slightly related: debugging coroutine stacks is not well supported at the moment. I found this script https://clang.llvm.org/docs/DebuggingCoroutines.html#async-stack-traces but since the addition of llvm/llvm-project#166664 / llvm/llvm-project#161870 it seems that we should be able to provide a script to generate a nicer looking frame, which could perhaps be visible in a debugger GUI (I use LLDB DAP VSCode extension). Not sure. This is a bit of a rabbit hole but I'll probably take a stab at it at some point. It would be another "killer feature" for this project to be able to provide a fully working debugger integration.

[solbjorn]

Re: Tracy I found this which sounds like a problem wolfpld/tracy#936 . Not 100% sure I understand but the suggestion appears to be to track the current coroutine in the worker thread, before every resume call. Not sure how I can support that.

Uff pity. I've read about their macros related to fibers, but I'd say they're poorly applicable to coroutines.
But yeah, I believe it should be neither your nor my problem. Your library shouldn't integrate with Tracy by default (it's a per-project decision) and from my side, I can't do much with the worker threads etc.
It's been almost 6 years since the introduction of the coroutines not as an experimental feature but a part of the standard, but most people still consider them experimental / not applicable to the real world usecases / etc >_<

Tracy zones are scope-based, it's a structure on the frame which has a constructor and a destructor which take care of the tracing info. But coroutines don't break the object scopes, otherwise it would be a hell,

    if (a)
    {
        scope_based_obj b;
        co_await do_smth_async();
        do_smth_sync(b);
    }
    // b gets destroyed only once and only when exiting the scope, the co_await doesn't affect its lifetime anyhow

so I don't really understand how Tracy gets multiple exits from the same zone when using coroutines.

I found this which is very cool https://github.com/tokio-rs/console but it implies that they built the entire thing from the ground up for Tokio. So we probably need a new tracing framework that is coroutine-aware for C++.

Correct, or at least that the devs of the already existing and popular frameworks like Tracy would start treating coroutines seriously and supporting it.

Slightly related: debugging coroutine stacks is not well supported at the moment. I found this script https://clang.llvm.org/docs/DebuggingCoroutines.html#async-stack-traces but since the addition of llvm/llvm-project#166664 / llvm/llvm-project#161870 it seems that we should be able to provide a script to generate a nicer looking frame, which could perhaps be visible in a debugger GUI (I use LLDB DAP VSCode extension). Not sure. This is a bit of a rabbit hole but I'll probably take a stab at it at some point. It would be another "killer feature" for this project to be able to provide a fully working debugger integration.

I had a couple crashes inside coroutines and got the following: the stack trace that gets printed to the log (generated by dbghelp.dll) showed me something like: TMC executor internals -> the call chain since the last suspension point. Same with minidumps generated by the same DLL. So the debug info is indeed poorer, but on the other hand, when I ran the exe under the VS debugger, all breakpoints and step-by-step execution worked as expected, so when a bug can be reproduced stably, it's not a problem to me.
^ all that was on a release binary built with -O3 -flto -fomit-frame-pointers and other crazy optimization flags, so it might be a bit better on debug builds.

(might be related / help you: https://developers.facebook.com/blog/post/2021/10/14/async-stack-traces-c-plus-plus-coroutines-folly-walking-async-stack/)

[tzcnt]

FYI since I've been recommending the use of fork_group: #186

[solbjorn]

I provided some info to the Tracy authors: wolfpld/tracy#936 (comment)
Will see whether they pay attention and treat that seriously.

[solbjorn]

Just curious:

  tmc::post(
    tmc::cpu_executor(),
    tmc::detail::client_main_awaiter(
      static_cast<tmc::task<int>&&>(ClientMainTask), &exitCode
    ),
    0, 0
  );

Why is the root coroutine run with thread hint == 0 in async_main()?

[tzcnt]

Since concurrentqueue creates an implicit producer queue for each unknown thread that posts, posting from the main thread creates one of these queues. If the app runs entirely on executors after that, that queue would remain empty but still need to be checked every iteration.

By using the thread hint, it goes to the group 0 inbox which is an already-existing queue instead. Since the new waking algorithm prefers to wake thread 0 when there are no working threads anyway, this isn't a functional change. And there is a check that the hinted thread is actually allowed to execute the chosen priority; if it can't, the work item goes into the normal submission path (via the implicit producer queue)

[tzcnt]

The other quirk of the async_main implementation is that it uses std::atomic::wait instead of a future. It would be cleaner if I just included tmc/sync.hpp and called post_waitable. But I wanted to avoid introducing a dependency on <future> for users that never want to actually use it. So I tried to keep that constrained to the sync.hpp file.

Edit: the above doesn't make sense any more since async_main is defined in ex_cpu.ipp which already includes sync.hpp. So I think I could make this change now. Posting from my phone, sorry

Someone also suggested having the main thread become a worker thread. I could do this too and it would eliminate that issue, but comes with its own complexity.

[solbjorn]

Ah I see. I did the same (thread hint) in the engine, since I needed to open-code async_main() to be able to pass more than one code via this atomic (the splash window is created by the "main" thread which runs WinMain(), so the WinAPI allows to destroy the splash window only from this thread).

(https://github.com/solbjorn/reaper-engine/blob/tmc/ogsr_engine/xr_3da/x_ray.cpp#L401)

I wouldn't change atomic wait/notify to a future in async_main(). Apart from the future overhead, I don't know exactly how future waiting is implemented in different C++ STLs / for different OSes (IIRC some STLs implement future waiting using a condition_variable + a mutex). OTOH I know for sure that atomic wait/notify is efficient now on both Linux and Windows. Moreover, as you said, it's all hidden in the .ipp and the user doesn't see or need to deal with this atomic.

As for making the main thread a worker thread, it could be me as well. But I remember you explained this would require inverting some logic upside down and wouldn't give any reasonable benefits. Apart from the logic details, I think you have less control from the code over the "main" thread than over the threads you create manually? This could also complicate things.

[tzcnt]

Since it looks like you've finished the initial pass of the migration, I'd love to try playing the game. I was able to get your branch to build in Release, but no luck in Debug. However I'm not able to run it - the splash screen only pops up for a second and then disappears. When running in the debugger I see it's unable to find fsgame.ltx. However it doesn't emit any logs so I'm not sure what the next step is. I tried moving the files around and passing -fsgame parameter but no luck.

I'm using the Steam 1.006 (?) version and was able to run the upstream OSGR engine just fine using the -steam parameter. Would you be willing to take some time to help me debug this? I joined the OGSR Engine and Open XRay discords so you can contact me there. If not I would appreciate some tips on getting it to find the fsgame / generate meaningful logs.

Also is there a specific mod / graphical overhaul that I should be using?

[solbjorn]

Hi, it was the initial pass and I'm glad it's done, but there are still lots of mutexes etc. which I need to convert, otherwise there'll most likely be deadlocks (the current tmc branch of my repo has a weird crash when loading the level).

I tried to build the Debug config in the past, but it was broken long before I forked the project and I didn't try to fix it properly (a Release build + game logs and sometimes the VS debugger have always been more than enough for me).

My fork is not compatible with the original assets in a couple places: all.spawn file, XMLs with translations, etc, but the most problematic is Luabind -> Sol transition (lots of Lua scripts rewritten). The easiest way would be to give you my mod which is already adjusted, but unfortunately it has only Russian localization (most of scenario etc. expansions which I took as a base had only it). That's why I didn't reply when you previously asked for this -- just didn't have an idea how to solve this =\

If you have a good automated way how to translate several Mbs of dialogs/texts with a decent quality (AI or so, I'm fluent in English, but a manual translation would take me like a month maybe, really tons of text), then the idea is that here:

https://github.com/solbjorn/reaper-build/tree/master/config/text

you have XMLs with translations. Each string has a node for a specific language ("rus", "eng" etc.). So you basically need to take the value from "rus", translate it and add a second node "eng" with the translated text.

After this is done, you'll also need several additional big files (with textures etc.) which I don't upload to the GH repo, and probably a couple adjustments in the main config, but it's not a big deal.

I had a small "fixup" archive in the past which you could just drop to the S.T. installation and the vanilla game would work with my fork; but after I converted the engine from Luabind to Sol, it doesn't work anymore and I didn't update it since it would be too much work (and I can't just use the scripts from the mod since they are changed and improved heavily and rely in other features from the mod).

What do you think?

[tzcnt]

I can AI translate the XMLs, should be good enough to make it playable. And if you are able to point me at where the other files are hosted and where to put them, then that should be all I need? After you get the crashes fixed, of course.

Just curious, are you planning on doing a public release of this at some point?

[solbjorn]

(BTW I'm not from Russia if that matters :D)

I didn't have any plans on public releases, just doing it for myself and having fun. The mod is also in ever-WIP state, the first map is playtested, but I don't promise there won't be script crashes / glitches etc. I got back to S.T. modding in 2023 after a long break (I was active in modding in 2008-2011, then in 2017-2018) and at first, I only wanted to play some old mod I felt nostalgic to, but then I started improving feature X, then Y, then... Step by step, deeper and deeper, and now my fork differs from OGSR by a couple times a thousand locs :D

(we had no engine source code back in 2008 and had to work around engine bugs in the scripts as good as we could and often resign from something that couldn't be done on the original exe, now that the source code is open and there are several public projects like OGSR and OpenXRay, it's so tempting to do crazy stuff and features)

Sure, I'll let you know when the tmc branch becomes stable, I hope there'll be no hard blockers.

[solbjorn]

I've seen you're planning to introduce an option to make TMC fully header-only, could you maybe make it a tri-state, where the third option would make it header-only, but leave the hwloc-related functionality in ipp which I'd need to define in a .cpp file? I wouldn't probably be so critical against it if hwloc headers didn't include this horrendous windows.h. I'm planning to get rid of including it project-wide in future (vanilla engine code legacy), but give a chance to header-only TMC. As project-wide windows.h is a very bad idea.

[solbjorn]

I know it's not your problem and I (and other users) should be more careful, so not asking for any changes, just curious...

Is operator bool() mandatory for tmc::task<>?

When I refactor something like

void process_events()
{
    // ...
}

// to

tmc::task<void> process_events()
{
    // ...
}

then the compiler will notify me if I missed a co_await somewhere.

But when changing

bool net_spawn()
{
    // ...
}

// to

tmc::task<bool> net_spawn()
{
    // ...
}

// later

if (!net_spawn())
    //

then unfortunately the compiler is not able to catch a missing co_await as the [[nodiscard]] attribute of tmc::task<> triggers only when the task struct is completely unused, but calling operator bool() as in the example above silences this warning even though the intention was to do if (!co_await net_spawn()).

I shot myself in the foot a couple times already, fortunately it was easy to find and fix.

[tzcnt]

Yeah, I already ran into some issues with task::operator bool being implicit, but it turns out explicit is still troublesome as you've shown. I think I will remove this operator bool, and the user can manually check for nullptr if they want to see if the task has been initialized.

[pfeatherstone]

I have seen operator bool be the source of so many subtle bugs.

[solbjorn]

I made operator bool() explicit everywhere in my code some time ago, but unfortunately this barely helps most of times. I thought that operator bool() in tmc::task<> is either required by the coroutine API or you use it a lot in the TMC internals and it would be less convenient to convert this operator to a method (like valid() or initialized() or empty() or so) or to remove it. But if this is only a user API, then yeah, I'd say it does more harm than good.

I wish C++ had more advanced [[nodiscard]] features. Then you could write tmc::task<> the way that the operator bool() would still be there, but the compiler would warn if the user didn't call co_await or .fork() or so (I mean, ability to specify particular methods/functions which MUST always be called).

[tzcnt]

I was just trying to match the API of std::coroutine_handle which has operator bool.

Yes, there's no way to create a linear type in C++ at compile time. However there is a debug assert that would fire at runtime in that scenario. I know your project's debug build is broken. Somewhere on my offline TODO list was a custom assert macro so the user can opt-in to having runtime asserts in release builds. I can promote this to an enhancement issue.

[solbjorn]

The engine contains a whole set of different debug macros; even if the Debug config doesn't work, I can undefine NDEBUG time to time on the Release builds to make sure everything works as expected, no worries ;)

[solbjorn]

Hey,

I've noticed the development of TMC has slowed down a bunch lately.
No worries, everything works great on my side, despite that I haven't converted everything to async control structures yet (but the actual functions which use standard mutexes etc. are not coroutines, so the engine works stably without deadlocks), and I've been (and still) busy with some other crazy stuff currently.

Just wanted to make sure you are fine, no burnouts, no motivation loss, no problems in real life etc. Take care!

(also seems like libfork has finally woken up, I'm curious what the new version will offer, although I'll stay with TMC with no doubts)

[tzcnt]

Taking a bit of a break after v1.4 release. I've started a new (secret) project that uses TMC so it's good to be dogfooding a bit - helps me find the rough edges or missing functionality. I'll come around and finish v1.5 (mostly finalizing the build system) at some point.

[solbjorn]

Just noticed the runtime-benchmarks repo update...

Man, WTH is this citor, how comes it's the fastest one? If I got it correctly, it's not even based on coroutines, right?
Do you know anything about this framework? Limitations in comparison to TMC etc? (if it's not coroutine-based, then it must at least suffer from context switches)

I'm not planning to migrate from TMC obviously, just curious. Also interesting if anything from citor could serve as inspiration for improvements in TMC.

[tzcnt]

I've only had a short time to read it so I'm not certain that I understand it fully, but it's got some advantages:

• custom arenas for allocation
• uses direct calls for inline continuations of groups which makes it perform well on recursive fork join benchmarks.
• I don't think it has continuation stealing so it can guarantee that tasks are deallocated where they were created.

Disadvantages:

• more special purpose, no multi-executor, async or any other kind of capabilities
• if A forks and B steals, then B forks, A cannot steal back from it. this can lead to lower utilization on unbalanced workloads.
• seems like it doesn't support dispatching multiple task groups at once - each root task dominates the entire executor until it completes
• executor is designed to spin (not sleep) when work is not ready
• can stack overflow on too-deep recursion due to inline calls rather than symmetric transfer tail calls

I hope I'm not misrepresenting it poorly here but I think that it would be a good choice for heavy data processing type work, but I wouldn't choose it for something like a game engine or backend server that needs to run many smaller dynamic workloads concurrently.

[solbjorn]

I noticed that rwlock is in progress -- nice! New Screen Space Shaders will be released soon, plus I need a resync with the upstream OGSR after merging the shaders, but after it's done, I'd go forth with making the engine completely lockless (currently there are still several "regular" locks in regular functions -- I've been waiting for rwlock to convert them the most optimal way right away).

[solbjorn]

What is CRL by the way? Anything useful there comparing to bare TMC?

[tzcnt]

Looks like a small wrapper Telegram devs use to abstract over libraries for different platforms. Seems like they're just using it with TMC for fire-and-forget tasks.

[solbjorn]

Looks like a small wrapper Telegram devs use to abstract over libraries for different platforms. Seems like they're just using it with TMC for fire-and-forget tasks.

Glad to hear they took TMC ^.^

[solbjorn]

Speaking of do-and-complete-task: is there a way to implement stuff like finally/defer, but which would work with coroutines? Normally it's done simply as a scoped object and the deferred lambda is called from the destructor, but since you implemented stuff like unlock_and_complete, I'm curious ;)

const auto do_on_exit = tmc::finally([] -> tmc::task<void> { co_await coro(); });
...
co_return; // do_on_exit is called here

[tzcnt]

I wish we had something like this. Async destructor / async finally that runs before the task completes cannot be implemented AFAIK. It's possible to chain a task to run after the task completes, and I think we could dynamically insert that task in the chain between the current task and its parent, but it wouldn't be able to access to the locals of the current task, because they are destroyed by the compiler before final_suspend is entered.

[solbjorn]

I see, thanks! Not that I really need this, just would allow for a bit cleaner code sometimes. I think the complexity of a theoretical implementation just isn't worth it.

So, unlock_and_return() is implemented somehow differently from what I thought?

[tzcnt]

Well, you can read the implementation :) it's just an interleaving of task's final_suspend and mutex's co_unlock. When I was building it, I had the same thought - can I implement async destructors by hooking a function pointer back into the coroutine's body into the final_suspend of task? It turns out this does not work, because as described in https://en.cppreference.com/cpp/language/coroutines, when a coroutine reaches the co_return statement, it "destroys all variables with automatic storage duration in reverse order they were created" BEFORE it calls promise.final_suspend() and co_awaits the result. So it's simply unsafe to access any locals of the coroutine at that point.

What's weird is that there's also a call to handle.destroy() inside the final_suspend. I had to do some digging on this, and it turns out that if you call destroy() earlier in the body of the function (as co_unlock_return does), then it both destroys the locals and frees the coroutine frame. Whereas if you call it inside final_suspend, the compiler has already destroyed the locals, so destroy() only frees the coroutine frame.

Essentially, the coroutine-body variables exist inside a scope, and that scope ends BEFORE final_suspend runs. co_unlock_return doesn't have this problem because it's just a regular awaitable that runs inside of the scope.

[tzcnt]

Just curious - would you have any use for coroutine functionality inside Lua? If so, what use cases?

[solbjorn]

Lua state is not thread-aware, so it can't cope with multiple simultaneous calls from several threads. Usually, when we call a Lua function from C++, we need it to be executed in full and immediately. This doesn't leave much space for coroutines with Lua.

On the other hand, since I'm planning to fully get rid of "classic" mutexes and convert everything to the async counterparts, there will be situations where I'll need to make some Lua logics async as well.
For example, we have script exports to perform a ray query -- generic term for everything related to object space etc. If you want to know whether the rain is falling to the actor's head, you query the space above him, and if there's no obstacles/ceilings/whatever, it is. Currently, there are mutexes inside the ray query code.
Converting them means converting these functions to coroutines. Since Lua can't call coroutines directly and can't preserve function coloring, I'd need to make this export a simple function which would queue a task to the executor. I call Lua functions on the ex_cpu, which means I can't (if I want to still be fully lockless) wait for the result right away. So the solution here would be to queue a task and immediately return, whereas this task will call a Lua callback at the end.

Script exports allow to have more fun with coroutines. I can export tmc::task to Lua as an opaque type which I can just receive from C++ code and pass to C++ functions, pretty much any primitive can be exported. I think it's just the original Lua limitation -- that we can interact with the Lua state from at most 1 thread -- made the developers design the interop strongly serialized and never think of something like this. Plus LuaJIT is extremely fast and barely takes 1/10 of a frame even though we have really complex scripts.
But I'll take a closer look later, maybe there are places which would benefit from this ;)

[solbjorn]

I've noticed you forked LLVM and created a couple branches. Any interesting upcoming optimizations/PRs?

BTW I was on the Netdevconf 0x1A and a couple folks from different organizations complained that their concurrency or coroutine libraries aren't NUMA-aware or slow or whatever, so I recommended them yours ^.^

[tzcnt]

There's 2 halves to the LLVM work:

• coroutine codegen optimization:
  • changing HALO to be size-bounded
  • making HALO work with recursion
  • making HALO work with spawn_many
  • removing unnecessary dynamic dispatch of known function pointers (std::noop_coroutine, coroutine ramp / resume functions that are loaded in the current function).
• safety improvements:
  • enforcing TMC's linear type rules to prevent leaked coroutines / double-join errors
  • making the above work with spawn_many
  • making the above work with mux
  • diagnosing capturing lambda coroutine lifetime errors

Realistically only the codegen optimization can be upstreamed, and maybe the capturing coroutine lambda - although I think that one may break existing code - I think there are too many variations on this in use to make it globally viable. But I'll try to get as much upstreamed as possible. For the TMC-specific safety improvements, I have yet to test whether they have to be implemented in the compiler, or if they could be implemented as a clang-tidy plugin, which would be preferable to maintaining a fork.

Right now everything's in vibe-coded / works on my machine status, so it's going to take some cleaning up before upstreaming.

[solbjorn]

I caught a rare crash inside my thread_init_hook(tmc::topology::thread_info info). It emplaces thread info into a vector. If I got it correctly, calls to this hook are not serialized, they can happen simultaneously from different threads when they're initialized, thus corrupting my vector sometimes.

If that's the case, maybe it would be a good idea to mention in the documentation that they're not thread-safe and the user must serialize them manually if needed. I guarded the vector with a std::mutex, if you have better ideas -- feel free to share ;)

[tzcnt]

Yes, it's called from inside each new thread, potentially in parallel. I could have the threads serialize themselves on startup but I suspect most users will use them for thread-local operations only. I'll add a comment regarding the potential need for manual serialization.

[solbjorn]

Yeah I don't think you need to serialize it inside the library, especially given that init hooks are not used frequently I guess.