Need a feature to yield to a new task immediately

[biddisco]

In certain areas like the parcelport code, we would like to execute tasks with 'immediate' priority so that the current task supends and the new task runs in it's place. This is similar to simply executing the new task on the current thread (as a plain function), but allows the possibility that the new task might in turn suspend and return control back to the current task which we assume does not need to wait for the return value.

one suggestion from @sithhell is

use the boost priority, call async, after it returned, do a this_thread::yield()

however that incurs the cost of putting the task on a/the queue, and then taking it off again (and runs the risk that something else is already on the high priority queue and gets taken first).

I can see several possible API's, the first assumes that the new task has no return value of interest to the caller and would look as follows

do work ...
hpx::yield( new task );
do work ...

or a second might look like this (more like a coroutine), using a special launch policy

do work ...
auto dummy = hpx::async( hpx::launch::immediate, new task );
do work ...
[optionally call dummy.get(); at some point if the results is needed]

The use case for this is during the sending of parcels via hpx::async(locality, ...) when we wish to immediately launch a task to serialize and enqueue the send data into the PP - however under certain circumstances (full send queues or no connection available) the task might suspend and leave the parcel until some later time when the high priority queue can retry the send.
In this case the calling function can resume and finish it's current work.

[hkaiser]

This may require to extend the coroutines scheme which underlies all of the HPX threading. Currently this is based on asymmetric coroutines, i.e. one 'main coroutine' - which is the thread scheduler, and many asymmetric coroutines to which the thread manager coroutine yields to and which always yield back to the thread manager.

It looks like that your request would be best implemented by a scheme where you explicitly yield back and forth between the user tasks without involving the thread manager directly.

[hkaiser]

FWIW, we currently implement this through async(launch::fork, f, ...), which schedules 'f' with boosted priority and puts the current thread to sleep right afterwards. This is very similar to one of your suggestions above, but has the mentioned disadvantages.

Also, in general such a facility would be very useful to properly implement 'parent-stealing' (as opposed to the current, 'child-stealing' scheme). The CILK-guys have shown that 'parent stealing' (after spawning a new task the current core continues executing the new task, leaving the parent task to be potentially stolen by another core) allows to put an upper bound on resource requirements for new tasks. On the other hand, 'child-stealing' (after spawning a new task the current core continues executing the parent task, leaving the child task to be potentially stolen by another core) is known to potentially require an unlimited amount of resources when spawning a large number of tasks.

[biddisco]

The phrase "parent-stealing" was unknown to me until now, but it is exactly what I have been looking for to avoid the problem of excessive resource consumption. The problem I have is that a single task can spawn thousands of child tasks, each of which might take a large memory chunk as a parameter (for a send for example), the process of generating each new task (reserving memory buffers) and then handing the args over as the task is queued can drain the system resources completely. Also when generating many many send operations, the network handler can become swamped with stacked up parcels resulting in thousands of them in the queue (which in turn creates different issues for the parcelport layer). Suspending the parent task - so that it stops spawning more and more child tasks - until some have been handled and resources are available again is exactly the use-case / scenario that I would like to solve.

I will pursue this further.