Rakudo hangs with shell processes in Promises after reaching max_threads

[molecules]

The Problem

When multiple Promises contain shell, rakudo hangs upon awaiting the n^th one, where n = max scheduler threads.

Expected Behavior

$ perl6 foo_bar_baz.p6
Planning on processing foo
Planning on processing bar
Planning on processing baz
processing foo
processing bar
processing baz
$ 

Actual Behavior

$ perl6 foo_bar_baz.p6
Planning on processing foo
Planning on processing bar
Planning on processing baz
processing foo
processing bar

Steps to Reproduce

Run the attached script (foo_bar_baz.p6) thus:

perl6 foo_bar_baz.p6

Here is the script as well:

#!/bin/env perl6
#foo_bar_baz.p6

my $*SCHEDULER = ThreadPoolScheduler.new( max_threads => 2 );

my @processes;

# The promises generated by this loop work as expected when awaited
my @items = <foo bar baz>;
for @items -> $item {
    @processes.append(
        start { say "Planning on processing $item" }
    );
}

# The nth Promise generated by this loop seems to hang when awaited
for @items -> $item {
    @processes.append(
        start { shell "echo 'processing $item'" }
    );
}

await(@processes);

Simply change max_threads to 4 and the problem disappears.

Environment

• Operating system:
  CentOS Linux release 7.4.1708 (Core)

• Compiler version (perl6 -v):
  Rakudo Star version 2018.06 built on MoarVM version 2018.06 implementing Perl 6.c.
  Rakudo Star version 2018.10 built on MoarVM version 2018.10 implementing Perl 6.c.
  Rakudo Star version 2019.03.1 built on MoarVM version 2019.03 implementing Perl 6.d.

[jnthn]

This is due to exhaustion of the thread pool, which is a shared resource. Proc is implemented in terms of Proc::Async, which uses the thread pool internally, together with the program that's running here. If the program exhausts all the threads itself, there's no way for any of Proc::Async's internal workings to make progress.

The default thread limit is not 2, but 64. I don't consider the problem here a particularly practical one, since by the time a program exhausts 64 OS threads just on running concurrent processes, that's well past the point where it should have been written in terms of Proc::Async, in which case it would likely use just a couple of OS threads, and therefore use a lot less memory and have a lot less GC overhead. Sometimes, it's probably good to have the implementation push back a bit when doing things in a suboptimal and unscalable way.

If one insists on using shell together with start without problems, it would be possible to make a threadpool instance independent of the core one (just assign it into a lexical $scheduler variable), and then use Promise.start({ shell ... }, :$scheduler) or similar.

One potential way to make this kind of program Just Work would be for us to decide that shell and run use non-blocking await internally. This would make the problem as provided Just Work. However, it's the usual trade-off: one then could not be sure that the line of code after the shell will run on the same OS thread as the line before. We've generally tried to keep our sync and async things straight in that sense, so they're predictable in how they interact with threads (even if that might mean predictably annoying :-)). That said, the inherent concurrent nature of the general case of process spawning may be an argument that Proc could be internally implemented in terms of non-blocking await.

Either way, so far as the language as defined today goes, there's not a Rakudo bug here.

[jnthn]

Ah, and since it looks like the only way anything is likely to change here is a language design change, the appropriate forum for further proposals on that would be the problem solving repo, so I'll close this one.

[molecules]

@jnthn If I use Proc::Async, is there a way to limit CPU usage (besides something external, like cgroups)?

[timo]

i don't see a simple way to figure out how much cpu resources the spawned processes are consuming - getrusage and times, for example, will only report on child processes that have already terminated - but if you find a sensible method of measurement, you can use SIGSTOP and SIGCONT to switch processes on and off to conserve cpu time

if your spawned processes can be made to cooperate, i.e. you can change their source code, setrlimit will allow you to set a soft limit on cpu time, which when exceeded will have the kernel send SIGXCPU to the child process; if you have a signal handler installed in that process, you can have it communicate with the spawning process that it will now wait for more cpu time to be allocated to it; the spawning process would then raise the soft cpu limit and tell the process to continue

tbh, i don't think there's anything simpler than cgroups or something like that

[tony-o]

@molecules you can create your own scheduler to use fewer threads.

my @x;
my $sched = ThreadPoolScheduler.new(max_threads => 2);
my $y = $*SCHEDULER; #save ref to current scheduler
$*SCHEDULER = $sched; # reassign while we do a mindless say/sleep
for 1..20 {
  my $l = $_;
  @x.push(start { $l.say; sleep .5; }); # these will only run two at a time
};
$*SCHEDULER = $y; #reset $*SCHEDULER to what it was when we started
await Promise.allof(@x);

[molecules]

Thanks @jnthn, @timo, and @tony-o! I'm going to experiment a little more and then I may have more questions. Thanks so much!

[jnthn]

@molecules If you're looking to limit the number of running procs, then see these slides from around slide 34.

[molecules]

@jnthn If you don't mind posting your answer (i.e. #2834 (comment)) I'll accept it for my question at Stackoverflow (https://stackoverflow.com/questions/55265176).

[jnthn]

@molecules Will do so shortly. :-)

[molecules]

@jnthn I was hoping you'd answer in time to claim raiph's bounty on the question. There is only seven or eight hours left. By the way, thank you for all of your work on Perl 6! It's definitely my favorite language.

[jnthn]

@molecules Done! :-)