How to launch multiple blocking processes #112
Labels
Comments
|
What do you mean by "But I can't get it to work." ? What is it doing? |
|
As in, I'm seeking advise how actually achieve this. As for what its doing: So I figured I got to give it a promise, which it will then collect for me. |
|
One of the problems I can see is that |
|
amphp/process seems to be more for commands like dig, ping, not specifically calling php scripts that are blocking, unless I fully misunderstood that. |
errors on |
|
amphp/process is to call anything at all... in fact amphp/parallel is built on top of amphp/process. |
|
Now you're not yielding the promises from send method. Also you can't call send before yielding the promise from start. What didn't you understand about "don't use wait"? |
|
🥺 Is there any chance you could guide me with (pseudo) code please?
|
|
Ok. Tomorrow. I need to sleep now. |
function runProcess($value) {
return call(
function () {
$process = new Process(__DIR__ . "/amp-runner.php");
yield $process->start();
yield $process->send($x);
}
)
}
Loop::run(
function () {
$promises = [];
for($x = 0; $x <= 10; $x++)
{
$promises[] = runProcess($x);
}
yield $promises;
}
); |
|
Thanks a bunch. I'm understanding it a bit more, this is my How would I now get the data from the children? |
|
@iNilo check this out: https://github.com/JanMikes/php-async-playground i think it is exactly what you are looking for :-) |
kelunik
changed the title
|
@iNilo I think @JanMikes provided the answer to your question. @JanMikes #112 (comment) might be helpful for you if you don't want to write your own protocol using STDOUT for communication, but rather a message based approach, where the library cares about the serialization. |
|
Thanks for this @JanMikes & @kelunik I would just love to get to use the frameworks way of sending things back parallel/examples/blocking-process.php Line 14 in 7f00b8e And grab it from the child as a parent. Line 31 in 7f00b8e A possible last request, to understand the framework better; Is there possibly any chance someone could write another example based on : https://github.com/amphp/parallel/blob/master/examples/process.php Use the same blocking process, but launch the child 20 times? ❤️ |
|
@iNilo You already linked the two relevant lines. What do you want to send / receive exactly? One or multiple messages? |
|
@kelunik thanks, was stuck for a bit, @JanMikes example used @JanMikes possibly check out my code too. Thanks for the support all of you ❤️
results: |
|
@iNilo looks good to me
Process i want to run is symfony/console command that knows nothing about amphp but is capable of writing json results to stdout, thats why i went this way. How would that process receive without amphp data from I tried it here: https://github.com/JanMikes/php-async-playground/blob/parallel-context/script.php If i understand it correctly, if i want to use channel/context to parent-child communication i need that process to be a php script which returns I am sure it could be done other the "amp way", instead of runnng directly symfony/console directly via <?php
return function(Channel $channel) {
$container = createSymfonyContainer(); // magic behind ...
$application = $container->get(Application::class);
$code = $application->run();
$service = $container->get(MyService::class);
yield $channel->send($service->getResults());
return $code;
};I dont fully understand what are all added values using channel to communicate instead of taking it directly from stdout and if my thinking is correct, though i like that i dont have to care about serialization/deserialization and just send data to channel. |
|
Whilst slightly offtopic: Was this the reason a delay was introduced in the examples? |
|
@iNilo Do you have error reporting enabled? Does PHP emit any notices / warnings? |
|
With xdebug on @kelunik
|
|
If you add |
|
@kelunik is there any chance I can do this in my |
|
You'll need to edit the file in |
|
Understood @kelunik So I add parallel/lib/Context/Internal/ProcessHub.php Line 122 in 2b418eb Also, is this an issue on my end? (host) or library? |
|
Editing my |
|
Yes, right before the linked line. I'm not sure what's causing the issue, I can't currently reproduce it the same way you're experiencing it. |
|
When adding a delay at the requested line: parallel/lib/Context/Internal/ProcessHub.php Line 122 in 2b418eb So it looks like:
|
|
How do things change if you disable xdebug? I can reproduce errors in case xdebug is enabled, but not the timeouts you see. |
|
On my side xdebug is not enabled and I am still getting those errors with branch issue-112-kelunik: |
|
@kelunik me butchering xdebug.
|
|
@kelunik do you test with a build of PHP with ZTS (Zend Thread Safety) enabled? On my side it's not. |
|
@kelunik @trowski When ZTS & krakjoe/parallel is installed does amphp/parallel automatically uses it instead of processes? If yes, does the tests you are doing on your side are are with ZTS & krakjoe/parallel? If yes, can it explains why you cannot reproduce the issue? Just brainstorming here... @iNilo did you try with ZTS & krakjoe/parallel installed? |
|
@bergmab I can reproduce it with |
|
Using a pool with 128 workers works fine now. 256 is not:
This probably another issue. Would you like me to open a new one for this? |
|
No, it's exactly that issue, but should have been solved by the retries, or so I thought. It's strange you didn't get that warning before if you get it now. |
|
That’s the first time I see this error. Do you need anything from me to help resolve that issue? |
|
Not at the moment, no, will try something later. |
|
@kelunik please advise 😢 Your latest fix is there (checked the code)
|
|
@bergmab Please retry with the latest @iNilo Please try it with an error handler like that: <?php
require __DIR__ . '/../vendor/autoload.php';
use function Amp\ParallelFunctions\parallelMap;
use function Amp\Promise\wait;
$pool = new Amp\Parallel\Worker\DefaultPool((int) $argv[1]);
$array = \array_fill(0, (int) $argv[2], 1);
set_error_handler(function (...$e) {
unset($e[4]);
var_dump($e);
});
// Parallel function execution is nice, but it's even better being able to use closures instead of having to write a
// function that has to be autoloadable.
try {
\var_dump(wait(parallelMap($array, function ($time) {
\sleep($time); // a blocking function call, might also do blocking I/O here
return $time * $time;
}, $pool)));
} catch (Amp\MultiReasonException $e) {
throw @reset($e->getReasons());
} |
|
Looks like the latest code is hanging with pool-size = 256. I can only see results from the first worker during the first 10 minutes of run but after that the only thing I see is the CPU usage increasing until I stop it. Note: it's still working nicely with pool-size = 128. |
|
Please test with the latest |
|
Latest master commit, same behavior. |
|
Just did some couple more tests with different pool-size value and with 200 I face these errors after 40K successful jobs: |
|
Please configure |
|
Here's the requested log: |
|
Did the following:
Seems to be working! 🦸 |
|
Still not on my side. Note that I am using parallel-functions, not directly parallel. |
|
@kelunik @bergmab
|
|
@kelunik thanks for your reply!
also I don't think I'm hitting that limit since its raised(?)
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Terribly sorry for making an issue, but I've been stuck for a while.
I'm in the need to launch multiple blocking php scripts in a concurrent fashion.
Its the same script 20 x times.
I've looked at https://github.com/amphp/parallel/blob/master/examples/process.php
and its child:
https://github.com/amphp/parallel/blob/master/examples/blocking-process.php
But I'm struggling to make it launch 20 children ( with different data )
I've got something like this:
But I can't get it to work.
The text was updated successfully, but these errors were encountered: