async/await multi curl stuck #7485
Comments
|
Why public function call(Vector<GateTask> $tasks):Map<string,mixed> {
$awaitable_tasks = Vector{};
foreach ($tasks as $task) {
$raw_data = $this->getCachedContent($task->hash());
if ($raw_data !== null) {
$this->callback_cache->set($task->contentAssocKey(), $this->applyCallback($task, $raw_data));
} else {
$awaitable_tasks->add($task);
}
}
$call_data = $this->callAsync($awaitable_tasks)->getWaitHandle()->join();
return $this->callback_cache;
} |
|
Well, I'm trying to use the original code from the docs :D https://docs.hhvm.com/hack/reference/function/curl_multi_await/ just modified run function <?hh
async function get_curl_content(Set<string> $urls): Awaitable<Vector<string>> {
$chs = Vector {};
foreach ($urls as $url) {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$chs[] = $ch;
}
$mh = curl_multi_init();
foreach ($chs as $ch) {
curl_multi_add_handle($mh, $ch);
}
$active = -1;
do {
$ret = curl_multi_exec($mh, $active);
} while ($ret == CURLM_CALL_MULTI_PERFORM);
while ($active && $ret == CURLM_OK) {
$select = await curl_multi_await($mh);
if ($select === -1) {
// https://bugs.php.net/bug.php?id=61141
await \HH\Asio\usleep(100);
}
do {
$ret = curl_multi_exec($mh, $active);
} while ($ret == CURLM_CALL_MULTI_PERFORM);
}
$content = Vector {};
foreach ($chs as $ch) {
$str = (string) curl_multi_getcontent($ch);
$content[] = substr($str, 0, 10);
curl_multi_remove_handle($mh, $ch);
}
curl_multi_close($mh);
return $content;
}
function run(): void {
$urls = Set {};
for($i=1;$i<10;$i++) $urls->add('http://example.com/?test='.$i);
$content = \HH\Asio\join(get_curl_content($urls));
var_dump($content);
}
while (true) {
run();
}
The result on Debian with HHVM 3.15 after about 10 cycles
The result on MacOSX with HHVM 3.11
|
|
However, if I change $select = await curl_multi_await($mh);to $select = curl_multi_select($mh);it works infinitely without any issues : |
|
Hi again! just took the code from the post above, which is originally a code from the docs, with
The results after 25 loops:
|
|
I continue to investigate.
|
…emlib one" fixes facebook#8156 refs facebook#7485 refs facebook#8043 The previous stability issues with the native impl were likely not actually an issue with async curl, but with async more generally with libevent2. facebook@93c31ec#diff-e9e6b8e9c00d40c3ec104ff79c2037a7 appears to have fixed the original stability issues (and the issues with stream_await()); it just looked like `curl_multi_await()` and `stream_await()` were buggy as: - FB rarely uses these functions (in particular, for our use case, async curl is generaly less performant than sync curl - but that depends on what other load you have, and how long requests take) - FB uses libevent1 - `curl_multi_await()` and `stream_await()` are among the most useful external features ... so, an HHVM issue when using libevent2 looks like a curl + stream async issue Test Plan: ran test programs from previous curl issues Crashes or gets stuck (doesn't on master or with this diff): ```Hack <?hh async function get_curl_content(Set<string> $urls): Awaitable<Vector<string>> { $chs = Vector {}; foreach ($urls as $url) { $ch = curl_init($url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); $chs[] = $ch; } $mh = curl_multi_init(); foreach ($chs as $ch) { curl_multi_add_handle($mh, $ch); } $active = -1; do { $ret = curl_multi_exec($mh, &$active); } while ($ret == CURLM_CALL_MULTI_PERFORM); while ($active && $ret == CURLM_OK) { $select = await curl_multi_await($mh); if ($select === -1) { // https://bugs.php.net/bug.php?id=61141 await \HH\Asio\usleep(100); } do { $ret = curl_multi_exec($mh, &$active); } while ($ret == CURLM_CALL_MULTI_PERFORM); } $content = Vector {}; foreach ($chs as $ch) { $str = (string) curl_multi_getcontent($ch); $content[] = substr($str, 0, 10); curl_multi_remove_handle($mh, $ch); } curl_multi_close($mh); return $content; } function run(): void { $urls = Set {}; for($i=1;$i<10;$i++) $urls->add('https://hhvm.com/?test='.$i); $content = \HH\Asio\join(get_curl_content($urls)); var_dump($content); } while (true) { run(); } ``` Spins 100% CPU (does on master, doesn't with this diff): ```Hack <?hh $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, "https://httpbin.org/delay/10"); // this line takes 100% cpu for 10 seconds \HH\Asio\join(\HH\Asio\curl_exec($ch)); // this line *does not* takes 100% cpu for 10 seconds // curl_exec($ch); curl_close($ch); ```
…emlib one" (#8313) Summary: fixes #8156 refs #7485 refs #8043 The previous stability issues with the native impl were likely not actually an issue with async curl, but with async more generally with libevent2. 93c31ec appears to have fixed the original stability issues (and the issues with stream_await()); it just looked like `curl_multi_await()` and `stream_await()` were buggy as: - FB rarely uses these functions (in particular, for our use case, async curl is generaly less performant than sync curl - but that depends on what other load you have, and how long requests take) - FB uses libevent1 - `curl_multi_await()` and `stream_await()` are among the most useful external features ... so, an HHVM issue when using libevent2 looks like a curl + stream async issue Pull Request resolved: #8313 Reviewed By: jano Differential Revision: D9648538 Pulled By: fredemmott fbshipit-source-id: 4c1b7bb9a77fbcdff9c28211c2f22f1b4ead5930
Here we go again:)
HHVM Version
HipHop VM 3.15.0 (rel)
Compiler: tags/HHVM-3.15.0-0-g92a682ebaa3c85b84857852d8621f528607fe27d
Repo schema: 225d4323575bbc8a498dc809a1c41354f6bca830
OS: Debian 8.3
Standalone code, or other way to reproduce the problem
Expected result
...infinite...
Actual result
Then just stuck.
strace
futex(0x7ffd8689ac74, FUTEX_WAIT_PRIVATE, 1, NULLAlso I did tests on another (old) hhvm version on MacOSX:
HipHop VM 3.11.0 (rel)
Compiler: 1455186033_N
Repo schema: 4d3d19cb46fb2651317d6269a2f601f2d8e29601
And there I have core dumped when it stuck:
PHP Stacktrace:
#0 HH\WaitHandle->join() called at [/Users/.../test_core_dump.hh:28]
#1 TestGate_Example->call(Vector) called at [/Users/.../test_core_dump.hh:14]
On a different servers with a different hardware scope it needs a different number of multi urls to catch this ($i in for cycle). The moment when it happens is very unstable, it could be iteration 2, or iteration 80, but you will definitely get it.
On the very slow urls (when response goes after 10-50 seconds, what is very common for some external APIs) this "stuck" will happens earlier, after several iterations.
Now, the questions is - what is wrong with this code?
p.s. And why this implementation is so strange you may ask - because in the original code there are also lambda callbacks in createCall function, some cache levels and additional logic. But they are not the reason of this issue because it happens without all this stuff.
I would greatly appreciate any suggestions.
The text was updated successfully, but these errors were encountered: