fork within cursor loop leads to RuntimeException or ServerExpection

[raffis]

Description

Creating forks within a mongodb cursor loop.
Having a hard time here. If the fork terminates, the cursor gets killed. Probably because the parents memory gets cloned into the fork and the driver kills the cursor after the fork exits.

Environment

Driver: 1.5.3
Lib: 1.4.2
php: 7.2.4
pcntl ext required

Test Script

<?php
require_once('vendor/autoload.php');

function fork($doc) {
    $pid = pcntl_fork();

     if (-1 === $pid) {
         throw new \Exception('failed to spawn');
     }

    if (!$pid) {
        var_dump("SPAWN", $doc);
        exit();
    }

    return $pid;
}

$mongodb = new \MongoDB\Client('mongodb://mongodb:27017/test');

try {
    $mongodb->test->createCollection(
    'test',
     [
      'capped' => true,
       'size' => 100000,
     ]
    );
} catch (RuntimeException $e) {
    if (48 !== $e->getCode()) {
        throw $e;
    }
}

$collection = $mongodb->test->test;
$collection->insertOne(['foo' => 'bar']);

$cursor = $collection->find([], [
    'cursorType' => \MongoDB\Operation\Find::TAILABLE_AWAIT
]);

$iterator = new \IteratorIterator($cursor);
$iterator->rewind();

while (true) {
   var_dump(getmypid(). "- loop");
   if ($iterator->valid()) {
        $document = $iterator->current();
        var_dump($document);
        fork($document);
   }

   $iterator->next();
}

Expected and Actual Behavior

Getting random cursor errors:

Fatal error: Uncaught MongoDB\Driver\Exception\ServerException: cursor id 92208694393 not found in /srv/t/test/test.php on line 53

MongoDB\Driver\Exception\ServerException: cursor id 92208694393 not found in /srv/t/test/test.php on line 53

Call Stack:
    0.0001     459800   1. {main}() /srv/t/test/test.php:0
    0.0652    2322016   2. IteratorIterator->next() /srv/t/test/test.php:53

or:

Fatal error: Uncaught MongoDB\Driver\Exception\RuntimeException: Invalid reply to getMore command. in /srv/t/test/test.php on line 53

MongoDB\Driver\Exception\RuntimeException: Invalid reply to getMore command. in /srv/t/test/test.php on line 53

Call Stack:
    0.0002     459800   1. {main}() /srv/t/test/test.php:0
    0.1092    2322016   2. IteratorIterator->next() /srv/t/test/test.php:53

I would expect, that the fork exits after doing some stuff with $doc and the parent process runs its while loop forever.

[jmikola]

MongoDB\Driver\Manager::__construct() includes the following note on forking:

On Unix platforms, the MongoDB driver is sensitive to scripts that use the fork() system call without also calling exec(). Users are advised not to re-use MongoDB\Driver\Manager instances in a forked child process.

When you fork(), the Manager and Cursor instances are being copied into the child process. When the Cursor object is garbage collected in that process, if it is not exhausted (i.e. there is still a cursor ID and a possibility that more results exist on the server) the driver will attempt to kill it and clean up server-side resources. That behavior occurs in mongoc_cursory_destroy(), which we call. Assuming the cursor is killed by a terminating child process, the parent process or another child could easily encounter a cursor id %d not found error attempting to work with it after that point.

This is not unique to the PHP driver's use of libmongoc, as libmongoc also notes that cursors are not thread-safe.

The Invalid reply to getMore command error is different, but I expect that relates to the underlying sockets becoming shared by multiple threads/processes. In that scenario, the threads are now sending separate messages on their respective sockets but reading data from a shared source. It becomes quite likely for one thread to read the reply intended for another (a possible cause of the error you observed), since there is no synchronization.

By design, the driver does not support these use cases. Any Manager and/or Cursor instances should be created after forking to avoid any resources being shared my multiple threads/processes. If you need to share query results with child threads, you would need to (a) spawn those children before creating a Manager or Cursor and (b) send documents to children using some form of IPC. You may be able to get away with creating the Manager before forking, provided the children never interact with that Manager (I would suggest unsetting it immediately in child processes); however, for reasons cited about you should not allow a Cursor to become shared by a child process, as even unsetting it could result in it being killed on the server during cleanup.

[raffis]

That's a shame. I have written quite a complex async task project using mongodb. v3 features forking.
Worker nodes may be bootstrapped before rotating a mongodb queue. But if a worker dies or for whatever other reason, a new worker needs to spawn within a task cycle. Therefore I have already a running manager instance. See (https://github.com/gyselroth/mongodb-php-task-scheduler/blob/dev/src/Queue.php#L376)

And yes the manager is created before forking. And also I do not need the manager in forks (Or just not the same instance).

What if the pid gets attached to the manager? If it gets cloned into a fork we have a different pid and if thats the case, the manager shall not destroy running cursors started by another pid.

[jmikola]

What if the pid gets attached to the manager? If it gets cloned into a fork we have a different pid and if thats the case, the manager shall not destroy running cursors started by another pid.

We already include the PID when creating a hash for persisting libmongoc clients, which are used by Manager objects. This ensures that a Manager constructed in a child process will not inadvertently use a libmongoc client belonging to another process. On a related note, we also do not destroy libmongoc clients that belong to other processes. This is effectively leaking memory; however, by this point the process is already being terminated so we're not too worried about it.

Looking at your application, the Queue directly contains a MongoDB\Database object (from the library), which internally has a MongoDB\Driver\Manager. The problem may be related to it also creating two MessageQueue instances and constructing MongoDB\Driver\Cursor objects from Queue::main(). I can't tell if any of the child processes you spawn interact with the driver explicitly, or if these errors are simply due to the Cursor object being garbage collected.

That said, we can look into whether it is possible to avoid the call to mongoc_cursor_destroy() if the current PID doesn't match the PID at the time the cursor was created. One option would be to simply not call the destroy method and allow libmongoc to leak memory. A more correct option would entail changing libmongoc to allow us to opt out of cleaning up server-side resources (i.e. issuing a killCursors command), although I'm not sure if they would be amenable to implementing that. Either way, the onus would still be on the application to unset any Cursor objects immediately after the child spawns or otherwise ensure they are never used in the child process.

Note: the driver has analogous behavior for ending server sessions via the endSessions command; however, that is triggered by mongoc_client_destroy(), which child process do not call for the reason cited in my first paragraph above.

[jmikola]

Opened PHPC-1274 to track the idea of having children not kill cursors created by their parents. There's no timeline for getting this implemented/released, so any immediate solution will require some design changes to your application.

One suggestion would be having the coordinating process responsible for maintaining the work pool (and spawning children) not interact with the driver and process the queue. If the queue consumption is done in a separate dedicated worker, which never forks, it can then pass workloads on to the children through some IPC mechanism. That will also require sharing some information about the worker pool to the queue consumer, although I suppose the coordinating process may be able to act as an intermediary there (either informing the consumer about how to reach children or relaying messages for everyone).

[raffis]

Looking at your application, the Queue directly contains a MongoDB\Database object (from the library), which internally has a MongoDB\Driver\Manager. The problem may be related to it also creating two MessageQueue instances and constructing MongoDB\Driver\Cursor objects from Queue::main(). I can't tell if any of the child processes you spawn interact with the driver explicitly, or if these errors are simply due to the Cursor object being garbage collected.

I'm pretty sure its just the fact that the cursor is garbage collected. This object gets in no way used in forks but the cursor gets killed as soon as a worker dies.

That said, we can look into whether it is possible to avoid the call to mongoc_cursor_destroy() if the current PID doesn't match the PID at the time the cursor was created.

That would be awesome 👍

For the time beeing, actually MongoDB was the kind of "IPC" I intended to use. And it already is.
I need to spawn workers within a loop of jobs (Running tailable cursor), there are also tasks which require to spawn a new worker only for one job.
What I did now is fork of a fork manager at the very beginning. A mongodb manager instance will be forked into the fork manager since one is required to create an instance of TaskScheduler\Queue.
But that should not mater since this instance gets not used in any way.
The queue (parent task) will spawn workers via a systemv message queue. And since it does it this way, a dying worker can't kill any started cursors by the parent process (hopefully) and new workers are forked from the fork manager process.

The good thing is it works that way as far as my tests are done. Will publish a new release as soon as I pushed the fixes. But: this requires another dependency (php compiled with --enable-sysvmsg) and it is just not the nice MongoDB only solution I intended to write.

[jmikola]

If you're looking for a MongoDB-only solution, I do have an example of implementing a job queue using findAndModify in this presentation. It's a few years old, but the principles still apply. This would entail using a collection as the queue and having each of the forked workers execute findAndModify commands to pluck a job from that queue. The atomic nature of the command ensures that two workers won't ever "check out" the same job and likewise that each worker can mark it "in progress" or "done" (or simply deleted) as needed.

I believe this is the same approach used by Bernard (a PHP job queue library); however, it doesn't look like the library has been updated to use the new mongodb extension. There may be an opportunity for you to contribute (the MongoDB driver is quite small) if the library itself looks suitable for your use case.

I'm not sure how many workers you're spawning, but I don't think you'd run into any server-side connection limits by having each worker open database connections. That may be worth a look before refactoring to utilize --enable-sysvmsg APIs.

Edit: Regarding Bernard, it looks like there's a pending PR to update it for the new MongoDB driveR: bernardphp/bernard#381

[raffis]

If you're looking for a MongoDB-only solution, I do have an example of implementing a job queue using findAndModify in this presentation. It's a few years old, but the principles still apply. This would entail using a collection as the queue and having each of the forked workers execute findAndModify commands to pluck a job from that queue. The atomic nature of the command ensures that two workers won't ever "check out" the same job and likewise that each worker can mark it "in progress" or "done" (or simply deleted) as needed.

You're missing the point of forking. The library features quite a lot of stuff. Everything you mentioned here is implemented. And yes I do need forking, one can also deploy blocking forever jobs like small daemons as tasks, and those would block the whole process forever. Also I want to have different options on how to run my tasks.

I believe this is the same approach used by Bernard (a PHP job queue library); however, it doesn't look like the library has been updated to use the new mongodb extension. There may be an opportunity for you to contribute (the MongoDB driver is quite small) if the library itself looks suitable for your use case.

I did not know this library exists.
But no, it does not, not for all features I need as far as I can see.
See https://github.com/gyselroth/mongodb-php-task-scheduler/blob/dev/README.md

I'm not sure how many workers you're spawning, but I don't think you'd run into any server-side connection limits by having each worker open database connections. That may be worth a look before refactoring to utilize --enable-sysvmsg APIs.

Well, too late :) gyselroth/php-task-scheduler@69ef89b

I just implemented this to get away of the fact that I can't spawn workers within a MongoDB cursor. Sadly it's quite a big deal since I need a local message queue besides MongoDB. But it works that way.
It is now a combination of MongoDB (Since I want clustered load balancing) and a systemv message queue to communicate between forks on cluster nodes.

It is probably the best way til the driver maybe support ignoring cursors to be garbage collected. Another option would have been to destroy a cursor in the main process before spawn a worker and the restart the cursor, but this kinda misses the point.

Well looks like it does work now. Just with a bit more complexity.
v3 is nearly stable now.

[jmikola]

Closing this out, as I don't think there's anything more to discuss. Please follow PHPC-1274 for updates. If we do end up requesting changes in libmongoc, a C driver ticket will end up linked to that issue as well.