Disable persistent connection

[dercoder]

Hello,

i am creating some threads using Parallel but after close this threads, the connections are still open and i have no idea how to close them.

After a while a "Too many open files" exception will be thrown. I know i can increase them but that will not solve the problem.

lsof -p 22489

php     22489 ec2-user   11u  IPv4 299183318       0t0    TCP localhost:41484->localhost:27017 (ESTABLISHED)
php     22489 ec2-user   12u  IPv4 299187424       0t0    TCP localhost:43650->localhost:27017 (ESTABLISHED)
php     22489 ec2-user   13u  IPv4 299187500       0t0    TCP localhost:43834->localhost:27017 (ESTABLISHED)
php     22489 ec2-user   14u  IPv4 299182643       0t0    TCP localhost:41910->localhost:27017 (ESTABLISHED)
php     22489 ec2-user   15u  IPv4 299189548       0t0    TCP localhost:45064->localhost:27017 (ESTABLISHED)
php     22489 ec2-user   16u  IPv4 299186431       0t0    TCP localhost:44190->localhost:27017 (ESTABLISHED)
php     22489 ec2-user   17u  IPv4 299187521       0t0    TCP localhost:43986->localhost:27017 (ESTABLISHED)
php     22489 ec2-user   18u  IPv4 299183598       0t0    TCP localhost:41918->localhost:27017 (ESTABLISHED)
php     22489 ec2-user   19u  IPv4 299187621       0t0    TCP localhost:44378->localhost:27017 (ESTABLISHED)
....

So how can I disable the persistent mode like on MySQL?
or how can i close the connection?

I am using

PHP 7.3.19 (cli) (built: Jun 13 2020 12:55:28) ( ZTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.19, Copyright (c) 1998-2018 Zend Technologies

MongoDB support => enabled
MongoDB extension version => 1.7.4
MongoDB extension stability => stable
libbson bundled version => 1.16.2
libmongoc bundled version => 1.16.2
libmongoc SSL => enabled
libmongoc SSL library => OpenSSL
libmongoc crypto => enabled
libmongoc crypto library => libcrypto
libmongoc crypto system profile => disabled
libmongoc SASL => enabled
libmongoc ICU => disabled
libmongoc compression => enabled
libmongoc compression snappy => disabled
libmongoc compression zlib => enabled
libmongocrypt bundled version => 1.0.3
libmongocrypt crypto => enabled
libmongocrypt crypto library => libcrypto

Directive => Local Value => Master Value
mongodb.debug => no value => no value

[jmikola]

The extension has no API for disabling connection persistence. This is mainly because, compared to a typical SQL driver, this extension has more networking overhead for things like topology discovery and socket monitoring. This has been asked a handful of times over the past few years, and I most recently provided some advice on manually patching the driver to disable the logic to persist internal libmongoc client objects (which own sockets). See: mongodb/mongo-php-driver#969 (comment). The reason we've not introduced this in a public API is for fear that it would be misused and ultimately result in degraded performance. That said, each request adds more weight to the idea and I'd be willing to revisit the discussion with @alcaeus once he returns in July.

Side note: The most recent driver change that may pertain to closing sockets was PHPC-1274 in 1.7.0, which added logic to clear connections if the driver detects as PID change (e.g. after forking). This was meant to address cases where users had created MongoDB\Driver\Manager objects in a parent process, which later forked or spawned threads.

Based on the lsof output you shared, it looks like all of those sockets are owned by the same PID (22489). I would be curious to learn why the driver isn't already re-using the same libmongoc client, since we cache and re-use those within the same PID by each Manager's constructor args (e.g. two Managers constructed with the same args in the same process will share the same libmongoc client and its sockets).

If possible, it'd be helpful for you to share a proof-of-concept application that uses the MongoDB driver/library with Parallel (ideally just a single script and optional composer.json file), which would allow me to reproduce what you're seeing and learn how Parallel interacts with another extension's lifecycle functions (e.g. global, module, and request initializers). From there, we may be able to come up with a fix to ensure persistent connections are closed when a Parallel.

[jmikola]

@remicollet @krakjoe: Short of reading the Parallel extension's source code, is there any documentation that might explain how it interacts with other extensions' lifecycle functions?

Specifically, I'd want to determine why the HashTable we initialize in GINIT is not re-used, assuming the OP is constructing MongoDB\Driver\Manager objects with the same arguments and from the same PID.

[dercoder]

I understand your fear but this should be developers decision how to use it. Even the wrong way.
MySQL is a good example. They support persistent connections but the developer can do a lot of mistakes: InnoDB transactions + persistent connections.

Maybe its possible to add a feature to disable this persisent feature by php.ini configuration?
I think a "close connection" function would be also helpful in my case.

The same issue happend with guzzle but here its possible to add a configuration to close the connection automatically.

For me its like this metaphor:
I would like to eat my soup with chopsticks.
In the most cases, it does not make any sense.
In my case, i want to eat a ramen soup ;)

I created a small demo:
https://github.com/dercoder/php-mongodb-parallel-issue

You will see that each loop, the size of mongodb connections will increase.

[jmikola]

I've opened PHPC-1645 to track the discussion of disabling persistence behavior, so feel free to watch that issue for any updates.

Thank you for providing the demo repository. I'll look into that separately when I get a chance.

[dercoder]

Great thx.

You are welcome.

[brendandebeasi]

This is an issue for a project I am working on as well. Is anybody aware of a workaround for the time being?

[Blount]

I confirm this problem.
In parallel extension, many connection keep opened.
I don't understand why manually close connection is a problem.

If I understand, when I use MongoDB in PHP FPM, your extension use the same socket between HTTP request ?

I join @dercoder, the developper must have the possibility to control the life of her code. The documentation are here to inform and if I read that closing connection can slowing my program so I close it only when it necessary.

[jmikola]

Is anybody aware of a workaround for the time being?

@brendandebeasi In #756 (comment), I alluded to mongodb/mongo-php-driver#969 (comment), which provided instructions for manually patching an earlier version of driver to completely disable persistence. Adapting that to the current version is left as an exercise to a reader comfortable with manually modifying/compiling the driver.

I don't understand why manually close connection is a problem. If I understand, when I use MongoDB in PHP FPM, your extension use the same socket between HTTP request ?

This is discussed in Connection handling and persistence within the driver's PHP.net documentation. The driver persists the entire libmongoc client object (including its sockets) in order to minimize the overhead of rediscovering (and reconnecting to) the MongoDB cluster on each request.

the developper must have the possibility to control the life of her code.

PHPC-1645 is currently being worked on by @alcaeus in mongodb/mongo-php-driver#1165. I've been away for a couple of months, but IIRC we discovered that providing the option to disable persistence will also require changes to the driver's command monitoring internals, which is part of the reason this is taking more time to implement. Rest assured, it is on the current roadmap.

[Blount]

Hey.
I make some tests.
When I use Parallel extension in CLI mode, the children have the same PID that the main process. We can see with "netstat -ap | grep 27017" that all MongoDB connection are the ESTABLISHED status and the PID are same for all threads.
When thread ended, connection stay with the same status et new connection are created for new threads.
When the main process die, all connection stay with the new status (TIME_WAIT) without process attached.
After some seconds, connections are clean.

If I understand, the driver use same connection for one process. So, I think the driver are not compatible with multi threads process because the main process and hers children doesn't share resource. PDO have the same problem, we can't use the same MySQL connection with all children and we need to create a new connection in all children. PDO has no "close" method to manually close connection, connection is closed when object lost all references or if thread is ended.

[jmikola]

If I understand, the driver use same connection for one process. So, I think the driver are not compatible with multi threads process because the main process and hers children doesn't share resource.

The driver's rules for re-using sockets (via libmongoc clients) within the same process is discussed in Connection handling and persistence. The decision to re-use a socket within the same process depends on the arguments used to construct the Manager (or Client in the library API).

In the case of a multi-threaded application, it's quite possible to create a Manager or Client in a parent thread and share it among child threads. Doing so will likely lead to unexpected errors since we utilize libmongoc's single-threaded API internally. I expect a multi-threaded app would likely experience this as a network error due to either writing corrupt data (e.g. two threads attempting to write to the same socket concurrently) or reading server replies out of order (e.g. the server responds to one thread's operation but it's picked up by a different thread).

The issues above are also possible when a Manager/Client is shared between forked child processes. In the case of forking, the driver does have logic in place to detect PID changes and help avoid some concurrency issues (see: PHPC-1274); however, there are no such protections for threading.

PDO have the same problem, we can't use the same MySQL connection with all children and we need to create a new connection in all children. PDO has no "close" method to manually close connection, connection is closed when object lost all references or if thread is ended.

The changes we're implementing in mongodb/mongo-php-driver#1187 (this replaces the previous PR I linked) will allow you to utilize the driver in the same manner as the PDO extension by constructing a Manager/Client with a disableClientPersistence option to ensure its libmongoc client is used exclusively by that PHP object and destroyed along with the PHP object. This will still require each thread to create its own Manager/Client, but you'll be able to guarantee that sockets are not inadvertently shared among threads.

Libmongoc does have an API for threaded applications, where server monitoring uses dedicated sockets in a background thread and application threads check out clients/sockets from a pool; however, that would likely require a completely separate PHP extension (and perhaps library). We have no plans/resources to do so at this time.

[jmikola]

FYI: mongodb/mongo-php-driver#1187 has been merged in case anyone in this thread would like to manually compile the extension and provide some feedback. The PR introduces a disableClientPersistence boolean driver option:

$manager = new MongoDB\Driver\Manager(URI, [], ['disableClientPersistence' => true]);

This defaults to false, which preserves the existing behavior. Specifying true will force the Manager to create a new libmongoc client, which will then be destroyed (i.e. sockets closed) when the Manager object is freed (e.g. garbage collection, end of PHP request).