Support refreshSession method

[719media]

Hello,
I am writing to request an enhancement to mongodb sessions by adding a method

Much like the relationship between
https://docs.mongodb.com/manual/reference/command/endSessions/
and
https://www.php.net/manual/en/mongodb-driver-session.endsession.php,

It would be nice if there was a method for
https://docs.mongodb.com/manual/reference/command/refreshSessions/
in the mongodb-php-driver to be able to refresh an individual session.

Thank you for your consideration.

[719media]

Hm, I see I was probably mistaken, and the endsession php method is probably based off of the session object itself, as documented loosely here: https://docs.mongodb.com/manual/reference/method/Session/#mongodb-method-Session.endSession

So, I suppose this ticket can be closed, as the php driver is probably just mirroring the session object. It's annoying, because trying to get behavior for allowing a given query to exceed the default 10 min idle timeout is really quite more difficult than I was expecting it to be...

[jmikola]

the endsession php method is probably based off of the session object itself, as documented loosely here: https://docs.mongodb.com/manual/reference/method/Session/#mongodb-method-Session.endSession

To clarify, drivers have their own specifications independent of the MongoDB shell. Our session API is documented by the Sessions spec.

Looking through JIRA, I dug up an old ticket from 2019 (SPEC-1268) where we proposed adding a refreshSessions helper but ultimately decided against doing so and resolved it as "Won't Fix". The ticket is private, but I can recap the conclusions here:

• Any operation using the session should refresh the session's lifetime. Originally, that only applied to authenticated commands but since MongoDB 4.0.7 (SERVER-39232) all commands should be covered. In practice, this means you can execute a ping command during a long-running cursor iteration routine and ensure the session (and cursor) stays alive for future getMore commands.
• Session::getLogicalSessionId() provides the session ID necessary to call refreshSessions directly.
• While Session::endSession is also an uncommon use case, it is not merely a wrapper for the the endSessions command (see: mongoc_client_session_destroy for more info).

There is another open ticket (DRIVERS-1602) discussing whether the driver or server should automate refreshing of sessions associated with long-running cursors.

As of MongoDB 5.0 (and 4.4.8 per a backport), cursors created with a session implicitly get noCursorTimeout enabled to ensure the cursor will not expire before the session. That is certainly relevant, as prior to that change executing refreshSessions alone would have been insufficient unless users explicitly specified noCursorTimeout.

To summarize:

• It's unlikely drivers will add an explicit helper for the refreshSessions command. The suggested workaround would be to periodically execute any command (e.g. ping) during long-running iteration or call refreshSessions using the ID returned by Session::getLogicalSessionId(). In both cases, you'll want to ensure your long-running query uses an explicit session so you have access to it to run your periodic command (be it ping or refreshSessions) with the same session.
• Feel free to subscribe to (DRIVERS-1602)) to get updates on whether automatic refresh behavior is designed and implemented, but there's currently not timeline for that.

[719media]

Wow thanks for the detailed response.

In my scenario, I am using aggregate(), which appears to have no option for noCursorTimeout (contrast https://docs.mongodb.com/php-library/current/reference/method/MongoDBCollection-aggregate/ with https://docs.mongodb.com/php-library/current/reference/method/MongoDBCollection-find/ to see this).

So, if my desire it to avoid the cursor timeout during an aggregate using mongodb 5.0, then currently I MUST:

1. start a session (to get the noCursorTimeout behavior)
2. periodically ping during iteration to avoid sessionTimeout (obviously this would assume that a single "loop" of the iteration wouldn't exceed the sessionTimeout, that would be a separate issue)

Makes sense. Although, can you clarify if the automatic refresh behavior that MAY be implemented in the future (DRIVERS-1602) is supposed to be something that happens on implicit sessions as well? Or just explicit sessions? In other words, is DRIVERS-1602 just supposed to make it so I only would use step 1 (start a session) and not step 2?

[jmikola]

@ShaneHarvey reminded me of another possible work-around: lowering the default batchSize would ensure getMore requests happen more frequently. This applies to both find and aggregate commands, so that's definitely worth a look.

Can you clarify if the automatic refresh behavior that MAY be implemented in the future (DRIVERS-1602) is supposed to be something that happens on implicit sessions as well? Or just explicit sessions?

I don't see any reason that couldn't apply to all sessions. The notion of implicit/explicit is purely a client-side concept. From the server's perspective there is no difference as everything is considered a logical session. Implicit sessions exist mainly to ensure that each operation issued by the driver is associated with a logical session ID, which allows operations to be traced server-side and also support features such as retryable writes. Explicit vs implicit sessions in the session spec goes into more detail about that.

[719media]

Thanks. One clarifying note. If it is required to start an explicit session before running an aggregate in order to avoid cursor timeout (by means of the session implicitly using noCursorTimeout in the aggregate), can you tell me why this behavior does NOT happen with IMPLICIT sessions already then?
I guess I'm a little confused as to the understanding of "starting a session will use noCursorTimeout" and the fact that every driver operation actually still uses an "implicit" session behind the scenes... as I don't understand then why these implicit sessions are also not utilizing this noCursorTimeout option. I'm missing some basic understanding somewhere...

[jmikola]

If it is required to start an explicit session before running an aggregate in order to avoid cursor timeout (by means of the session implicitly using noCursorTimeout in the aggregate), can you tell me why this behavior does NOT happen with IMPLICIT sessions already then?

As of MongoDB 4.4.8+, any cursor associated with a session should get noCursorTimeout automatically applied to it (per SERVER-6036). No matter whether the operation uses an explicit or implicit session, the find or aggregate command will include an lsid field. That will denote the session on the cursor yielded by the command, and should be all the cursor cares about when deciding to apply noCursorTimeout. Apologies if that wasn't clear from this comment.

Offhand, I don't know what would happen if an outgoing find or aggregate omitted an lsid field; however, that cannot happen in any driver that implements implicit sessions.

don't understand then why these implicit sessions are also not utilizing this noCursorTimeout option. I'm missing some basic understanding somewhere...

Note that the noCursorTimeout option and the session lifetime are entirely separate concepts. The default lifetime for an idle cursor is ten minutes, and the noCursorTimeout option only disables that. Sessions have their own idle timeout, which defaults to 30 minutes (see: localLogicalSessionTimeoutMinutes). If either of these timeouts are hit, the cursor is terminated on the server side. This is discussed in more detail in Session Idle Timeout within the find command documentation, but

If a session is killed, all operations (including cursors) associated with that session are also killed, and that is the reason for the aforementioned tickets proposing the driver or server automatically refresh sessions. In the meantime, issuing any command (e.g. ping, iterating the cursor such that a getMore is issued) will also ensure the session is refreshed.

Similarly, if the cursor didn't have noCursorTimeout applied, any getMore on the cursor would be sufficient to extend its lifetime. Before sessions were implemented in MongoDB 3.6, it was quite possible to accumulate zombie cursors on the server if a driver created them with noCursorTimeout and abandoned the connection. For more context on that, see this old slide deck from 2017: https://speakerdeck.com/jmikola/its-10pm-do-you-know-where-your-writes-are?slide=61

The only reason I suggest using an explicit session is that you can then use that for both the find or aggregate operation and later to issue your own ping command. If the cursor happened to use an implicit session, the application wouldn't have any way to know what session was being used behind the scenes (short of command monitoring, but that'd be a needlessly complicated approach).

[719media]

Note that the noCursorTimeout option and the session lifetime are entirely separate concepts.

Yep, totally clear on that, just to confirm.

As of MongoDB 4.4.8+, any cursor associated with a session should get noCursorTimeout automatically applied to it (per SERVER-6036). No matter whether the operation uses an explicit or implicit session, the find or aggregate command will include an lsid field.

OK, so following that explanation, the noCursorTimeout option in the mongo php library (for example, see https://docs.mongodb.com/php-library/current/reference/method/MongoDBCollection-find/) really only has application to mongo databases prior to 4.4.8?

Because from 4.4.8+, the find() command would either be using an implicit session, or an explicit session (via the session option in that same command, as an example), either of which would be basically issuing the noCursorTimeout: true option to the database. Is that correct?

[jmikola]

Because from 4.4.8+, the find() command would either be using an implicit session, or an explicit session (via the session option in that same command, as an example), either of which would be basically issuing the noCursorTimeout: true option to the database. Is that correct?

I believe that is correct. I've opened DOCS-15181 to suggest that the server manual clarify this in the find command documentation. Depending on how that issue is triaged, I will also suggest that drivers update their downstream documentation for Collection::find() methods.

[719media]

OK cool. I think this issue can be closed then! Thank you