Cursor not found in GridFS/StreamWrapper.php

[fastmanu]

Is there a way to pass nocursorTimout option (like on the cursor object) while getting a download stream from a gridfs bucket ? Asking this because i faced following error today, that makes me think about a cursor timeout :
Warning: MongoDB\Driver\Exception\RuntimeException: Cursor not found, cursor id: 245969474954 in /code/vendor/mongodb/mongodb/src/GridFS/StreamWrapper.php on line 134

Here's the code that triggered the exception :

$grid = $con->dbbame->selectGridFSBucket();
$id_file = new MongoDB\BSON\ObjectID($id_file);
return $grid->openDownloadStream($id_file);

returned ressource is then used in fgetcsv...

[jmikola]

Per Closure of Inactive Cursors in the MongoDB manual, the server closes a cursor after 10 minutes of inactivity. In this case, "activity" will be the initial query and any subsequent getMore operations to return additional batches of documents from the fs.chunks collection. Per Cursor Batches, the first batch defaults to 101 documents and subsequent batches may be up to 16MB in size.

Since the PHP library keeps a cursor for fs.chunks in the stream object, it could become inactive if you did not read 16MB worth of data from the stream within 10 minutes. Can you confirm whether your code may have satisfied that condition while using fgetcsv() to read the GridFS file? If so, this seems a like a legitimate use case we should address in the GridFS specification by allowing the openDownloadStream() and related methods to support a noCursorTimeout option.

The specification currently only supports a noCursorTimeout option on the Bucket::find() method, which queries the fs.files collection and is unrelated to reading the GridFS file stream.

[fastmanu]

"while fgetcsv" was satisified during 18 000 csv lines, it then started throwing mentionned errors...

here is a code snippet, where $this->getFile() refers to the code previously mentionned :

if (($handle = $this->getFile($id_file)) !== FALSE) {
			
	// loop through the file line-by-line
	while(($data = fgetcsv($handle, 2000, $delimiter, $enclosure)) !== false) {
		[ultra crazy stuff like inserts && updates]
	}
}

[jmikola]

"while fgetcsv" was satisified during 18 000 csv lines, it then started throwing some errors...

Is it likely that more than ten minutes elapsed between the time the while loop began and the first "Cursor not found" exception was thrown?

Without knowing the contents of the GridFS file being read, it's hard to saw how 18,000 CVS lines translates into megabytes and batches on the fs.chunks cursor. Can you estimate how many megabytes would have been read between the first while iteration and the "Cursor not found" exception?

I'm trying to establish whether it's reasonable that you actually hit the cursor inactivity limit, which would make a strong case for supporting a noCursorTimeout option for the stream cursors. That said, getting a "Cursor not found" exception under some other circumstances (e.g. iterating for only 90 seconds) would hint at a bug of some sort.

[fastmanu]

More than 10 min elapsed in the loop, i confirm that.

Csv file in grid is 45443817 bytes or something arround 43,3mb.

wc -l filename returns 46624 lines and code started throwing errors at line 18019.

[jmikola]

Thanks for confirming. This should be enough to conclude there's no bug and this is just a feature request. I'll follow up once I have more feedback from the other driver developers about how we can address this in the specification.

If this is currently blocking your work and you're willing to modify the library directly, modifying MongoDB\GridFS\CollectionWrapper::findChunksByFileId() to specify the noCursorTimeout option on the internal cursor should work around the issue.

[fastmanu]

No worries, sorry for having filled the issue as "bug" since it wasn't one. Thanks for your help Jeremy ;)

[jmikola]

No worries. Our internal spec tickets in JIRA are private, so I've opened PHPLIB-274 as a public tracking ticket. Feel free to follow that for updates.