Fix memory leak in _cbson._element_to_dict

[messa]

The memory leak happens when I use document_class=RawBSONDocument, because its __inflated property calls bson._iterate_elements that in turn calls _element_to_dict. The memory leak is present only in the C extension; if it is disabled everything becomes OK.

When the default document_class is used (i.e. dict) there is no memory leak, because that uses different _cbson functions where the ref counts are handled properly (_elements_to_dict in _cbsonmodule.c).

[behackett]

Thanks for the bug report and patch, I've done a bit of triage and opened https://jira.mongodb.org/browse/PYTHON-1171.

[behackett]

@messa, can you change your PR to switch from OO to NN in the Py_BuildValue call instead of moving the Py_DECREF calls? That should be slightly more efficient, avoiding two calls to Py_INCREF and two calls to Py_DECREF in the common case.

Also, I'm curious why you are inflating the raw documents?

[messa]

Changed and force-pushed (hope that's OK; previous version here).

I'm not inflating the raw documents myself. The low-level cursor batch document is inflated when the result documents are retrieved from it:

mongo-python-driver/pymongo/cursor.py

Line 992 in 8fdb581

documents = cursor['firstBatch']

It's just a "shallow inflate" but it copies all the document contents so when memory leaked it eats memory pretty fast.

[messa]

Btw. my use case for RawBSONDocument:

• right now I've just needed a quick pre-heat script that reads all documents in a collection, without doing anything with them. I just tried raw documents to compare how much faster such script will be so I discovered this issue.
• we have some data processing component implemented in Python with the core parts in C++ via Cython. I'm thinking about using pymongo raw documents so I can move document loading from Python to pure C++ (with C/C++ bson lib) so the Python part just transfers byte blobs from pymongo to Cython/C++ code.

[behackett]

I see, we leak two references for each cursor batch regardless of the application explicitly inflating the individual documents. Sigh. Sadly, coverity doesn't appear to understand Python's reference counting at all and I completely missed this in code review back in 3.2.

Our thinking for this feature was to allow various optional BSON decoders, possibly written in languages other than Python (but with slim python wrappers). python-bsonjs was the first project of that type we wrote. Your use case is very similar.

[behackett]

I've confirmed the problem, and that your patch fixes it. Patch merged to master. Thanks again!

[behackett]

We've release PyMongo 3.3.1, with your fix for this issue.

https://pypi.python.org/pypi/pymongo/3.3.1