Memory leak with yajl2_c backend

[mhugo]

Hi,

Thank you very much for the ijson library.

I think I may have found a memory leak when using the yajl2_cbackend. I've reused a code similar to the one found in a previous issue:

https://gist.github.com/mhugo/dec469223e578ea7ec94946edcd43e6f

With yajl2_c:

using backend yajl2_c
using ijson version 3.1.1
starting memory usage: 11.736 MB
spent time: 5.27
memory usage after ijson calls: 204.432 MB
memory usage after garbage collection: 204.432 MB

With yajl2_cffi:

using backend yajl2_cffi
using ijson version 3.1.1
starting memory usage: 18.556 MB
spent time: 16.25
memory usage after ijson calls: 18.556 MB
memory usage after garbage collection: 18.568 MB

[mhugo]

Tested with

• python 2.7.13 on linux
• python 3.5.3 on linux

[rtobar]

@mhugo thanks for spotting this. I indeed managed to reproduce the error in all versions since 3.0. The rate of growth is thankfully very slow, or at least much slower than memory leaks found due to the parsing itself -- so probably is a memory leak on the generator objects themselves.

[rtobar]

@mhugo could you try the memory-leak branch? I think I found all the tiny leaks, but I'd feel more comfortable with more testing.

Note that this affected both sync and async generators (but not the coros). There were also a few different problems in different places, so all of basic_parse, parse, kvitems and items were affected at different degrees. I expanded on your test script to test all of these (only construction though, which is what I think was leaking):

https://gist.github.com/rtobar/4ee73e673b999d40c7c13381f1bf5b51

[mhugo]

@rtobar Thank you very much for your fast answer ! I'll have a look at your branch and let you know.

[mhugo]

I confirm the memory-leak branch gives better results, thanks ! Using my initial test (I locally changed the version to 3.1.2 to make sure the right version was executed):

(I am on debian 9, sorry for the old python versions)

With yajl2_c, python 2.7.13

using backend yajl2_c
using ijson version 3.1.2
starting memory usage: 19.96 MB
spent time: 3.54
memory usage after ijson calls: 31.924 MB
memory usage after garbage collection: 31.924 MB

With yajl_cffi, python 2.7.13:

using backend yajl2_cffi
using ijson version 3.1.2
starting memory usage: 17.412 MB
spent time: 16.17
memory usage after ijson calls: 30.1 MB
memory usage after garbage collection: 30.1 MB

With yajl2_c, python 3.5.3

using backend yajl2_c
using ijson version 3.1.2
starting memory usage: 11.588 MB
spent time: 4.77
memory usage after ijson calls: 11.588 MB
memory usage after garbage collection: 11.588 MB

With yajl_cffi, python 3.5.3:

using backend yajl2_cffi
using ijson version 3.1.2
starting memory usage: 18.604 MB
spent time: 16.04
memory usage after ijson calls: 18.604 MB
memory usage after garbage collection: 18.62 MB

So there is still a small leak with python 2 (confirmed by increasing the number of calls), and it happens whatever the backend ... but yeah I know Python 2 ...
With Python 3 this is all good !

Side note: I've tried your test script. It works well on the first function (and confirms there is no memory leak). However it generates infinite exceptions on *_coro functions:

Exception ignored in: <generator object basic_parse_basecoro at 0x7f49e2510f10>
Traceback (most recent call last):
  File "/opt/venv/lib/python3.5/site-packages/ijson-3.1.2-py3.5-linux-x86_64.egg/ijson/backends/yajl2_cffi.py", line 225, in basic_parse_basecoro
    yajl_parse(handle, buffer)
  File "/opt/venv/lib/python3.5/site-packages/ijson-3.1.2-py3.5-linux-x86_64.egg/ijson/backends/yajl2_cffi.py", line 196, in yajl_parse
    raise exception(error)
ijson.common.IncompleteJSONError: parse error: premature EOF
                                       
                     (right here) ------^

[rtobar]

@mhugo Thanks for reporting back. Sounds a bit suspicious that both backends leak in 2, so that could be yet something else... I might have some time during the weekend to experiment, but otherwise this will have to wait until next week.

I wrote the modified code in my gist in a hurry and tried it only with 3.8. With some dedication I could probably add something like that to unit test suite though, which has been on my mind for a while.

[rtobar]

Further testing shows that the python 2.7 memory growth is independent from ijson.

https://gist.github.com/rtobar/b1a83a4712f810e6aa227cff988609fc

Running the code above gives me:

Python 2.7.18rc1 (default, Apr  7 2020, 12:05:55)
[GCC 9.3.0]
Memory usage at start: 8.12 MB
spent time: 0.25
Memory usage after loop: 135.36 MB
Memory usage after GC: 135.36 MB

Python 3.8.2 (default, Jul 16 2020, 14:00:26)
[GCC 9.3.0]
Memory usage at start: 9.52 MB
spent time: 0.15
Memory usage after loop: 9.52 MB
Memory usage after GC: 9.52 MB

This seems like a deep, core python behavior that changed in 3. This SO answer seems to hit the nail. Also note that using xrange instead of range in 2 also makes the problem go away, both in this latest gist and in the original one testing ijson.

Let me know if his makes sense and if you can see the same results. If you do then we can declare this solved.

[mhugo]

Thank you very much for your investigation !

Also note that using xrange instead of range in 2 also makes the problem go away

I confirm that (and learned something). So this has nothing to do with ijson. Problem solved ! :-)

Thanks again for your fast reaction. Do you plan a patch release ? Or do you have more to include before a release ?

[rtobar]

@mhugo I leaned something new too, so that's too of us :-)

Yes, I'll make a patch release containing this fix and will release it to PyPI. I'll update this ticket once that's done.

[rtobar]

A new 3.1.2 version of ijson containing the fix for this problem is now available on PyPI.

[LuisAlejandro]

Thank you very much for this effort @rtobar! ijson is a great piece of software and has saved a lot of memory and time for me.

[rtobar]

@LuisAlejandro I'm glad this package has been helpful :-). And we both have @isagalaev to thank, as he created it and came up with an intuitive design that makes it easy to use.

[mhugo]

@rtobar thanks again for your quick reaction and the fix release !