PyCall use-after free

[Keno]

Relating to the problem we had yesterday. Running

using PyCall
@pyimport pylab as plt
plt.plot(rand(10))
plt.plot(rand(10))
plt.plot(rand(10))
while true
    plt.gcf()["canvas"][:print_figure](IOBuffer(),format="png")
end

under valgrind gives https://gist.github.com/loladiro/6134837 which has a bunch of instances of use-after-free even those that weren't allocated by Pycall. Are you at any point (in the I/O code perhaps) passing raw pointers to python?

[Keno]

Here are the interesting bits immediately le leading up to the segfault:

==17778== Invalid read of size 8
==17778==    at 0x2749EA75: Py_DecRef (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==17778==    by 0x410B4D3: ???
==17778==    by 0x53196D4: jl_apply (julia.h:1017)
==17778==    by 0x531DE93: jl_apply_generic (gf.c:1402)
==17778==    by 0x5394E84: jl_apply (julia.h:1017)
==17778==    by 0x539574E: run_finalizer (gc.c:290)
==17778==    by 0x53958E8: run_finalizers (gc.c:309)
==17778==    by 0x539750F: jl_gc_collect (gc.c:926)
==17778==    by 0x5395A66: alloc_big (gc.c:342)
==17778==    by 0x53976BC: alloc_2w (gc.c:987)
==17778==    by 0x5380655: jl_box_int64 (alloc.c:747)
==17778==    by 0x410B1B5: ???
==17778==  Address 0x3c218a30 is not stack'd, malloc'd or (recently) free'd
==17778== 
==17778== Conditional jump or move depends on uninitialised value(s)
==17778==    at 0x2749EA79: Py_DecRef (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==17778==    by 0x410B4D3: ???
==17778==    by 0x53196D4: jl_apply (julia.h:1017)
==17778==    by 0x531DE93: jl_apply_generic (gf.c:1402)
==17778==    by 0x5394E84: jl_apply (julia.h:1017)
==17778==    by 0x539574E: run_finalizer (gc.c:290)
==17778==    by 0x53958E8: run_finalizers (gc.c:309)
==17778==    by 0x539750F: jl_gc_collect (gc.c:926)
==17778==    by 0x5395A66: alloc_big (gc.c:342)
==17778==    by 0x53976BC: alloc_2w (gc.c:987)
==17778==    by 0x5380655: jl_box_int64 (alloc.c:747)
==17778==    by 0x410B1B5: ???
==17778==  Uninitialised value was created by a heap allocation
==17778==    at 0x4C2CD7B: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==17778==    by 0x5395AC7: alloc_big (gc.c:348)
==17778==    by 0x5397685: allocobj (gc.c:977)
==17778==    by 0x5386438: _new_array (array.c:80)
==17778==    by 0x538718B: jl_alloc_array_1d (array.c:297)
==17778==    by 0x4058939: ???
==17778==    by 0x409F564: ???
==17778==    by 0x4105E8F: ???
==17778==    by 0x53196D4: jl_apply (julia.h:1017)
==17778==    by 0x531DE93: jl_apply_generic (gf.c:1402)
==17778==    by 0x4104A48: ???
==17778==    by 0x53223D9: jl_apply (julia.h:1017)
==17778== 
==17778== Invalid read of size 8
==17778==    at 0x2749EA80: Py_DecRef (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==17778==    by 0x410B4D3: ???
==17778==    by 0x53196D4: jl_apply (julia.h:1017)
==17778==    by 0x531DE93: jl_apply_generic (gf.c:1402)
==17778==    by 0x5394E84: jl_apply (julia.h:1017)
==17778==    by 0x539574E: run_finalizer (gc.c:290)
==17778==    by 0x53958E8: run_finalizers (gc.c:309)
==17778==    by 0x539750F: jl_gc_collect (gc.c:926)
==17778==    by 0x5395A66: alloc_big (gc.c:342)
==17778==    by 0x53976BC: alloc_2w (gc.c:987)
==17778==    by 0x5380655: jl_box_int64 (alloc.c:747)
==17778==    by 0x410B1B5: ???
==17778==  Address 0x3a4c7398 is not stack'd, malloc'd or (recently) free'd
==17778== 
==17778== Invalid read of size 8
==17778==    at 0x2749EA84: Py_DecRef (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==17778==    by 0x410B4D3: ???
==17778==    by 0x53196D4: jl_apply (julia.h:1017)
==17778==    by 0x531DE93: jl_apply_generic (gf.c:1402)
==17778==    by 0x5394E84: jl_apply (julia.h:1017)
==17778==    by 0x539574E: run_finalizer (gc.c:290)
==17778==    by 0x53958E8: run_finalizers (gc.c:309)
==17778==    by 0x539750F: jl_gc_collect (gc.c:926)
==17778==    by 0x5395A66: alloc_big (gc.c:342)
==17778==    by 0x53976BC: alloc_2w (gc.c:987)
==17778==    by 0x5380655: jl_box_int64 (alloc.c:747)
==17778==    by 0x410B1B5: ???
==17778==  Address 0x40 is not stack'd, malloc'd or (recently) free'd
==17778== 
==17778== Jump to the invalid address stated on the next line
==17778==    at 0xFF08EC8348F88948: ???
==17778==    by 0x53196D4: jl_apply (julia.h:1017)
==17778==    by 0x531DE93: jl_apply_generic (gf.c:1402)
==17778==    by 0x5394E84: jl_apply (julia.h:1017)
==17778==    by 0x539574E: run_finalizer (gc.c:290)
==17778==    by 0x53958E8: run_finalizers (gc.c:309)
==17778==    by 0x539750F: jl_gc_collect (gc.c:926)
==17778==    by 0x5395A66: alloc_big (gc.c:342)
==17778==    by 0x53976BC: alloc_2w (gc.c:987)
==17778==    by 0x5380655: jl_box_int64 (alloc.c:747)
==17778==    by 0x410B1B5: ???
==17778==    by 0x53196D4: jl_apply (julia.h:1017)
==17778==  Address 0xff08ec8348f88948 is not stack'd, malloc'd or (recently) free'd
==17778== 
==17778== 
==17778== Process terminating with default action of signal 11 (SIGSEGV)
==17778==  Bad permissions for mapped region at address 0xFF08EC8348F88948
==17778==    at 0xFF08EC8348F88948: ???
==17778==    by 0x53196D4: jl_apply (julia.h:1017)
==17778==    by 0x531DE93: jl_apply_generic (gf.c:1402)
==17778==    by 0x5394E84: jl_apply (julia.h:1017)
==17778==    by 0x539574E: run_finalizer (gc.c:290)
==17778==    by 0x53958E8: run_finalizers (gc.c:309)
==17778==    by 0x539750F: jl_gc_collect (gc.c:926)
==17778==    by 0x5395A66: alloc_big (gc.c:342)
==17778==    by 0x53976BC: alloc_2w (gc.c:987)
==17778==    by 0x5380655: jl_box_int64 (alloc.c:747)
==17778==    by 0x410B1B5: ???
==17778==    by 0x53196D4: jl_apply (julia.h:1017)
==17778== 

[stevengj]

I pass lots of pointers to Python, but I try to be careful to pass pointers only to const things or to things that I keep a reference to in Julia so that they won't be garbage collected until after any Python references to them are garbage collected.

I suspect that I'm forgetting to increment a reference count somewhere, probably in the new IO code or the Vector{Uint8} conversions.

[Keno]

The use after free thing turns out to be normal and python just hopes it's ok. Here's the relevant part with python debug symbols. You should be able to figure out what object is missing the reference count from where it's deallocated:

==28485== 
==28485== Invalid read of size 8
==28485==    at 0x26A9EA75: Py_DecRef (object.c:214)
==28485==    by 0x40EABF3: ???
==28485==    by 0x531DC34: jl_apply (julia.h:1017)
==28485==    by 0x53223F3: jl_apply_generic (gf.c:1402)
==28485==    by 0x53988D4: jl_apply (julia.h:1017)
==28485==    by 0x539919E: run_finalizer (gc.c:290)
==28485==    by 0x5399338: run_finalizers (gc.c:309)
==28485==    by 0x539AF5F: jl_gc_collect (gc.c:926)
==28485==    by 0x53994B6: alloc_big (gc.c:342)
==28485==    by 0x539B10C: alloc_2w (gc.c:987)
==28485==    by 0x53840A5: jl_box_int64 (alloc.c:747)
==28485==    by 0x4108FD5: ???
==28485==  Address 0x37080270 is 0 bytes inside a block of size 8,229 free'd
==28485==    at 0x4C2BA6C: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==28485==    by 0x26A7AFE9: tupledealloc.24990 (tupleobject.c:222)
==28485==    by 0x2699649E: PyObject_CallFunction (abstract.c:2563)
==28485==    by 0x2F5DBC5B: write_png_data(png_struct_def*, unsigned char*, unsigned long) (_png.cpp:79)
==28485==    by 0x2F8018BC: png_write_chunk_data (in /lib/x86_64-linux-gnu/libpng12.so.0.49.0)
==28485==    by 0x2F801A46: png_write_chunk (in /lib/x86_64-linux-gnu/libpng12.so.0.49.0)
==28485==    by 0x2F80210D: ??? (in /lib/x86_64-linux-gnu/libpng12.so.0.49.0)
==28485==    by 0x2F8043EF: ??? (in /lib/x86_64-linux-gnu/libpng12.so.0.49.0)
==28485==    by 0x2F80487A: ??? (in /lib/x86_64-linux-gnu/libpng12.so.0.49.0)
==28485==    by 0x2F808433: png_write_row (in /lib/x86_64-linux-gnu/libpng12.so.0.49.0)
==28485==    by 0x2F808657: png_write_image (in /lib/x86_64-linux-gnu/libpng12.so.0.49.0)
==28485==    by 0x2F5DC23B: _png_module::write_png(Py::Tuple const&) (_png.cpp:229)
==28485== 
==28485== Invalid read of size 8
==28485==    at 0x26A9EA80: Py_DecRef (object.c:214)
==28485==    by 0x40EABF3: ???
==28485==    by 0x531DC34: jl_apply (julia.h:1017)
==28485==    by 0x53223F3: jl_apply_generic (gf.c:1402)
==28485==    by 0x53988D4: jl_apply (julia.h:1017)
==28485==    by 0x539919E: run_finalizer (gc.c:290)
==28485==    by 0x5399338: run_finalizers (gc.c:309)
==28485==    by 0x539AF5F: jl_gc_collect (gc.c:926)
==28485==    by 0x53994B6: alloc_big (gc.c:342)
==28485==    by 0x539B10C: alloc_2w (gc.c:987)
==28485==    by 0x53840A5: jl_box_int64 (alloc.c:747)
==28485==    by 0x4108FD5: ???
==28485==  Address 0x238e5ca8 is 8 bytes inside a block of size 48 free'd
==28485==    at 0x4C2BA6C: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==28485==    by 0x29CE6903: array_dealloc (arrayobject.c:419)
==28485==    by 0x26AB1652: meth_dealloc.19993 (methodobject.c:134)
==28485==    by 0x2696B6ED: PyEval_EvalFrameEx (ceval.c:4055)
==28485==    by 0x26A032F9: PyEval_EvalCodeEx (ceval.c:3253)
==28485==    by 0x26A034A5: function_call.15545 (funcobject.c:526)
==28485==    by 0x26A35D8D: PyObject_Call (abstract.c:2529)
==28485==    by 0x269A506C: instancemethod_call.8833 (classobject.c:2602)
==28485==    by 0x26A35D8D: PyObject_Call (abstract.c:2529)
==28485==    by 0x26A2942F: slot_tp_init.26169 (typeobject.c:5688)
==28485==    by 0x26A390EB: type_call.26146 (typeobject.c:741)
==28485==    by 0x26A35D8D: PyObject_Call (abstract.c:2529)
==28485== 
==28485== Invalid read of size 8
==28485==    at 0x26A9EA84: Py_DecRef (object.c:214)
==28485==    by 0x40EABF3: ???
==28485==    by 0x531DC34: jl_apply (julia.h:1017)
==28485==    by 0x53223F3: jl_apply_generic (gf.c:1402)
==28485==    by 0x53988D4: jl_apply (julia.h:1017)
==28485==    by 0x539919E: run_finalizer (gc.c:290)
==28485==    by 0x5399338: run_finalizers (gc.c:309)
==28485==    by 0x539AF5F: jl_gc_collect (gc.c:926)
==28485==    by 0x53994B6: alloc_big (gc.c:342)
==28485==    by 0x539B10C: alloc_2w (gc.c:987)
==28485==    by 0x53840A5: jl_box_int64 (alloc.c:747)
==28485==    by 0x4108FD5: ???
==28485==  Address 0x32 is not stack'd, malloc'd or (recently) free'd
==28485==