Ubuntu 12.10 x86_64 with v0.8.6 deb installed from repo crashes hard in x264lib.c

[totaam]

Issue migrated from trac ticket # 271

component: platforms | priority: critical | resolution: fixed

2013-02-26 11:05:48: antoine created the issue

---

(gdb) bt
#0  0x00007ffff68f4425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff68f7b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff693239e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ffff693cb96 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007ffff36991a2 in set_decoder_csc_format ()
   from /usr/lib/python2.7/dist-packages/xpra/x264/codec.so
#5  0x00007ffff3690004 in ?? () from /usr/lib/python2.7/dist-packages/xpra/x264/codec.so
#6  0x0000000000497ea4 in PyEval_EvalFrameEx ()

First suspect:

• r2661 - thought the Ubuntu 12.10 binaries should have old-libav.patch applied

[totaam]

2013-02-26 16:06:51: antoine changed status from new to assigned

[totaam]

2013-02-26 16:06:51: antoine commented

---

confirmed as being caused by r2661, this comes from #244 and us moving to newer libav version for some builds - the code changes that have been suggested are not backwards compatible by the looks of things :(

[totaam]

2013-02-26 17:33:55: antoine changed status from assigned to closed

[totaam]

2013-02-26 17:33:55: antoine changed resolution from ** to fixed

[totaam]

2013-02-26 17:33:55: antoine commented

---

A better stacktrace:

*** glibc detected *** /usr/bin/python: free(): invalid pointer: 0x00007f5f70002010 ***
===#### Backtrace:=====
/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f5f8a4a4b96]
xpra/x264/codec.so(do_clean_decoder+0x49)[0x7f5f872c3f39]
xpra/x264/codec.so(set_decoder_csc_format+0x22)[0x7f5f872c41b2]
xpra/x264/codec.so(+0x8004)[0x7f5f872bb004]

Fixed in r2821 + r2823 for trunk and 2824 for v0.8.x.
The API docs do state (taken from the Ubuntu version which has the crash):

/**
 - Allocate an AVFrame and set its fields to default values.  The resulting
 - struct can be deallocated by simply calling av_free().
 -
 - @return An AVFrame filled with default values or NULL on failure.
 - @see avcodec_get_frame_defaults
 -/
AVFrame *avcodec_alloc_frame(void);

But having tried very hard to make it work, this does not seem to be the case.
Despite the same recommendation to use av_free() in 0.8 and earlier also found in #244 - this just does not work.