Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory corruption with 6.10.0 on Ubuntu 18.04 #948

Closed
quozl opened this issue Apr 21, 2018 · 21 comments
Closed

Memory corruption with 6.10.0 on Ubuntu 18.04 #948

quozl opened this issue Apr 21, 2018 · 21 comments

Comments

@quozl
Copy link

quozl commented Apr 21, 2018

G'day, I'm with One Laptop per Child. I'm working a problem on the Sugar Labs activity Music Keyboard, and would like some Csound expert eyes. Music Keyboard uses Csound from Python, as well as GStreamer and GTK+. With the upgrade to CSound 6.10.0 from 6.09, the program fails with a reproducible segmentation fault.

/usr/lib/libcsound64.so.6.0(+0x3f7e3) [0x7f70d65d77e3]
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7f70e9767f20]
/usr/lib/libcsound64.so.6.0(+0x258014) [0x7f70d67f0014]
/usr/lib/libcsound64.so.6.0(+0x272601) [0x7f70d680a601]
/usr/lib/libcsound64.so.6.0(+0x26ee34) [0x7f70d6806e34]
/usr/lib/libcsound64.so.6.0(+0x26eecb) [0x7f70d6806ecb]
/usr/lib/libcsound64.so.6.0(csoundParseOrc+0x2d7) [0x7f70d68126b7]
/usr/lib/libcsound64.so.6.0(+0x27a087) [0x7f70d6812087]
/usr/lib/libcsound64.so.6.0(csoundCompileCsdText+0x5f) [0x7f70d673e37f]
/usr/lib/libcsound64.so.6.0(csoundCompileCsd+0x3b) [0x7f70d673e47b]
/usr/lib/python2.7/dist-packages/_csnd6.x86_64-linux-gnu.so(+0x4bc6e) [0x7f70d6d43c6e]
...
Csound tidy up: Segmentation fault

and gdb backtrace

gdb `which python`
(gdb) run /usr/bin/sugar-activity
(gdb) bt
#0  0x00007fffe8b5c014 in make_label (csound=csound@entry=0x55555669f800, s=s@entry=0x5555569f0680 "__synthetic_300") at ./Engine/csound_orc.lex:586 
#1  0x00007fffe8b76601 in create_synthetic_label (count=300, csound=0x55555669f800) 
    at ../Engine/csound_orc_expressions.c:848 
#2  0x00007fffe8b76601 in expand_if_statement (csound=csound@entry=0x55555669f800, current=current@entry=0x5555568b2410, typeTable=typeTable@entry=0x55555680e470) 
    at ../Engine/csound_orc_expressions.c:1156 
#3  0x00007fffe8b72e34 in verify_tree (csound=csound@entry=0x55555669f800, root=<optimised out>, typeTable=typeTable@entry=0x55555680e470) at ../Engine/csound_orc_semantics.c:1813 
#4  0x00007fffe8b72ecb in verify_tree (csound=csound@entry=0x55555669f800, root=<optimised out>, typeTable=typeTable@entry=0x55555680e470) at ../Engine/csound_orc_semantics.c:1787 
#5  0x00007fffe8b7e6b7 in csoundParseOrc (csound=csound@entry=0x55555669f800, str=str@entry=0x0) at ../Engine/new_orc_parser.c:258 
#6  0x00007fffe8b7e087 in csoundCompileOrcInternal (csound=csound@entry=0x55555669f800, str=str@entry=0x0, async=async@entry=0) at ../Engine/csound_orc_compile.c:1827 
#7  0x00007fffe8aaa37f in csoundCompileCsdText (csound=csound@entry=0x55555669f800, csd_text=<optimised out>) at ../Top/main.c:530 
#8  0x00007fffe8aaa47b in csoundCompileCsd (csound=0x55555669f800, str=<optimised out>) 
    at ../Top/main.c:516 
#9  0x00007fffe90afc6e in Csound::CompileCsd(char const*) (csd=<optimised out>, this=0x5555564fbb80) at .././include/csound.hpp:259 
#10 0x00007fffe90afc6e in _wrap_Csound_CompileCsd(PyObject*, PyObject*) (args=<optimised out>) at python_interfacePYTHON_wrap.cxx:25886 
#11 0x000055555564901a in PyEval_EvalFrameEx () 
...

Other segmentation faults are easily reproduced using python, with the simplest reproducer being

import csnd6
a = csnd6.Csound()
a.Start()
b = csnd6.CsoundPerformanceThread(a)
b.Play()

which generates this repeatable gdb backtrace

#0  0x00007ffff5a36c04 in spoutsf (csound=0x555555c4c3c0) at ../InOut/libsnd.c:79
#1  0x00007ffff5a066fb in kperf_nodebug (csound=0x555555c4c3c0) at ../Top/csound.c:1761
#2  0x00007ffff5a07f2e in csoundPerformKsmps (csound=0x555555c4c3c0)
    at ../Top/csound.c:2113
#3  0x00007ffff5f124b6 in CsoundPerformanceThread::Perform (this=0x555555e3bf40)
    at ../interfaces/csPerfThread.cpp:434
#4  0x00007ffff5f12778 in CsPerfThread_PerformScore::Perform (this=<synthetic pointer>)
    at ../interfaces/csPerfThread.cpp:483
#5  csoundPerformanceThread_ (userData=<optimised out>)
    at ../interfaces/csPerfThread.cpp:500
#6  0x00007ffff77cc6db in start_thread (arg=0x7fffe8505700) at pthread_create.c:463
#7  0x00007ffff7b0588f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

So my questions;

  • is the reproducer above a correct use of your API? i.e. is it something I shall fix?
  • does anyone else reproduce a similar segmentation fault on Ubuntu 18.04? i.e. is this really a distribution, library or compiler problem?
  • what has been corrupted and why?

Thanks!
@jpffitch
Copy link
Contributor

jpffitch commented Apr 21, 2018 via email

@gogins
Copy link
Contributor

gogins commented Apr 21, 2018 via email

@quozl
Copy link
Author

quozl commented Apr 21, 2018

Thanks for replies.

@jpffitch;

Hard to say without a csd file

For the first backtrace, there are three csd files. But as a segmentation fault can also be reproduced by CsoundPerformanceThread.Play() when single stepped under pdb, before the csd files are opened, I don't think the csd files are part of the problem.

For the second backtrace, there's no csd file used. A null performance. File test.wav could not be opened. Changing the current directory to be writable (it is normally not) allows Play() to proceed, and does instead segmentation fault during cleanup from Python REPL;

>>> ^D
/usr/lib/libcsound64.so.6.0(+0x3f7e3) [0x7fe254c3d7e3]
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7fe256c5bf20]
Csound tidy up: Segmentation fault

I'll call that the third backtrace.

A writable directory does not solve the first backtrace.

Have you tried valgrind?

No, good idea. It has a lot to say.

For the first backtrace, wrapping it with valgrind ceases to reproduce, and the program starts without segmentation fault. Here's the log. It doesn't have much in the Csound call stacks.

For the third backtrace during cleanup; see invalid reads inside blocks freed by CsoundPerformanceThread.

@gogins:

What's the CPU architecture?

x86_64. Both Csound and Python are the distributed versions from Ubuntu, so they match architecture.

may wish to switch to the ctcsound interface ... better maintained.

Thanks. As I don't know the size of the work to fix this problem compared to the size of the work to port to the new interface, I'll keep looking at it.

@vlazzarini
Copy link
Member

vlazzarini commented Apr 21, 2018 via email

@quozl
Copy link
Author

quozl commented Apr 23, 2018

My first traceback;

  • is unchanged after porting to ctcsound, porting patch,
  • happens on Ubuntu 18.04 Bionic and Debian Buster with Csound 6.10,
  • does not happen with valgrind, see log,
  • happens during my compileCsd() call for the csd file tamtamorc.csd.

When I try to replicate at the Python prompt without GTK+, or with valgrind, there are problems reported with the csd file;

guest@debian:/usr/share/sugar/activities/MusicKeyboard.activity$ python
Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctcsound
>>> a = ctcsound.Csound()
0dBFS level = 32768.0
--Csound version 6.10 (double samples) 2018-01-27 
[commit: none]
libsndfile-1.0.27
>>> a.compileCsd('ttcommon/Resources/tamtamorc.csd')
STARTING FILE
Creating options
Creating orchestra
closing tag
Creating score
error:  Variable '__synthetic_300' used before defined
Line -1
error:  Variable type for __synthetic_300 could not be determined.
Parsing failed due to syntax errors
Stopping on parser failure
-1
>>>

I don't know enough about the syntax to figure out what is wrong. Any clues? The file is inherited from a much older activity, TamTam.

My second traceback is solved; it was caused by non-writable directory.

My third traceback is solved; it was probably violation of API, fixed by calling stop() on the performance thread and then stop() Csound before exit.

Incidentally, to use import ctcsound on Debian one must install libcsnd-dev and libcsound64-dev, because the library name given to ctypes.CDLL has no version. Not sure if this is a bug. @fsateler?

Also incidentally, @vlazzarini's book Computer Musical Instruments, foundations, design and development, 2017, was very useful for learning the ctcsound API. I wasn't able to find a better documentation source. Listing 2.2 "Using a performance thread" has a small error; self.cs.csound() should be cs.csound().

@vlazzarini
Copy link
Member

vlazzarini commented Apr 23, 2018 •
edited

Ok, it's much clearer now. We need to look into the CSD to see what is going on, but it is the case that the CSD is failing to compile. I think it is something in the parser that is not quite right. I see in valgrind that it is point to the places where we are parsing if statements, and generating labels.

However, on 6.11 (beta), which we're just about to release, it does not seem to fail (at least on MacOs)

MacBook-Pro:debug victor$ python
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 26 2016, 12:10:39) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctcsound
>>> a = ctcsound.Csound()
time resolution is 1000.000 ns
0dBFS level = 32768.0
--Csound version 6.11 beta (double samples) Mar 30 2018 
[commit: ee0954349844e12371e489d189bde9c2444035e2]
libsndfile-1.0.28
>>> a.compileCsd("tamtamorc.csd")
STARTING FILE
Creating options
Creating orchestra
closing tag
Creating score
0

So it might be the case that there was something wrong in 6.10 which we have fixed. @jpffitch @kunstmusik ring any bells? I seem to recall something only vaguely.

@vlazzarini
Copy link
Member

By the way, it is great to see you are updating the OLPC work. We're very happy to support it in any way we can.

@quozl
Copy link
Author

quozl commented Apr 23, 2018

Oh, good. Might I build from git and iterate with git bisect? I'll look for build instructions.

@vlazzarini
Copy link
Member

Building on Linux should be straightforward (see BUILD.md). However, I just tried csound 6.10 on MacOs and it did not fail either. So I am wondering about this. In any case, if you can build on Linux at least we would know for sure.

@quozl
Copy link
Author

quozl commented Apr 23, 2018

Thanks. Tried to git bisect, but both f5bcd83 (head) and 71eedf0 (6.10.0) work fine. 9c6a1ec is the point of failure; one of the Debian packaging patches. Yet to find the commit that fixes it; will do an upside-down bisect next.

By the way, the build doesn't install the Python API, or rather it installs in wrong place; /root/.local . After make install the import ctcsound says no module by that name. Do you want that logged somewhere?

@jpffitch
Copy link
Contributor

jpffitch commented Apr 23, 2018 via email

@jpffitch
Copy link
Contributor

jpffitch commented Apr 23, 2018 via email

@vlazzarini
Copy link
Member

vlazzarini commented Apr 23, 2018 •
edited

do you mean tamtamorc.csd?

@vlazzarini
Copy link
Member

@quozl could you log the installer issue separately? thx

@vlazzarini
Copy link
Member

I guess we ought to release 6.11 soon so that the olpc code can work with it.

@jpffitch
Copy link
Contributor

jpffitch commented Apr 23, 2018 via email

@vlazzarini
Copy link
Member

vlazzarini commented Apr 23, 2018 via email

@quozl
Copy link
Author

quozl commented Apr 24, 2018

Thanks everyone, I've figured it out. My traceback is solved by df5658c on the develop branch. It's not a Csound upstream problem, it's a Debian problem.

@fsateler, the Debian package has 9c6a1ec ("Fix format string attributes and usages to allow -Wformat"), and needs df5658c ("fixed crashing due to colon that was left out in PR 880"). I'll solve this with a custom package for my Ubuntu build, so I've no need to raise a Debian bug. There's no Debian package that reproduces the problem; it depends on specific syntax in the csd file.

@quozl quozl closed this as completed Apr 24, 2018
@fsateler
Copy link
Contributor

@quozl could you please file a downstream debian bug? Otherwise I might forget. Even better if you post a proper patch 😉

Btw, I would be very happy to get other people looking at the debian package. If you are looking after csound in Ubuntu it may be beneficial to join efforts.

@quozl
Copy link
Author

quozl commented Apr 25, 2018

@fsateler, thanks for the offer. No, I'm not looking after Csound in Ubuntu. I'm deriving a distribution from Debian and Ubuntu. See Derivatives Census - OLPC. Let me know how I can help.

I'll look into filing a downstream Debian bug, but I too might forget;

  • there's no way to file a bug in a browser,
  • the bug will not be necessary once 6.11 is released and packaged; @vlazzarini says soon,
  • it is unlikely anybody else will need the bug fixed,
  • our need is immediate, we can't wait for a new Debian release or package.

I could also file a downstream Ubuntu bug, but roughly the same reasoning applies.

The proper patch is df5658c and can be directly applied to the Debian package. I used git-buildpackage like this;

# get the debian source package
apt-get source csound
# import into git-buildpackage
gpb import-dsc csound_6.10.0~dfsg-1.dsc
cd csound
# import existing patch queue
gbp pq import
# download the patch that fixes the problem and apply it
wget https://github.com/csound/csound/commit/df5658cd30f60f7caf50650bfdbfd0f480139f43.patch
git am df5658cd30f60f7caf50650bfdbfd0f480139f43.patch 
rm df5658cd30f60f7caf50650bfdbfd0f480139f43.patch 
# export the patch queue; updates debian/patches
gbp pq export
# add change log entry with local version
dch --newversion 1:6.10.0~dfsg-2~quozl0 --distribution bionic Fix crash due to colon, df5658c
# commit changes
git add debian/changelog debian/patches
git commit -m "1:6.10.0~dfsg-2~quozl0 Fix crash due to colon, df5658c"

then I built the package on the target distribution, and used reprepro to add it to the archive we use for private building of our manufacturing disk image. Hope that helps!

@quozl
Copy link
Author

quozl commented Apr 25, 2018

Filed Debian bug #896848.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@fsateler @quozl @jpffitch @vlazzarini @gogins