Segfault after obspy.read #1658

krischer · 2017-03-23T09:41:30Z

Hi devs, running the QC we are sometimes running in to a segfault reading the record headers of some files. It's been quite a challenge but I traced it back to the obspy.read subroutine... it really blows my mind.

from obspy import read
from obspy.io.mseed.util import get_flags

# Reading the file before getting the records causes a segfault
# when getting header flags.
read("broken_records.mseed")
get_flags("broken_records.mseed")

>>> Warning: Number of blockettes in fixed header (2) does not match the number parsed (1)
>>> Segmentation fault (core dumped)

When we do not read the file first the segmentation fault does not occur.:

from obspy import read
from obspy.io.mseed.util import get_flags

get_flags("broken_records.mseed")

>>> Warning: Number of blockettes in fixed header (2) does not match the number parsed (1)
>>> Warning: Number of blockettes in fixed header (2) does not match the number parsed (1)
>>> ... identical for all 16 records

Here is the file: ftp://www.orfeus-eu.org/pub/outgoing/broken_records.mseed (8 kb).

Thanks for the help and hope you understand this behaviour better than me..

megies · 2017-02-08T13:57:36Z

I can confirm the problem.. no idea what's going on though..

megies · 2017-03-22T08:36:28Z

Any new insights for this one @Jollyfant?

Jollyfant · 2017-03-22T11:16:41Z

Nope.. I'm putting my money on @krischer for this one.

megies · 2017-03-22T11:20:53Z

Might be connected to #1728..?

Might be a bit overengineered but is transparent to the user and it works.

krischer · 2017-03-23T09:43:23Z

This is now A PR fixing the issue.

The problem was that the call to read_mseed() hooked up libmseed's logging to some Python callbacks. Leaving the read function the Python functions were garbage collected and a call the next time naturally resulted in a segfault.

There is now a wrapper around the ctypes lib to always setup the logging so this can no longer happen and it is completely transparent to anyone directly using libmseed from Python.

Does not fix #1728 which is another issue.

krischer · 2017-03-23T22:00:17Z

IMHO ready to go.

Now called InternalMSEEDError instead of InternalMSEEDReadingError and InternalMSEEDWarning instead of InternalMSEEDReadingWarnings as both can now also be raised in non-reading contexts.

Jollyfant

Wrapper appears to be good and is covered by tests.

QuLogic

There were still a few issues...

QuLogic · 2017-03-24T21:58:57Z

CHANGELOG.txt

@@ -92,6 +92,9 @@ master: (doi: 10.5281/zenodo.165135)
     some rare segfaults. (see #1658)
   * Update to libmseed v2.19.2 (see #1703).
   * Correctly read MiniSEED files with a data offset of 48 bytes (see #1540).
+   * InternalMSEEDReadingError now called InternalMSEEDError and
+	 InternalMSEEDReadingWarning now called of InternalMSEEDWarning as both


QuLogic · 2017-03-24T22:00:50Z

obspy/io/mseed/headers.py

@@ -581,11 +497,11 @@ class MSTraceSeg(C.Structure):
    ('samprate', C.c_double),         # Nominal sample rate (Hz)
    ('samplecnt', C.c_int64),         # Number of samples in trace coverage
    ('datasamples', C.c_void_p),      # Data samples, 'numsamples' of type
-                                      # 'sampletype'
+    # 'sampletype'


This is wrapped from the previous line; looks pretty weird unindented like this.

QuLogic · 2017-03-24T22:02:28Z

obspy/io/mseed/headers.py

+
+        def _wrapper(*args):
+            # Collect exceptions. They cannot be raised in the callback as
+            # they could never be caught then. They are collected an raised


QuLogic · 2017-03-24T22:05:21Z

obspy/io/mseed/headers.py

+                    _warns.append(msg)
+
+            diag_print = \
+                C.CFUNCTYPE(C.c_void_p, C.c_char_p)(log_error_or_warning)


Return type is None, not c_void_p.

QuLogic · 2017-03-24T22:05:35Z

obspy/io/mseed/headers.py

+            def log_message(msg):
+                if self.verbose:
+                    print(msg[6:].strip())
+            log_print = C.CFUNCTYPE(C.c_void_p, C.c_char_p)(log_message)


Return type is None, not c_void_p.

QuLogic · 2017-03-24T22:15:33Z

obspy/io/mseed/headers.py

+# this is completely different to C.c_char_p which is a string
+__clibmseed.mst_packgroup.argtypes = [
+    C.POINTER(MSTraceGroup), C.CFUNCTYPE(
+        C.c_void_p, C.POINTER(C.c_char), C.c_int, C.c_void_p),


None, not c_void_p (the first time).

QuLogic · 2017-03-24T22:16:56Z

obspy/io/mseed/headers.py

+__clibmseed.mst_packgroup.argtypes = [
+    C.POINTER(MSTraceGroup), C.CFUNCTYPE(
+        C.c_void_p, C.POINTER(C.c_char), C.c_int, C.c_void_p),
+    C.c_void_p, C.c_int, C.c_short, C.c_short, C.POINTER(C.c_int), C.c_short,


C.POINTER(C.c_int64), actually

QuLogic · 2017-03-24T22:17:24Z

obspy/io/mseed/headers.py

+    C.c_int8,
+    C.CFUNCTYPE(C.c_void_p, C.c_char_p),
+    C.CFUNCTYPE(C.c_void_p, C.c_char_p)]
+__clibmseed.setupLogging.restype = C.c_void_p


QuLogic · 2017-03-24T22:17:35Z

obspy/io/mseed/headers.py


+__clibmseed.msr_free.argtypes = [C.POINTER(C.POINTER(MSRecord))]
+__clibmseed.msr_free.restype = C.c_void_p


QuLogic · 2017-03-24T22:18:51Z

obspy/io/mseed/headers.py


+__clibmseed.lil_free.argtypes = [C.POINTER(LinkedIDList)]
+__clibmseed.lil_free.restype = C.c_void_p


megies · 2017-03-25T11:15:08Z

Thanks for the review @QuLogic, I've opened a follow up issue to make sure your comments don't go unnoticed..

Jollyfant added .io.mseed bug confirmed bug labels Feb 1, 2017

megies assigned krischer Feb 8, 2017

megies added this to the 1.1.0 milestone Feb 8, 2017

megies added the release blocker critical issues blocking the corresponding future release label Feb 27, 2017

krischer added 4 commits March 23, 2017 10:26

Adding test for logging related segfault.

4f60e4d

Hook up the libmseed logging for every libmseed call.

69606c7

Might be a bit overengineered but is transparent to the user and it works.

Catch the warnings in the tests.

2c761bb

Changelog

00c2912

krischer added 3 commits March 23, 2017 11:45

Removing duplicate import

5f5fdbd

py2k compat

60f2f39

Forgot to add new function to the windows definition file.

cd1d275

krischer mentioned this pull request Mar 24, 2017

Error on mst_pack during conversion of a GSE2 file to mseed #1724

Closed

krischer added 2 commits March 24, 2017 11:15

Renamed internal mseed exception + warning classes.

3b259e3

Now called InternalMSEEDError instead of InternalMSEEDReadingError and InternalMSEEDWarning instead of InternalMSEEDReadingWarnings as both can now also be raised in non-reading contexts.

Changelog

c1ce337

Jollyfant reviewed Mar 24, 2017

View reviewed changes

krischer merged commit c147e47 into master Mar 24, 2017

QuLogic reviewed Mar 24, 2017

View reviewed changes

megies mentioned this pull request Mar 25, 2017

Segfault after obspy.read (follow-up cleanup) #1730

Merged

megies deleted the segfault-get-flags branch March 25, 2017 11:15

flixha mentioned this pull request Jul 28, 2022

Occasional Segmentation fault on threaded MSEED-reading from SDS-client #3114

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segfault after obspy.read #1658

Segfault after obspy.read #1658

krischer commented Mar 23, 2017

megies commented Feb 8, 2017

megies commented Mar 22, 2017

Jollyfant commented Mar 22, 2017

megies commented Mar 22, 2017

krischer commented Mar 23, 2017

krischer commented Mar 23, 2017

Jollyfant left a comment

QuLogic left a comment

QuLogic Mar 24, 2017

QuLogic Mar 24, 2017

QuLogic Mar 24, 2017

QuLogic Mar 24, 2017

QuLogic Mar 24, 2017

QuLogic Mar 24, 2017

QuLogic Mar 24, 2017

QuLogic Mar 24, 2017

QuLogic Mar 24, 2017

QuLogic Mar 24, 2017

megies commented Mar 25, 2017


		__clibmseed.msr_free.argtypes = [C.POINTER(C.POINTER(MSRecord))]
		__clibmseed.msr_free.restype = C.c_void_p


		__clibmseed.lil_free.argtypes = [C.POINTER(LinkedIDList)]
		__clibmseed.lil_free.restype = C.c_void_p

Segfault after obspy.read #1658

Segfault after obspy.read #1658

Conversation

krischer commented Mar 23, 2017

megies commented Feb 8, 2017

megies commented Mar 22, 2017

Jollyfant commented Mar 22, 2017

megies commented Mar 22, 2017

krischer commented Mar 23, 2017

krischer commented Mar 23, 2017

Jollyfant left a comment

Choose a reason for hiding this comment

QuLogic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

megies commented Mar 25, 2017