Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSEED: Segfault reading truncated file #1728

Merged
merged 6 commits into from Mar 28, 2017
Merged

Conversation

@krischer
Copy link
Member

krischer commented Mar 23, 2017

While trying to work around a problem when reasding truncated files (in SDS client while reading files that are currently being appended to by a different program), I came across a segfault when reading truncated MiniSEED files:

import copy
from io import BytesIO
from obspy import read
from obspy.core.util import get_example_file

file_ = get_example_file('BW.BGLD.__.EHE.D.2008.001.first_10_records')

with open(file_, 'rb') as fh: 
    data = fh.read()

# for i in range(1, 1000):
for i in [257]:
    print(i)
    bio = BytesIO(copy.deepcopy(data[:-i]))
    read(bio, format='MSEED')
$ python read_mseed_truncated.py 
257
Segmentation fault
@megies megies added the .io.mseed label Mar 22, 2017
@QuLogic

This comment has been minimized.

Copy link
Member

QuLogic commented Mar 23, 2017

Seems to be crashing in our code, not libmseed:

0x00007fffdda816aa in readMSEEDBuffer (mseed=0x18ffc10 "763445D BGLD   EHEBW", <incomplete sequence \330>, buflen=4863, selections=0x0, 
    unpack_data=1 '\001', reclen=-1, verbose=0 '\000', details=0 '\000', header_byteorder=-1, allocData=0x7ffff7fae048, diag_print=0x7ffff7fae080, 
    log_print=0x7ffff7fae0b8) at obspy/io/mseed/src/obspy-readbuffer.c:472
472	        if ((unpack_data != 0) && (msr->fsdh->data_offset >= 48) &&
@megies megies added the bug label Mar 23, 2017
@megies

This comment has been minimized.

Copy link
Member Author

megies commented Mar 23, 2017

Maybe @krischer can have a look when he's got some time, no hurry though..

@krischer

This comment has been minimized.

Copy link
Member

krischer commented Mar 23, 2017

This branch contains a fix: https://github.com/obspy/obspy/tree/mseed-fix-segfault-truncated-file

Not sure why I cannot convert this issue to a PR right now but I'll try again later tonight or tomorrow. Or maybe somebody else can try?


Some other types of record corruption where already caught by libmseed and correctly bubble up to the Python warnings. I'm not entirely sure why this one does not but maybe its just because its truncated fairly late in the file?

In any case: now works as expected and it raises a nice warning (but still reads all previous records).

@krischer

This comment has been minimized.

Copy link
Member

krischer commented Mar 23, 2017

Hmm...looks like one of my tries did convert it to a PR in the end? Or did someone else do it?

Anyways - IMHO good to go. Feel free to review and merge :)

@megies

This comment has been minimized.

Copy link
Member Author

megies commented Mar 25, 2017

Thanks for the fix @krischer, checking again, there's still some truncation scenarios that end in segfaults though..

Can you maybe have a look at these two byte offset:

  • 256
  • 5066

These seem to be different issues.. the latter one I've seen in real live reading mseed files that currently also get appended to in other threads (checking data latency).

import copy
from io import BytesIO
from obspy import read
from obspy.core.util import get_example_file

file_ = get_example_file('BW.BGLD.__.EHE.D.2008.001.first_10_records')

with open(file_, 'rb') as fh: 
    data = fh.read()

for i in range(1, 10000):
    # this seems to be a different issue than the already covered one:
    if i == 256:
        continue
    # these seem to be the same issue as with 256, as there just offset by 512
    # bytes..
    if i % 512 == 256:
        continue
    # this is finally the issue I was looking after: :-)
    if i == 5066:
        continue
    print(i)
    bio = BytesIO(copy.deepcopy(data[:-i]))
    read(bio, format='MSEED')
krischer added 5 commits Mar 23, 2017
We already caught a couple of other variants of this but not this
particular one. Now works correctly and raises a proper warning.
@krischer krischer force-pushed the mseed-fix-segfault-truncated-file branch from 5475060 to 52d109d Mar 27, 2017
@krischer

This comment has been minimized.

Copy link
Member

krischer commented Mar 27, 2017

All fixed, rebased and force pushed.

The 256 + 512 bytes offsets were just because I forgot the <= case. The larger truncation because you passed a file with less than 128 bytes - this now raises a much better error message.

@megies

This comment has been minimized.

Copy link
Member Author

megies commented Mar 27, 2017

Thanks for the fix(es)! 🎉

@krischer

This comment has been minimized.

Copy link
Member

krischer commented Mar 27, 2017

IMHO ready to be merged.

Copy link
Member

megies left a comment

Works like a charm, thanks!
(somehow I can't 'approve' this PR, seems like there's a problem with the review button..)

@megies megies merged commit 062b241 into master Mar 28, 2017
5 checks passed
5 checks passed
ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
docker-deb-buildbot Deb packaging and testing succeeded
Details
docker-testbot Docker tests succeeded
Details
@megies megies deleted the mseed-fix-segfault-truncated-file branch Mar 28, 2017
megies added a commit that referenced this pull request Mar 28, 2017
too small file size

also see #639 and #1728
megies added a commit that referenced this pull request Mar 28, 2017
too small file size

also see #639 and #1728
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.