Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MiniSEED output of int64 data #2356

Open
megies opened this issue Mar 15, 2019 · 4 comments

Comments

Projects
None yet
3 participants
@megies
Copy link
Member

commented Mar 15, 2019

When creating custom integer data with numpy and trying to write to MiniSEED, we can end up in problems because miniseed has no way to handle int64 data. In most cases it might be possible to simply downcast the data to int32 without changing the data before writing it I guess, maybe we should consider doing that @krischer? Or give some better information how to work around this when encountering int64 data arrays..

import numpy as np
from obspy import Trace, Stream, UTCDateTime


data = [ 
    ("2016-01-07T00:00:00.008300Z", 3010),
    ("2016-01-07T00:00:02.668393Z", 686),
    ("2016-01-07T00:00:09.528393Z", 4086),
    ("2016-01-07T00:00:30.108393Z", 2028),
    ("2016-01-07T00:00:50.388393Z", 6158),
    ("2016-01-07T00:00:57.248393Z", 1370),
    ("2016-01-07T00:01:31.458393Z", 4107),
    ("2016-01-07T00:01:51.968393Z", 2056)]

x = np.arange(30000)
tr = Trace(x)
tr.stats.starttime = UTCDateTime("2016-01-07T00:00:00.008300Z")
tr.stats.sampling_rate = 100 

st = Stream()
for i, (start, numsamp) in enumerate(data):
    tr_ = tr.slice(starttime=UTCDateTime(start))
    tr_.data = tr_.data[:numsamp]
    st.append(tr_)

import pdb; pdb.set_trace()
st.write('/tmp/buggy.mseed', 'MSEED')
Traceback (most recent call last):
  File "buggy.py", line 26, in <module>
    st.write('/tmp/buggy.mseed')
  File "/home/megies/git/obspy/obspy/core/stream.py", line 1452, in write
    write_format(self, filename, **kwargs)
  File "/home/megies/git/obspy/obspy/io/mseed/core.py", line 791, in _write_mseed
    raise Exception(msg)
Exception: Unsupported data type int64 in Stream[0].data

@megies megies added the .io.mseed label Mar 15, 2019

@megies

This comment has been minimized.

Copy link
Member Author

commented Mar 28, 2019

@LMurrayBergquist you could work on this one. Basically just (in a test case)

  • create a dummy stream with e.g. np.arange(20, dtype=np.int64)
  • make sure it can be written to mseed (writing to a io.BytesIO() object in the test, search in other test cases how that's done if unclear)
  • in the miniseed writer routine add a check when encountering int64 data for output whether the data is still the same if downcast to int32 and then write as int32 or show a proper error message

If anything is unclear, please get in touch with @DominikStr

@LMurrayBergquist

This comment has been minimized.

Copy link
Contributor

commented Apr 1, 2019

@DominikStr I have a test case but I'm actually having trouble running it, I have been using the Github Windows app and Visual Studio 2015 for code, and I have obspy_dev set up following the instructions here: https://github.com/obspy/obspy/wiki/Developer-Installation, I already have a clone of the obspy repository that I have been working with, but when I run the test case which starts out with importing some things from obspy I get the message "no module named 'obspy'", I see that previous tests also import from obspy, do you know how I might be able to fix my set up so that I can run my test case?

@DominikStr

This comment has been minimized.

Copy link
Contributor

commented Apr 1, 2019

I followed the Developer-Installation only in the beginning to install all the dependencies and uninstall obspy again. Instead of pip install I used python setup.py develop as shown here. It didn't work directly but after a few tries of uninstalling the environment completely and installing it again, it worked. And at the beginning, importing obspy while being in the folder didn't work. Maybe try changing the working directory and have a look if it is working there.

As this is not directly related to this issue I would suggest that you write me directly via E-Mail (dominikstrutz@freenet.de) or maybe @megies can add you to the gitter conversation if there are any further questions.

LMurrayBergquist added a commit to LMurrayBergquist/obspy that referenced this issue Apr 4, 2019

Fix mseed output of int64 data obspy#2356
Changes obspy/io/mseed/core.py so that int64 data is accepted if it can be safely downcast to type int32 data, otherwise it creates an error message. A test for this has been added to test_mseed_reading_and_writing.py to check that an input array of int64 data is the same after being written to mseed.

@LMurrayBergquist LMurrayBergquist referenced this issue Apr 4, 2019

Open

Mini seed output of int64 data #2356 #2373

0 of 9 tasks complete
@megies

This comment has been minimized.

Copy link
Member Author

commented Apr 4, 2019

maybe @megies can add you to the gitter conversation if there are any further questions.

no need to add, anybody can join that chat. to write in the gitter chat you need to login to gitter with your github account

LMurrayBergquist added a commit to LMurrayBergquist/obspy that referenced this issue Apr 4, 2019

Update CHANGELOG.txt
Added lines:
 "- obspy.io.mseed:
   * Add add ability to write int64 data to mseed if it can safely be downcast to int32 data, otherwise raises ObsPyMSEEDError. (see obspy#2356)"

LMurrayBergquist added a commit to LMurrayBergquist/obspy that referenced this issue Apr 6, 2019

Fix mseed output of int64 data obspy#2356
Changes obspy/io/mseed/core.py so that int64 data is accepted if it can be safely downcast to type int32 data, otherwise it creates an error message. A test for this has been added to test_mseed_reading_and_writing.py to check that an input array of int64 data is the same after being written to mseed.

LMurrayBergquist added a commit to LMurrayBergquist/obspy that referenced this issue Apr 6, 2019

LMurrayBergquist added a commit to LMurrayBergquist/obspy that referenced this issue May 10, 2019

megies added a commit to LMurrayBergquist/obspy that referenced this issue May 15, 2019

Fix mseed output of int64 data obspy#2356
Changes obspy/io/mseed/core.py so that int64 data is accepted if it can be safely downcast to type int32 data, otherwise it creates an error message. A test for this has been added to test_mseed_reading_and_writing.py to check that an input array of int64 data is the same after being written to mseed.

megies added a commit to LMurrayBergquist/obspy that referenced this issue May 15, 2019

Update CHANGELOG.txt
Added lines:
 "- obspy.io.mseed:
   * Add add ability to write int64 data to mseed if it can safely be downcast to int32 data, otherwise raises ObsPyMSEEDError. (see obspy#2356)"

megies added a commit to LMurrayBergquist/obspy that referenced this issue May 15, 2019

Fix mseed output of int64 data obspy#2356
Changes obspy/io/mseed/core.py so that int64 data is accepted if it can be safely downcast to type int32 data, otherwise it creates an error message. A test for this has been added to test_mseed_reading_and_writing.py to check that an input array of int64 data is the same after being written to mseed.

LMurrayBergquist added a commit to LMurrayBergquist/obspy that referenced this issue May 15, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.