Add support for INGV DMX format

[ThomasLecocq]

What does this PR do?

Adds support to read INGV's DMX format

Why was it initiated? Any relevant Issues?

To rule the World & to read soooo many small DMX files from INGV Volcano Observatories, among others.

PR Checklist

• Correct base branch selected? master for new features, maintenance_... for bug fixes
• This PR is not directly related to an existing issue (which has no PR yet): it is Closes DMX read #2438
• If the PR is making changes to documentation, docs pages can be built automatically.
  Just remove the space in the following string after the + sign: "+DOCS"
• All tests still pass.
• Any new features or fixed regressions are be covered via new tests.
• Any new or changed features have are fully documented.
• Significant changes have been added to CHANGELOG.txt .

[megies]

See comments, also needs docs skeletons, see https://github.com/obspy/obspy/wiki/How-to%3A-add-a-new-submodule

[megies]

When making changes, would be good to rebase on current master in the process, to get cleaner CI results @ThomasLecocq

[ThomasLecocq]

Ok boss :-)

[ThomasLecocq]

@megies not sure I managed to rebase... please do it if you have 5min ?

[megies]

Rebase was OK, I added some minor fixes and rebased on most recent master again

[megies]

See comments, also needs docs skeletons, see https://github.com/obspy/obspy/wiki/How-to%3A-add-a-new-submodule

• This still applies, docs skeletons missing, see wiki link above what needs be done. Also +DOCS to get a automatic build
• the test file needs changing. 4MB is not acceptable here. I'm assuming you have some software to make a smaller DMX test file?
• test cases never test the unpacked sample data, there should be a test checking like at least the first 5 samples and maybe the last as well.

@ThomasLecocq I force-pushed, so you will need to resync your local checkout

[ThomasLecocq]

I have maybe forgotten to do the doc skeleton

Asking ingv for the test file

[megies]

no such file I believe?

[megies]

Commit history is messed up now, you merged branches together locally it seems, instead of hard resetting your local history to match the new history upstream after my last commit.

I'll fix it.

[megies]

Cleaned up history, force-pushed. Also rebased the 4MB out of history

[ThomasLecocq]

@megies thanks a lot and sorry for the mess :(

Why do tests fail ? nothing to do with dmx, right ?

[megies]

I think the problem is that your _is_dmx routine is recognizing any given file as DMX. That while loop in there that tries to find a data block finishes just fine not finding anything and then goes to return True at end.

I don't understand tbh, assuming the file is not DMX, you are jumping to a supposedly next block position based on some random bytes parsed from the file? int(structtag.len_struct) + int(structtag.len_data) this position shift will contain random numbers if the file is not DMX.

I don't know the format but it looks like the block size can vary, so it will be impossible to check anything but the first data block on unknown files. Is it not possible to check something that always has to be there in the first data portion in the file?

@ThomasLecocq

[ThomasLecocq]

hummmm.. OK, checking this now...

[krischer]

@ThomasLecocq Any chance to get this done within the next view days?

[ThomasLecocq]

yep :-)

[megies]

ping @ThomasLecocq

[megies]

Rebased and force-pushed @ThomasLecocq

[ThomasLecocq]

Added some more clever checks in the _is_dmx method, let's see how CI like this...

[megies]

Rebased once more to get clean(er) CI results

[megies]

@ThomasLecocq there are some problems on our minimum dependency build, maybe you can have a look?

https://travis-ci.org/obspy/obspy/jobs/618544621?utm_medium=notification&utm_source=github_status

http://tests.obspy.org/106920/#2

environment is listed here:
https://travis-ci.org/obspy/obspy/jobs/618544621#L1746

[ThomasLecocq]

@megies there is 1 unrelated error (next_fast_len missing in stack) and then

data = np.fromfile(fid, structtag_dtypes, 1)
IOError: first argument must be an open file

because the oooooold numpy is not happy... I'll check how this is handled in other formats... Or it's maybe due to my SpooledTemporaryFile ?

[megies]

Or it's maybe due to my SpooledTemporaryFile ?

maybe.. not sure. we have some compat code for frombuffer i saw, maybe you get some ideas from that (although it doesnt allow specifying the object count atm). i would probably just recommend reading as many bytes as you want and use that compat layer from_buffer function.

io/mseed/util.py
232-    for file in files:
233-
234-        # If it's a file name just read it.
235-        if isinstance(file, (str, native_str)):
236-            # Read to NumPy array which is used as a buffer.
237:            bfr_np = np.fromfile(file, dtype=np.int8)
238-        elif hasattr(file, 'read'):
239-            bfr_np = from_buffer(file.read(), dtype=np.int8)
240-
241-        offset = 0
242-

Nevermind actually, I'll push the fix..

[megies]

One single unrelated CI test fail, merging

[ThomasLecocq]

Yeah !! Thanks a bunch for your help & time @megies !!! 🍻 🍻