Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apollo seismometer format support: obspy.io.alsep #2281

Open
wants to merge 1 commit into
base: master
from

Conversation

@yamamo-to
Copy link

commented Jan 9, 2019

Purpose
This module is an extension to read seismic data obtained by Apollo missions. The module supports three tape formats: PSE Tape, WTN Tape, and WTH Tape. The module only has the read function.

Background
The Apollo mission was known as the human activity on the moon, but science was also great. A series of experiments was called an Apollo Lunar Surface Experiments Packages (ALSEP). As a part of ALSEP, the seismometers were mounted on the moon, and they measured the lunar seismic events from 1969 to 1977.

Data
The data can be downloaded from the DARTS website:

Tutorial
Let's draw the same figure in figure 8-4 in the following paper:
http://an.rsl.wustl.edu/apollo/data/a15/station8/pse/docs/pse_psr.pdf

from obspy import read, UTCDateTime

st = read('http://darts.isas.jaxa.jp/pub/apollo/pse/p15s/pse.a15.1.2')
st_spz = st.select(id='XA.S15..SPZ')
st_spz.plot(starttime=UTCDateTime('1971-08-02T17:11:22.6'),
            endtime=UTCDateTime('1971-08-02T17:11:35.8'))

Problem

  1. Takes several minutes to download and decode ALSEP tapes
  2. Test code uses the real data and all unit tests require more than 20 minutes.

Reference

@megies

This comment has been minimized.

Copy link
Member

commented Jan 9, 2019

Cool. I just tried it and indeed it takes rather long to decode. Are the packets always the same size and all headers etc at the same byte position in the packet? If yes, I think it could be made much faster similar to the io.reftek approach, i.e. unpacking in one numpy.from_buffer call with a complex dtype ending up with an structured array.

@megies megies added this to the Future release milestone Jan 9, 2019

@megies

This comment has been minimized.

Copy link
Member

commented Jan 9, 2019

I assume there will be license problems so we can't include test files in our repo.. (we'd need to be able to include it under LGPL)?!

@isas-yamamoto

This comment has been minimized.

Copy link

commented Jan 9, 2019

Cool. I just tried it and indeed it takes rather long to decode. Are the packets always the same size and all headers etc at the same byte position in the packet? If yes, I think it could be made much faster similar to the io.reftek approach, i.e. unpacking in one numpy.from_buffer call with a complex dtype ending up with an structured array.

The tape format is based on 10-bit ALSEP word, and bit-field operation is essential. I will check the io.reftek approach with bit-field.

@isas-yamamoto

This comment has been minimized.

Copy link

commented Jan 9, 2019

I assume there will be license problems so we can't include test files in our repo.. (we'd need to be able to include it under LGPL)?!

Not the license problem, but the filesize. Each test file is about 10 MB. I did not want to include files of this size in the repository. I had to create dummy data for unittest.

@megies

This comment has been minimized.

Copy link
Member

commented Jan 9, 2019

The tape format is based on 10-bit ALSEP word, and bit-field operation is essential. I will check the io.reftek approach with bit-field.

I mean.. it really depends.. since this data format is exclusively for the old Apollo data I guess, it might not be worth it to spend too much time into optimization though? If somebody needed to read the data loads of times, they could still read it once and convert it to e.g. MiniSEED.

@megies

This comment has been minimized.

Copy link
Member

commented Jan 9, 2019

Not the license problem, but the filesize. Each test file is about 10 MB. I did not want to include files of this size in the repository. I had to create dummy data for unittest.

dummy data is fine if you can be bothered to mimick the encoding.. otherwise if the data is in fully self-contained packets, you could just trim one original file to the first few packets?

@isas-yamamoto isas-yamamoto force-pushed the isas-yamamoto:feature/alsep_format_support branch from 8d9a2ba to 4db5c69 May 24, 2019

@isas-yamamoto isas-yamamoto force-pushed the isas-yamamoto:feature/alsep_format_support branch from 269142c to 5efb07f Jun 24, 2019

@megies

This comment has been minimized.

Copy link
Member

commented Jun 25, 2019

@isas-yamamoto if you are working on this, you need to eventually rebase on current master branch. This would also get rid of some of the CI test fails.

@isas-yamamoto isas-yamamoto force-pushed the isas-yamamoto:feature/alsep_format_support branch from 5efb07f to d51cb54 Jun 26, 2019

@isas-yamamoto isas-yamamoto force-pushed the isas-yamamoto:feature/alsep_format_support branch from 3967662 to c747f5e Jul 14, 2019

Added ALSEP PSE Tape reader
Rebased:
- Fixed to work with both Python 2 and Python 3
- Added Work Tape Normal reader
- Added Work Tape High reader
- Changed the maximum line length to 78 characters
- Added URL list for unittest
- Removed unnecessary backslash
- Separated definition and added PSE tape structure
- Checked PEP8 using flake8 and updated
- Renamed class names and function names considering accessibility
- Added docs module skeleton
- Fixed bug of PSE new format processing
- Fixed bug of time skipping
- Fixed frame rate bug
  - Frame rate is always the same because slow rate sent a half ALSEP words.
- Added test code for file with many time skipping
- Fixed a bug of dictionary key check in WTN process
- Removed unnecessary test codes
- Added year overwrite option in _pse_read
- Added date validation code
- Fixed unexpected changes
- Fixed unexpected changes
- Added sync pattern barker code validation
- Used collections.deque instead of list for speeding up
- Fixed integer overflow on Windows
  On Win64 platform, default integer type `np.int_` is int32.
  On Linux and MacOSX, the type is int64.
  Therefore, milliseconds of year to describe datetime has overflowed
  only on Win64 platform.
- Removed functions with double leading underscores
- Added dummy data for unittest
- Solved PEP8 warnings
  - F401 'io' imported but unused
  - E127 continuation line over-indented for visual indent
- Improved doctest results
  The command `obspy-runtests obspy.io.alsep.tests.suite` works fine.
- Included future import in every python code
  The following test case is solved:
  - FAIL: test_future_imports_in_every_file

@isas-yamamoto isas-yamamoto force-pushed the isas-yamamoto:feature/alsep_format_support branch from c747f5e to 47c78af Sep 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.