Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IO Updates #1476

Merged
merged 20 commits into from Oct 5, 2020
Merged

IO Updates #1476

merged 20 commits into from Oct 5, 2020

Conversation

dopplershift
Copy link
Member

@dopplershift dopplershift commented Aug 24, 2020

Description Of Changes

Some clean-ups and updates for GINI and NEXRAD Levels 2 and 3. Definitely a bunch of updates due to NEXRAD ORPG Build 19.

  • Add some support for passing in pathlib.Path objects (starts to address Check support for pathlib.Path #224)
  • Add support for Power Removed Control product
  • Refactor and improve message 31 reading for better speed and robustness to format changes
  • Various Build 19 product and data format updates
  • Add missing GINI Percent Normal TPW product support
  • Lots of new test data files
  • Fix problems reading compressed data from file-like objects (such as those returned when using Boto3 to read data from AWS archive)

Checklist

@dopplershift dopplershift added this to the 1.0 milestone Aug 24, 2020
@dopplershift dopplershift added Area: IO Pertains to reading data Type: Bug Something is not working like it should Type: Enhancement Enhancement to existing functionality labels Aug 24, 2020
@lgtm-com
Copy link

lgtm-com bot commented Aug 24, 2020

This pull request introduces 1 alert when merging 7f51825 into 071c700 - view on LGTM.com

new alerts:

  • 1 for Missing call to __init__ during object initialization

@dopplershift dopplershift marked this pull request as draft August 24, 2020 16:29
@dopplershift
Copy link
Member Author

Marking draft because I want to take a shot at using from_bytes (#1367) to read in integers.

@lgtm-com
Copy link

lgtm-com bot commented Aug 26, 2020

This pull request introduces 1 alert when merging ad3cf42 into 931c114 - view on LGTM.com

new alerts:

  • 1 for Missing call to __init__ during object initialization

@lgtm-com
Copy link

lgtm-com bot commented Aug 26, 2020

This pull request introduces 1 alert when merging 62f174a into cdeb8c7 - view on LGTM.com

new alerts:

  • 1 for Missing call to __init__ during object initialization

@lgtm-com
Copy link

lgtm-com bot commented Aug 27, 2020

This pull request introduces 1 alert when merging f919a42 into cdeb8c7 - view on LGTM.com

new alerts:

  • 1 for Missing call to __init__ during object initialization

This just requires a one-line change in open_as_needed to force
conversion to string. This is a nice addition for our pure Python 3
world.
This was found based on information in SCN20-03; channel 60 was
previously unknown, and not even found by Unidata's ingest code.
Also fixes __repr__ to work for GSM.
@dopplershift dopplershift force-pushed the io-update branch 4 times, most recently from 75cfa40 to 81891ed Compare October 2, 2020 07:01
@dopplershift dopplershift marked this pull request as ready for review October 2, 2020 07:03
@lgtm-com
Copy link

lgtm-com bot commented Oct 2, 2020

This pull request introduces 1 alert when merging 3441003 into 69f86f9 - view on LGTM.com

new alerts:

  • 1 for Missing call to __init__ during object initialization

This, for instance, makes it possible to use Level2File with the
realtime chunks available on AWS.
Update product specific fields to include new metadata relating to
SAILS/MRLE.
Puts a basic lookup table in the base mapper class; reduces some code
duplication as well as silences some PyCharm warnings.
Add some sample test files, add support for a couple new products, and
fix a bug in the mapping of width data.
Two main changes:
* Add read_array function for reading data directly into a numpy array
  using from_buffer. This is noticably faster (~25%)
* Don't assume any blocks are present or their order--this is
  technically more correct, as well as more robust to future format
  adjustments.
Update it to use Python 3's int.from_bytes() method.
This allows them to be accessed outside methods.
This paves the way towards writing instances of NamedStructs.
@lgtm-com
Copy link

lgtm-com bot commented Oct 2, 2020

This pull request introduces 1 alert when merging efcd1c3 into 69f86f9 - view on LGTM.com

new alerts:

  • 1 for Missing call to __init__ during object initialization

@dcamron dcamron merged commit ea2bf54 into Unidata:master Oct 5, 2020
@dopplershift dopplershift deleted the io-update branch October 5, 2020 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: IO Pertains to reading data Type: Bug Something is not working like it should Type: Enhancement Enhancement to existing functionality
Projects
None yet
2 participants