ENH: new GNSS TEC instrument #11

aburrell · 2020-08-11T17:12:51Z

Adding a new instrument for vertical and slant TEC from MIT Haystack.

Renamed import nickname so that the source of two routines would not be confused.

Added a GNSS TEC instrument.

aburrell · 2020-08-11T17:14:31Z

There's a problem listing files, which I don't know how to resolve. Files download as expected, but the file list is not working. Additionally, specifying a file to load does not work. This needs to be fixed before I can check to see that the Datasets for the two data sets are formatted correctly.

This may be related to #5

pysatMadrigal/instruments/gnss_tec.py

rstoneback

I'm currently waiting for the load command on 'vtec' to finish.

pysatMadrigal/instruments/gnss_tec.py

rstoneback · 2020-08-11T21:16:22Z

I've started a pull on pysat with the bug fix for finding files.
pysat/#510
pysat/pysat#510

rstoneback · 2020-08-11T21:25:29Z

Loads data just fine.

In [8]: vtec.load(date=dt.datetime(2017, 11, 18))                                                                                                                                                                       

In [9]: vtec.data                                                                                                                                                                                                       
Out[9]: 
<xarray.Dataset>
Dimensions:   (gdalt: 1, gdlat: 180, glon: 360, time: 288)
Coordinates:
  * time      (time) datetime64[ns] 2017-11-18T00:02:30 ... 2017-11-18T23:57:30
  * gdlat     (gdlat) float64 -90.0 -89.0 -88.0 -87.0 ... 86.0 87.0 88.0 89.0
  * glon      (glon) float64 -180.0 -179.0 -178.0 -177.0 ... 177.0 178.0 179.0
  * gdalt     (gdalt) float64 350.0
Data variables:
    year      (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    month     (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    day       (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    hour      (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    min       (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    sec       (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    recno     (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    kindat    (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    kinst     (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    ut1_unix  (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    ut2_unix  (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    tec       (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan
    dtec      (time, gdlat, glon, gdalt) float64 nan nan nan nan ... nan nan nan

aburrell · 2020-08-12T12:04:30Z

@rstoneback can you provide your system and version details? Because it won't load on mine.

Applied code review feedback and new style standards, which include: - spelling, - fixing filename to use 2-digit year, split between 1900s and 2000s, - init and clean docstrings, - correct join string for acknowledgements, and - attaching acknowledgements and references.

Updated to: - remove python 2.7 support, - improve docstrings, - remove unnecessary comments, and - apply new standards for acknowledgements and references.

Fixed logging call and extraction of list of keys.

aburrell · 2020-08-12T20:13:49Z

Applied suggested changes and now the load works with the bugfix branch on pysat. Currently working on:

defining xarray with all variables except tec and dtec only depending on the time coordinate
determining best way to load los data, and how to best use sat_id to define method.

For help with the second point, refer to: http://cedar.openmadrigal.org/static/siteSpecific/programming_tips.pdf

Adapted the general load routine for loading data from an HDF file to allow different data variables to depend on different coordinates. This will make the loaded data set smaller, since less padding will be needed and non-unique values will not need to be maintained.

Removed the initial support for 'los' data and updated 'vtec' load to use new xarray loading capabilities.

To have a tuple with a length attribute, a comma needs to be included after the variable.

Updated example in docstring and fixed bug where variable name retrieval from xarray wasn't correctly obtaining a list of keys.

Fixed bug introduced by assuming merge was done in place.

Remove the madrigal keywords as cooridinates for data variables, but keep them as float coordinates.

Created docs directory and added VTEC figure for future examples.

aburrell · 2020-08-14T13:08:54Z

Moved the LOS task to issue #12. Added a figure that begins to address issue #14. Still need to fix one thing:

Update metadata 'units' after loading, as they are incorrectly specified in the HDF file.

aburrell · 2020-08-14T13:19:16Z

@rstoneback tried to figure out how to fix metadata, but failed miserably. Advice?

rstoneback · 2020-08-14T14:38:48Z

I'd change the units in the gnss_tec.py load method, after you get everything back from the general methods.

In [50]:  vtec.meta.data                                                                                                                                                                                                
Out[50]: 
              units long_name                                             desc     label      axis   scale notes value_min value_max fill
year              y      YEAR                            Year (universal time)      year      year  linear             NaN       NaN  NaN

In [51]: vtec.meta['year'] = {'units': 'new_units'}                                                                                                                                                                     

In [52]: vtec.meta.data                                                                                                                                                                                                 
Out[52]: 
              units long_name                                             desc     label      axis   scale notes value_min value_max fill
year      new_units      YEAR                            Year (universal time)      year      year  linear             NaN       NaN  NaN

I thought we could do something like,

vtec.meta['year', 'units'] = 'new_units'

but I was wrong and I need to go fix some recent code.

rstoneback · 2020-08-14T14:40:14Z

Correction for generality

In [51]: vtec.meta['year'] = {vtec.units_label: 'new_units'}

Fixed incorrectly specified units in the VTEC meta data.

aburrell · 2020-08-14T15:04:42Z

OK, this PR is ready to convert and review pending pysat/pysat#510

Updated the PiPy distribution name for netCDF4-Python.

Develop update for PR #11

Have Travis use develop-3, since enough breaking changes have been made.

aburrell · 2020-10-09T14:27:09Z

@jklenzing any idea what's going on with Travis?? I am very confused.

jklenzing · 2020-10-09T14:36:42Z

mm_gen needs to be updated to ps_gen on line 93 of the dmsp file.

Module nickname was not carried through entire file, fixed this.

Removed hopefully the last instance of sat_id.

Replaced print statements with logger warnings.

Added a file_format flag option to JRO ISR load and download routine.

Added ability to remove expected coordinates from HDF5 xarray loading if they aren't present. Now only errors if all variables are absent.

Flake8 correction for hanging indent.

aburrell · 2020-10-13T12:04:37Z

I did it! NetCDF solved the problem! And I managed not to break the other instruments!!!

Updated general madrigal routines by: - fixing incorrect use of `self` in `filter_data_single_date`, - changing `Notes` to `Note`, - ensuring local files have the right extension, - changing variable `file_format` to `file_type` to remove any confusion with the Instrument attribute, - changing `netcdf4` to `netCDF4` everywhere, and - specifying accepted file types.

Allow pysat to handlee multiple TEC file formats, though instruments will only load one type at a time.

Fixed bug assigning time index, now is an index of datetime values.

Fixed bug that tried to set file_type multiple times.

Adapted code to allow flexible file types, removed unused default function call, and updated the style and docstrings in clean.

Updated jro_isr to use flexible file type throughout instrument, and changed old variable name.

More efficient transfer from unix to timestamp. Fixed bug where index does not need to be assigned.

rstoneback

Thanks @aburrell! Comments posted. I ran into some issues so I wasn't able to test the hdf support.

I was able to load multiple days (netcdf files) using the appropriate pysat branch for multiple days. Thanks for the additional improvements!

pysatMadrigal/instruments/gnss_tec.py

pysatMadrigal/instruments/jro_isr.py

rstoneback · 2020-10-14T21:18:03Z

pysatMadrigal/instruments/gnss_tec.py

+# use the default pysat method
+dname = '{year:02d}{month:02d}{day:02d}'
+vname = '.{version:03d}'
+supported_tags = {ss: {'vtec': "gps{:s}g{:s}.hdf5".format(dname, vname)}


This is hardcoded to .hdf5.

pysatMadrigal/instruments/methods/madrigal.py

rstoneback · 2020-10-14T21:24:25Z

pysatMadrigal/instruments/gnss_tec.py

+        Password for data download. (default=None)
+    url : string
+        URL for Madrigal site (default='http://cedar.openmadrigal.org')
+    file_format : string


The file_format keyword conflicts with the built-in keyword at the Instrument level. I remember you said something about this during the meeting today but unfortunately I don't remember what. I'm having trouble in this area.

I tried checking out 'hdf5' support. Using the branch over in pysat (without the keyword) I get,

gps = pysat.Instrument('gnss', 'tec', tag='vtec') --------------------------------------------------------------------------- IndexError Traceback (most recent call last) <ipython-input-8-c7e28318904d> in <module> ----> 1 gps = pysat.Instrument('gnss', 'tec', tag='vtec') ~/Code/pysat/pysat/_instrument.py in _get_supported_keywords(local_func) 2950 for pop in pop_list[::-1]: 2951 args.pop(pop) -> 2952 defaults.pop(pop) 2953 2954 out_dict = {} IndexError: pop index out of range

With the keyword,

In [9]: gps = pysat.Instrument('gnss', 'tec', tag='vtec', file_format='hdf5') --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-9-06a989677482> in <module> ----> 1 gps = pysat.Instrument('gnss', 'tec', tag='vtec', file_format='hdf5') ~/Code/pysat/pysat/_instrument.py in __init__(self, platform, name, tag, inst_id, clean_level, update_files, pad, orbit_info, inst_module, multi_file_day, manual_org, directory_format, file_format, temporary_file_list, strict_time_flag, ignore_empty_files, units_label, name_label, notes_label, desc_label, plot_label, axis_label, scale_label, min_label, max_label, fill_label, **kwargs) 252 estr = 'file format set to default, supplied string must be ' 253 estr = '{:s}iteratable [{:}]'.format(estr, self.file_format) --> 254 raise ValueError(estr) 255 256 # set up empty data and metadata ValueError: file format set to default, supplied string must be iteratable [hdf5]

I was using the wrong pysat branch. You had the correct link above. My bad. Trying again...

My test setup was good the first time. Clearly I got myself turned around a bit.

The test for 'hdf5' was done with the 'delimiter_bug` branch. I confirmed that I get the pop issue or the ValueError.

Yeah, I apparently changed this bug in the following PR. Will update it here, too. Problems of finding too many related problems...

rstoneback · 2020-10-14T21:26:09Z

pysatMadrigal/instruments/gnss_tec.py

+    return
+
+
+def download(date_array, tag='', inst_id='', data_path=None, user=None,


I downloaded a few days for testing but the files I got are labeled stuff.hdf. The load time is really quick though, suggesting they are actually netcdf files. Poking at the code the 'netcdf' path is indeed being run.

yeah, regardless of the format you ask for and get, the remote files are listed as .hdf5. Not sure if I fixed the rename here or in the other PR.

pysatMadrigal/instruments/gnss_tec.py

rstoneback

I forgot a couple consistency suggestions to go along with one of my inputs.

pysatMadrigal/instruments/methods/madrigal.py

pysatMadrigal/instruments/gnss_tec.py

Several suggestions from code review, including code style and docstring typos. Co-authored-by: Russell Stoneback <rstoneba@utdallas.edu>

aburrell · 2020-10-15T14:01:15Z

This instrument will now be added with the merge of #20

aburrell added 2 commits August 11, 2020 12:52

ENH: rename import nickname

42d487f

Renamed import nickname so that the source of two routines would not be confused.

ENH: added GNSS TEC instrument

5cdeffe

Added a GNSS TEC instrument.

aburrell added the enhancement New feature or request label Aug 11, 2020

aburrell self-assigned this Aug 11, 2020

aburrell marked this pull request as draft August 11, 2020 17:13

aburrell requested a review from rstoneback August 11, 2020 17:14

jklenzing reviewed Aug 11, 2020

View reviewed changes

pysatMadrigal/instruments/gnss_tec.py Outdated Show resolved Hide resolved

rstoneback reviewed Aug 11, 2020

View reviewed changes

aburrell added 3 commits August 12, 2020 15:28

MAINT: new standards

aeab2f7

Updated to: - remove python 2.7 support, - improve docstrings, - remove unnecessary comments, and - apply new standards for acknowledgements and references.

BUG: list from keys and logging

1fb3fc4

Fixed logging call and extraction of list of keys.

aburrell added 7 commits August 13, 2020 14:35

MAINT: removed 'los' and updated 'vtec' load

d599f62

Removed the initial support for 'los' data and updated 'vtec' load to use new xarray loading capabilities.

BUG: tuple with length attribute

12afe51

To have a tuple with a length attribute, a comma needs to be included after the variable.

BUG: final xarray check

0e0f347

Updated example in docstring and fixed bug where variable name retrieval from xarray wasn't correctly obtaining a list of keys.

BUG: merge not done in place

3cfa630

Fixed bug introduced by assuming merge was done in place.

ENH: Squeeze out the madrigal keywords

d32b71c

Remove the madrigal keywords as cooridinates for data variables, but keep them as float coordinates.

DOC: VTEC example figure for docs

e8a714a

Created docs directory and added VTEC figure for future examples.

ENH: fixed metadata units

5684bb0

Fixed incorrectly specified units in the VTEC meta data.

aburrell marked this pull request as ready for review August 14, 2020 15:05

aburrell and others added 4 commits October 9, 2020 08:35

BUG: netCDF4-Python pip name

f5b451a

Updated the PiPy distribution name for netCDF4-Python.

Merge branch 'develop' into tec_inst

bdbaee6

Merge pull request #19 from pysat/develop

6dd3e57

Develop update for PR #11

TST: Travis uses develop-3

1380341

Have Travis use develop-3, since enough breaking changes have been made.

aburrell added 6 commits October 9, 2020 10:41

BUG: update module nickname

59c5983

Module nickname was not carried through entire file, fixed this.

BUG: lingering sat_id

6908d09

Removed hopefully the last instance of sat_id.

STY: removed print statement

5df53b1

Replaced print statements with logger warnings.

STY: file_format flag

ecf0403

Added a file_format flag option to JRO ISR load and download routine.

ENH: HDF5 coordinate flexibility

dcf32ef

Added ability to remove expected coordinates from HDF5 xarray loading if they aren't present. Now only errors if all variables are absent.

STY: flake8

d9f3247

Flake8 correction for hanging indent.

aburrell requested a review from rstoneback October 13, 2020 12:04

aburrell added 4 commits October 13, 2020 12:50

BUG: allow multiple TEC file formats

e94aec6

Allow pysat to handlee multiple TEC file formats, though instruments will only load one type at a time.

BUG: time index

3029d4a

Fixed bug assigning time index, now is an index of datetime values.

BUG: file_type can only be set once

15dd2b3

Fixed bug that tried to set file_type multiple times.

aburrell mentioned this pull request Oct 13, 2020

ENH: Fixes for multiple file types and adds GNSS_TEC instrument #20

Merged

3 tasks

aburrell added 3 commits October 14, 2020 08:57

STY: dmsp_ivm file_type cleanup

54ab698

Adapted code to allow flexible file types, removed unused default function call, and updated the style and docstrings in clean.

STY: jro_isr file_type update

d7e64a3

Updated jro_isr to use flexible file type throughout instrument, and changed old variable name.

BUG: index

9f25389

More efficient transfer from unix to timestamp. Fixed bug where index does not need to be assigned.

rstoneback reviewed Oct 14, 2020

View reviewed changes

pysatMadrigal/instruments/methods/madrigal.py Show resolved Hide resolved

pysatMadrigal/instruments/methods/madrigal.py Show resolved Hide resolved

aburrell commented Oct 15, 2020

View reviewed changes

pysatMadrigal/instruments/gnss_tec.py Outdated Show resolved Hide resolved

aburrell and others added 2 commits October 15, 2020 09:48

STY: Apply suggestions from code review

f316ad9

Several suggestions from code review, including code style and docstring typos. Co-authored-by: Russell Stoneback <rstoneba@utdallas.edu>

Merge branch 'inst_kwarg_updates' into tec_inst

fc50297

aburrell closed this Oct 15, 2020

aburrell deleted the tec_inst branch December 18, 2020 15:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: new GNSS TEC instrument #11

ENH: new GNSS TEC instrument #11

aburrell commented Aug 11, 2020

aburrell commented Aug 11, 2020 •

edited

rstoneback left a comment

rstoneback commented Aug 11, 2020

rstoneback commented Aug 11, 2020

aburrell commented Aug 12, 2020

aburrell commented Aug 12, 2020 •

edited

aburrell commented Aug 14, 2020 •

edited

aburrell commented Aug 14, 2020

rstoneback commented Aug 14, 2020

rstoneback commented Aug 14, 2020

aburrell commented Aug 14, 2020 •

edited

aburrell commented Oct 9, 2020

jklenzing commented Oct 9, 2020

aburrell commented Oct 13, 2020

rstoneback left a comment

rstoneback Oct 14, 2020

rstoneback Oct 14, 2020

rstoneback Oct 14, 2020

rstoneback Oct 14, 2020

aburrell Oct 15, 2020

rstoneback Oct 14, 2020

aburrell Oct 15, 2020

rstoneback left a comment

aburrell commented Oct 15, 2020

		return


		def download(date_array, tag='', inst_id='', data_path=None, user=None,

ENH: new GNSS TEC instrument #11

ENH: new GNSS TEC instrument #11

Conversation

aburrell commented Aug 11, 2020

aburrell commented Aug 11, 2020 • edited

rstoneback left a comment

Choose a reason for hiding this comment

rstoneback commented Aug 11, 2020

rstoneback commented Aug 11, 2020

aburrell commented Aug 12, 2020

aburrell commented Aug 12, 2020 • edited

aburrell commented Aug 14, 2020 • edited

aburrell commented Aug 14, 2020

rstoneback commented Aug 14, 2020

rstoneback commented Aug 14, 2020

aburrell commented Aug 14, 2020 • edited

aburrell commented Oct 9, 2020

jklenzing commented Oct 9, 2020

aburrell commented Oct 13, 2020

rstoneback left a comment

Choose a reason for hiding this comment

rstoneback Oct 14, 2020

Choose a reason for hiding this comment

rstoneback Oct 14, 2020

Choose a reason for hiding this comment

rstoneback Oct 14, 2020

Choose a reason for hiding this comment

rstoneback Oct 14, 2020

Choose a reason for hiding this comment

aburrell Oct 15, 2020

Choose a reason for hiding this comment

rstoneback Oct 14, 2020

Choose a reason for hiding this comment

aburrell Oct 15, 2020

Choose a reason for hiding this comment

rstoneback left a comment

Choose a reason for hiding this comment

aburrell commented Oct 15, 2020

aburrell commented Aug 11, 2020 •

edited

aburrell commented Aug 12, 2020 •

edited

aburrell commented Aug 14, 2020 •

edited

aburrell commented Aug 14, 2020 •

edited