add batch conversion rnx2hdf5.py

robust test and batch naming

README.md

@@ -63,30 +63,29 @@ make install -C rnxcmp
 
 #### Windows
 Windows as usual is more difficult to compile code on.
-For optional Hatanaka converter, here are some tips.
-
-> cc -O2 source/rnx2crx.c -o rnx2crx
-process_begin: CreateProcess(NULL, cc -O2 source/rnx2crx.c -o rnx2crx, ...) failed.
-make (e=2): The system cannot find the file specified.
-
-Assuming you have 
+For optional Hatanaka converter on Windows, assuming you have
 [installed MinGW compiler on Windows](https://www.scivision.co/windows-gcc-gfortran-cmake-make-install/):
 ```posh
 set CC=gcc
 mingw32-make -C rnxcmp
 ```
 
-
 ## Usage
 
 The simplest command-line use is through the top-level `ReadRinex` script.
 
--   Read RINEX3 or RINEX 2 Obs or Nav file: `ReadRinex myrinex.XXx`
-    * you can read multiple files bounded by `--tlim` like
-      ```sh
-      ReadRinex ~/data/ --tlim 2017-01-02-T21 2017-01-03T01
-      ```
--   Read NetCDF converted RINEX data: `ReadRinex myrinex.nc`
+* Read single RINEX3 or RINEX 2 Obs or Nav file:
+  ```sh
+  ReadRinex myrinex.XXx
+  ```
+* Read NetCDF converted RINEX data:
+  ```sh
+  ReadRinex myrinex.nc
+  ```
+* Batch convert RINEX to NetCDF4 / HDF5 (this example for RINEX 2 OBS):
+  ```sh
+  ./rnx2hdf5.py -o ~/data -g "*.*o"
+  ```
 
 By default all plots and status messages are off, unless using the `-v --verbose` option to save processing time.
 
@@ -100,21 +99,36 @@ the following examples. Each example assumes you have first done:
 import georinex as gr
 ```
 
+### Time limits
+Time bounds can be set for reading -- load only data between those time bounds with the
+```sh
+--tlim start stop
+```
+option, where `start` and `stop` are formatted like `2017-02-23T12:00`
 
+```python
+dat = gr.load('my.rnx', tlim=['2017-02-23T12:59', '2017-02-23T13:13'])
+```
 
-### Benchmark
+### Measurement selection
+Further speed increase can arise from reading only wanted measurements:
+```sh
+--meas C1C L1C
+```
 
-An Intel Haswell i7-3770 CPU with plain uncompressed RINEX 2 OBS processes in about:
-* [6 MB file](ftp://data-out.unavco.org/pub/rinex/obs/2018/021/ab140210.18o.Z): 5 seconds
-* [13 MB file](ftp://data-out.unavco.org/pub/rinex/obs/2018/021/ab180210.18o.Z): 10 seconds
 
-This processing speed is about within a factor of 5 of compiled RINEX parsers, with the convenience of Python, Xarray, Pandas and HDF5 / NetCDF4.
+```python
+dat = gr.load('my.rnx', meas=['C1C', 'L1C'])
+```
+
+### Use Signal and Loss of Lock indicators
+By default, the SSI and LLI (loss of lock indicators) are not loaded to speed up the program and save memory.
+If you need them, the `-useindicators` option loads SSI and LLI for OBS 2/3 files.
 
 
-### read RINEX
+## read RINEX
 
-This convenience function reads any possible RINEX 2/3 OBS/NAV or .nc
-file:
+This convenience function reads any possible format (including compressed, Hatanaka) RINEX 2/3 OBS/NAV or `.nc` file:
 
 ```python
 obs = gr.load('tests/demo.10o')
@@ -138,7 +152,7 @@ times = gr.gettimes('~/my.rnx')
 ```
 
 
-### read Obs
+## read Obs
 
 If you desire to specifically read a RINEX 2 or 3 OBS file:
 
@@ -159,26 +173,13 @@ Not every receiver receives every type of GNSS system.
 Most Android devices in the Americas receive at least GPS and GLONASS.
 
 
-
-
-#### Time limits
-For very large files, time bounds can be set -- load only data between those time bounds with the
-```python
---tlim start stop
-```
-option, where `start` and `stop` are formatted like `2017-02-23T12:00`
-
-#### Use Signal and Loss of Lock indicators
-By default, the SSI and LLI (loss of lock indicators) are not loaded to speed up the program and save memory.
-If you need them, the `-useindicators` option loads SSI and LLI.
-
-#### get OBS header
+### read OBS header
 To get a `dict()` of the RINEX file header:
 ```python
 hdr = gr.rinexheader('myfile.rnx')
 ```
 
-#### Index OBS data
+### Index OBS data
 
 assume the OBS data from a file is loaded in variable `obs`.
 
@@ -206,7 +207,7 @@ Eind = obs.sv.to_index().str.startswith('E')  # returns a simple Numpy Boolean 1
 Edata = obs.isel(sv=Eind)  # any combination of other indices at same time or before/after also possible
 ```
 
-####  Plot OBS data
+###  Plot OBS data
 
 Plot for all satellites L1C:
 ```python
@@ -221,7 +222,7 @@ Suppose L1C pseudorange plot is desired for `G13`:
 obs['L1C'].sel(sv='G13').dropna(dim='time',how='all').plot()
 ```
 
-### read Nav
+## read Nav
 
 
 If you desire to specifically read a RINEX 2 or 3 NAV file:
@@ -233,7 +234,7 @@ This returns an `xarray.Dataset` of the data within the RINEX 3 or RINEX 2 Navig
 Indexed by time x quantity
 
 
-#### Index NAV data
+### Index NAV data
 
 assume the NAV data from a file is loaded in variable `nav`.
 Select satellite(s) (here, `G13`) by
@@ -273,7 +274,19 @@ Although Pandas DataFrames are 2-D, using say `df = nav.to_dataframe()` will res
 Satellites can be selected like `df.loc['G12'].dropna(0, 'all')` using the usual
 [Pandas Multiindexing methods](http://pandas.pydata.org/pandas-docs/stable/advanced.html).
 
-## Benchmarks
+## Benchmark
+
+An Intel Haswell i7-3770 CPU with plain uncompressed RINEX 2 OBS processes in about:
+* [6 MB file](ftp://data-out.unavco.org/pub/rinex/obs/2018/021/ab140210.18o.Z): 5 seconds
+* [13 MB file](ftp://data-out.unavco.org/pub/rinex/obs/2018/021/ab180210.18o.Z): 10 seconds
+
+This processing speed is about within a factor of 2 of compiled RINEX parsers, with the convenience of Python, Xarray, Pandas and HDF5 / NetCDF4.
+
+OBS2 and NAV2 currently have the fast pure Python read that has C-like speed.
+
+### Obs3
+OBS3 / NAV3 are not yet updated to new fast pure Python method.
+
 Done on 5 year old Haswell laptop:
 ```sh
 time ./ReadRinex.py tests/CEDA00USA_R_20182100000_23H_15S_MO.rnx.gz -u E
@@ -300,6 +313,8 @@ and `ipython`:
 ```
 shows that `np.genfromtxt()` is consuming about 30% of processing time, and `xarray.concat` and xarray.Dataset` nested inside `concat` takes over 60% of time.
 
+
+
 ## Notes
 
 RINEX 3.03 [specification](ftp://igs.org/pub/data/format/rinex303.pdf)

ReadRinex.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 """
-Reads RINEX 2/3 OBS/NAV files and plots.
+Reads RINEX 2/3 OBS/NAV file and plot (or convert to NetCDF4 / HDF5).
 Returns data as xarray.Dataset, think of it like an N-dimensional Numpy NDarray with lots of metadata and
 very fancy indexing methods.
 Xarray can be thought of as an analytically-tuned Pandas.
@@ -10,62 +10,43 @@
     * GZIP .gz
     * ZIP .zip
     * LZW .Z
+    * Hatanaka .crx / .crx.gz
 are handled seamlessly via TextIO stream.
 
 Examples:
 
+# read RINEX files (NAV/OBS, Rinex 2 or 3, Hatanaka, etc.)
 ./ReadRinex.py ~/data/VEN100ITA_R_20181580000_01D_MN.rnx.gz
 ./ReadRinex.py ~/data/ABMF00GLP_R_20181330000_01D_30S_MO.zip
 
+# read a limited range of time in a RINEX file
 ./ReadRinex.py ~/data/PUMO00CR__R_20180010000_01D_15S_MO.rnx -t 2018-01-01 2018-01-01T00:30
 
 """
-from pathlib import Path
 from argparse import ArgumentParser
 import georinex as gr
-import georinex.plots as grp
-try:
-    from matplotlib.pyplot import show
-except ImportError:
-    show = None
 
 
 def main():
     p = ArgumentParser(description='example of reading RINEX 2/3 Navigation/Observation file')
     p.add_argument('rinexfn', help='path to RINEX 2 or RINEX 3 file')
-    p.add_argument('-g', '--glob', help='file glob pattern', default='*')
-    p.add_argument('-o', '--outfn', help='write data as NetCDF4 file')
+    p.add_argument('-o', '--out', help='write data to path or file as NetCDF4')
     p.add_argument('-v', '--verbose', action='store_true')
+    p.add_argument('-p', '--plot', help='display plots', action='store_true')
     p.add_argument('-u', '--use', help='select which GNSS system(s) to use', nargs='+')
     p.add_argument('-m', '--meas', help='select which GNSS measurement(s) to use', nargs='+')
     p.add_argument('-t', '--tlim', help='specify time limits (process part of file)', nargs=2)
     p.add_argument('-useindicators', help='use SSI, LLI indicators (signal, loss of lock)',
                    action='store_true')
     P = p.parse_args()
 
-    verbose = P.verbose
-
-    fn = Path(P.rinexfn).expanduser()
-
-    if fn.is_file():
-        data = gr.load(P.rinexfn, P.outfn, use=P.use, tlim=P.tlim,
-                       useindicators=P.useindicators, meas=P.meas, verbose=verbose)
-    elif fn.is_dir():
-        flist = [f for f in fn.glob(P.glob) if f.is_file()]
-        for f in flist:
-            try:
-                data = gr.load(f, P.outfn, use=P.use, tlim=P.tlim,
-                               useindicators=P.useindicators, meas=P.meas, verbose=verbose)
-            except Exception as e:
-                print(f'{f.name}: {e}')
-                continue
-
-            if verbose:
-                grp.timeseries(data)
-    else:
-        raise FileNotFoundError(f'{fn} is not a path or file')
+    data = gr.load(P.rinexfn, P.out, use=P.use, tlim=P.tlim,
+                   useindicators=P.useindicators, meas=P.meas, verbose=P.verbose)
 # %% plots
-    if verbose and show is not None:
+    if P.plot:
+        import georinex.plots as grp
+        from matplotlib.pyplot import show
+
         grp.timeseries(data)
         show()
 

georinex/__init__.py

@@ -1,4 +1,5 @@
-from .base import load, rinexnav, rinexobs, gettime, rinextype, rinexheader  # noqa: F401
+from .base import load, rinexnav, rinexobs, batch_convert  # noqa: F401
+from .utils import gettime, rinextype, rinexheader  # noqa: F401
 from .io import rinexinfo  # noqa: F401
 from .obs2 import rinexobs2, obsheader2, obstime2  # noqa: F401
 from .obs3 import rinexobs3, obsheader3, obstime3  # noqa: F401