Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Reader for SWI_TS product of CGLOPS - update testdata repo. #13

Merged
merged 5 commits into from
Jan 17, 2017
Merged

Add Reader for SWI_TS product of CGLOPS - update testdata repo. #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
30fa763
Add Reader for SWI_TS product of CGLOPS - update testdata repo.
Jan 17, 2017
52b5141
Update requirements and environment.yml to new pygeogrids version.
Jan 17, 2017
a7c52c1
Fix netCDF4 requirement to version 1.2.4 for now.
Jan 17, 2017
e94d137
Handle porosity of old ascat versions properly.
Jan 17, 2017
a55a156
Rename clgops module to cgls and update documentation.
Jan 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGES.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@
Changelog
=========

Version 0.3
===========

- Add reader for Copernicus Global Land SWI_TS products.

Version 0.2
===========

Expand Down
6 changes: 6 additions & 0 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,12 @@ Time Series Products
<http://hsaf.meteoam.it/soil-moisture.php>`_ (H25 product)


* CGLS SWI(Soil Water Index) Time Series (SWI_TS)

Available from the `Copernicus Global Land Service (CGLS)
<http://land.copernicus.eu/global/products/swi>`_


* ASCAT SWI(Soil Water Index) Time Series

Available in binary format from `TU Wien <http://rs.geo.tuwien.ac.at/products/>`_
Expand Down
104 changes: 104 additions & 0 deletions ascat/cgls.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# Copyright (c) 2017, Vienna University of Technology (TU Wien),
# Department of Geodesy and Geoinformation (GEO).
# All rights reserved.
#
# All information contained herein is, and remains the property of Vienna
# University of Technology (TU Wien), Department of Geodesy and Geoinformation
# (GEO). The intellectual and technical concepts contained herein are
# proprietary to Vienna University of Technology (TU Wien), Department of
# Geodesy and Geoinformation (GEO). Dissemination of this information or
# reproduction of this material is forbidden unless prior written permission
# is obtained from Vienna University of Technology (TU Wien), Department of
# Geodesy and Geoinformation (GEO).

'''
Module for reading CGLOPS SWI TS products
'''

import pynetcf.time_series as netcdf_dataset
import pygeogrids.netcdf as netcdf

import os
import numpy as np


class SWI_TS(netcdf_dataset.GriddedNcOrthoMultiTs):

"""
SWI TS reader for timeseries data from CGLOPS

Parameters
----------
data_path: string
path to the netCDF files
parameters: list
list of parameters to read from netCDF file
dt: string, optional
datetime in the filenames of the cells
version: string, optional
version number of the files
grid_fname: string, optional
filename + path of the grid netCDF file,
default is the standard grid file (c_gls_SWI-STATIC-DGG_201501010000_GLOBE_ASCAT_V3.0.1.nc)
in the same folder as the data
read_bulk: boolean, optional
if set to true then a complete 5x5 degree cell will be read at once
providing speedup if the complete data is needed.
fname_template: string, optional
Filename template. Has to have two slots for {dt} and {version} and a
slot for the cell number that is available for further formatting.
Because of this the cell number location has to be written as '{{:04d}}'.
The has to be without the .nc ending since this is added during reading.
"""

def __init__(self, data_path, parameters=['SWI_001', 'SWI_005', 'SWI_010',
'SWI_015', 'SWI_020', 'SWI_040',
'SWI_060', 'SWI_100', 'SSF'],
dt='201612310000', version='3.0.1',
grid_fname=None, read_bulk=True,
fname_template='c_gls_SWI-TS_{dt}_C{{:04d}}_ASCAT_V{version}'):

if grid_fname is None:
grid_fname = os.path.join(
data_path, 'c_gls_SWI-STATIC-DGG_201501010000_GLOBE_ASCAT_V3.0.1.nc')
grid = netcdf.load_grid(grid_fname, location_var_name='location_id')

scale_factors = {'SWI_001': 0.5,
'SWI_005': 0.5,
'SWI_010': 0.5,
'SWI_015': 0.5,
'SWI_020': 0.5,
'SWI_040': 0.5,
'SWI_060': 0.5,
'SWI_100': 0.5,
'SSF': 1}

dtypes = {'SWI_001': np.uint8,
'SWI_005': np.uint8,
'SWI_010': np.uint8,
'SWI_015': np.uint8,
'SWI_020': np.uint8,
'SWI_040': np.uint8,
'SWI_060': np.uint8,
'SWI_100': np.uint8,
'SSF': np.uint8}

super(SWI_TS, self).__init__(
data_path, grid,
fn_format=fname_template.format(dt=dt, version=version),
parameters=parameters, scale_factors=scale_factors,
dtypes=dtypes, autoscale=False,
automask=False, ioclass_kws={'read_bulk': read_bulk,
'loc_ids_name': 'locations'})

def read_ts(self, gpi, period=None, mask_frozen=True):
data = super(SWI_TS, self).read_ts(gpi, period=period)

if mask_frozen is True:
unfrozen = data['SSF'].values <= 1
data = data[unfrozen]

for column in data:
data.loc[data[column] > 100, column] = np.nan

return data
6 changes: 5 additions & 1 deletion ascat/timeseries.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,8 +294,12 @@ def _read_gp(self, gpi, **kwargs):
# read porosity values
porosity = {}
for por_source in ['gldas', 'hwsd']:
porosity[por_source] = ncfile.variables[
ncfile.variables['por_%s' % por_source].set_auto_mask(False)
por_value = ncfile.variables[
'por_%s' % por_source][gpi_index]
if por_value == ncfile.variables['por_%s' % por_source].missing_value:
por_value = np.nan
porosity[por_source] = por_value

if 'absolute_values' in kwargs:

Expand Down
13 changes: 13 additions & 0 deletions docs/examples.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,19 @@ documentation of :class:`ascat.timeseries.AscatH25_SSM`
.. include::
read_ASCAT_H25.rst

Reading and plotting CGLS SWI_TS data from netCDF format
========================================================

This example script reads the SWI_TS product of the Copernicus Global Land
Service.

If the standard file names assumed by the script have changed this can be
specified during initialization of the SWI_TS object. Please see the
documentation of :class:`ascat.cgls.SWI_TS`.

.. include::
read_cgls_swi_ts.rst

Reading and plotting H-SAF images
=================================

Expand Down
135 changes: 135 additions & 0 deletions docs/read_cgls_swi_ts.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@

First we import the necessary class

.. code:: python

import os
from ascat.cgls import SWI_TS

By default we should have the grid file from the SWI-STATIC collection
and the unzipped SWI-TS products in one folder like so:

.. code:: python

ls tests/test-data/sat/cglops/swi_ts


.. parsed-literal::

c_gls_SWI-STATIC-DGG_201501010000_GLOBE_ASCAT_V3.0.1.nc
c_gls_SWI-TS_201612310000_C0375_ASCAT_V3.0.1.nc


Now we can initialize the data reader

.. code:: python

data_path = os.path.join('tests', 'test-data', 'sat', 'cglops', 'swi_ts')
rd = SWI_TS(data_path)

With this object we can read the data

.. code:: python

data = rd.read_ts(3002621)
print(data)


.. parsed-literal::

SSF SWI_001 SWI_005 SWI_010 SWI_015 SWI_020
2007-05-10 12:00:00 1.0 70.0 71.0 71.0 71.0 71.0
2007-05-14 12:00:00 1.0 71.5 71.5 71.5 71.5 71.5
2007-05-15 12:00:00 1.0 82.0 78.0 77.0 76.5 76.5
2007-05-16 12:00:00 1.0 82.5 79.0 78.0 77.5 77.0
2007-05-17 12:00:00 1.0 77.5 77.5 77.0 76.5 76.5
2007-05-18 12:00:00 1.0 75.0 76.5 76.0 76.0 76.0
2007-05-19 12:00:00 1.0 75.5 76.0 76.0 76.0 76.0
2007-05-20 12:00:00 1.0 86.0 81.0 79.5 79.5 79.0
2007-05-21 12:00:00 1.0 84.5 81.0 80.0 79.5 79.5
2007-05-22 12:00:00 1.0 81.5 80.5 80.0 79.5 79.0
2007-05-23 12:00:00 1.0 81.5 80.5 80.0 79.5 79.5
2007-05-24 12:00:00 1.0 81.0 81.0 80.0 80.0 79.5
2007-05-25 12:00:00 1.0 79.5 80.0 79.5 79.5 79.5
2007-05-26 12:00:00 1.0 81.5 80.5 80.0 79.5 79.5
2007-05-27 12:00:00 1.0 76.5 79.5 79.5 79.0 79.0
2007-05-28 12:00:00 1.0 80.5 79.5 79.5 79.5 79.5
2007-05-29 12:00:00 1.0 82.0 80.5 80.0 79.5 79.5
2007-05-30 12:00:00 1.0 83.0 81.0 80.5 80.0 80.0
2007-05-31 12:00:00 1.0 79.0 80.5 80.0 80.0 80.0
... ... ... ... ... ... ...
2016-10-01 12:00:00 1.0 90.0 84.0 81.0 81.0 81.0
2016-10-02 12:00:00 1.0 84.0 83.5 81.0 80.5 81.0
2016-10-03 12:00:00 1.0 72.0 80.0 79.5 80.0 80.0
2016-10-04 12:00:00 1.0 64.5 76.0 77.5 78.5 79.0
2016-10-05 12:00:00 1.0 57.5 71.5 75.0 76.5 78.0
2016-10-06 12:00:00 1.0 65.0 70.5 74.0 75.5 77.0
2016-10-07 12:00:00 1.0 66.0 70.0 73.5 75.0 76.5
2016-10-08 12:00:00 1.0 68.0 69.5 73.0 75.0 76.0
2016-10-09 12:00:00 1.0 67.5 69.0 72.5 74.5 76.0
2016-10-10 12:00:00 1.0 65.5 68.5 71.5 74.0 75.5
2016-10-11 12:00:00 1.0 60.0 66.0 70.0 72.5 74.0
2016-10-12 12:00:00 1.0 54.0 63.5 68.5 71.0 73.0
2016-10-13 12:00:00 1.0 52.5 61.0 66.5 69.5 72.0
2016-10-14 12:00:00 1.0 55.0 60.0 65.5 69.0 71.0
2016-10-15 12:00:00 1.0 56.5 59.0 64.5 68.0 70.5
2016-10-16 12:00:00 1.0 61.5 60.0 64.5 67.5 70.0
2016-10-26 12:00:00 1.0 12.5 42.5 59.0 64.5 68.0
2016-10-28 12:00:00 1.0 69.5 54.0 60.0 64.5 67.5

SWI_040 SWI_060 SWI_100
2007-05-10 12:00:00 71.0 NaN NaN
2007-05-14 12:00:00 71.5 71.5 NaN
2007-05-15 12:00:00 76.0 76.0 76.0
2007-05-16 12:00:00 77.0 77.0 76.5
2007-05-17 12:00:00 76.0 76.0 76.0
2007-05-18 12:00:00 76.0 76.0 76.0
2007-05-19 12:00:00 76.0 75.5 75.5
2007-05-20 12:00:00 79.0 78.5 78.5
2007-05-21 12:00:00 79.0 79.0 79.0
2007-05-22 12:00:00 79.0 79.0 78.5
2007-05-23 12:00:00 79.0 79.0 79.0
2007-05-24 12:00:00 79.5 79.5 79.0
2007-05-25 12:00:00 79.0 79.0 79.0
2007-05-26 12:00:00 79.5 79.0 79.0
2007-05-27 12:00:00 79.0 79.0 79.0
2007-05-28 12:00:00 79.0 79.0 79.0
2007-05-29 12:00:00 79.5 79.5 79.0
2007-05-30 12:00:00 80.0 79.5 79.5
2007-05-31 12:00:00 79.5 79.5 79.5
... ... ... ...
2016-10-01 12:00:00 82.5 83.0 83.5
2016-10-02 12:00:00 82.5 83.0 83.5
2016-10-03 12:00:00 82.0 82.5 83.0
2016-10-04 12:00:00 81.5 82.5 83.0
2016-10-05 12:00:00 80.5 81.5 82.5
2016-10-06 12:00:00 80.0 81.5 82.0
2016-10-07 12:00:00 79.5 81.0 82.0
2016-10-08 12:00:00 79.5 81.0 82.0
2016-10-09 12:00:00 79.0 80.5 81.5
2016-10-10 12:00:00 79.0 80.5 81.5
2016-10-11 12:00:00 78.0 80.0 81.0
2016-10-12 12:00:00 77.5 79.5 81.0
2016-10-13 12:00:00 77.0 79.0 80.5
2016-10-14 12:00:00 76.5 78.5 80.0
2016-10-15 12:00:00 75.5 78.0 80.0
2016-10-16 12:00:00 75.5 78.0 79.5
2016-10-26 12:00:00 74.5 77.0 79.0
2016-10-28 12:00:00 74.0 76.5 79.0

[1603 rows x 9 columns]


Since the returned value is a pandas.DataFrame we can plot the data
easily.

.. code:: python

import matplotlib.pyplot as plt
data[['SWI_001', 'SWI_010']].plot()
plt.show()



.. image:: read_cgls_swi_ts_files/output_9_0.png

Binary file added docs/read_cgls_swi_ts_files/output_9_0.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ dependencies:
- ptyprocess==0.5.1
- pybufr-ecmwf==0.81
- pygeobase==0.3.9
- pygeogrids==0.2.0
- pygeogrids==0.2.2
- pygments==2.1.3
- pykdtree==1.1.1
- pytest-cov==2.3.0
Expand Down
Loading