Skip to content
/ ismn Public

Readers for the data from the International Soil Moisture Network

License

Notifications You must be signed in to change notification settings

TUW-GEO/ismn

Repository files navigation

ismn

ci cov pip doc

Readers for data from the International Soil Moisture Network (ISMN) https://ismn.earth.

This package is installable through pip (verified for python>=3.9)

pip install ismn

This should also install all dependencies on Window, Linux or MacOS.

Quickstart

Initialise an ISMN_Interface by passing the path to your downloaded data. The interface shows you available ISMN networks, stations and sensors. You can load sensor time series as pandas DataFrames as well as metadata on the stations surroundings, soil conditions and probes (depths, sensor type, etc.).

>> from ismn.interface import ISMN_Interface
""" .zip archives are downloaded from https://ismn.earth """
>> ds = ISMN_Interface('Data_separate_files_header_20090101_20201231_9289_Cwpc_20221201.zip')
""" Read time series from your previously downloaded ISMN archive as pandas DataFrames """
>> ds["REMEDHUS"]["Canizal"][0].data

    Out[0]:
                             soil_moisture soil_moisture_flag soil_moisture_orig_flag
    date_time
    2009-01-01 00:00:00          0.372                  G                       M
    2009-01-01 01:00:00          0.372                  G                       M
    ...                          ...                   ...                     ...
    2020-12-31 22:00:00          0.285                  G                       M
    2020-12-31 23:00:00          0.285                  G                       M

""" Each ISMN sensor comes with additional information on soil/landcover/climate etc. """
>> ds["REMEDHUS"]["Canizal"][0].metadata.to_pd()

    Out[0]:
    variable        key
    climate_KG      val                           BSk
    instrument      val           Stevens-Hydra-Probe
                    depth_from                    0.0
                    depth_to                     0.05
    ...             ...                           ...
    latitude        val                      41.19603
    lc_2010         val                            20
    longitude       val                      -5.35997
    network         val                      REMEDHUS
    station         val                       Canizal

Many more features to e.g. visualise, select or transform data are available. See the full documentation.

Documentation

The full documentation is available at https://ismn.readthedocs.io/en/latest and includes a tutorial on reading ISMN data in python after downloading it from https://ismn.earth

The following tutorials are also available as ipython notebooks in docs/examples:

  1. ISMN reader basic functionality
  2. Adding custom metadata readers

Data used in the tutorials is not provided in this package. Please create an account at ismn.earth to download the required files.

For a detailed description of the ISMN, technical data aspects (properties, coverage, etc.) and correct usage (applications), see

W. Dorigo et al. The International Soil Moisture Network: serving Earth system science for over a decade, Hydrol. Earth Syst. Sci., 25, 5749–5804, https://doi.org/10.5194/hess-25-5749-2021, 2021.

Optional dependencies

The cartopy and matplotlib packages are only needed when creating data visualisations. They can be installed separately via

conda install -c conda-forge matplotlib cartopy

or pip install ismn[plot] for most operating systems if you already have geos installed.

If you want to convert ISMN data into xarray objects, please install xarray and dask via

conda install -c conda-forge xarray dask

or pip install ismn[xr] for most operating systems.

Citation

If you use the software in a publication then please cite it using the Zenodo DOI. Be aware that this badge links to the latest package version.

Please select your specific version at https://doi.org/10.5281/zenodo.855308 to get the DOI of that version. You should normally always use the DOI for the specific version of your record in citations. This is to ensure that other researchers can access the exact research artefact you used for reproducibility.

You can find additional information regarding DOI versioning at http://help.zenodo.org/#versioning

Description

ISMN data can be downloaded for free after creating an account on the ISMN Website

ISMN data can be downloaded in two different formats:

  • Variables stored in separate files (CEOP formatted)
  • Variables stored in separate files (Header+values) (default format)

Both formats are supported by this package.

If you downloaded ISMN data in one of the supported formats in the past it can be that station names are not recognized correctly because they contained the '_' character which is supposed to be the separator. If you experience problems because of this please download new data from the ISMN since this issue should be fixed.

Variables and Units

The following variables are available in the ISMN. Note that not every station measures all of the variables. You can use this package to read only data for locations where one or multiple of the variables were measured.

Temporally dynamic variables and their units in ISMN
Variable Units
Soil Moisture m3/m3
Soil Suction kPa
Soil Temperature °C
Air Temperature °C
Surface Temperature °C
Precipitation mm
Snow Depth mm
Snow Water Equivalent mm

Temporally static variables and their units in ISMN
Variable Units
Climate classification None
Land cover classification None
Soil classification None
Bulk density g/cm³
Sand fraction % weight
Silt fraction % weight
Clay fraction % weight
Organic carbon % weight
Saturation % vol
Field capacity % vol
Potential plant available water % vol
Permanent wilting point % vol

Landcover Classification

The ISMN data comes with information about landcover classification from the ESA CCI land cover project (years 2000, 2005 and 2010) as well as from in-situ measurements. To use ESA CCI land cover variables for filtering the data in the get_dataset_ids function, set the keyword parameters (landcover_2000, landcover_2005 or landcover_2010) to the corresponding integer values (e.g. 10) in the list below. To get a list of possible values for filtering by in-situ values (keyword parameter: "landcover_insitu"), call the get_landcover_types method of your ISMN_Interface object and set landcover='landcover_insitu'.

ISMN Landcover classes and meanings
Value Meaning
10 Cropland, rainfed
11 Cropland, rainfed / Herbaceous cover
12 Cropland, rainfed / Tree or shrub cover
20 Cropland, irrigated or post-flooding
30 Mosaic cropland (>50%) / natural vegetation (tree, shrub, herbaceous)
40 Mosaic natural vegetation (>50%) / cropland (<50%)
50 Tree cover, broadleaved, evergreen, Closed to open (>15%)
60 Tree cover, broadleaved, deciduous, Closed to open (>15%)
61 Tree cover, broadleaved, deciduous, Closed (>40%)
62 Tree cover, broadleaved, deciduous, Open (15-40%)
70 Tree cover, needleleaved, evergreen, Closed to open (>15%)
71 Tree cover, needleleaved, evergreen, Closed (>40%)
72 Tree cover, needleleaved, evergreen, Open (15-40%)
80 Tree cover, needleleaved, deciduous, Closed to open (>15%)
81 Tree cover, needleleaved, deciduous, Closed (>40%)
82 Tree cover, needleleaved, deciduous, Open (15-40%)
90 Tree cover, mixed leaf type (broadleaved and needleleaved)
100 Mosaic tree and shrub (>50%) / herbaceous cover (<50%)
110 Mosaic herbaceous cover (>50%) / tree and shrub (<50%)
120 Shrubland
121 Shrubland / Evergreen Shrubland
122 Shrubland / Deciduous Shrubland
130 Grassland
140 Lichens and mosses
150 Sparse vegetation (tree, shrub, herbaceous cover) (<15%)
152 Sparse vegetation (<15%) / Sparse shrub (<15%)
153 Sparse vegetation (<15%) / Sparse herbaceous cover (<15%)
160 Tree cover, flooded, fresh or brackish water
170 Tree cover, flooded, saline water
180 Shrub or herbaceous cover, flooded, fresh/saline/brackish water
190 Urban areas
200 Bare areas
201 Consolidated bare areas
202 Unconsolidated bare areas
210 Water
220 Permanent snow and ice

Climate Classification

The ISMN data comes with information about climate classification from the Koeppen-Geiger Climate Classification (2007) as well as in-situ measurements. To use Koeppen-Geiger variable for filtering the data in the get_dataset_ids function, set the keyword parameter "climate" to the corresponding keys (e.g. 'Af') in the list below. To get a list of possible values for filtering by in-situ values (keyword parameter: "climate_insitu"), call the get_climate_types method of your ISMN_Interface object and set climate='climate_insitu'.

Climate Classes and Meanings
Class Meaning
Af Tropical Rainforest
Am Tropical Monsoon
As Tropical Savanna Dry
Aw Tropical Savanna Wet
BWk Arid Desert Cold
BWh Arid Desert Hot
BWn Arid Desert With Frequent Fog
BSk Arid Steppe Cold
BSh Arid Steppe Hot
BSn Arid Steppe With Frequent Fog
Csa Temperate Dry Hot Summer
Csb Temperate Dry Warm Summer
Csc Temperate Dry Cold Summer
Cwa Temperate Dry Winter, Hot Summer
Cwb Temperate Dry Winter, Warm Summer
Cwc Temperate Dry Winter, Cold Summer
Cfa Temperate Without Dry Season, Hot Summer
Cfb Temperate Without Dry Season, Warm Summer
Cfc Temperate Without Dry Season, Cold Summer
Dsa Cold Dry Summer, Hot Summer
Dsb Cold Dry Summer, Warm Summer
Dsc Cold Dry Summer, Cold Summer
Dsd Cold Dry Summer, Very Cold Winter
Dwa Cold Dry Winter, Hot Summer
Dwb Cold Dry Winter, Warm Summer
Dwc Cold Dry Winter, Cold Summer
Dwd Cold Dry Winter, Very Cold Winter
Dfa Cold Dry Without Dry Season, Hot Summer
Dfb Cold Dry Without Dry Season, Warm Summer
Dfc Cold Dry Without Dry Season, Cold Summer
Dfd Cold Dry Without Dry Season, Very Cold Winter
ET Polar Tundra
EF Polar Eternal Winter
W Water

Contribute

We are happy if you want to contribute. Please raise an issue explaining what is missing or if you find a bug. We will also gladly accept pull requests against our master branch for new features or bug fixes.

Guidelines

If you want to contribute please follow these steps:

  • Fork the ismn repository to your account
  • Clone the repository
  • make a new feature branch from the ismn master branch
  • Add your feature
  • Please include tests for your contributions in one of the test directories. We use pytest so a simple function called test_my_feature is enough
  • submit a pull request to our master branch

Code Formatting

To apply pep8 conform styling to any changed files [we use yapf](https://github.com/google/yapf). The correct settings are already set in setup.cfg. Therefore the following command should be enough:

yapf file.py --in-place

Release new version

To release a new version of this package, make sure all tests are passing on the master branch and the CHANGELOG.rst is up-to-date, with changes for the new version at the top.

Then draft a new release at https://github.com/TUW-GEO/ismn/releases. Create a version tag following the v{MAJOR}.{MINOR}.{PATCH} pattern. This will trigger a new build on GitHub and should push the packages to pypi after all tests have passed.

If this does not work (tests pass but upload fails) you can download the whl and dist packages for each workflow run from https://github.com/TUW-GEO/ismn/actions (Artifacts) and push them manually to https://pypi.org/project/ismn/ (you need to be a package maintainer on pypi for that).

In any case, pip install ismn should download the newest version afterwards.