Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,26 @@ from the workbench or Globus.
## Getting started

After installing terrautils, you should be able to import the *product* module.
```
```{python}
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After installing terrautils

how do you install terrautils?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

terrautils could also be pre-installed ... I drafted this for the Rstudio dockerfile based on the terrautils docker file but haven't tested it:

# Install terrautils
RUN cd && git clone https://github.com/terraref/terrautils && cd terrautils 
COPY setup.py requirements.txt MANIFEST.in /tmp/terrautils/
RUN pip install --upgrade  -r /tmp/terrautils/requirements.txt

COPY terrautils /tmp/terrautils/terrautils
RUN pip install --upgrade /tmp/terrautils \
&& rm -rf /tmp/terrautils

from terrautils.products import get_sensor_list, unique_sensor_names
from terrautils.products import get_file_listing, extract_file_paths
```

The get\_sensor\_list and get\_file\_listing both require connection, url,
The `get_sensor_list` and `get_file_listing` functions both require connection, url,
and key parameters. *Connection* can be 'None', the *url* (called host in the
code) should be something like https://terraref.ncsa.illinois.edu/clowder/.
The *key* is a unique access key for the Clowder api.

## Getting the sensor list

The first thing to get is the sensor name. This can be retreived using the
get\_sensor\_list function. This function returns the full record which may
`get_sensor_list` function. This function returns the full record which may
be useful in some cases but primarily includes sensor names that include
a plot id number. The utility function unique_sensor_names accpets the
a plot id number. The utility function `unique_sensor_names` accpets the
sensor list and provides a list of names suitable for use in the
get_file_listing function.
`get_file_listing` function.

```
```{python}
sensors = get_sensor_list(None, url, key)
names = unique_sensor_names(sensors)
```
Expand All @@ -36,16 +37,16 @@ geostreams API. The currently available sensors are:
* IR Surface Temperature
* Thermal IR GeoTIFFs Datasets
* flirIrCamera Datasets
* (EL) sensor\_weather\_station
* (EL) sensor_weather_station
* Irrigation Observations
* Canopy Cover
* Energy Farm Observations SE
* (EL) sensor\_par
* (EL) sensor_par
* scanner3DTop Datasets
* Weather Observations
* Energy Farm Observations NE
* RGB GeoTIFFs Datasets
* (EL) sensor\_co2
* (EL) sensor_co2
* stereoTop Datasets
* Energy Farm Observations CEN

Expand All @@ -55,28 +56,30 @@ The geostreams API can be used to get a list of datasets that overlap a
specific plot boundary and, optionally, limited by a time range. Iterating
over the datasets allows the paths to all the files to be extracted.

```
```{python}
sensor = 'Thermal IR GeoTIFFs Datasets'
sitename = 'MAC Field Scanner Season 1 Field Plot 101 W'
datasets = get_file_listing(None, url, key, sensor, sitename)
files = extract_file_paths(datasets)
```

Datasets can be further filtered using the *since* and *until* parameters
of get\_file\_listing with a date string.
of `get_file_listing` with a date string.

```
```{python}
dataset = get_file_listing(None, url, key, sensor, sitename,
since='2016-06-01', until='2016-06-10')
```


# Alternative method

The following method demonstrates the same approach using the Clowder API. This
approach is useful for understanding the data layout and when the Python
terrautils package is not available.

## Finding plot ID

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jterstriep How are these commands run? I get syntax errors in both bash and python

```
SENSOR_NAME = "MAC Field Scanner Season 1 Field Plot 101 W"
GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors?sensor_name={SENSOR_NAME}
Expand All @@ -85,7 +88,9 @@ GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors?sensor_nam
This returns a JSON object with an 'id' parameter. You can use this ID parameter to specify the right data stream.

## Finding stream ID within a plot

The names are formatted as "<Sensor Group> Datasets (<Sensor ID>)".

```
SENSOR_ID = 3355
STREAM_NAME = "Thermal IR GeoTIFFs Datasets ({SENSOR_ID})"
Expand All @@ -95,13 +100,15 @@ GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/streams?stream_nam
This returns a JSON object with an 'id' parameter. You can use this ID parameter to get the right datapoints.

## Listing Clowder file IDs for that plot & sensor stream

```
STREAM_ID = "11586"
GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id={STREAM_ID}
```

This returns a list of datapoint JSON objects, each with a 'properties' parameter that looks like:
```

```{python}
properties: {
dataset_name: "Thermal IR GeoTIFFs - 2016-05-09__12-07-57-990",
source_dataset: "https://terraref.ncsa.illinois.edu/clowder/datasets/59fc9e7d4f0c3383c73d2905"
Expand All @@ -111,19 +118,23 @@ properties: {
The source_dataset URL can be used to view the dataset in Clowder.

You can also filter the datapoints by date:

```
GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id={STREAM_ID}&since=2017-01-02&until=2017-06-10
```

## Getting ROGER file path from dataset

Given a source dataset URL, we can call the API to get the files and their paths.

```
SOURCE_DATASET = "https://terraref.ncsa.illinois.edu/clowder/datasets/59fc9e7d4f0c3383c73d2905"
# Add /api after /clowder, and add /files at the end of the URL
GET "https://terraref.ncsa.illinois.edu/clowder/api/datasets/59fc9e7d4f0c3383c73d2905/files"
```

This returns a list of files in the dataset and their paths if available:

```
[
{
Expand Down