Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 25 additions & 5 deletions sensors/06-list-datasets-by-plot.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,21 @@ The terrautils python package has a new `products` module that aids in connectin
plot boundaries stored within betydb with the file-based data products available
from the workbench or Globus.

* if are using Rstudio and want to run the Python code chunks, the R package "reticulate" is required
* use `pip3 install terrautils` to install the terrautils Python library

## Getting started

After installing terrautils, you should be able to import the `products` module.
```{python eval=FALSE}
```{python}
from terrautils.products import get_sensor_list, unique_sensor_names
from terrautils.products import get_file_listing, extract_file_paths
```

The `get_sensor_list` and `get_file_listing` functions both require the *connection*,
*url*, and *key* parameters. The *connection* can be 'None'. The *url* (called host in the
code) should be something like `https://terraref.ncsa.illinois.edu/clowder/`.
The *key* is a unique access key for the Clowder api.
The *key* is a unique access key for the Clowder API.

## Getting the sensor list

Expand All @@ -32,11 +35,21 @@ a plot id number. The utility function `unique_sensor_names` accpets the
sensor list and provides a list of names suitable for use in the
`get_file_listing` function.

```{python eval=FALSE}
To use this tutorial you will need to sign up for Clowder, have your
account approved, and then get an API key from the [Clowder web interface](https://terraref.ncsa.illinois.edu/clowder).

```{python}
url = 'https://terraref.ncsa.illinois.edu/clowder/'
key = 'ENTER YOUR KEY HERE'
```

```{python}
sensors = get_sensor_list(None, url, key)
names = unique_sensor_names(sensors)
print(names)
```


Names will now contain a list of sensor names available in the Clowder
geostreams API. The list of returned sensor names could be something like the
following:
Expand All @@ -55,9 +68,10 @@ The geostreams API can be used to get a list of datasets that overlap a
specific plot boundary and, optionally, limited by a time range. Iterating
over the datasets allows the paths to all the files to be extracted.

```{python eval=FALSE}
```{python eval = FALSE}
sensor = 'Thermal IR GeoTIFFs Datasets'
sitename = 'MAC Field Scanner Season 1 Field Plot 101 W'
key = 'INSERT YOUR KEY HERE'
datasets = get_file_listing(None, url, key, sensor, sitename)
files = extract_file_paths(datasets)
```
Expand All @@ -73,6 +87,10 @@ dataset = get_file_listing(None, url, key, sensor, sitename,

# Querying the API

<!--
TODO: move this to a separate tutorial page focused on using curl
-->

The source files behind the data are available for downloading through the API. By executing a series
of requests against the API it's possible to determine the files of interest and then download them.

Expand All @@ -82,7 +100,7 @@ to see how then end of the URL changes depending upon the reuqest.

Below is what the API looks like as a URL. Try pasting it into your browser.

https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors?sensor_name=MAC Field Scanner Season 1 Field Plot 101 W
[https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors?sensor_name=MAC Field Scanner Season 1 Field Plot 101 W](https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors?sensor_name=MAC Field Scanner Season 1 Field Plot 101 W)

This will return data for the requested plot including its id. This id (or identifier) can then be used for
additional queries against the API.
Expand All @@ -93,6 +111,8 @@ to retrieve its data.

## A Word of Caution

We are no longer using the python terrautils package, which is a python library that provides helper functions that simplify interactions with the Clowder API. One of the ways it makes the interface easier is by using function names that make sense in the scope of the project. The API and the Clowder database have different names and _this is confusing_ since the same names are used for different parts of the database.

The names and meanings of variables in this section don't necessarily match the ones in the section
above and it may be easy to get them confused. The API queries the database directly and thereby reflects
the database structure. This is the main reason for the naming differences between the API and the terraref
Expand Down