-
Notifications
You must be signed in to change notification settings - Fork 11
minor updates to listing datasets #27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,25 +7,26 @@ from the workbench or Globus. | |
| ## Getting started | ||
|
|
||
| After installing terrautils, you should be able to import the *product* module. | ||
| ``` | ||
| ```{python} | ||
| from terrautils.products import get_sensor_list, unique_sensor_names | ||
| from terrautils.products import get_file_listing, extract_file_paths | ||
| ``` | ||
|
|
||
| The get\_sensor\_list and get\_file\_listing both require connection, url, | ||
| The `get_sensor_list` and `get_file_listing` functions both require connection, url, | ||
| and key parameters. *Connection* can be 'None', the *url* (called host in the | ||
| code) should be something like https://terraref.ncsa.illinois.edu/clowder/. | ||
| The *key* is a unique access key for the Clowder api. | ||
|
|
||
| ## Getting the sensor list | ||
|
|
||
| The first thing to get is the sensor name. This can be retreived using the | ||
| get\_sensor\_list function. This function returns the full record which may | ||
| `get_sensor_list` function. This function returns the full record which may | ||
| be useful in some cases but primarily includes sensor names that include | ||
| a plot id number. The utility function unique_sensor_names accpets the | ||
| a plot id number. The utility function `unique_sensor_names` accpets the | ||
| sensor list and provides a list of names suitable for use in the | ||
| get_file_listing function. | ||
| `get_file_listing` function. | ||
|
|
||
| ``` | ||
| ```{python} | ||
| sensors = get_sensor_list(None, url, key) | ||
| names = unique_sensor_names(sensors) | ||
| ``` | ||
|
|
@@ -36,16 +37,16 @@ geostreams API. The currently available sensors are: | |
| * IR Surface Temperature | ||
| * Thermal IR GeoTIFFs Datasets | ||
| * flirIrCamera Datasets | ||
| * (EL) sensor\_weather\_station | ||
| * (EL) sensor_weather_station | ||
| * Irrigation Observations | ||
| * Canopy Cover | ||
| * Energy Farm Observations SE | ||
| * (EL) sensor\_par | ||
| * (EL) sensor_par | ||
| * scanner3DTop Datasets | ||
| * Weather Observations | ||
| * Energy Farm Observations NE | ||
| * RGB GeoTIFFs Datasets | ||
| * (EL) sensor\_co2 | ||
| * (EL) sensor_co2 | ||
| * stereoTop Datasets | ||
| * Energy Farm Observations CEN | ||
|
|
||
|
|
@@ -55,28 +56,30 @@ The geostreams API can be used to get a list of datasets that overlap a | |
| specific plot boundary and, optionally, limited by a time range. Iterating | ||
| over the datasets allows the paths to all the files to be extracted. | ||
|
|
||
| ``` | ||
| ```{python} | ||
| sensor = 'Thermal IR GeoTIFFs Datasets' | ||
| sitename = 'MAC Field Scanner Season 1 Field Plot 101 W' | ||
| datasets = get_file_listing(None, url, key, sensor, sitename) | ||
| files = extract_file_paths(datasets) | ||
| ``` | ||
|
|
||
| Datasets can be further filtered using the *since* and *until* parameters | ||
| of get\_file\_listing with a date string. | ||
| of `get_file_listing` with a date string. | ||
|
|
||
| ``` | ||
| ```{python} | ||
| dataset = get_file_listing(None, url, key, sensor, sitename, | ||
| since='2016-06-01', until='2016-06-10') | ||
| ``` | ||
|
|
||
|
|
||
| # Alternative method | ||
|
|
||
| The following method demonstrates the same approach using the Clowder API. This | ||
| approach is useful for understanding the data layout and when the Python | ||
| terrautils package is not available. | ||
|
|
||
| ## Finding plot ID | ||
|
|
||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jterstriep How are these commands run? I get syntax errors in both bash and python |
||
| ``` | ||
| SENSOR_NAME = "MAC Field Scanner Season 1 Field Plot 101 W" | ||
| GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors?sensor_name={SENSOR_NAME} | ||
|
|
@@ -85,7 +88,9 @@ GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors?sensor_nam | |
| This returns a JSON object with an 'id' parameter. You can use this ID parameter to specify the right data stream. | ||
|
|
||
| ## Finding stream ID within a plot | ||
|
|
||
| The names are formatted as "<Sensor Group> Datasets (<Sensor ID>)". | ||
|
|
||
| ``` | ||
| SENSOR_ID = 3355 | ||
| STREAM_NAME = "Thermal IR GeoTIFFs Datasets ({SENSOR_ID})" | ||
|
|
@@ -95,13 +100,15 @@ GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/streams?stream_nam | |
| This returns a JSON object with an 'id' parameter. You can use this ID parameter to get the right datapoints. | ||
|
|
||
| ## Listing Clowder file IDs for that plot & sensor stream | ||
|
|
||
| ``` | ||
| STREAM_ID = "11586" | ||
| GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id={STREAM_ID} | ||
| ``` | ||
|
|
||
| This returns a list of datapoint JSON objects, each with a 'properties' parameter that looks like: | ||
| ``` | ||
|
|
||
| ```{python} | ||
| properties: { | ||
| dataset_name: "Thermal IR GeoTIFFs - 2016-05-09__12-07-57-990", | ||
| source_dataset: "https://terraref.ncsa.illinois.edu/clowder/datasets/59fc9e7d4f0c3383c73d2905" | ||
|
|
@@ -111,19 +118,23 @@ properties: { | |
| The source_dataset URL can be used to view the dataset in Clowder. | ||
|
|
||
| You can also filter the datapoints by date: | ||
|
|
||
| ``` | ||
| GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id={STREAM_ID}&since=2017-01-02&until=2017-06-10 | ||
| ``` | ||
|
|
||
| ## Getting ROGER file path from dataset | ||
|
|
||
| Given a source dataset URL, we can call the API to get the files and their paths. | ||
|
|
||
| ``` | ||
| SOURCE_DATASET = "https://terraref.ncsa.illinois.edu/clowder/datasets/59fc9e7d4f0c3383c73d2905" | ||
| # Add /api after /clowder, and add /files at the end of the URL | ||
| GET "https://terraref.ncsa.illinois.edu/clowder/api/datasets/59fc9e7d4f0c3383c73d2905/files" | ||
| ``` | ||
|
|
||
| This returns a list of files in the dataset and their paths if available: | ||
|
|
||
| ``` | ||
| [ | ||
| { | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how do you install terrautils?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
terrautils could also be pre-installed ... I drafted this for the Rstudio dockerfile based on the terrautils docker file but haven't tested it: