From 1541181648db6108230671ca422109d0e579dfef Mon Sep 17 00:00:00 2001 From: Chris Schnaufer Date: Fri, 25 Jan 2019 15:19:59 -0700 Subject: [PATCH 1/5] Fleshed out images vignette --- vignettes/03-get-images-python.Rmd | 101 ++++++++++++++++++++++++++++- 1 file changed, 100 insertions(+), 1 deletion(-) diff --git a/vignettes/03-get-images-python.Rmd b/vignettes/03-get-images-python.Rmd index bc5fed0..c326c53 100644 --- a/vignettes/03-get-images-python.Rmd +++ b/vignettes/03-get-images-python.Rmd @@ -1 +1,100 @@ -# Images Vignette \ No newline at end of file +--- +title: "Get Source Image Files" +output: html_document +--- + +# Objective: To be able to demonstrate how to locate and retrieve RGB image files + +This vignette shows how to locate and retrieve image files associated with growing Season 6 +from the University of Arizona's [Maricopa Agricultural Center](http://cals-mac.arizona.edu/) +using Python. The files are stored online on the data management system Clowder, +which is accessed using an API. We will be working with the image files generated during the +month of May by limiting the requests to that time period. + +After completing this vignette it should be possible to search for and retrieve other +files through the use of the API. + +As an added bonus we've also included an exmple of how to retrieve the list of available +sensor names through the API. By using the sensor names returned, it's possible to retrieve +other files containing the data the sensors have collected. + +## Locating the images + +To begin looking for files, a sensor name and site name are needed. We will be using +'RGB GeoTIFFs Datasets' as the sensor name and '' as the site name. Later in this +vignette we show how to retrieve the list of available sensors. + +As mentioned in the overview, the url string will point to the API to use. In this case +we'll be using "https://terraref.ncsa.illinois.edu/clowder/api" and the key will be the +one you received in an email. + +```{python eval=FALSE} +from terrautils.products import get_file_listing + +url = 'https://terraref.ncsa.illinois.edu/clowder/api' +key = 'YOUR_KEY_GOES_HERE' +sensor = 'RGB GeoTIFFs Datasets' +sitename = '' +files = get_file_listing(None, url, key, sensor, sitename, + since='2018-05-01', until='2018-05-31') +``` + +The `files` variable now contains an array of all the file in the datasets that match the +sensor in the plot for the month of May. When performing you own queries it's possible that there +are no matches found and the `files` array would be empty. + +# Retrieving the images + +Now that we have a list of files we can retrieve them one-by-one. We do this by creating a URL +that identifies the file to retrieve, making the API call to retrieve the file contents, and writing +the contents to disk. + +To create the correct URL we start with the one defined before and attach the keyword '/files/' +followed by the ID of each file. Assuming we have a file ID of '111', the final URL for retrieving +the file would be: + +``` {sh eval=FALSE} +https://terraref.ncsa.illinois.edu/clowder/api/files/111 +``` + +By looping through each of the returned files from the previous example, and using their ID and +filename, we can retrieve the files from the server and store them locally. + +We are streaming the data returned from our server request (`stream=True` in the code below) due to +the high probability of large file sizes. If the `stream=True` parameter was omitted the file's entire +contents would be in the `r` variable which could then be written to the local file. + +```{python eval=FALSE} +# We are using the same `url` and `key` variables declared in the previous example above. +filesurl = url + '/files/' +params={ 'key': key } + +for f in files: + r = requests.get(fileurl + f.id, params=params, stream=True) + with open(f.filename, 'wb') as o: + for chunk in r.iter_content(chunk_size=1024): + if chunk: + o.write(chunk) + +``` + +The images are now stored on the local file system. + +# Retrieving sensor names + +In this section we retrieve the names of different sensor types that are available. This will +allow you to retrieve files other than those containing RBG image data. + +```{python eval=FALSE} +# We are using the same `url` and `key` variables declared in the previous example above. +from terrautils.products import get_sensor_list, unique_sensor_names + +sensors = get_sensor_list(None, url, key) +names = unique_sensor_names(sensors) +``` + +The variable `names` will now contain the list of all available sensors. Using these sensor +names it's possible to use the above search to locate and then retrieve additional data files. +Substitute the new sensor name for 'RGB GeoTIFFs Datasets' where the variable `sensor` is +assigned above. + From ad5b12c840fbc1b23b769142a2491e6565d44745 Mon Sep 17 00:00:00 2001 From: David LeBauer Date: Mon, 28 Jan 2019 14:31:06 -0700 Subject: [PATCH 2/5] Update vignettes/03-get-images-python.Rmd Co-Authored-By: Chris-Schnaufer --- vignettes/03-get-images-python.Rmd | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/vignettes/03-get-images-python.Rmd b/vignettes/03-get-images-python.Rmd index c326c53..2657d19 100644 --- a/vignettes/03-get-images-python.Rmd +++ b/vignettes/03-get-images-python.Rmd @@ -18,6 +18,12 @@ As an added bonus we've also included an exmple of how to retrieve the list of a sensor names through the API. By using the sensor names returned, it's possible to retrieve other files containing the data the sensors have collected. +**requirements** +* Python 3 +* the terrautils library + * this can be installed from pypi by running `pip install terrautils` in the terminal +* an API key to access these data + ## Locating the images To begin looking for files, a sensor name and site name are needed. We will be using From ea0874f07a922bedc7e68939d1c9b35d5bcb1236 Mon Sep 17 00:00:00 2001 From: Chris Schnaufer Date: Mon, 28 Jan 2019 14:50:29 -0700 Subject: [PATCH 3/5] Added information on getting API keys --- vignettes/03-get-images-python.Rmd | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/vignettes/03-get-images-python.Rmd b/vignettes/03-get-images-python.Rmd index 2657d19..d3cb1c8 100644 --- a/vignettes/03-get-images-python.Rmd +++ b/vignettes/03-get-images-python.Rmd @@ -24,6 +24,16 @@ other files containing the data the sensors have collected. * this can be installed from pypi by running `pip install terrautils` in the terminal * an API key to access these data +The API key is a string that gets generated upon request through your Clowder account. Existing +API keys will work with this vignette. To get a new API key it is necessary to first register +with Clowder at "https://terraref.ncsa.illinois.edu/clowder/". First click the `Login` button and +wait for the login screen to appear. Then select the `Sign up` button and enter an email +address you have access to. An email is sent to the entered address with instructions for +completing the registration process. Once registration is complete, log +into Clowder and select the `View profile` menu option from the drop-down that is near the search +control. By clicking the `+ Add` button under "User API Keys" heading in the profile page, a new +key is gnerated that can be used. + ## Locating the images To begin looking for files, a sensor name and site name are needed. We will be using From 413a538ab2f322c825eb6da7692183d2eb95a37d Mon Sep 17 00:00:00 2001 From: Chris Schnaufer Date: Mon, 28 Jan 2019 14:55:05 -0700 Subject: [PATCH 4/5] Cleaned up an API key related spot --- vignettes/03-get-images-python.Rmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vignettes/03-get-images-python.Rmd b/vignettes/03-get-images-python.Rmd index d3cb1c8..01ea82c 100644 --- a/vignettes/03-get-images-python.Rmd +++ b/vignettes/03-get-images-python.Rmd @@ -42,7 +42,7 @@ vignette we show how to retrieve the list of available sensors. As mentioned in the overview, the url string will point to the API to use. In this case we'll be using "https://terraref.ncsa.illinois.edu/clowder/api" and the key will be the -one you received in an email. +one you created for your Clowder account. ```{python eval=FALSE} from terrautils.products import get_file_listing From f4f25d3e7582025eefb401e1d6a9619383597c46 Mon Sep 17 00:00:00 2001 From: Chris Schnaufer Date: Mon, 28 Jan 2019 14:58:28 -0700 Subject: [PATCH 5/5] Removed some extra words --- vignettes/03-get-images-python.Rmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vignettes/03-get-images-python.Rmd b/vignettes/03-get-images-python.Rmd index 01ea82c..2fac28e 100644 --- a/vignettes/03-get-images-python.Rmd +++ b/vignettes/03-get-images-python.Rmd @@ -32,7 +32,7 @@ address you have access to. An email is sent to the entered address with instruc completing the registration process. Once registration is complete, log into Clowder and select the `View profile` menu option from the drop-down that is near the search control. By clicking the `+ Add` button under "User API Keys" heading in the profile page, a new -key is gnerated that can be used. +key is gnerated. ## Locating the images