Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/default_template.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
*Fill out this issue by completing the items below. Not every item may be relevent to you, so feel free to make changes*
*Fill out this issue by completing the items below. Not every item may be relevant to you, so feel free to make changes*

#### The tutorial or vignette name
Name:
Expand Down
14 changes: 7 additions & 7 deletions index.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,24 @@ documentclass: book

This book is intended to quickly introduce users to TERRA REF data through a series of tutorials. TERRA REF has many types of data, and most can be accessed in multiple ways. Although this makes it more complicated to learn (and teach!), the objective is to provide users with the flexibility to access data in the most useful way.

The first section walks the user through the steps of downloading and combining three different types of data: plot level phenotypes, meteorological data, and images. Subesquent sections provide more detailed examples that show how to access a larger variety of data and meta-data.
The first section walks the user through the steps of downloading and combining three different types of data: plot level phenotypes, meteorological data, and images. Subsequent sections provide more detailed examples that show how to access a larger variety of data and meta-data.

For those who are eager to see what is on deck, there are additional tutorials under development in the [tutorials repository](https://github.com/terraref/tutorials) on GitHub. These can be found in the traits, sensors, and genomics subfolders. Users are encouraged to revise and contribute new content.

## Pre-requisites

While we assume that readers will have some familiarity with the nature of the problem - remote sensing of crop plants - for the most part, these tutorials assume that the user will bring their own scientific questions and a sense of curiosity and are eager to learn.

These tutorials are aimed at users who are familiar with or willing to learn programming languages including R (particularly for accessing plot level trait data) and Python (primarily for accessing environmental data and sensor data). In addition, there are examples of using SQL for more sophisticated database queries as well as the bash terminal.
These tutorials are aimed at users who are familiar with or willing to learn programming languages including R (particularly for accessing plot level trait data) and Python (primarily for accessing environmental data and sensor data). In addition, there are examples of using SQL for more sophisticated database queries, as well as the Bash terminal.

Some of the lessons only require a web browser; others will assume familarity with programming at the command line in (typically only one of) Python, R, and / or SQL. You should be willing to find help (see finding help, below).
Some of the lessons only require a web browser; others will assume familiarity with programming at the command line in (typically only one of) Python, R, and / or SQL. You should be willing to find help (see finding help, below).

### Technical Requirements

At a minimum, you should have:

* An internet connection
* Web Browser
* Web browser
* Access to the data that you are using
+ The tutorials will state which databases you will need access to
* Software:
Expand All @@ -39,12 +39,12 @@ At a minimum, you should have:

The first few chapters in the 'vignettes' section use publicly available sample data sets.
Subsequent sections are also written to use publicly available data sets, but some of the examples require data that requires users to sign up.
To sign up, you will need to 1) fill out the TERRA REF Beta user questionaire ([terraref.org/beta](https://terraref.org/beta)) and 2) request access to specific databases.
To sign up, you will need to 1) fill out the TERRA REF Beta user questionnaire ([terraref.org/beta](https://terraref.org/beta)) and 2) request access to specific databases.

<!-- Not sure where this goes, either in documentation or perhaps in an appendix. But I don't think this belongs in the introduction. Perhaps after the vignettes chaper
<!-- Not sure where this goes, either in documentation or perhaps in an appendix. But I don't think this belongs in the introduction. Perhaps after the vignettes chapter
-->

## Ways of Acessing Data
## Ways of Accessing Data

* Web Interfaces
+ [Clowder](https://docs.terraref.org/user-manual/how-to-access-data/using-clowder-sensor-and-genoomics-data) (sensor and genomic data)
Expand Down
2 changes: 1 addition & 1 deletion metadata/metadata_tutorial_addition.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ URL to a public published google sheet (you will need to provide this url in the

# How to Run this Rmd

Knitting this markdown file will not upload any data to the test database. Being able to knit this file means that you will not encounter any errors when uploading your metadata. All experiments, sites, treatments, cutlivars, and citations can be uploaded successfully.
Knitting this markdown file will not upload any data to the test database. Being able to knit this file means that you will not encounter any errors when uploading your metadata. All experiments, sites, treatments, cultivars, and citations can be uploaded successfully.

To actually commit additions to the test database, you will need to make an edit to this markdown file.

Expand Down
6 changes: 3 additions & 3 deletions metadata/test_database_tutorial.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The purpose of this tutorial is to demonstrate how to set up a local instance of

You will want to set up this test database when preparing to upload data to the master database.

You should make sure that your data can be succesfully uploaded to the test database before trying to upload to master.
You should make sure that your data can be successfully uploaded to the test database before trying to upload to master.

# Requirements:

Expand Down Expand Up @@ -41,7 +41,7 @@ After initial set up, you will only need to run step 4 to restart the database.

## Step 1: Clone bety repository

In a directory of your chioce, clone the bety repository from github and cd to the bety folder:
In a directory of your choice, clone the bety repository from github and cd to the bety folder:

```sh
git clone https://github.com/PecanProject/bety.git
Expand Down Expand Up @@ -107,7 +107,7 @@ Open a new tab in the terminal and connect to the test database using:
```sh
psql -h localhost -p 5433 -U bety
```
You should see the following prompt if you have succesfully logged into the test database:
You should see the following prompt if you have successfully logged into the test database:
```sh
psql (11.2, server 9.5.16)
Type "help" for help.
Expand Down
2 changes: 1 addition & 1 deletion sensors/01-meteorological-data.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ The data represent 5 minute summaries aggregated from 1/s observations.
### Download data using the command line

Data can be downloaded from Clowder using the command line program Curl. If the
following is typed into the commmand line, it will download the datapoints data
following is typed into the command line, it will download the datapoints data
that we're interested in as a file which we have chosen to call `spectra.json`.

```{sh eval=FALSE}
Expand Down
10 changes: 5 additions & 5 deletions sensors/06-list-datasets-by-plot.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ The *key* is a unique access key for the Clowder API.

## Getting the sensor list

The first thing to get is the sensor name. This can be retreived using the
The first thing to get is the sensor name. This can be retrieved using the
`get_sensor_list` function. This function returns the full record which may
be useful in some cases but primarily includes sensor names that include
a plot id number. The utility function `unique_sensor_names` accpets the
a plot id number. The utility function `unique_sensor_names` accepts the
sensor list and provides a list of names suitable for use in the
`get_file_listing` function.

Expand Down Expand Up @@ -96,7 +96,7 @@ of requests against the API it's possible to determine the files of interest and

Each of the API URL's have the same beginning (https://terraref.ncsa.illinois.edu/clowder/api),
followed by the data needed for a specific request. As we step through the process you will be able
to see how then end of the URL changes depending upon the reuqest.
to see how then end of the URL changes depending upon the request.

Below is what the API looks like as a URL. Try pasting it into your browser.

Expand Down Expand Up @@ -156,7 +156,7 @@ STREAM_ID=11586
curl -o datasets.json -X GET "https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id=${STREAM_ID}"
```

After the call succeeds, a file named *datasets.json* is created containing the returned JSON onject. As part of the
After the call succeeds, a file named *datasets.json* is created containing the returned JSON object. As part of the
JSON object there are one or more `properties` fields containing *source_dataset* parameters.

```{javascript eval=FALSE}
Expand Down Expand Up @@ -211,7 +211,7 @@ of the files in the dataset with their IDs, and other data if available:

### Retrieving the files

Given that a large number of files may be contained in a dataset, it may be desireable to automate the process of pulling down files
Given that a large number of files may be contained in a dataset, it may be desirable to automate the process of pulling down files
to the local system.

For each file to be retrieved, the unique file ID is needed on the URL.
Expand Down
2 changes: 1 addition & 1 deletion sensors/09-vnir-radiometer.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ output: html_document

An Ocean Optics STS Spectrometer measures downwelling solar spectral radiance every 5s on top of the Gantry.

Lets look at the output from this sensor over the course of the day (and then we will see how to acces the data):
Lets look at the output from this sensor over the course of the day (and then we will see how to access the data):

```{r one-downwelling-spectra}
library(tidyverse)
Expand Down
2 changes: 1 addition & 1 deletion traits/00-BETYdb-getting-started.Rmd
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# (PART\*) Secton 2: Traits {-}
# (PART\*) Section 2: Traits {-}

# Getting Started with BETYdb

Expand Down
2 changes: 1 addition & 1 deletion traits/06-agronomic-metadata.Rmd
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Querying Agronomic Meta-data

In previous tutorials you have learned how to query trait data using a variety of different methods, including the web interface, an API, and the R traits package. Here you will continute to use the R traits package, and learn how to access meta-data from other tables in the database.
In previous tutorials you have learned how to query trait data using a variety of different methods, including the web interface, an API, and the R traits package. Here you will continue to use the R traits package, and learn how to access meta-data from other tables in the database.

While the basic search query that we have used in previous sections provides the key information that you may need for an analysis - the genotype name, the location, date, and method, there are other tables that contain more specific metadata.

Expand Down
2 changes: 1 addition & 1 deletion traits/07-betydb-sql-access.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ will be derived from https://github.com/pi4-uiuc/2017-bootcamp/blob/master/conte

## On workbench

On the TERRA REF Workbench, you have access to the database. These connnections will only work if you are on the workbench.
On the TERRA REF Workbench, you have access to the database. These connections will only work if you are on the workbench.


## Using PostgresSQL Studio
Expand Down
6 changes: 3 additions & 3 deletions traits/10-simulated-sorghum.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ The 'genotypes' are based on five-hundred quasi-random parameterizations of a bi

_Note that these data sets contain numerical artifacts and scientific reinterpretations for illustrative purpose._

All of these simulated datasets are released with an unrestrive [copyright](https://creativecommons.org/publicdomain/zero/1.0/). This means you can copy, modify, and share the data. Please keep in mind that the data sets are not production quality - they have been developed solely to inspire and solicit feedback.
All of these simulated datasets are released with an unrestrictive [copyright](https://creativecommons.org/publicdomain/zero/1.0/). This means you can copy, modify, and share the data. Please keep in mind that the data sets are not production quality - they have been developed solely to inspire and solicit feedback.


### Design of Simulation Experiment
Expand Down Expand Up @@ -72,7 +72,7 @@ Two-hundred and twenty-seven lines were grown at each site. Each line is identif

The phenotypes associated with each genotype is in the file `phenotypes.csv`.

These 'phenotypes' are used as input parameters to the simulation model. We often refer to these as 'traits' (as opposed to biomass or growth rates, which are states and proceses). In this example, we assume that 'phenotypes' are time-invariant.
These 'phenotypes' are used as input parameters to the simulation model. We often refer to these as 'traits' (as opposed to biomass or growth rates, which are states and processes). In this example, we assume that 'phenotypes' are time-invariant.


| variable_id|name |standard_name |units |Description |
Expand Down Expand Up @@ -293,7 +293,7 @@ Now, join the traits data frame with the new max_height data frame trait data we
traits_height <- traits_wide %>% left_join(max_height, by = 'genotype')

```
Which traits are related to height? We can discover this in a few way, for example, a pairs plot that shows correltations:
Which traits are related to height? We can discover this in a few way, for example, a pairs plot that shows correlations:

```{r trait_pairs, fig.height = 8, fig.width = 8, warning = FALSE}
ggpairs(traits_height %>% select(-genotype),
Expand Down
4 changes: 2 additions & 2 deletions videos/first_walkthrough.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ These live walkthroughs will be turned into video tutorials that can be viewed a
#### Project overview
TERRA REF is a project funded by the Advanced Research Projects Agency for Energy (ARPA-E). The objective is to advance the science and technology of high throughput field phenotyping, which is the automated measurement of plants. TERRA REF has deployed an exceptionally large robot capable of collecting data from a range of sensors at 1 mm spatial resolution every other day. You can learn more about the project from the website, terraref.org. The suite of sensors include visible light cameras, hyperspectral cameras, a laser 3d scanner, and environmental data. In addition there are weather stations and high resolution genome sequences for hundreds of varieties of Sorghum being monitored.

These tutorials are to make these data accessible and useable. I will provide an overview of how to use computational tools to get these data, and combine, plot, and analyze them.
These tutorials are to make these data accessible and usable. I will provide an overview of how to use computational tools to get these data, and combine, plot, and analyze them.

We have written online tutorials [here](https://terraref.github.io/tutorials/). These walkthroughs and videos complement those materials, which are growing and improving continually.

Expand Down Expand Up @@ -249,7 +249,7 @@ We'll calculate a metric from temperature called growing degree days. Basically

First we just want the date, not including time of day. Pull out date using built-in `as.Date` and put into new column `day`.

Also create new column with air temperature in Celcius instead of Kelvin.
Also create new column with air temperature in Celsius instead of Kelvin.

```{r}
weather_daily <- weather_all %>%
Expand Down
4 changes: 2 additions & 2 deletions videos/first_walkthrough.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ cameras, hyperspectral cameras, a laser 3d scanner, and environmental
data. In addition there are weather stations and high resolution genome
sequences for hundreds of varieties of Sorghum being monitored.

These tutorials are to make these data accessible and useable. I will
These tutorials are to make these data accessible and usable. I will
provide an overview of how to use computational tools to get these data,
and combine, plot, and analyze them.

Expand Down Expand Up @@ -415,7 +415,7 @@ over time during the year.
First we just want the date, not including time of day. Pull out date
using built-in `as.Date` and put into new column `day`.

Also create new column with air temperature in Celcius instead of
Also create new column with air temperature in Celsius instead of
Kelvin.

``` r
Expand Down
2 changes: 1 addition & 1 deletion videos/second_walkthrough.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ These data are summarized from camera data, can see that by unselecting "Heat Ma

We can download these individual files. TERRA REF data are on two platforms. We pulled data from the first platform Clowder last week. Other platform is called Globus, let's work with that first.

All data are on Globus. Go to [website](globus.org) and log in. Need to set up account and get acccess to Terraref collection. Instructions are [here](https://docs.terraref.org/user-manual/how-to-access-data/using-globus-sensor-and-genomics-data).
All data are on Globus. Go to [website](globus.org) and log in. Need to set up account and get access to Terraref collection. Instructions are [here](https://docs.terraref.org/user-manual/how-to-access-data/using-globus-sensor-and-genomics-data).

In File Manager section. Click "Start here", "Shared With You" tab, and then "Terraref" option. This gives access to a bunch of files for the project.

Expand Down
2 changes: 1 addition & 1 deletion vignettes/00-introduction.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Information for getting a beta user account is available in the section on [User
### R

In order to use the vignettes, you will need to have R installed along with the extra packages that are used.
All of the packages except 'trait' are available on CRAN and can be installed using the `install.packages` funciton:
All of the packages except 'trait' are available on CRAN and can be installed using the `install.packages` function:

```r
install.packages('tidyverse', 'jsonlite', 'knitr', 'lubridate', 'raster', 'sf')
Expand Down
4 changes: 2 additions & 2 deletions vignettes/01-get-trait-data-R.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ In this chapter you will learn:

In this chapter, we go over how to query TERRA REF trait data using the [`traits`](https://github.com/ropensci/traits) package. The `traits` package is a way to query for various sources of species trait data, including [BETYdb](https://www.betydb.org/), [NCBI](http://www.ncbi.nlm.nih.gov/), [Coral Traits Disease](http://coraltraits.org/) and others. In this chapter we use BETYdb as our trait source, as it contains the TERRA REF data that we are interested in.

Our example will show how to query for season 6 data <!-- what is a season? need to define this for users> --> and visualize canopy height. In addition to the `traits` package we will also be using some of the [`tidyverse`](https://www.tidyverse.org/) packages, which allow us to maniupulate the data in an efficient, understandable way. If you are unfamiliar with `tidyverse` syntax, we recommend checking out some of the resources [here](https://www.tidyverse.org/learn/).
Our example will show how to query for season 6 data <!-- what is a season? need to define this for users> --> and visualize canopy height. In addition to the `traits` package we will also be using some of the [`tidyverse`](https://www.tidyverse.org/) packages, which allow us to manipulate the data in an efficient, understandable way. If you are unfamiliar with `tidyverse` syntax, we recommend checking out some of the resources [here](https://www.tidyverse.org/learn/).



Expand Down Expand Up @@ -99,7 +99,7 @@ kable(traits)

### Querying season 6 canopy height data

You may find after constructing a general query as above that you want to only query a specific trait. Here, we query for the canopy height trait by adding the key-value pair `trait = "canopy_height"` to our query function. Note that the limit is also set to return only 250 records, shown here for demostration purposes.
You may find after constructing a general query as above that you want to only query a specific trait. Here, we query for the canopy height trait by adding the key-value pair `trait = "canopy_height"` to our query function. Note that the limit is also set to return only 250 records, shown here for demonstration purposes.

```{r traits-vig-canopy-query, message = FALSE}

Expand Down
2 changes: 1 addition & 1 deletion vignettes/03-get-images-python.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ All the `terrautils` functions are now available in Python, although we will onl
## Retrieving sensor names

In this section we retrieve the names of different sensor types that are available.
This will allow you to understand what files may be avaialable other than just those containing RBG image data.
This will allow you to understand what files may be available other than just those containing RBG image data.

In order to run Python functions, including those from the `terrautils` library, within this Rmarkdown, we have to install and set up `reticulate`.

Expand Down
Loading