Reorganized tutorials book to include vignette section

_bookdown.yml

@@ -3,7 +3,7 @@ output_dir: "docs"
 language:
   ui:
     chapter_name: "Chapter "
-rmd_files: ["index.Rmd", "traits/00-BETYdb-getting-started.Rmd", "traits/01-web-access.Rmd", 
-"traits/02-betydb-api-access.Rmd", "traits/03-access-r-traits.Rmd", "traits/04-danforth-indoor-phenotyping-facility.Rmd",
-"traits/06-agronomic-metadata.Rmd", "traits/07-betydb-sql-access.Rmd", "traits/10-simulated-sorghum.Rmd"]#, "10-simulated-sorghum.Rmd"
+rmd_files: ["index.Rmd", "vignettes/00-introduction.Rmd", "vignettes/01-get-trait-data-R.Rmd", "vignettes/02-get-weather-data-R.Rmd",
+"vignettes/03-get-images-python.Rmd", "vignettes/04-synthesis-data.Rmd", "traits/03-access-r-traits.Rmd","sensors/01-meteorological-data.Rmd",
+"sensors/06-list-datasets-by-plot.Rmd"]
 

index.Rmd

@@ -11,15 +11,38 @@ output:
 
 # Overview
 
-This book is intended to introduce users to TERRA REF data as quickly as possible. 
+This book is intended to quickly introduce users to TERRA REF data through a series of tutorials. TERRA REF has many types of data, and most can be  accessed in multiple ways. Although this makes it more complicated to learn (and teach!), the objective is to provide users with the flexibility to access data in the most useful way.
 
-It introduces to the wide range of phenomics datasets generated by the TERRA Reference program. Not only does TERRA REF have a large number of data sets, but many of the databases can be accessed in a number of different ways. While this makes it more complicated to learn, the goal is to provide users with the flexibility to access data in the most useful way.
+
+## Contents
+
+The first section walks the user through the steps of downloading and combining three different types of data: plot level phenotypes, meteorological data, and images. Subesquent sections provide more detailed examples that show how to access a larger variety of data and meta-data.
+
+## Pre-requisites
+
+While we assume that readers will have some familiarity with the nature of the problem - remote sensing of crop plants - for the most part, these tutorials assume that the user will bring their own scientific questions and a sense of curiosity and are eager to learn. 
+
+These tutorials are aimed at users who are familiar with or willing to learn programming languages including R (particularly for accessing plot level trait data) and Python (primarily for accessing environmental data and sensor data). In addition, there are examples of using SQL for more sophisticated database queries as well as the bash terminal.
+
+Some of the lessons only require a web browser; others will assume familarity with programming at the command line in (typically only one of) Python, R, and / or SQL. You should be willing to find help (see finding help, below).
+
+## Technical Requirements
+
+At a minimum, you should have:
+
+* An internet connection
+* Web Browser
+* Access to the data that you are using
+    + The tutorials will state which databases you will need access to
+* Software:
+    + Software requirements vary with the tutorials, and may be complex
 
 ## User Accounts and permission to access TERRA REF data
 
-TODO: link to relevant parts of docs.terraref.org
+We have tried to write these tutorials using open access sample data sets. However, access to much of the data will require you to 1) fill out the TERRA REF Beta user questionaire ([terraref.org/beta](terraref.org/beta)) and 2) request access to specific databases.
 
-* Info on how to [request access to data](https://docs.terraref.org/user-manual/how-to-access-data/using-betydb-trait-data-experimental-metadata) 
+<!-- Not sure where this goes, either in documentation or perhaps in an appendix. But I don't think this belongs in the introduction. Perhaps after the vignettes chaper 
+-->
 
 ## Ways of Acessing Data
 
@@ -40,41 +63,21 @@ The TERRA REF website: [terraref.org](http://terraref.org/)
 
 The TERRA REF Technical Documentation: [docs.terraref.org](docs.terraref.org)
 
-## Contents
-
-Scope ...
-
-Audience ...
-
-
-## Pre-requisites
-
-While we assume that readers will have some familiarity with the nature of the problem - remote sensing of crop plants - for the most part, these tutorials assume that the user will bring their own scientific questions and a sense of curiosity and are eager to learn. 
-
-Some of the lessons only require a web browser; others will assume familarity with programming at the command line in (typically only one of) Python, R, and / or SQL. You should be willing to find help (see finding help, below).
-
-## Technical Requirements
-
-At a minimum, you should have:
-
-* An internet connection
-* Web Browser
-* A TERRA REF Beta User account
-    + If you have not done so, please sign up at [terraref.org/beta](terraref.org/beta)
-* Access to the data that you are using
-    + The tutorials will state which databases you will need access to
-* Software:
-    + Software requirements vary with the tutorials, and may be complex
-
 
 ## Finding help
 
-- [Slack](terra-ref.slack.com)
-- [GitHub](https://github.com/terraref/tutorials)
-- [Google](https://www.google.com/)
+- Slack at terra-ref.slack.com ([signup](https://terraref-slack-invite.herokuapp.com/))
+- Browse issues and repositories in GitHub: 
+  - search the organization at github.com/terraref
+  - questions about the tutorials in the [tutorials repository](https://github.com/terraref/tutorials/issues)
+  - about the data in the [reference-data repository](https://github.com/terraref/reference-data/issues)
 
 ```{r, include = FALSE}
-knitr::opts_chunk$set(echo = FALSE, cache = TRUE)
+knitr::opts_chunk$set(echo = FALSE,
+                      engine.path = list(
+                        python = 'python3'
+                      ))
+
 options(warn = -1)
 ```
 

traits/00-BETYdb-getting-started.Rmd

@@ -1,4 +1,4 @@
-# (PART\*) Secton 1: Traits {-}
+# (PART\*) Secton 2: Traits {-}
 
 # Getting Started with BETYdb
 

traits/02-betydb-api-access.Rmd

@@ -26,18 +26,19 @@ The first step toward reproducible pipelines is to automate the process of searc
 ### Using Your API key to Connect
 
 An API key is like a password. It allows you to access data, and should be kept private. 
-Therefore, we are not going to put it in code that we share. The one exception is the key 9999999999999999999999999999999999999999 that will allow you to access metadata tables (all tables except _traits_ and _yields_). It will also allow you to access all of the simulated data in the https://terraref.ncsa.illinois.edu/bety-test database.
 
-A common way of handling private API keys is to place it in a text file in your current directory. 
-Don't put it in a project directory where it might be inadvertently shared.
+Therefore, we are not going to put it in code that we share. 
+
+A common way of handling private API keys is to place it in a text file in your current directory. Don't put it in a project directory where it might be inadvertently shared.
 
 Here is how to find and save your API key:
 
 * click file --> new --> text file
 * copy the api key that was sent when you registered into the file
 * file --> save as '.betykey'
 
-For the public key, you can call this file `.betykey_public`. 
+An API key is not needed to access public data. This includes metadata tables and simulated data in the https://terraref.ncsa.illinois.edu/bety-test database.
+
 
 
 ## Accessing data using a URL query
@@ -49,7 +50,9 @@ For the public key, you can call this file `.betykey_public`.
 * path to the api: `/api/v1`
 * api endpoint: `/search` or `traits` or `sites`. For BETYdb, these are the names of database tables. 
 * Query parameters: `genus=Sorghum`
-* Authentication: `key=9999999999999999999999999999999999999999` is the public key for the TERRA REF traits database. 
+
+* Authentication: `key=api_key` is your assigned API key. This will only be needed when querying trait data. No key is needed to access the public metadata tables.
+
 
 ### Constructing a URL query
 
@@ -62,17 +65,17 @@ First, lets construct a query by putting together a URL.
 3. Add the name of the table you want to query. Lets start with `variables`
   * terraref.ncsa.illinois.edu/bety/api/v1/variables
 4. add query terms by appending a `?` and combining with `&`, for example:
-  * `key=9999999999999999999999999999999999999999`
   * `type=trait` where the variable type is 'trait'
   * `name=~height` where the variable name contains 'height'
 5. This is your complete query:
-  * `terraref.ncsa.illinois.edu/bety/api/v1/variables?type=trait&name=~height&key=9999999999999999999999999999999999999999`
+  * `terraref.ncsa.illinois.edu/bety/api/v1/variables?type=trait&name=~height`
   * it will query all variables that are type trait and have 'height' in the name
   * Does it return the expected values?
 
 ## Your Turn
 
-> What will the URL https://terraref.ncsa.illinois.edu/bety/api/v1/species?genus=Sorghum&key=9999999999999999999999999999999999999999 return?
+> What will the URL https://terraref.ncsa.illinois.edu/bety/api/v1/species?genus=Sorghum return?
+
 
 > Write a URL that will query the database for sites with "Field Scanner" in the name field. Hint: combine two terms with a `+` as in `Field+Scanner`
 
@@ -84,23 +87,30 @@ Type the following command into a bash shell (the `-o` option names the output f
 
 ```sh
 curl -o sorghum.json \
-   "https://terraref.ncsa.illinois.edu/bety/api/v1/species?genus=Sorghum&key=9999999999999999999999999999999999999999"
+   "https://terraref.ncsa.illinois.edu/bety/api/v1/species?genus=Sorghum"
 ```
 
 If you want to write the query without exposing the key in plain text, you can construct it like this:
 
 ```sh
 curl -o sorghum.json \
-    "https://terraref.ncsa.illinois.edu/bety/api/v1/species?genus=Sorghum&key=`cat .betykey_public`"
+    "https://terraref.ncsa.illinois.edu/bety/api/v1/species?genus=Sorghum"
 ```
 
 ## Using the R jsonlite package to access the API with a URL query 
 
+
+```{r 02-jsonlite-load, include = FALSE}
+
+library(jsonlite)
+
+```
+
 ```{r text-api, warning = FALSE}
 sorghum.json <- readLines(
   paste0("https://terraref.ncsa.illinois.edu/bety/api/v1/species?genus=Sorghum&key=", 
-         readLines('traits/.betykey')))
-
+         readLines('.betykey')))
+         
 ## print(sorghum.json) 
 ## not a particularly useful format
 ## lets convert to a data frame

traits/03-access-r-traits.Rmd

@@ -8,59 +8,61 @@ The rOpenSci traits package makes it easier to query the TERRA REF trait databas
 
 Install the traits package
 
-The traits package is on CRAN, and can therefore be installed using the following command:  
+The traits package can be installed through github using the following command: 
 
 ```{r install_traits, echo = TRUE, message = FALSE}
-install.packages('traits', repos = 'http://cran.rstudio.com/')
+
+if(packageVersion("traits") == '0.2.0'){
+  devtools::install_github('ropensci/traits')
+}
+
 ```
 
 Load other packages that we will need to get started.
 
-```{r 00-setup, message = FALSE, echo = TRUE}
+```{r 00-setup, message = FALSE, echo = TRUE, warning = FALSE}
 library(traits)
 library(ggplot2)
 library(ggthemes)
 theme_set(theme_bw())
 library(dplyr)
 ```
+Create a file that contains your API key. If you have signed up for access to the TERRA REF database, your API key will have been sent to you in an email. You will need this personal key _and_ permissions to access the trait data. If you receive empty (NULL) datasets, it is likely that you do not have permissions. 
 
-Create a file that contains your API key. If you have signed up for access to the TERRA REF database, your API key will have been sent to you in an email. The public key will provide access to all metadata; you will need a personal key _and_ permissions to access the trait data. If you receive empty (NULL) datasets, it is likely that you do not have permissions. 
 
 ```{r writing-key, echo = TRUE}
 # This should be done once with the key sent to you in your email
-# writeLines('abcdefg_rest_of_key_sent_in_email', 
+
+# Example:
+#writeLines('abcdefg_rest_of_key_sent_in_email',
 #            con = '.betykey')
 
-# Example with the public key:
-writeLines('9999999999999999999999999999999999999999', 
-           con = '.betykey_public')
 ```
 
+
 #### R - using the traits package
 
 The R traits package is an API 'client'. It does two important things:
 1. It makes it easier to specify the query parameters without having to construct a URL
 2. It returns the results as a data frame, which is easier to use within R
 
-Lets start with the query of information about Sorghum from species table from above
+Lets start with the query of information about Sorghum from the species table
 
-```{r query-species, echo = TRUE}
+```{r query-species, results = 'hide', echo = TRUE}
 
 sorghum_info <- betydb_query(table = 'species',
-                            genus = "Sorghum",
-                            api_version = 'v1',
-                            limit = 'none',
-                            betyurl = "https://terraref.ncsa.illinois.edu/bety/", 
-                            key = readLines('.betykey', warn = FALSE))
+                             genus = "Sorghum",
+                             api_version = 'v1',
+                             limit = 'none',
+                             betyurl = "https://terraref.ncsa.illinois.edu/bety/", 
+                             key = readLines('.betykey', warn = FALSE))
 
 ```
 
 #### R - setting options for the traits package
 
 Notice all of the arguments that the `betydb_query` function requires? We can change this by setting the default connection options thus:
 
-
-
 ```{r 03-set-up, echo = TRUE}
 options(betydb_key = readLines('.betykey', warn = FALSE),
         betydb_url = "https://terraref.ncsa.illinois.edu/bety/",
@@ -69,7 +71,8 @@ options(betydb_key = readLines('.betykey', warn = FALSE),
 
 Now the same query can be reduced to:
 
-```{r query-species-reduce, echo = TRUE, results = FALSE}
+```{r query-species-reduce, message = FALSE, echo = TRUE}
+
 sorghum_info <- betydb_query(table = 'species',
                              genus = "Sorghum",
                              limit = 'none')
@@ -78,20 +81,23 @@ sorghum_info <- betydb_query(table = 'species',
 ### Time series of height
 
 Now let's query some trait data.
-```{r canopy_height, echo = TRUE, results = FALSE}
-sorghum_height <- betydb_query(table     = 'search',
+
+```{r canopy_height, echo = TRUE, message = FALSE}
+canopy_height <- betydb_query(table     = 'search',
                                trait     = "canopy_height",
-                               sitename  = "~Season 6",
+                               sitename  = "~Season 2",
                                limit     = 'none')
 ```
 
 ```{r plot_height}
-ggplot(data = sorghum_height,
+
+ggplot(data = canopy_height,
        aes(x = lubridate::yday(lubridate::ymd_hms(raw_date)), y = mean)) +
   geom_point(size = 0.5, position = position_jitter(width = 0.1)) +
 # scale_x_datetime(date_breaks = '6 months') +
   xlab("Day of Year") + ylab("Plant Height") +
   guides(color = guide_legend(title = 'Genotype')) +
   theme_bw()
+
 ```
 

traits/04-danforth-indoor-phenotyping-facility.Rmd

@@ -1,6 +1,7 @@
 # Danforth Indoor Phenotype Analysis
 
 ```{r 02-setup, include=FALSE}
+
 knitr::opts_chunk$set(echo = TRUE, cache = TRUE)
 library(jsonlite)
 library(dplyr)
@@ -21,7 +22,8 @@ library(traits)
 Unlike the first two tutorials, now we will be querying real data from the public TERRA REF database. So we will use a new URL, https://terraref.ncsa.illinois.edu/bety/, and we will need to use our own private key.
 
 ```{r terraref-connect-options}
-options(betydb_key = readLines('traits/.betykey', warn = FALSE),
+
+options(betydb_key = readLines('.betykey', warn = FALSE),
         betydb_url = "https://terraref.ncsa.illinois.edu/bety/",
         betydb_api_version = 'v1')
 ```
@@ -92,7 +94,7 @@ ggplot(data = danforth_sorghum) +
 
 ### Growth rate over time
 
-```{r danforth-phenotypes, fig.width=8, fig.height=4}
+```{r danforth-phenotypes, fig.width=8, fig.height=4, message = FALSE}
 
 ggplot(data = danforth_sorghum, aes(x = date, y = mean, color = cultivar)) +
  # geom_line(aes(group = entity), size = 0.1) +