shiny_app_use.Rmd

---
title: "GeneKeepR Tutorial"
subtitle: "A Shiny Application for the Genetic Management of Colonies"
author: "R. Mark Sharp"
output:
  html_document:
    df_print: paged
  pdf_document: default
vignette: >
  %\VignetteIndexEntry{GeneKeepR Tutorial} 
  %\usepackage[UTF-8]{inputenc}
  %\VignetteEngine{knitr::rmarkdown_notangle} 
---

```{r setup, include=FALSE}
library(png)
library(kableExtra)
library(grid)
library(stringi)
library(nprcgenekeepr)
knitr::opts_chunk$set(eval = FALSE, echo = TRUE, results = "markup", cache = FALSE)
pdf.options(useDingbats = TRUE)
start_time <- proc.time()

```

## Introduction
This tutorial demonstrates the major functions used 
within the Shiny application provided by the __nprcgenekeepr__ package.
This is a brief tutorial that illustrates a typical workflow and does
not explore all possible workflows.

Please provide any comments, questions, or bug reports through the GitHub
issue tracker at 
[https://github.com/rmsharp/nprcgenekeepr/issues](
https://github.com/rmsharp/nprcgenekeepr/issues).

## Installation and Help

To get the most recent version you can install **nprcgenekeepr** from
GitHub with the following code.

```{r gh-installation, include = TRUE}
install.packages("devtools")
devtools::install_github("rmsharp/nprcgenekeepr")

```

All missing package dependencies should be automatically installed.


```{r child = "./manual_components/_online_documentation.Rmd", ref = "online_documentation", include = TRUE, eval = TRUE}

```

This document is the `Application for Genetic Management of Colonies` vignette.

```{r child = "./manual_components/_running_shiny_application.Rmd", ref = "running_shiny_application", include = TRUE, eval = TRUE}

```

This will result in the opening screen where you tell the application how to 
find the pedigree you will be using.

## Input

### Pedigree File Structure

Most of the screen is filled with information about formatting a text or
Excel worksheet pedigree file.

```{r eopening-screen-top, eval = TRUE, fig.width = 6.5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/opening_screen_top.png")
grid::grid.raster(img)

```

*********

Scrolling down to the middle of the opening screen exposes a table that 
describes a pedigree file and further instructions.

```{r eopening-screen-middle, eval = TRUE, fig.width = 6.5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/opening_screen_middle.png")
grid::grid.raster(img)

```

*********

Scrolling down to the bottom of the opening screen exposes more pedigree file 
instructions, a table that describes a genotype file and instructions regarding
use of a genotype file. 

```{r eopening-screen-bottom, eval = TRUE, fig.width = 5.5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/opening_screen_bottom.png")
grid::grid.raster(img)

```

*********

The following is an example of the pedigree file format.

Without genotypes:  

```{r eexamplePedigreeTutorial, eval = TRUE, fig.width = 5.5, fig.height = 2, echo = FALSE}
img <- readPNG("./shiny_app_use/examplePedigreeTutorial.png")
grid.raster(img)

```
  
With genotypes:  

```{r eexamplePedigreeTutorial-with_alleles, eval = TRUE, fig.width = 5.5, fig.height = 2, echo = FALSE}
img <- readPNG("./shiny_app_use/examplePedigreeTutorial_with_alleles.png")
grid.raster(img)

```

*********

### Reading in the Pedigree

In this introductory tutorial, we will use an Excel file containing 
a hypothetical pedigree of macaques.
We will work with the gray box on the left at the top of the screen.

```{r eopening-screen-top-red-oval, eval = TRUE, fig.width = 5.5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/opening_screen_top_red_oval.png")
grid::grid.raster(img)

```

*********

A Microsoft Excel workbook with a single worksheet is the default file 
type; though comma (.csv), semi-colon (.txt), and tab (.txt) separated value 
files are all acceptable formats.

The _Example_Pedigree.xlsx_ file we are using is from a CSV file created as 
shown below and then saved in an Excel format.

```{r make-example-file}
makeExamplePedigreeFile()

```

Select the __Browse__ button and select the pedigree file from your file system.


```{r example-pedigree, eval = TRUE, fig.width = 2.5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/input_example_pedigree_xlsx.png")
grid::grid.raster(img)

```

*********

It is important to make sure the minimum parent age is low enough for the 
animals in your pedigree. For our example pedigree, we are changing it from
4 years to 2 years of age since these macaques may reproduce as early as two 
years of age.
This is shown below in three progressive images with the center image 
demonstrating how the hovertext provides an explanation of how this value is 
used.


```{r example-pedigree-minParentAgeSequence, eval = TRUE, fig.width = 11, echo = FALSE}
img <- png::readPNG("./shiny_app_use/input_minParentAgeSequence.png")
grid::grid.raster(img)

```

*********

### Reading in a Pedigree and Testing for Errors

Selected __Read and Check Pedigree__ will read in the file and test to see
if the pedigree file has all of the columns needed and the pedigree is 
internally consistent.


```{r read-and-check-pedigree, eval = TRUE, fig.width = 2.5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/read_and_check_pedigree.png")
grid::grid.raster(img)

```

*********

Several error types, shown below, are detected the application.

```{r make-errorList-definition-tbl, echo = FALSE, eval=TRUE}
errorTypes <- names(getEmptyErrorLst())
errorDescriptions <- c(
  "Database connection failed: configuration or permissions are invalid",
  "Columns that must be within the pedigree file are missing.",
  "Values, which are supposed to be dates, cannot be interpreted as a date.",
  "Parents were too young on the date of birth of to have been the parent.",
  "Individuals listed as female or hermaphroditic and as a sire.",
  "Individuals are listed as male and as a dam.", 
  "Individuals who are listed as both a sire and a dam.",
  "IDs listed more than once.",
  stri_c("Fatal Errors. These are unanticipated errors that should be ", 
         "reported to the package maintainer"),
  stri_c("Columns that have been changed to conform to internal naming ",
  "conventions and what they were changed to.")
)
errorTbl <- data.frame(`Error` = errorTypes, Definition = errorDescriptions, 
                       stringsAsFactors = FALSE) 

```
```{r print-error-definition-tbl, eval = TRUE, echo=FALSE, include = TRUE, results='markup'}
knitr::kable(errorTbl)  %>%
  kable_styling(full_width = F) %>%
  column_spec(1, bold = T, color = "blue") %>%
  column_spec(2, width = "30em")

```

*********

## Pedigree Browser

The __Pedigree Browser__ tab defaults to displaying 10 rows of the pedigree
at a time, but you can choose to display 10, 25, 50, or 100 rows.
You can choose to display UNKNOWN IDs in the rows displayed.
UNKNOWN IDs (UIDs) are used to label unknown parents of animals with one known
parent.
The program calculates additional columns based on the input pedigree.


```{r pb-10-rows-display-unknown-ids, eval = TRUE, fig.width = 5.5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/pb_10_rows_display_unknown_ids.png")
grid::grid.raster(img)

```

*********

### Unknown IDs

I have placed red lines under the UNKNOWN IDs in the partial pedigree list
below. UNKNOWN IDs are used to label unknown parents of animals with one known
parent. (Note these are found near the end of the pedigree list,)
These IDs have no meaning other than they all begin with the letter
_U_ and are following with a left alphanumeric string of five places. 

```{r unknown-displayed, eval = TRUE, fig.width = 5, fig.height = 2, echo = FALSE}
img <- png::readPNG("./shiny_app_use/pb_unknown_displayed.png")
grid::grid.raster(img)

```

*********

In this example pedigree, when you deselect the __Display Unknown IDs__
checkbox.
The number of rows reduces from 3,694 to 2,322, because there were 1,372
UNKNOWN animals generated when constructing the pedigree to provide sire
and dam placeholders for all animals.


```{r no-unknown-displayed, eval = TRUE, fig.width = 5, fig.height = 2, echo = FALSE}
img <- png::readPNG("./shiny_app_use/pb_no_unknown_displayed.png")
grid::grid.raster(img)

```

*********

### Selecting a Pedigree Subset --- Focal Animals

The __Pedigree Browser__ tab displays the full pedigree by default but allows
you to select a subset of the pedigree by entering a list of 
animals of interest (_focal animals_).


```{r focal-animal-text-box, eval = TRUE, fig.width = 5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/pb_focal_animal_text_box.png")
grid::grid.raster(img)
                                                                                      
```

*********

You can enter in the animal IDs by typing them into the text box directly as
shown below (FJS7RQ, H6T2FF, HEVL3L, I04JZV, S63QDN). 
Deselect the __Display Unknown IDs__ checkbox and select the
__Trim pedigree based on focal animals__ checkbox.
(See top right of image below).

Trimming the pedigree based on focal animals will keep only animals in the pedigree
that are related to the focal animals selected.

Select the __Update Focal Animals__ button to tell the application to read
your list of animals, trim the pedigree based on that list, and display the 
trimmed pedigree below.
You will end up with 54 animals in your pedigree.



```{r pedigree-browser-5-focal-animals-small, eval = TRUE, fig.width = 8.0, echo = FALSE}
img <- png::readPNG("./shiny_app_use/pb_5_focal_animals_small.png")
grid::grid.raster(img)

```

*********

Also, you can import a list of focal animals by selecting the __Browse__ button
under __Choose CSV file with focal animals__. 
This file can be constructed by creating a simple text file with commas between
animal IDs or by placing individual animal IDs on separate lines.

Focal animals are the list of animals that will be used in the following analysis. 
In most cases, we recommend using all alive animals in the breeding population.
By selecting focal animals, the number of pedigree entrees does not change, but 
the population membership flag will be set to `TRUE` for the focal animals, and
`FALSE` for all other animals.


```{r selection-large-focal-group, eval = TRUE, fig.width = 4, fig.height = 2, echo = FALSE}
img <- png::readPNG("./shiny_app_use/pb_selection_large_focal_group.png")
grid::grid.raster(img)

```                                                                                      

*********

After entering your list of focal animals, you can select to trim the 
pedigree so that it will only include relatives of the focal animals you have 
selected.
This will reduce the number of members within the pedigree to all 
animals required to connect all of the focal animals in the pedigree.


```{r select-trim-for-focal-animals, eval = TRUE, fig.width = 6, echo = FALSE}
img <- png::readPNG("./shiny_app_use/pb_select_trim_for_focal_animals.png")
grid::grid.raster(img)

```                                                                                      

*********

A pedigree trimmed based on focal animals will have only the relatives of those
animals remaining.
In this instance there are only a total of 522 focal animals and their relatives.

_Note: focal animals and their relatives will only be included in the same 
pedigree when the original pedigree file uploaded indicates a common 
ancestor for them. Otherwise, focal animals and their relatives will 
be sorted into separate pedigrees in the output, with each separate
pedigree indicated by its own number._

```{r trimmed-for-focal-animals, eval = TRUE, fig.width = 4, fig.height = 2, echo = FALSE}
img <- png::readPNG("./shiny_app_use/pb_trimmed_for_focal_animals.png")
grid::grid.raster(img)

```                                                                                      

*********

You can remove the animals from the list of focal animals by selecting the 
__Clear Focal Animals__ checkbox and selecting the __Update Focal Animals__
button. This will read in an empty ID list, clear the box of IDs, and bring back
all of the trimmed away IDs.

```{r cleared-of-focal-animals, eval = TRUE, fig.width = 6, echo = FALSE}
img <- png::readPNG("./shiny_app_use/pb_cleared_focal_animals_combined.png")
grid::grid.raster(img)

```                                                                                      

Deselect the __Clear Focal Animals__ checkbox and reselect the 
__Update Focal Animals__ button before continuing with the tutorial so that
we will be working with the trimmed pedigree.

*********

## Pedigree Age Plot

The __Pedigree Age Plot__ tab displays a standard pyramid plot for the pedigree
as selected in the __Pedigree Browser__ tab.
This is showing 332 living animals from the entire example pedigree.


```{r age-plot, eval = TRUE, fig.width = 8.0, echo = FALSE}
img <- png::readPNG("./shiny_app_use/age_plot.png")
grid::grid.raster(img)

```                                                                                      

*********

## Genetic Value Analysis

Selecting the __Genetic Value Analysis__ tab and enter the number of 
simulations and genome uniqueness threshold desired. 
See the __Genetic Value Analysis and Breeding Group Description__ tab for 
a breakdown of the calculation. 

We recommend trying multiple numbers of simulations to arrive at an ideal
number that produces consistent results (i.e., 1,000).
Genome uniqueness values are calculated using a gene-drop simulation 
according to MacCluer et al. (1986) and Ballou & Lacy (1995), by assigning 
unique alleles to all pedigree founders, and simulating their segregation 
throughout the pedigree according to Mendelian rules. Genome uniqueness is 
a measure of the probability that an animal possesses founder alleles that
are present in at most x other animals (usually 0-3), and thus are rare 
and at risk of being lost from the population. A range of 2 to 100,000 
simulations may be selected. A minimum of 1,000 simulations is recommended.
A genome uniqueness threshold value between 0-3 should also be selected,
as desired.

Select the __Begin Analysis__ button to start the gene dropping 
process, which you can monitor with the progress meter in the 
lower right corner of the display. 

We are not aware of a systematic study of pedigree structure with this 
algorithm and have not performed extensive studies with pedigrees of various 
structures, but 1000 iterations has seemed to provide reproducible results for
out pedigrees. It must be emphasized that pedigree structure is expected to 
affect precision when iterations are held constant.

<!--  Come up wit a definition of reproducible and see if we can run and -->
<!-- automated test to find the needed number of iterations.-->


```{r gva-calculating, eval = TRUE, fig.width = 8, echo = FALSE}
img <- png::readPNG("./shiny_app_use/gva_calculating.png")
grid::grid.raster(img)

```                                                                                      

*********

As soon as the calculations are completed, a table showing
the results of the analysis is displayed in 10 record increments. 
The calculations for 1000 iterations of the gene dropping algorithm took 1 
minute 38 seconds with the example pedigree of 3,691
animals using a MacBook Pro (Mid 2014), 2.8 GHz Intel Core i7 with 16 GB of
1600 MHz DDR3 memory.

Again you can select how many rows to display at once by changing the 
values in the  __Show entries__ selection tool.



```{r gva-first-high-value, eval = TRUE, fig.width = 8.0, echo = FALSE}
img <- png::readPNG("./shiny_app_use/gva_first_high_value.png")
grid::grid.raster(img)

```                                                                                      

*********

Searching down the table of results in the __Value Designation__ column you can 
see starting at row 268 the values change from _High Value_ to _Low Value_.
Though not shown here, the value of _Undetermined_ in the __Value Designation__ 
column means the animal did not have parentage information.
Infants or very young animals without assigned parents are given an 
"Undetermined" designation. Founders also do not have parentage information 
but are high value by definition.


```{r gva-high-and-low-value, eval = TRUE, fig.width = 5.5, fig.height = 1.5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/gva_high_and_low_value.png")
grid::grid.raster(img)

```                                                                                      

*********

## Summary Statistics

The __Summary Statistics and Plots__ tab used results from the 
__Genetic Value Analysis__ tab.
Definitions of genome uniqueness and kinship are located 
in the __Genetic Value Analysis and Breeding Group Description__ tab. 
Additionally, definitions of founder equivalents and founder genome 
equivalents are located at the bottom of the __Summary Statistics and
Plots__ tab.

```{r ss-first-view, eval = TRUE, fig.width = 8.0, echo = FALSE}
img <- png::readPNG("./shiny_app_use/ss_first_view.png")
grid::grid.raster(img)

```                                                                                      

*********

The __Export Kinship Matrix__ button creates a CSV file that has a row and
column for each individual in the genetic analysis plus a first row 
and first column each containing the IDs.

The first few rows of such a file are shown below.

```{r ss-kinship-matrix, eval = TRUE, fig.width = 6, fig.height = 2, echo = FALSE}
img <- png::readPNG("./shiny_app_use/ss_kinship_matrix.png")
grid::grid.raster(img)

```                                                                                      

*********

The __First-Order Relationships__ button creates a CSV file that has the 
following columns defined: an unnamed column for row number, _id_, _parents_,
_offspring_, _siblings_, and _total_. 
Counts are based off known relationships.
The first few rows of such a file are shown below.

```{r first-order-relationships, eval = TRUE, fig.width = 5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/ss_first_order_relationships.png")
grid::grid.raster(img)

```

*********

The __Export Female Founders__ and __Export Male Founders__ buttons creates a
CSV file that has the following columns defined: an unnamed column for 
row number, _id_, _sires_, _dam_, _sex_, _gen_, _birth_, _exit_, _age_, 
_recordStatus_, _population_, and _pedNum_.
The first few rows of such a file are shown below.


```{r female-founders, eval = TRUE, fig.width = 5, fig.height = 2, echo = FALSE}
img <- readPNG("./shiny_app_use/ss_female_founders.png")
grid.raster(img)

```

*********

The six plots provide histograms and boxplots for the kinship coefficients,
the Z-scores of the kinship coefficients, and the genome uniqueness scores.

```{r ss-trimmed-all-plots, eval = TRUE, fig.width = 8, echo = FALSE}
img <- png::readPNG("./shiny_app_use/ss_trimmed_all_plots.png")
grid::grid.raster(img)

```                                                                                      

*********

Each of the plots can be exported as a PNG file to a directory you choose.

```{r ss-export-mean-kinship-coef-hist-plot, eval = TRUE, fig.width = 5, echo = FALSE}
img <- png::readPNG(
  "./shiny_app_use/ss_export_mean_kinship_coefficient_histogram.png")
grid::grid.raster(img)

```                                                                                      

*********

## Breeding Group Formation

Selecting the __Breeding Group Formation__ tab brings forward the screen shown
below. 
In this screen you can form breeding groups using one of three ways based on
your source of animals selected under __Choose one group formation workflow:__.

Additionally, you must specify how you want to construct the breeding groups 
with regard to the groups' sex ratios. 
The third of the three options (_User specified sex ratio of breeders_) causes
the appearance of the field where you can fill in the sex ratio (F/M) that 
you want to have in the formed breeding groups.
The sex ratio algorithm will form a group as nearly to the selected ratio as 
possible given the size of the group. 
Limits in the availability of either sex will restrict the size of the groups 
formed.

```{r breeding-group-first-view, eval = TRUE, fig.width = 7, echo = FALSE}
img <- png::readPNG("./shiny_app_use/breeding_group_first_view.png")
grid::grid.raster(img)

```                                                                                      

*********

The __Make Groups__ button appears once you select the source of animals you 
are going to use.
However, you probably will be making additional selections using other controls
on the screen.

The most common source of animals will be the high-value animals found by the 
genetic analysis.

You can either type in the number of groups that you want to form or select the
number of groups using the arrows on the right edge of the 
__Number of Groups Desired__ field, which is outlined in blue in the image 
below.

```{r breeding-group-1, eval = TRUE, fig.width = 8, echo = FALSE}
img <- png::readPNG("./shiny_app_use/breeding_group_1.png")
grid::grid.raster(img)

```                                                                                      

*********

There are often behavioral constraints, such as preexisting social groups, 
that dictate the need to have some animals maintained together. 
This need is readily accommodated by pre-seeding groups with those social 
groups.
You may select the __Optional: Seed Groups with Specific Animals__ field if
you decide to place some animals together within the groups because you know
them to be compatible with each other.

This has been done in the example below using six groups with differing numbers
of seed animals. Note the selection of having animals below the minimum parent 
age of two being grouped with their mother.

*********

```{r breeding-group-6-infants-with-dam, eval = TRUE, fig.width = 6, echo = FALSE}
img <- png::readPNG("./shiny_app_use/breeding_group_6_infants_with_dam.png")
grid::grid.raster(img)

```                                                                                      

*********

Each group has all of the seed animals that were assigned to it plus additional
animals that could be added while satisfying the requirements imposed by the 
selected settings. I have indicated the seed animals for the first group with
red rectangles.

```{r breeding-group-first-group-no-kinship-seeds-indicated, eval = TRUE, fig.width = 8.5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/breeding_group_first_group_no_kinship_seeds_indicated.png")
grid::grid.raster(img)

```                                                                                      

*********

Display of kinship values requires that the
__Include kinship in display of groups__ checkbox be selected prior to group
formation.

A group of ten animals was formed in the next run after choosing to include 
kinship and selecting the __Make Groups__ button.


```{r breeding-group-6-seed-grps-grp-6-kinship, eval = TRUE, fig.width = 8.5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/breeding_group_6_seed_grps_grp_6_kinship.png")
grid::grid.raster(img)

```                                                                                      

*********

The option to select a desired sex ratio allows you to select any ratio
desired.
However, the ratio obtained is limited by the availability of animals that
meet all criteria you have set.

```{r breeding-group-sex-ratio-specification, eval = TRUE, fig.width = 5.5, echo = FALSE}
img <- png::readPNG("./shiny_app_use/breeding_group_sex_ratio_specification.png")
grid::grid.raster(img)

```                                                                                      

Selecting a sex ratio of 2.5 with 6 groups as is illustrated resulted in 5 
groups of 20 with a ratio of 14:6 (2.3) and 1 group of 23 with a ratio of 16:7 
(2.3).

Groups can be individually exported into file names and locations of your 
choosing. The corresponding kinship matrix for each group can also be exported.