5_unify_information.Rmd

# Unify related information

In this chapter we unify related information for each verified taxon.

## Read data

```{r unify_information-1, echo = TRUE}
# Read checklists
checklists <- read_csv(here("data", "raw", "checklists.csv"))

# Read data
taxa <- read_csv(here("data", "interim", "taxa_with_verification.csv"))
distributions <- read_csv(
  here("data", "raw", "distributions.csv"),
  col_types = cols(
    .default = col_character(),
    taxonKey = col_double()
  )
)
speciesprofiles <- read_csv(here("data", "raw", "speciesprofiles.csv"))
descriptions <- read_csv(
  here("data", "raw", "descriptions.csv"),
  col_types = cols(
    .default = col_character(),
    key = col_double(),
    taxonKey = col_double()
  )
)
```

## Assign checklist order

1. Get checklist keys as (ordered) vector.

```{r unify_information-2}
checklist_keys <-
  checklists %>%
  pull(datasetKey)
```

2. Assign the checklist order (= index of checklist keys) as an extra column to taxa.

```{r unify_information-3}
taxa <-
  taxa %>%
  rowwise() %>%
  mutate(checklistOrder = which(checklist_keys == datasetKey)[1])
```

## Unify distribution {#unify-distribution}

1. Parse `temporal` (eventDate) information.

```{r unify_information-4, echo = TRUE}
distributions_unified <-
  distributions %>%

  # Split temporal on "/" into startYear and endYear
  # If temporal only contains a single year, then endYear will be empty
  separate(
    temporal,
    into = c("startYear", "endYear"),
    sep = "/",
    remove = FALSE,
    convert = TRUE,
    extra = "drop",
    fill = "right"
  ) %>%

  # Only keep the first 4 characters: 1968-11-21 -> 1968
  mutate(
    startYear = as.integer(str_sub(startYear, 1, 4)),
    endYear = as.integer(str_sub(endYear, 1, 4))
  ) %>%

  # If endYear is empty (no range), populate it with startYear: 2018 & 2018
  mutate(endYear = if_else(is.na(endYear), startYear, endYear))
```

2. Filter distributions: this was already done in \@ref(filter-on-distribution), where only distributions of interest were kept.

3. Convert specific localities (from WRIMS) to regional or national localities:

```{r unify_information-5, echo = TRUE}
distributions_unified <-
  distributions_unified %>%
  mutate(locality = recode(locality,
    "Belgian part of the North Sea" = "Belgium",
    "Belgian Exclusive Economic Zone" = "Belgium",
    "Baai van Heist" = "Flemish Region",
    "Belgian Coast" = "Flemish Region",
    "Blankenberge Harbour" = "Flemish Region",
    "Bredene" = "Flemish Region",
    "Coastal Polders" = "Flemish Region",
    "De Panne" = "Flemish Region",
    "Haven van Antwerpen" = "Flemish Region",
    "Kanaal Gent - Terneuzen" = "Flemish Region",
    "Knokke-Heist" = "Flemish Region",
    "Koksijde" = "Flemish Region",
    "Nieuwpoort Harbour" = "Flemish Region",
    "Oostende" = "Flemish Region",
    "Oostende" = "Flemish Region",
    "Oostende Harbour" = "Flemish Region",
    "port of Bruges-Zeebrugge" = "Flemish Region",
    "Port of Ghent" = "Flemish Region",
    "Spuikom" = "Flemish Region",
    "Zeebrugge" = "Flemish Region",
    "Flanders" = "Flemish Region",
    "Wallonia" = "Walloon Region"
  )) %>%
  
  # Standardize locationId for all
  mutate(locationId = case_when(
    locality == "Belgium" ~ "ISO_3166:BE",
    locality == "Brussels-Capital Region" ~ "ISO_3166:BE-BRU",
    locality == "Flemish Region" ~ "ISO_3166:BE-VLG",
    locality == "Walloon Region" ~ "ISO_3166:BE-WAL",
    TRUE ~ NA_character_
  ))
```

4. Add a Belgian distribution from regional distributions within a checklist if not present.

```{r unify_information-6, echo = TRUE}
distributions_belgium_from_regions <-
  distributions_unified %>%

  # Group by taxonKey within a checklist
  group_by(taxonKey) %>%

  # Filter on those groups without a national distribution
  filter(!"Belgium" %in% locality) %>%

  # Take earliest and latest year from regional distributions
  # Don't copy source and remarks
  summarize(
    startYear = min(startYear, na.rm = TRUE),
    endYear = max(endYear, na.rm = TRUE)
  ) %>%
  ungroup() %>%

  # Convert dates to integers and set Inf to NA
  mutate(
    startYear = ifelse(!is.infinite(startYear), as.integer(startYear), NA_integer_),
    endYear = ifelse(!is.infinite(endYear), as.integer(endYear), NA_integer_)
  ) %>%

  # Add locality and locationId
  mutate(
    locality = "Belgium",
    locationId = "ISO_3166:BE"
  ) %>%

  # Order columns
  select(one_of(names(distributions_unified)))

# Add newly created Belgian distributions to distributions
distributions_unified <-
  distributions_unified %>%
  bind_rows(distributions_belgium_from_regions) %>%

  # Sort by taxonKey
  arrange(taxonKey)
```

5. Choose a single distribution per locality within a checklist.

**Note**: In contrast to the other extensions, the distribution information can come from multiple taxa within the same checklist. For example:

scientific name | eventDate
--- | ---
[https://www.gbif.org/species/141266662/verbatim](Medicago monantha subsp. incisa (Benth.) Verloove & Lambinon) | 1885/1901
[Medicago monantha subsp. noeana (Boiss.) Greuter & Burdet](https://www.gbif.org/species/141266791/verbatim) | 1886/1955

Both are considered (verified) synonyms of [Medicago monantha Trautv.](https://www.gbif.org/species/2965226), so their distribution information is merged to `1885/1955`. We note both in `taxonKeys` and `scientificNames`.

```{r unify_information-7, echo = TRUE}
distributions_unified <-
  distributions_unified %>%

  # Join distribution with taxon to get verificationKey and checklistOrder
  left_join(taxa, by = "taxonKey") %>%

  # Remove records that have no verificationKey (e.g. one wasn't assigned yet)
  filter(!is.na(verificationKey)) %>%

  # Group by verificationKey and locality within a checklist
  group_by(
    datasetKey,
    checklistOrder,
    verificationKey,
    locality,
    locationId
  ) %>%

  # Take earliest year, latest year and note taxonKey(s) and scientificName(s)
  summarize(
    startYear = min(startYear, na.rm = TRUE),
    endYear = max(endYear, na.rm = TRUE),
    taxonKeys = paste(unique(taxonKey), collapse = "|"),
    scientificNames = paste(unique(scientificName), collapse = "|")
  ) %>%
  ungroup() %>%

  # Convert dates to integers and set Inf to NA
  mutate(
    startYear = ifelse(!is.infinite(startYear), as.integer(startYear), NA_integer_),
    endYear = ifelse(!is.infinite(endYear), as.integer(endYear), NA_integer_)
  )
```

6. Choose a single distribution per locality across checklists.

```{r unify_information-8, echo = TRUE}
distributions_unified <-
  distributions_unified %>%

  # Sort by checklist order (trustworthiness)
  arrange(checklistOrder) %>%

  # Group by verificationKey and locality across checklists
  group_by(
    verificationKey,
    locality,
    locationId
  ) %>%

  # Select year of most trustworthy checklist (first one)
  # and note that checklist and its taxonKey(s) and scientificName(s)
  summarize(
    startYear = first(startYear),
    endYear = first(endYear),
    datasetKey = first(datasetKey),
    taxonKeys = first(taxonKeys),
    scientificNames = first(scientificNames)
  ) %>%
  ungroup() %>%

  # Sort by verificationKey
  arrange(verificationKey)
```

7. Save to [CSV](https://github.com/trias-project/unified-checklist/blob/master/data/interim/distributions_unified.csv).

```{r unify_information-9}
write_csv(distributions_unified, here("data", "interim", "distributions_unified.csv"), na = "")
```

## Unify species profiles

1. Filter species profiles.

```{r unify_information-10, echo = TRUE}
speciesprofiles_unified <-
  speciesprofiles %>%

  # Remove species profiles that contain NA for any of the attributes
  # This is rare: normally all attributes are populated or there just isn't a
  # species profile for that species
  filter(
    !is.na(marine) &
      !is.na(freshwater) &
      !is.na(terrestrial)
  )
```

2. Choose a single species profile within a checklist.

```{r unify_information-11, echo = TRUE}
speciesprofiles_unified <-
  speciesprofiles_unified %>%

  # Join species profile with taxon to get verificationKey and checklist order
  left_join(taxa, on = taxonKey) %>%

  # Remove records that have no verificationKey (e.g. one wasn't assigned yet)
  filter(!is.na(verificationKey)) %>%

  # Group by verificationKey within checklist
  group_by(
    datasetKey,
    checklistOrder,
    verificationKey
  ) %>%

  # Take first species profile and note taxonKey
  summarize(
    marine = first(marine),
    freshwater = first(freshwater),
    terrestrial = first(terrestrial),
    taxonKey = first(taxonKey),
    scientificName = first(scientificName)
  )
```

3. Choose a single species profile across checklists.

```{r unify_information-12, echo = TRUE}
speciesprofiles_unified <-
  speciesprofiles_unified %>%

  # Sort by checklist order (trustworthiness)
  arrange(checklistOrder) %>%

  # Group by verificationKey across checklists
  group_by(verificationKey) %>%

  # Select species profile of most trustworthy checklist (first one)
  # and note that checklist and its taxonKey
  summarize(
    marine = first(marine),
    freshwater = first(freshwater),
    terrestrial = first(terrestrial),
    datasetKey = first(datasetKey),
    taxonKey = first(taxonKey),
    scientificName = first(scientificName)
  ) %>%

  # Sort by verificationKey
  arrange(verificationKey)
```

4. Save to [CSV](https://github.com/trias-project/unified-checklist/blob/master/data/interim/speciesprofiles_unified.csv).

```{r unify_information-13}
write_csv(speciesprofiles_unified, here("data", "interim", "speciesprofiles_unified.csv"), na = "")
```

## Unify descriptions

### Native range

1. Retrieve native range.

```{r unify_information-14}
native_range <-
  descriptions %>%
  filter(type == "native range")
```

2. Standardize native range:

```{r unify_information-15, echo = TRUE}
native_range <-
  native_range %>%
  mutate(description = recode(description,
    "Africa" = "Africa",
    "Africa (WGSRPD:2)" = "Africa",
    "America" = "Americas",
    "Arctic" = "",
    "Asia" = "Asia",
    "Asia And Pacific Islands" = "Asia",
    "Australasia (WGSRPD:5)" = "Australia and New Zealand",
    "Australia" = "Australia and New Zealand",
    "Australia (WGSRPD:50)" = "Australia and New Zealand",
    "Bolivia, Chile, Argentina" = "South America",
    "Brazil, Uruguay, Paraguay, Argentina" = "South America",
    "Canada To Peru, Caribbean" = "Americas",
    "Canary Is. To Turkey" = "",
    "Canary Islands (WGSRPD:21_CNY)" = "",
    "Cape Verde Is., Africa, Madagascar, Comoros" = "",
    "Caribbean (WGSRPD:81)" = "Caribbean",
    "Caribbean To Brazil" = "Caribbean | Caribbean",
    "Carpathians" = "",
    "Caucasus region" = "",
    "Central America (WGSRPD:80)" = "Central America",
    "Central And South America" = "Central America | Central America",
    "Central Asia" = "Central Asia",
    "central Italy" = "Southern Europe",
    "central Mediterranean coastal areas" = "Southern Europe | Southern Europe",
    "Central, South America" = "Central America | Central America",
    "China" = "Eastern Asia",
    "China (WGSRPD:36)" = "Eastern Asia",
    "circum western Mediterranean coastal areas" = "Southern Europe | Southern Europe",
    "circum-Mediterranean" = "Southern Europe | Southern Europe",
    "coastal areas of the western Balkan" = "Southern Europe",
    "Cosmopolitan" = "",
    "Costa Rica (WGSRPD:80_COS)" = "Central America",
    "cultivated origin" = "",
    "Cyprus, Turkey, Middle East, Egypt, Sudan" = "",
    "East Asia" = "Eastern Asia",
    "eastern Africa" = "Eastern Africa",
    "Eastern Africa" = "Eastern Africa",
    "Eastern Asia (WGSRPD:38)" = "Eastern Asia",
    "Eastern Asian Russia" = "Eastern Asia | Eastern Asia",
    "Eastern Europe" = "Eastern Europe",
    "Eastern Europe (WGSRPD:14)" = "Eastern Europe",
    "Eastern North America" = "Northern America",
    "Eastern Palearctic Including China, Korea, Japan" = "Eastern Asia",
    "Egypt, Ethiopia, Kenya, Uganda, Rwanda" = "Northern Africa | Northern Africa",
    "Ethiopia" = "Northern Africa",
    "Eurasia" = "Europe | Europe",
    "Eurasia Africa" = "Europe | Europe | Africa",
    "Europe (WGSRPD:1)" = "Europe",
    "Europe To Azerbaijan, Egypt" = "",
    "Europe To Northern Africa" = "Europe | Europe",
    "Europe, Africa, Yemen, Pakistan, Philippines" = "",
    "Europe, North Africa To Near East, Turkey, Caucasus, Russia To Central Asia, Iran, Afghanistan, China, Mongolia, Korea" = "",
    "Galapagos (WGSRPD:GAL)" = "South America",
    "Hawaiian Islands (WGSRPD:HI)" = "",
    "hybrid origin" = "",
    "Iberia" = "Southern Europe",
    "Iberian Peninsula" = "Southern Europe",
    "India, Southeast Asia To Australia, New Zealand" = "",
    "Indian Subcontinent (WGSRPD:40)" = "Southern Asia",
    "Indo-Pacific" = "",
    "Indonesia (WGSRPD:ID)" = "South-eastern Asia",
    "Italy" = "Southern Europe",
    "Japan (WGSRPD:38_JAP)" = "Eastern Asia",
    "Japan (WGSRPD:JAP)" = "Eastern Asia",
    "Macaronesia" = "Northern Africa",
    "Macaronesia, Europe, North Africa To Turkey, Caucasus, Turkmenistan, Iran" = "",
    "Macaronesia, North Africa, Europe, Turkey, Caucasus, Near East, Kazakhstan" = "",
    "Madagascar (WGSRPD:MDG)" = "Eastern Africa",
    "Mallorca" = "Southern Europe",
    "Mediterranean" = "Southern Europe",
    "Mediterranean & Portugal" = "Southern Europe",
    "Mediterranean To Central Asia" = "Southern Europe | Southern Europe",
    "Mediterranean To Russia" = "Southern Europe | Southern Europe",
    "Mediterranean To Turkey, Israel" = "Southern Europe | Southern Europe",
    "Mediterranean, Africa, India, Myanmar, China" = "",
    "Mediterranean, Northern Africa" = "Southern Europe | Southern Europe",
    "Middle Africa" = "Middle Africa",
    "Middle East" = "Western Asia",
    "Nam" = "",
    "Near East" = "",
    "Nearctic" = "",
    "Neotropic" = "",
    "New Zealand" = "Australia and New Zealand",
    "New Zealand (WGSRPD:51)" = "Australia and New Zealand",
    "North Africa" = "Northern Africa",
    "North America" = "Northern America",
    "North America, Europe, North Africa, Turkey, Israel, Caucasus, Russia (European To Far East), Central Asia, China, Korea" = "",
    "North Pacific Ocean" = "",
    "North, Central And South America" = "Americas",
    "Northeast Asia" = "",
    "Northern Africa" = "Northern Africa",
    "Northern Africa And Middle East" = "Northern Africa | Northern Africa",
    "Northern Africa, Southern Europe To Turkey" = "",
    "Northern America" = "Northern America",
    "Northern America (WGSRPD:7)" = "Northern America",
    "northern Balkan" = "",
    "northern coastal areas of the western Mediterranean" = "",
    "northwestern Croatia" = "",
    "nortwestern Africa" = "Northern Africa",
    "Palearctic" = "",
    "pan-American" = "",
    "Panama (WGSRPD:PAN)" = "Central America",
    "Pantropical" = "",
    "Papua New Guinea" = "Melanesia",
    "Philippines (WGSRPD:PHI)" = "South-eastern Asia",
    "Ponto-Caspian" = "",
    "Portugal (WGSRPD:POR)" = "Southern Europe",
    "Probably Native To North America Only" = "Northern America",
    "Russia" = "Eastern Europe",
    "Slovenia" = "Eastern Europe",
    "South America" = "South America",
    "Southeast Asia" = "South-eastern Asia",
    "southeastern Alps" = "Southern Europe",
    "Southeastern Asia" = "South-eastern Asia",
    "Southeastern Europe (WGSRPD:13)" = "Southern Europe",
    "southeastern France" = "Western Europe",
    "Southern Africa" = "Southern Africa",
    "Southern Africa (WGSRPD:27)" = "Southern Africa",
    "southern Alps" = "Southern Europe",
    "Southern America (WGSRPD:8)" = "South America",
    "Southern Asia" = "Southern Asia",
    "southern Europe" = "Southern Europe",
    "Southern Europe" = "Southern Europe",
    "Southern Europe, Africa, Turkey, Caucasus, Near East, Iran" = "",
    "southern France" = "Western Europe",
    "Southern Hemisphere" = "",
    "Southwestern Africa" = "Southern Africa",
    "southwestern France" = "Western Europe",
    "Spain (WGSRPD:SPA)" = "Southern Europe",
    "Spain, Italy, North Africa" = "Southern Europe | Southern Europe",
    "Sub-Saharan Africa" = "",
    "Tanzania (WGSRPD:TAN)" = "Eastern Africa",
    "Tasmania (WGSRPD:50_TAS)" = "Australia and New Zealand",
    "Temperate Asia" = "",
    "temperate Asia (WGSRPD:3)" = "Eastern Asia | Eastern Asia",
    "Tropical and warm seas" = "",
    "Tropical Asia" = "",
    "tropical Asia (WGSRPD:4)" = "",
    "tropical western Africa" = "Western Africa",
    "unclear" = "",
    "United States" = "Northern America",
    "Usa To Guatemala" = "Northern America | Northern America",
    "West Africa" = "Western Africa",
    "Western Africa" = "Western Africa",
    "Western Asia (WGSRPD:34)" = "Western Asia",
    "Western Atlantic" = "",
    "western circum-Mediterranean" = "Southern Europe | Southern Europe",
    "western Italy" = "Southern Europe",
    "western Mediterranean" = "Southern Europe",
    "western Mediterranean coastal areas" = "Southern Europe",
    .default = "" # Change to description to discover new values
  ))
```

### Combine and process descriptions

1. Combine descriptions.

```{r unify_information-16}
descriptions_unified <-
  native_range %>%
  bind_rows(
    descriptions %>% filter(type != "native range")
  )
```

2. Split and gather descriptions on `|`.

```{r unify_information-17}
descriptions_unified <-
  descriptions_unified %>%
  
  # Split on " | "
  separate(
    description,
    into = c("description_1", "description_2", "description_3"),
    sep = " \\| ",
    remove = FALSE,
    convert = TRUE,
    extra = "drop",
    fill = "right"
  ) %>%
  
  # Gather in maximum 3 rows
  gather(
    key,
    description,
    description_1, description_2, description_3,
    na.rm = TRUE,
    convert = TRUE
  ) %>%
  
  # Remove key column
  select(-key)
```

3. Remove `NA` and empty descriptions.

```{r unify_information-18}
descriptions_unified <-
  descriptions_unified %>%
  mutate(description = na_if(description, "")) %>% # Change "" to NA
  filter(!is.na(description))
```

4. Select unique descriptions (within their type) within a checklist.

```{r unify_information-19, echo = TRUE}
descriptions_unified <-
  descriptions_unified %>%

  # Join species profile with taxon to get verificationKey and checklist order
  left_join(taxa, on = taxonKey) %>%

  # Remove records that have no verificationKey (e.g. one wasn't assigned yet)
  filter(!is.na(verificationKey)) %>%

  # Group by type and verificationKey within checklist
  group_by(
    datasetKey,
    checklistOrder,
    verificationKey,
    type,
    description
  ) %>%

  # Take first taxonKey and scientificName
  summarize(
    taxonKey = first(taxonKey),
    scientificName = first(scientificName)
  )
```

5. Select unique descriptions (within their type) across checklists:

```{r unify_information-20, echo = TRUE}
descriptions_unified <-
  descriptions_unified %>%

  # Sort by checklist order (trustworthiness)
  arrange(checklistOrder) %>%

  # Group by type and verificationKey across checklists
  group_by(
    type,
    description,
    verificationKey
  ) %>%

  # Select first datasetKey, taxonKey and scientificName
  summarize(
    datasetKey = first(datasetKey),
    taxonKey = first(taxonKey),
    scientificName = first(scientificName)
  ) %>%

  # Move verificationKey to beginning and drop checklist_order
  select(verificationKey, everything()) %>%

  # Sort by verificationKey and type
  arrange(verificationKey, type)
```

6. Save to [CSV](https://github.com/trias-project/unified-checklist/blob/master/data/interim/descriptions_unified.csv).

```{r unify_information-21}
write_csv(descriptions_unified, here("data", "interim", "descriptions_unified.csv"), na = "")
```