dwc_mapping.Rmd

---
title: "Darwin Core mapping"
subtitle: "For: Ad hoc checklist of alien species in Belgium"
author:
- Lien Reyserhove
- Sanne Govaert
- Peter Desmet
- Quentin Groom
- Tim Adriaens
date: "`r Sys.Date()`"
output:
  html_document:
    df_print: paged
    number_sections: true
    toc: true
    toc_depth: 3
    toc_float: true
---

This document describes how we map the checklist data to Darwin Core. The source file for this document can be found [here](https://docs.google.com/spreadsheets/d/1LeXXbry2ArK2rngsmFjz_xErwE1KwQ8ujtvHNmTVA6E/edit#gid=0).

```{r setup, include = FALSE}
knitr::opts_chunk$set(echo = TRUE, warning = FALSE, message = FALSE)
```

Load libraries:

```{r}
library(tidyverse)      # To do data science
library(here)           # To find files
library(janitor)        # To clean input data
```

# Read source data

The data is maintained in [this Google Spreadsheet](https://docs.google.com/spreadsheets/d/1LeXXbry2ArK2rngsmFjz_xErwE1KwQ8ujtvHNmTVA6E/edit#gid=0).

Read the relevant worksheet (published as csv):

```{r read_source_data}
raw_data <- read_csv("https://docs.google.com/spreadsheets/d/e/2PACX-1vTl8IEk2fProQorMu5xKQPdMXl3OQp-c0f6eBXitv0BiVFZ3JSJCde0PtbFXuETgguf6vK8b43FDX1C/pub?gid=0&single=true&output=csv", show_col_types = FALSE)
```
Copy the source data to the repository to keep track of changes:

```{r}
write_csv(raw_data, here("data", "raw", "ad_hoc_checklist_dump.csv"), na = "")
```

Preview data: 

```{r}
raw_data %>% head()
```
# Preprocessing: tidy data and add taxon ID's

To link taxa with information in the extension(s), each taxon needs a unique and relatively stable `taxonID`. We have created one in the form of `dataset_shortname:taxon:hash`, where `hash` is unique code based on scientific name and kingdom. Once this is created, it is added to the source data. 

```{r}
input_data <-
  raw_data %>%
  remove_empty("rows") %>%
  clean_names() %>%
  mutate(
    taxon_id = paste(
      "ad-hoc-checklist",
      "taxon",
      .data$taxon_id_hash,
      sep = ":"
    )
  )
```

# Darwin Core mapping

Map the data to [Darwin Core Taxon](http://rs.gbif.org/core/dwc_taxon_2015-04-24.xml).

## Taxon core

```{r}
taxon <-
  input_data %>% 
    distinct(.data$taxon_id, .keep_all = TRUE) %>% 
    mutate(
      language = "en",
      license = "http://creativecommons.org/publicdomain/zero/1.0/",
      rightsHolder = "INBO",
      accessRights = "https://www.inbo.be/en/norms-data-use",
      datasetID = "https://doi.org/10.15468/3pmlxs",
      institutionCode = "INBO",
      datasetName = "Ad hoc checklist of alien species in Belgium",
      taxonID = taxon_id,
      scientificName = .data$scientific_name,
      kingdom = .data$kingdom,
      phylum = .data$phylum,
      class = .data$class,
      order = .data$order,
      family = .data$family,
      genus = .data$genus,
      taxonRank = .data$taxon_rank,
      nomenclaturalCode = .data$nomenclatural_code
      ) %>% 
    arrange(taxonID) %>% 
    select("language", "license", "rightsHolder", "accessRights", "datasetID",
           "institutionCode", "datasetName", "taxonID", "scientificName",
           "kingdom", "phylum", "class", "order", "family", "genus",
           "taxonRank", "nomenclaturalCode")
```

## Distribution extension

Map the data to [Species Distribution](http://rs.gbif.org/extension/gbif/1.0/distribution.xml)

```{r}
distribution <- 
  input_data %>% 
  # pathway mapping
    pivot_longer(
      names_to = "key",
      values_to = "pathway",
      starts_with("introduction_pathway"),
      values_drop_na = FALSE
    ) %>% 
    filter(
      !is.na(pathway) |
        (is.na(pathway) & key == "introduction_pathway_1")
    ) %>% 
    mutate(
      taxonID = .data$taxon_id,
      locationID = case_when(
        is.na(location) & country_code == "BE" ~ "ISO_3166-2:BE",
        location == "Flanders" ~ "ISO_3166-2:BE-VLG",
        location == "Wallonia" ~ "ISO_3166-2:BE-WAL",
        location == "Brussels" ~ "ISO_3166-2:BE-BRU"
      ),
      locality = case_when(
        is.na(location) & country_code == "BE" ~ "Belgium",
        location == "Flanders" ~ "Flemish Region",
        location == "Wallonia" ~ "Walloon Region",
        location == "Brussels" ~ "Brussels-Capital Region"),
      countryCode = .data$country_code,
      occurrenceStatus = .data$occurrence_status,
      establishmentMeans = establishment_means,
      degreeOfEstablishment = .data$degree_of_establishment,
      first_observation_clean = 
        str_remove_all(.data$date_first_observation, ">"),
      eventDate = case_when(
        is.na(first_observation_clean) & is.na(date_last_observation) ~ NA,
        is.na(first_observation_clean) ~ paste0("/", date_last_observation),
        !is.na(date_last_observation) ~ paste0(first_observation_clean, "/"),
        !is.na(first_observation_clean) & !is.na(date_last_observation) ~ 
           paste(first_observation_clean, date_last_observation, sep = "/")
      ),
      source = .data$source
    ) %>%
  select(
    "taxonID", "locationID", "locality", "countryCode", "occurrenceStatus",
    "establishmentMeans", "degreeOfEstablishment", "pathway", "eventDate",
    "source"
  ) %>%
  arrange(taxonID)
```

## Species profile extension

In this extension we will express broad habitat characteristics of the species (e.g. `isTerrestrial`).

Create a dataframe with unique taxa only (ignoring multiple distribution rows).
Only keep records for which `terrestrial`, `marine` and `freshwater` is not empty.

Map the data to [Species Profile](http://rs.gbif.org/extension/gbif/1.0/speciesprofile.xml).


```{r}
species_profile <- 
  input_data %>% 
    distinct(taxon_id, .keep_all = TRUE) %>% 
    filter(
      !is.na(terrestrial) |
      !is.na(marine) |
      !is.na(freshwater)
    ) %>% 
  mutate(
    .keep = "none",
    taxonID = .data$taxon_id,
    isMarine = .data$marine,
    isFreshwater = .data$freshwater,
    isTerrestrial = .data$terrestrial
  ) %>% 
  arrange(taxonID)
```

## Description extension

In the description extension we want to include the native range of a species

```{r}
description <- 
  input_data %>% 
    distinct(taxon_id, .keep_all = TRUE) %>% 
  mutate(native_range = strsplit(native_range, "\\|")) %>% 
  unnest(native_range) %>% 
  filter(!is.na(native_range)) %>% 
  mutate(
    .keep = "none",
    taxonID = .data$taxon_id,
    description = str_trim(native_range),
    type = "native range",
    language = "en"
  ) %>% 
  arrange(taxonID)
```

# Save to CSV:

Save to CSV:

```{r}
write_csv(taxon, here("data", "processed", "taxon.csv"), na = "")
write_csv(distribution, here("data", "processed", "distribution.csv"), na = "")
write_csv(species_profile, here("data", "processed", "speciesprofile.csv"), na = "")
write_csv(description, here("data", "processed", "description.csv"), na = "")
```