7_transform.qmd

---
execute:
  echo: true
---

::: {.content-visible when-format="pdf"}

```{=latex}
\setDOI{10.4324/9781003393764.7}
\thispagestyle{chapterfirstpage}
```

:::

# Transform {#sec-transform-chapter}

```{r}
#| label: setup-options
#| child: "../_common.qmd"
#| cache: false
```

::: {.callout}
**{{< fa regular list-alt >}} Outcomes**

- Understand the role of data transformation in a text analysis project.
- Identify the main types of transformations used to prepare datasets for analysis.
- Recognize the importance of planning and documenting the transformation process.
:::

In this chapter, we will focus on transforming curated datasets to refine and possibly expand their relational characteristics to align with our research. I will approach the transformation process by breaking it down into two sub-processes: preparation and enrichment. The preparation process involves normalizing and tokenizing text. The enrichment process involves generating, recoding, and integrating variables. These processes are not sequential but may occur in any order based on the researcher's evaluation of the dataset characteristics and the desired outcome.

::: {.callout}
**{{< fa terminal >}} Lessons**

**What**: Reshape Datasets by Rows, Reshape datasets by Columns\
**How**: In an R console, load {swirl}, run `swirl()`, and follow prompts to select the lesson.\
**Why**: Explore data preprocessing skills to manipulate rows and columns using powerful packages like {dplyr} and {tidytext} to normalization, tokenize, and integrate datasets equipping you with the essential techniques to structure datasets for analysis.
:::

## Preparation {#sec-transform-preparation}

In this section, we will cover the processes of normalization and tokenization. These processes are particularly relevant for text analysis, as text conventions can introduce unwanted variability in the data and, therefore, the unit of observation may need to be adjusted to align with the research question.

To illustrate these processes, we will use a curated version of the Europarl Parallel Corpus [@Koehn2005]\index{Europarl Parallel Corpus}. This dataset contains transcribed source language (Spanish) and translated target language (English) from the proceedings of the European Parliament. The unit of observation\index{unit of observation} is the `lines` variable whose values are lines of dialog. We will use this dataset to explore the normalization and tokenization processes.\index{text normalization}\index{text tokenization}

The contents of the data dictionary for this dataset appears in @tbl-transform-europarl-dd.

```{r}
#| label: tbl-transform-europarl-dd
#| tbl-cap: "Data dictionary for the curated Europarl Parallel Corpus."
#| tbl-colwidths: [7, 22, 13, 58]
#| echo: false

read_csv(file = "data/curate-europarl_curated_dd.csv") |>
  tt(width = 1)
```

Let's read in the dataset CSV file with `read_csv()` and inspect the first lines of the dataset with `slice_head()` in @exm-transform-europarl-preview.

::: {#exm-transform-europarl-preview}

```r
# Read in the dataset
europarl_tbl <-
  read_csv(file = "../data/derived/europarl_curated.csv")

# Preview the first 10 lines
europarl_tbl |>
  slice_head(n = 10)
```
\index{R packages!readr}\index{R packages!dplyr}
\cindex{read_csv()}\cindex{slice_head()}
```{r}
#| label: transform-europarl-read
#| echo: false

# Read in the dataset
europarl_tbl <-
  read_csv(file = "data/europarl_curated.csv")

# Preview the first 10 lines
europarl_tbl |>
  slice_head(n = 10)
```
:::

This dataset includes `r format(nrow(europarl_tbl), big.mark = ",")` observations\index{observations} and four variables.\index{variables} The key variable for our purposes is the `lines` variable. This variable contains the text we will be working with. The other variables are metadata\index{metadata} that may be of interest for our analyses.

### Normalization {#sec-transform-normalization}

The process of normalizing datasets in essence is to sanitize the values of a variable or set of variables such that there are no artifacts that will contaminate subsequent processing. It may be the case that non-linguistic metadata may require normalization, but more often than not, linguistic information is the most common target for normalization as text often includes artifacts from the acquisition process which will not be desired in the analysis.\index{text normalization}\index{transform datasets}

Simply looking at the first 10 lines of the dataset from @exm-transform-europarl-preview gives us a clearer sense of the dataset structure, but, in terms of normalization procedures we might apply, it is likely not sufficient. We want to get a sense of any potential inconsistencies in the dataset, in particular in the `lines` variable. Since this is a large dataset with `r format(nrow(europarl_tbl), big.mark = ",")` observations, we will need to explore the dataset in more detail using procedures for summarizing and filtering data.

After exploring variations in the `lines` variable, I identified a number of artifacts in this dataset that we will want to consider addressing. These are included in @tbl-transform-europarl-normalization.

::: {#tbl-transform-europarl-normalization tbl-colwidths="[35, 65]"}

| Description | Examples |
|:---|:---|
| Non-speech annotations | `(Abucheos)`, `(A4-0247/98)`, `(The sitting was opened at 09:00)` |
| Inconsistent whitespace (<code>␣</code>) | `5`<code>␣</code>`%`<code>␣</code>, <code>␣</code>, `Palacio'`<code>␣</code>`s` |
| Non-sentence punctuation | `-`, `:`, `...`, `;` |
| (Non-)Abbreviations | `Mr.`, `Sr.`, `Mme.`, `Mr`, `Sr`, `Mme`, `Mister`, `Señor`, `Madam` |
| Text case | `The`, `the`, `White`, `white` |

Characteristics of the Europarl Parallel Corpus dataset that may require normalization
:::

These artifacts either may not be of interest or may introduce unwanted variability that could prove problematic for subsequent processing (*e.g* tokenization, calculating frequencies, *etc.*).

The majority of text normalization procedures incorporate {stringr} [@R-stringr]. This package provides a number of functions for manipulating text strings. The workhorse functions we will use for our tasks are the `str_remove()` and `str_replace()` functions. These functions give us the ability to remove or replace text based on literal strings, actual text, or Regular Expressions.\index{regular expression (regex)} **Regular Expressions** (regex, *pl.* regexes) are a powerful tool for pattern matching and text manipulation which employs a syntax that allows for the specification of complex search patterns.\index{R packages!stringr}

Our first step, however, is to identify the patterns we want to remove or replace. For demonstration purposes, let's focus on removing non-speech annotations from the `lines` variable. To develop a search pattern to identify these annotations, there are various possibilities, `str_view()`, `str_detect()` inside a `filter()` call, or `str_extract()` inside a `mutate()` call. No matter which approach we choose, we need to be sure that our search pattern does not over- or under-generalize the text we want to remove or replace. If we are too general, we may end up removing or replacing text that we want to keep. If we are too specific, we may not remove or replace all the text we want to remove or replace.\index{R packages!dplyr}

In @exm-transform-europarl-search-non-speech, I've used `str_detect()` which detects a pattern in a character vector and returns a logical vector, `TRUE` if the pattern is detected and `FALSE` if it is not. In combination with `filter()` we can identify a variable with rows that match a pattern. I've added the `slice_sample()` function at the end to return a small, random sample of the dataset to get a better sense how well our pattern works across the dataset.

Now, how about our search pattern? From the examples in @tbl-transform-europarl-normalization above, we can see that the instances of non-speech are wrapped with parentheses `(` and `)`. The text within the parentheses can vary, so we need a regex to do the heavy lifting. To start out we can match any one or multiple characters with `.+`. But it is important to recognize the `+` (and also the `*`) operators are '**greedy**'\index{regular expression (regex)!greedy}, meaning that if there are multiple matches (in this case multiple sets of parentheses) in a single line of text, the longest match will be returned. That is, the match will extend as far as possible. This is not what we want in this case. We want to match the shortest match. To do this we can append the `?` operator to make the `+` operator '**lazy**'. This will match the shortest match.\index{regular expression (regex)!lazy}

```{r}
#| label: set-seed-sample-3
#| echo: false

set.seed(111) # for reproducibility
```

::: {#exm-transform-europarl-search-non-speech}
```r
# Load package
library(stringr)

# Identify non-speech lines
europarl_tbl |>
  filter(str_detect(lines, "\\(.+?\\)")) |>
  slice_sample(n = 10)
```
:::

::: {.content-visible when-format="pdf"}
\vspace{4em}
:::

```{r}
#| label: transform-europarl-search-non-speech
#| echo: false

# Load package
library(stringr)

# Identify non-speech lines
europarl_tbl |>
  filter(str_detect(lines, "\\(.+?\\)")) |>
  slice_sample(n = 10)
```
\index{regular expression (regex)!lazy}
\cindex{library()}\cindex{str_detect()}\cindex{filter()}\cindex{slice_sample()}

The results from @exm-transform-europarl-search-non-speech show that we have identified the lines that contain at least one of the parliamentary session description annotations. A more targeted search to identify specific instances of the parliamentary session descriptions can be accomplished adding the `str_extract_all()` function as seen in @exm-transform-europarl-search-non-speech-2.

```{r}
#| label: set-seed-sample-4
#| echo: false

set.seed(111) # for reproducibility
```
::: {#exm-transform-europarl-search-non-speech-2}

```{r}
#| label: transform-europarl-search-non-speech-2

# Extract non-speech fragments
europarl_tbl |>
  filter(str_detect(lines, "\\(.+?\\)")) |>
  mutate(non_speech = str_extract_all(lines, "\\(.+?\\)")) |>
  slice_sample(n = 10)
```

\cindex{mutate()}\cindex{str_extract_all()}
:::

OK, that might not be what you expected. The `str_extract_all()` function returns a list of character vectors. This is because for any given line in `lines` there may be a different number of matches. To maintain the data frame as rectangular, a list is returned for each value of `non_speech`. We could expand the list into a data frame with the `unnest()` function if our goal were to work with these matches. But that is not our aim. Rather, we want to know if we have multiple matches per line. Note that the information provided for the `non_speech` column by the data frame object tells use that we have some lines with multiple matches, as we can see in line 6 of our small sample. So good thing we checked!

Let's now remove these non-speech annotations from each line in the `lines` column. We turn to `str_remove_all()`, a variant of `str_remove()`, that, as you expect, will remove multiple matches in a single line. We will use the `mutate()` function to overwrite the `lines` column with the modified text. The code is seen in @exm-transform-europarl-remove-non-speech.

::: {#exm-transform-europarl-remove-non-speech}

```{r}
#| label: transform-europarl-remove-non-speech

# Remove non-speech fragments
europarl_tbl <-
  europarl_tbl |>
  mutate(lines = str_remove_all(lines, "\\(.+?\\)"))
```

\cindex{mutate()}\cindex{str_remove_all()}
:::

I recommend spot checking the results of this normalization step by running the code in @exm-transform-europarl-search-non-speech again, if nothing appears we've done our job.

When you are content with the results, drop the observations that have no text in the `lines` column. These were rows where the entire line was non-speech annotation. This can be done with the `is.na()` function and the `filter()` function as seen in @exm-transform-europarl-drop-empty-lines.

::: {#exm-transform-europarl-drop-empty-lines}

```{r}
#| label: transform-europarl-drop-empty-lines

# Drop empty lines
europarl_tbl <-
  europarl_tbl |>
  filter(!is.na(lines))
```
\cindex{filter()}\cindex{is.na()}
:::

Normalization goals will vary from dataset to dataset but the procedures often follow a similar line of attack to those outlined in this section.

### Tokenization {#sec-transform-tokenization}

Tokenization is the process of segmenting units of language into components relevant for the research question. This includes breaking text in curated datasets into smaller units, such as words, ngrams, sentences, *etc.* or combining smaller units into larger units.\index{text tokenization}

The process of tokenization is fundamentally row-wise. Changing the unit of observation\index{unit of observation} changes the number of rows. It is important both for the research and the text processing to operationalize\index{operationalize} our language units beforehand For example, while it may appear obvious to you what 'word' or 'sentence' means, a computer, and your reproducible research, needs a working definition. This can prove trickier than it seems. For example, in English, we can segment text into words by splitting on whitespace. This works fairly well, but there are some cases where this is not ideal. For example, in the case of contractions, such as `don't`, `won't`, `can't`, *etc.* the apostrophe is not a whitespace character. If we want to consider these contractions as separate words, then perhaps we need to entertain a different tokenization strategy.

::: {.callout}
**{{< fa regular lightbulb >}} Consider this**

Consider the following paragraph:

> "As the sun dipped below the horizon, the sky was set ablaze with shades of orange-red, illuminating the landscape. It's a sight Mr. Johnson, a long-time observer, never tired of. On the lakeside, he'd watch with friends, enjoying the ever-changing hues—especially those around 6:30 p.m.—and reflecting on nature's grand display. Even in the half-light, the water's glimmer, coupled with the echo of distant laughter, created a timeless scene. The so-called 'magic hour' was indeed magical, yet fleeting, like a well-crafted poem; it was the essence of life itself."

What text conventions would pose issues for word tokenization based on a whitespace criterion?
:::

Furthermore, tokenization strategies can vary between languages. In German, for example, words are often compounded together, meaning many 'words' will not be captured by the whitespace convention. Whitespace may not even be relevant for word tokenization in logographic writing systems, such as Chinese characters. The take-home message is there is no one-size-fits-all tokenization strategy.

::: {.callout}
**{{< fa medal >}} Dive deeper**

For processing Chinese text, including tokenization, see {jiebaR} [@R-jiebaR] and {gibasa} [@R-gibasa].
:::

Let's continue to work with the Europarl Parallel Corpus\index{Europarl Parallel Corpus} dataset to demonstrate tokenization. We will start by tokenizing the text into words. If we envision what this should look like, we might imagine something like @tbl-transform-europarl-tokenization-words-example.

::: {#tbl-transform-europarl-tokenization-words-example tbl-colwidths="[25, 25, 25, 25]"}

|  doc_id   | type | line_id | token |
|----------|--------|----------|------|
| 1 | Target | 2 | I |
| 1 | Target | 2 | declare |
| 1 | Target | 2 | resumed |
| 1 | Target | 2 | the |
| 1 | Target | 2 | session |

Example of tokenizing the `lines` variable into word tokens
:::

Comparing @tbl-transform-europarl-tokenization-words-example to the fourth row of the output of @exm-transform-europarl-preview, we can see that we want to segment the words in `lines` and then have each segment appear as a separate observation, retaining the relevant metadata variables.\index{observations}\index{metadata}

Tokenization that maintains the tidy dataset structure is a very common task in text analysis using R.\index{tidy format} So common, in fact, that {tidytext} [@R-tidytext] includes a function, `unnest_tokens()` that tokenizes text in just such a way. Various tokenization types can be specified including 'characters', 'words', 'ngrams', 'sentences' among others. We will use the 'word' tokenization type to recreate the structure we envisioned in @tbl-transform-europarl-tokenization-words-example.\index{R packages!tidytext}

In @exm-transform-europarl-tokenization-words-tidytext, we set our output variable to `token` and our input variable to `lines`.

::: {#exm-transform-europarl-tokenization-words-tidytext}

```{r}
#| label: transform-europarl-subset
#| echo: false

# Subset the dataset
europarl_tbl <-
  slice_head(europarl_tbl, n = 10)
```

```{r}
#| label: transform-europarl-tokenization-words-tidytext

# Load package
library(tidytext)

# Tokenize the lines into words
europarl_unigrams_tbl <-
  europarl_tbl |>
  unnest_tokens(
    output = token,
    input = lines,
    token = "words"
  )
```

\cindex{unnest_tokens()}\cindex{library()}
:::

Let's preview the very same lines we modeled in @tbl-transform-europarl-tokenization-words-example to see the results of our tokenization.

::: {#exm-transform-europarl-tokenization-words-preview}

```{r}
#| label: transform-europarl-tokenization-words-preview

# Preview
europarl_unigrams_tbl |>
  filter(type == "Target", line_id == 2) |>
  slice_head(n = 10)
```

\cindex{==}
\cindex{filter()}\cindex{slice_head()}
:::

In @exm-transform-europarl-tokenization-words-preview, the `token` column now contains our word tokens. One thing to note, however, is that text is lowercased and punctuation is stripped by default. If we want to retain the original case or punctuation, keep the original variable, or change the tokenization strategy, we can update the `to_lower`, `strip_punct`, `drop`, or `token` parameters, respectively.

As we derive datasets to explore, let's also create bigram tokens. We can do this by changing the `token` parameter to `"ngrams"` and specifying the value for $n$ with the `n` parameter. I will assign the result to `europarl_bigrams_tbl` as we will have two-word tokens, as seen in @exm-transform-europarl-tokenization-bigrams-tidytext.\index{ngrams}

::: {#exm-transform-europarl-tokenization-bigrams-tidytext}

```{r}
#| label: transform-europarl-tokenization-bigrams-tidytext

# Tokenize the lines into bigrams
europarl_bigrams_tbl <-
  europarl_tbl |>
  unnest_tokens(
    output = token,
    input = lines,
    token = "ngrams",
    n = 2
  )
# Preview
europarl_bigrams_tbl |>
  filter(type == "Target", line_id == 2) |>
  slice_head(n = 10)
```
\cindex{==}
\cindex{unnest_tokens()}
:::

\pagebreak

::: {.callout}
**{{< fa medal >}} Dive deeper**

{tidytext} is one of a number of packages that provide tokenization functions. Some other notable packages include {tokenizers} [@R-tokenizers]\index{R packages!tokenizers} and {textrecipes} [@R-textrecipes]\index{R packages!textrecipes}. In fact, the functions from {tokenizers} are used under the hood in {tidytext}. {textrecipes} is part of the Tidymodels framework\index{Tidyverse} and is designed to work with a suite of packages for computational modeling. It is particularly useful for integrating tokenization with other preprocessing steps and machine learning models\index{machine learning}, as we will see in @sec-predict-chapter.
:::

The most common tokenization strategy is to segment text into smaller units, often words. However, there are times when we may want text segments to be larger than the existing token unit, effectively collapsing over rows. Let's say that we are working with a dataset like the one we created in `europarl_unigrams_tbl` and we want to group the words into sentences. We can again turn to the `unnest_tokens()` function to accomplish this. In @exm-transform-europarl-tokenization-sentences, we use the `token = "sentences"` and `collapse = c("type", "line_id")` parameters to group the words into sentences by the `type` and `line_id` variables.

::: {#exm-transform-europarl-tokenization-sentences}

```{r}
#| label: transform-europarl-tokenization-sentences

# Tokenize the lines into sentences
europarl_sentences_tbl <-
  europarl_unigrams_tbl |>
  unnest_tokens(
    output = token,
    input = token,
    token = "sentences",
    collapse = c("type", "line_id")
  )

# Preview
europarl_sentences_tbl |>
  slice_head(n = 5)
```

\cindex{unnest_tokens()}\cindex{slice_head()}
:::

In this example, we have collapsed the word tokens into sentences. But note, the `token` column contained no punctuation so all the tokens grouped by `type` and `line_id` were concatenated together. This works for our test dataset as lines are sentences. However, in other scenarios, we would need punctuation to ensure that the sentences are properly segmented ---if, in fact, punctuation is the cue for sentence boundaries.

## Enrichment {#sec-transform-enrichment}

Where preparation steps are focused on sanitizing and segmenting the text, enrichment steps are aimed towards augmenting the dataset either through recoding, generating, or integrating variables.\index{dataset enrichment} These processes can prove invaluable for aligning the dataset with the research question and facilitating the analysis.

As a practical example of these types of transformations, we'll posit that we are conducting translation research. Specifically, we will set up an investigation into the effect of translation on the syntactic simplification of text. The basic notion is that when translators translate text from one language to another, they subconsciously simplify the text, relative to native texts [@Liu2021].\index{translation studies}\index{research question}

To address this research question, we will use the ENNTT corpus\index{Europarl corpus of native, non-native and translated texts (ENNTT)}, introduced in @sec-curate-semi-structured-orientation. This data contains European Parliament proceedings and the type of text (native, non-native, or translation) from which the text was extracted. There is one curated dataset for each of the text types.

The data dictionary for the curated native dataset appears in @tbl-transform-enntt-native-dd.

```{r}
#| label: tbl-transform-enntt-native-dd
#| tbl-cap: "Data dictionary for the curated native ENNTT dataset."
#| tbl-colwidths: [13, 22, 13, 52]
#| echo: false

read_csv(file = "data/enntt_natives_curated_dd.csv") |>
  tt(width = 1)
```

All three curated datasets have the same variables. The unit of observation\index{unit of observation} for each dataset is the `text` variable.

Before we get started, let's consider what the transformed dataset might look like and what its variables mean. First, we will need to operationalize\index{operationalize} what we mean by syntactic simplification There are many measures of syntactic complexity [@Szmrecsanyi2004]. For our purposes, we will focus on two measures of syntactic complexity: number of T-units and sentence length (in words). A T-unit is a main clause and all of its subordinate clauses. To calculate the number of T-units, we will need to identify the main clauses and their subordinate clauses. The sentence length is straightforward to calculate after word tokenization.

An idealized transformed dataset dictionary for this investigation should look something like @tbl-transform-generation-idealized.

::: {#tbl-transform-generation-idealized tbl-colwidths="[15, 20, 10, 55]"}

| variable | name | type | description |
|----------|------|------|-------------|
| doc_id | Document ID | integer | Unique identifier for each document. |
| type | Type | character | Type of text (native or translated). |
| t_units | T-units | integer | Number of T-units in the text. |
| word_len | Word Length | integer | Number of words in the text. |

Idealized transformed dataset for the syntactic simplification investigation
:::

We will be using the native and translated datasets for our purposes, so let's go ahead and read in these datasets.

```{r}
#| label: transform-enntt-dataset-read

# Read in curated natives
enntt_natives_tbl <-
  read_csv("data/enntt_natives_curated.csv")

# Read in curated translations
enntt_translations_tbl <-
  read_csv("data/enntt_translations_curated.csv")
```

\index{R packages!readr}
\cindex{read_csv()}

### Generation {#sec-transform-generation}

The process of generation involves the addition of information to a dataset.\index{variable generation} This differs from other transformation procedures in that instead of manipulating, classifying, and/ or deriving information based on characteristics explicit in a dataset, generation involves deriving new information based on characteristics implicit in a dataset.

The most common type of operation involved in the generation process is the addition of **linguistic annotation**.\index{linguistic annotation} This process can be accomplished manually by a researcher or research team or automatically through the use of pre-trained linguistic resources and/ or software. Ideally, the annotation of linguistic information can be conducted automatically.

To identify the main clauses and their subordinate clauses in our datasets, we will need to derive syntactic annotation information from the ENNTT `text` variable.

As fun as it would be to hand-annotate the ENNTT corpus, we will instead turn to automatic linguistic annotation. {udpipe} [@R-udpipe] provides an interface for annotating text using pre-trained models from the [Universal Dependencies](https://universaldependencies.org/) (UD) project [@Nivre2020]. The UD project is an effort to develop cross-linguistically consistent syntactic annotation for a variety of languages.\index{R packages!udpipe}\index{Natural Language Processing (NLP)}

Our first step, then, is to peruse the available pre-trained models for the languages we are interested in and selected the most register-aligned models.\index{language models} The models, model names, and licensing information are documented in the {udpipe} documentation for the `udpipe_download_model()` function.\index{terms of use} For illustrative purposes, we will use the `english` treebank model from the *<https://github.com/bnosac/udpipe.models.ud>* repository which is released under a [CC-BY-SA license](https://creativecommons.org/licenses/by-sa/4.0/). This model is trained on various sources including news, Wikipedia, and web data of various genres.

Let's set the stage by providing an overview of the annotation process.

1. Load {udpipe}.
2. Select the pre-trained model to use and the directory where the model will be stored in your local environment.
3. Prepare the dataset to be annotated (if necessary). This includes ensuring that the dataset has a column of text to be annotated and a grouping column. By default, the names of these columns are expected to be `text` and `doc_id`, respectively. The `text` column needs to be a character vector and the `doc_id` column needs to be a unique index for each text to be annotated.
4. Annotate the dataset. The result returns a data frame.

Since we are working with two datasets, steps 3 and 4 will be repeated. For brevity, I will only show the code for the dataset for the natives. Additionally, I will subset the dataset to 10,000 randomly selected lines for both datasets. Syntactic annotation is a computationally expensive operation and the natives and translations datasets contain 116,341 and 738,597 observations, respectively. This subset will suffice for illustrative purposes.

::: {.callout}
**{{< fa medal >}} Dive deeper**

In your own research computationally expensive operations cannot be avoided, but they can be managed.\index{computing environment} One strategy is to work with a subset of the data until your code is working as expected. Once you are confident that your code is working as expected, then you can scale up to the full dataset. If you are using Quarto, you can use the `cache: true` metadata field in your code blocks to cache the results of computationally expensive code blocks. This will allow you to run your code once and then use the cached results for subsequent runs. Parallel processing is another strategy for managing computationally expensive code. Some packages, such as {udpipe}, have built-in support for parallel processing. Other packages, such as {tidytext}, do not. In these cases, you can use {future} [@R-future] to parallelize your code.
:::

```{r}
#| label: set-seed-sample-7
#| include: false

set.seed(1234) # for reproducibility
```

```{r}
#| label: transform-generation-subset-natives
#| include: false

# Subset the natives ENNTT dataset
enntt_natives_tbl <-
  enntt_natives_tbl |>
  slice_sample(n = 10000)
```

```{r}
#| label: set-seed-sample-8
#| include: false

set.seed(1234) # for reproducibility
```

```{r}
#| label: transform-generation-subset-translations
#| include: false

# Subset the translations ENNTT dataset
enntt_translations_tbl <-
  enntt_translations_tbl |>
  slice_sample(n = 10000)
```

With the subsetted `enntt_natives_tbl` object, let's execute steps 1 through 4, as seen in @exm-transform-generation-udpipe-natives.

::: {#exm-transform-generation-udpipe-natives}

```r
# Load package
library(udpipe)

# Model and directory
model <- "english"
model_dir <- "../data/"

# Prepare the dataset to be annotated
enntt_natives_prepped_tbl <-
  enntt_natives_tbl |>
  mutate(doc_id = row_number()) |>
  select(doc_id, text)

# Annotate the dataset
enntt_natives_ann_tbl <-
  udpipe(
    x = enntt_natives_prepped_tbl,
    object = model,
    model_dir = model_dir
  ) |>
  tibble()

# Preview
glimpse(enntt_natives_ann_tbl)
```

\cindex{library()}\cindex{udpipe()}\cindex{mutate()}\cindex{select()}\cindex{glimpse()}\cindex{tibble()}\cindex{row_number()}

```{r}
#| label: transform-generation-udpipe-natives
#| echo: false
#| cache: true

# Load package
library(udpipe)

# Model and directory
model <- "english"
model_dir <- "data/"

# Prepare the dataset to be annotated
enntt_natives_prepped_tbl <-
  enntt_natives_tbl |>
  mutate(doc_id = row_number()) |>
  select(doc_id, text)

# Annotate the dataset
enntt_natives_ann_tbl <-
  udpipe(
    x = enntt_natives_prepped_tbl,
    object = model,
    model_dir = model_dir,
    parallel.cores = 4
  ) |>
  tibble()

# Preview
glimpse(enntt_natives_ann_tbl)
```

:::

```{r}
#| label: transform-generation-udpipe-translations
#| include: false
#| cache: true

# Load package
library(udpipe)

# Model and directory
model <- "english"
model_dir <- "data/"

# Prepare the dataset to be annotated
enntt_translations_prepped_tbl <-
  enntt_translations_tbl |>
  mutate(doc_id = row_number()) |>
  select(doc_id, text)

# Annotate the dataset
enntt_translations_ann_tbl <-
  udpipe(
    x = enntt_translations_prepped_tbl,
    object = model,
    model_dir = model_dir,
    parallel.cores = 4
  ) |>
  tibble()

# Preview
glimpse(enntt_translations_ann_tbl)
```

There is quite a bit of information which is returned from `udpipe()`. Note that the input lines have been tokenized by word.\index{text tokenization} Each token includes the `token`, `lemma`, part of speech\index{part-of-speech tagging} (`upos` and `xpos`)[^upos], morphological features\index{morphological features} (`feats`), and syntactic relationships\index{syntactic constituents} (`head_token_id` and `dep_rel`). The `token_id` keeps track of the token's position in the sentence and the `sentence_id` keeps track of the sentence's position in the original text. Finally, the `doc_id` column and its values correspond to the `doc_id` in the `enntt_natives_tbl` dataset.

[^upos]: The Universal POS tags are defined by the Universal Dependencies project. The `upos` tag is a coarse-grained POS tag and the `xpos` (Penn) tag is a fine-grained POS tag. For more information, see the UD Project <https://universaldependencies.org/u/pos/>.

The number of variables in the `udpipe()` annotation output is quite overwhelming. However, these attributes come in handy for manipulating, extracting, and plotting information based on lexical and syntactic patterns. See the dependency tree in @fig-transform-generation-udpipe-english-plot-tree for an example of the syntactic information that can be extracted from the `udpipe()` annotation output.

```{r}
#| label: fig-transform-generation-udpipe-english-plot-tree
#| fig-cap: "Plot of the syntactic tree for a sentence in the ENNTT natives dataset"
#| fig-alt: "A syntactic dependency parse tree which allows to inspect the annotation output from the {udpipe} annotation."
#| fig-width: 6
#| fig-asp: 0.75
#| echo: false

# Load package
library(rsyntax)

# Plot the syntactic tree
enntt_natives_ann_tbl |>
  filter(doc_id == 165) |>
  plot_tree(
    token,
    upos,
    use_color = FALSE,
    viewer_mode = FALSE,
    textsize = 1
  )
```

::: {.callout .halfsize}
**{{< fa medal >}} Dive deeper**

The plot in @fig-transform-generation-udpipe-english-plot-tree was created using {rsyntax} [@R-rsyntax]. In addition to creating dependency tree plots, {rsyntax} can be used to extract syntactic patterns from the `udpipe()` annotation output. 
:::

In @fig-transform-generation-udpipe-english-plot-tree we see the syntactic tree for a sentence in the ENNTT natives dataset. Each node is labeled with the `token_id` which provides the linear ordering of the sentence. Above the nodes the `dep_relation`, or dependency relationship label is provided. These labels are based on the UD project's dependency relations. We can see that the 'ROOT' relation is at the top of the tree and corresponds to the verb 'brought'. 'ROOT' relations mark predicates in the sentence. Not seen in the example tree, 'cop' relation is a copular, or non-verbal predicate and should be included. These are the key syntactic pattern we will use to identify main clauses for T-units.

### Recoding {#sec-transform-recoding}

Recoding processes can be characterized by the creation of structural changes which are derived from values in variables effectively recasting values as new variables to enable more direct access in our analyses.\index{variable recoding}

Specifically, we will need to identify\index{feature extraction} and count the main clauses and their subordinate clauses to create a variable `t_units` from our natives and translations annotations objects.\index{feature calculation} In the UD project's listings, the relations 'ccomp' (clausal complement), 'xcomp' (open clausal complement), and 'acl:relcl' (relative clause) are subordinate clauses. Furthermore, we will also need to count the number of words in each sentence to create a variable `word_len`.

To calculate T-units and words per sentence we turn to {dplyr}. We will use the `group_by()` function to group the dataset by `doc_id` and `sentence_id` and then use the `summarize()` function to calculate the number of T-units and words per sentence, where a T-unit is the combination of the sum of main clauses and sum of subordinate clauses. The code is seen in @exm-transform-generation-udpipe-natives-tunits-words.

::: {#exm-transform-generation-udpipe-natives-tunits-words}

```{r}
#| label: transform-generation-udpipe-natives-tunits-words

# Calculate the number of T-units and words per sentence
enntt_natives_syn_comp_tbl <-
  enntt_natives_ann_tbl |>
  group_by(doc_id, sentence_id) |>
  summarize(
    main_clauses = sum(dep_rel %in% c("ROOT", "cop")),
    subord_clauses = sum(dep_rel %in% c("ccomp", "xcomp", "acl:relcl")),
    t_units = main_clauses + subord_clauses,
    word_len = n()
  ) |>
  ungroup()

# Preview
glimpse(enntt_natives_syn_comp_tbl)
```

\index{R packages!dplyr}
\cindex{group_by()}\cindex{summarize()}\cindex{ungroup()}\cindex{n()}\cindex{sum()}
:::

```{r}
#| label: transform-generation-udpipe-translations-tunits-words
#| include: false

# Calculate the number of T-units and words per sentence
enntt_translations_syn_comp_tbl <-
  enntt_translations_ann_tbl |>
  group_by(doc_id, sentence_id) |>
  summarize(
    main_clauses = sum(dep_rel %in% c("ROOT", "cop")),
    subord_clauses = sum(dep_rel %in% c("ccomp", "xcomp", "acl:relcl")),
    t_units = main_clauses + subord_clauses,
    word_len = n()
  ) |>
  ungroup()
```

A quick spot check of some sentences calculations `enntt_natives_syn_comp_tbl` dataset against the `enntt_natives_ann_tbl` is good to ensure that the calculation is working as expected. In @fig-transform-generation-udpipe-natives-tunits we see a sentence that has a word length of 13 and a T-unit value of 5.

```{r}
#| label: fig-transform-generation-udpipe-natives-tunits
#| fig-cap: "Sentence with a word length of 13 and a T-unit value of 5"
#| fig-alt: "A syntactic dependency parse tree which allows to inspect and verify that there are 13 words and 5 T-units in the sentence."
#| fig-width: 6
#| fig-asp: 0.75
#| echo: false

# Preview
enntt_natives_ann_tbl |>
  filter(doc_id == 1468) |>
  plot_tree(
    token,
    use_color = FALSE,
    viewer_mode = FALSE,
    textsize = 0.8
  )
```

Now we can drop the intermediate columns we created to calculate our key syntactic complexity measures using `select()` to indicate those that we do want to keep, as seen in @exm-transform-generation-udpipe-natives-tunits-words-select.

\pagebreak

::: {#exm-transform-generation-udpipe-natives-tunits-words-select}

```{r}
#| label: transform-generation-udpipe-natives-tunits-words-select

# Select columns
enntt_natives_syn_comp_tbl <-
  enntt_natives_syn_comp_tbl |>
  select(doc_id, sentence_id, t_units, word_len)
```

\cindex{select()}
:::

```{r}
#| label: transform-generation-udpipe-translations-tunits-words-select
#| include: false

# Select columns
enntt_translations_syn_comp_tbl <-
  enntt_translations_syn_comp_tbl |>
  select(doc_id, sentence_id, t_units, word_len)
```

Now we can repeat the process for the ENNTT translated dataset. I will assign the result to `enntt_translations_syn_comp_tbl`. The next step is to join the `sentences` from the annotated data frames into our datasets so that we have the information we set out to generate for both datasets. Then, we will combine the native and translations datasets into a single dataset. These steps are part of the transformation process and will be covered in the next section.

### Integration {#sec-transform-integration}

One final class of transformations that can be applied to curated datasets to enhance their informativeness for a research project is the process of integrating two or more datasets.\index{variable integration} There are two integration types: joins and concatenation. **Joins**\index{joining datasets} can be row- or column-wise operations that combine datasets based on a common attribute or set of attributes. **Concatenation**\index{concatenating datasets} is exclusively a row-wise operation that combines datasets that share the same attributes.

Of the two types, joins are the most powerful and sometimes more difficult to understand. When two datasets are joined at least one common variable must be shared between the two datasets. The common variable(s) are referred to as **keys**.\index{join keys} The keys are used to match observations in one dataset with observations in another dataset by serving as an index.

There are a number of join types. The most common are left, full, semi, and anti. The type of join determines which observations are retained in the resulting dataset. Let's see this in practice. First, let's create two datasets to join with a common variable `key`, as seen in @exm-transform-merging-join-dfs.

::: {#exm-transform-merging-join-dfs}

::: {layout="[50, 50]" layout-valign="top"}

::: {#first}

```{r}
#| label: transform-merging-join-a-df

a_tbl <-
  tibble(
    key = c(1, 2, 3, 5, 8),
    a = letters[1:5]
  )

a_tbl
```

:::

::: {#second}

```{r}
#| label: transform-merging-join-b-df

b_tbl <-
  tibble(
    key = c(1, 2, 4, 6, 8),
    b = letters[6:10]
  )

b_tbl
```

:::

:::
:::

The `a_tbl` and the `b_tbl` datasets share the `key` variable, but the values in the `key` variable are not identical. The two datasets share values `1`, `2`, and `8`. The `a_tbl` dataset has values `3` and `5` in the `key` variable and the `b_tbl` dataset has values `4` and `6` in the `key` variable.

If we apply a **left join**\index{joining datasets!left join} to the `a_tbl` and `b_tbl` datasets, the result will be a dataset that retains all of the observations in the `a_tbl` dataset and only those observations in the `b_tbl` dataset that have a match in the `a_tbl` dataset. The result is seen in @exm-transform-merging-join-left.

::: {#exm-transform-merging-join-left}

```{r}
#| label: transform-merging-join-left

left_join(x = a_tbl, y = b_tbl, by = "key")
```

\index{R packages!dplyr}
\cindex{left_join()}
:::

Now, if the key variable has the same name, R will recognize and assume that this is the variable to join on and we don't need the `by =` argument, but if there are multiple potential key variables, we use `by =` to specify which one to use.

A **full join**\index{joining datasets!full join} retains all observations in both datasets, as seen in @exm-transform-merging-join-full.

::: {#exm-transform-merging-join-full}

```{r}
#| label: transform-merging-join-full

full_join(x = a_tbl, y = b_tbl)
```
\cindex{full_join()}
:::

Left and full joins maintain or increase the number of observations. On the other hand, semi and anti joins aim to decrease the number of observations. A **semi join**\index{joining datasets!semi join} retains only those observations in the left dataset that have a match in the right dataset, as seen in @exm-transform-merging-join-semi.

::: {#exm-transform-merging-join-semi}

```{r}
#| label: transform-merging-join-semi

semi_join(x = a_tbl, y = b_tbl)
```

\cindex{semi_join()}
:::

And an **anti join**\index{joining datasets!anti join} retains only those observations in the left dataset that do not have a match in the right dataset, as seen in @exm-transform-merging-join-anti.

::: {#exm-transform-merging-join-anti}

```{r}
#| label: transform-merging-join-anti

anti_join(x = a_tbl, y = b_tbl)
```

\cindex{anti_join()}
:::

Of these join types, the left join and the anti join are some of the most common to encounter in research projects.

::: {.callout .halfsize}
**{{< fa regular lightbulb >}} Consider this**

In addition to datasets that are part of an acquired resource or derived from a corpus resource, there are also a number of datasets that are included in R packages that are particularly relevant for text analysis. For example, {tidytext} includes `sentiments` and `stop_words` datasets. {lexicon} [@R-lexicon] includes large number of datasets that include sentiment lexicons, stopword lists, contractions, and more.
:::

With this in mind, let's return to our syntactic simplification investigation. Recall that we started with two curated ENNTT datasets: the natives and translations.\index{Europarl corpus of native, non-native and translated texts (ENNTT)} We manipulated these datasets subsetting them to 10,000 randomly selected lines, prepped them for annotation by adding a `doc_id` column and dropping all columns except `text`, and then annotated them using {udpipe}. We then calculated the number of T-units and words per sentence and created the variables `t_units` and `word_len` for each.

These steps produced two datasets for both the natives and for the translations. The first dataset for each is the annotated data frame. The second is the data frame with the syntactic complexity measures we calculated. The `enntt_natives_ann_tbl` and `enntt_translations_ann_tbl` contain the annotations and `enntt_natives_syn_comp_tbl` and `enntt_translations_syn_comp_tbl` the syntactic complexity measures. In the end, we want a dataset that take the form in @tbl-transform-integration-idealized.

::: {#tbl-transform-integration-idealized tbl-colwidths="[10, 15, 10, 13, 52]"}

| doc_id | type        | t_units | word_len | text                                                   |
| ------ | ----------- | ------- | -------- | ------------------------------------------------------ |
| 1      | natives     | 1       | 5        | I am happy right now.                                  |
| 2      | translation | 3       | 11       | I think that John believes that Mary is a good person. |

Idealized integrated dataset for the syntactic simplification 
:::

To create this unified dataset, we will need to apply joins and concatenation. First, we will join the prepped datasets with the annotated datasets. Then, we will concatenate the two resulting datasets.

Let's start first by joining the annotated datasets with the datasets with the syntactic complexity calculations. In these joins, we can see that the prepped and calculated datasets share a couple of variables, `doc_id` and `sentence_id`, in @exm-transform-merging-join-prepped-syn-comp.\index{joining datasets}

::: {#exm-transform-merging-join-prepped-syn-comp}

```{r}
#| label: transform-merging-join-prepped-syn-comp
#| results: hold

# Preview datasets to join
glimpse(enntt_natives_ann_tbl)
glimpse(enntt_natives_syn_comp_tbl)
```
\cindex{slice_head()}
:::

The `doc_id` and `sentence_id` variables are both keys that we will use to join the datasets. The reason being that if we only use one of the two we will not align the two datasets at the sentence level. Only the combination of `doc_id` and `sentence_id` isolates the sentences for which we have syntactic complexity measures.

Beyond having a common variable (or variables in our case), we must also ensure that join key variables are of the same vector type\index{vector types} in both data frames and that we are aware of any differences in the values. From the output in @exm-transform-merging-join-prepped-syn-comp, we can see that the `doc_id` and `sentence_id` variables aligned in terms of vector type; `doc_id` is character and `sentence_id` is integer in both data frames. If they happened not to be, their types would need to be adjusted.

Now, we need to check for differences in the values. We can do this by using the `setequal()` function. This function returns `TRUE` if the two vectors are equal and `FALSE` if they are not. If the two vectors are not equal, the function will return the values that are in one vector but not the other. So if one has `10001` and the other doesn't we will get `FALSE`. Let's see this in practice, as seen in @exm-transform-merging-join-prepped-syn-comp-check.

::: {#exm-transform-merging-join-prepped-syn-comp-check}

```{r}
#| label: transform-merging-join-prepped-syn-comp-check
#| results: hold

# Check for differences in the values
setequal(
  enntt_natives_ann_tbl$doc_id,
  enntt_natives_syn_comp_tbl$doc_id
)

setequal(
  enntt_natives_ann_tbl$sentence_id,
  enntt_natives_syn_comp_tbl$sentence_id
)
```

\cindex{setequal()}
:::

So the values are the same. The final check is to see if the vectors are of the same length. We know the values are the same, but we don't know if the values are repeated. We do this by simply comparing the length of the vectors, as seen in @exm-transform-merging-join-prepped-syn-comp-length.

::: {#exm-transform-merging-join-prepped-syn-comp-length}

```{r}
#| label: transform-merging-join-prepped-syn-comp-length
#| results: hold

# Check for differences in the length
length(enntt_natives_ann_tbl$doc_id) ==
  length(enntt_natives_syn_comp_tbl$doc_id)

length(enntt_natives_ann_tbl$sentence_id) ==
  length(enntt_natives_syn_comp_tbl$sentence_id)
```

\cindex{length()}\cindex{==}
:::

So they are not the same length. Using the `nrow()` function, I can see that the annotated dataset has `r format(nrow(enntt_natives_ann_tbl), big.mark = ",")` observations and the calculated dataset has `r format(nrow(enntt_natives_syn_comp_tbl), big.mark = ",")` observations. The annotation data frames will have many more observations due to the fact that the unit of observations is word tokens. The recoded syntactic complexity data frames' unit of observation is the sentence.\cindex{nrow()}

To appreciate the difference in the number of observations, let's look at the first 10 observations of the natives annotated frame for just the columns of interest, as seen in @exm-transform-merging-join-prepped-syn-comp-ann.

::: {#exm-transform-merging-join-prepped-syn-comp-ann}

```{r}
#| label: transform-merging-join-prepped-syn-comp-ann

# Preview the annotated dataset
enntt_natives_ann_tbl |>
  select(doc_id, sentence_id, sentence, token) |>
  slice_head(n = 10)
```

\cindex{slice_head()}\cindex{select()}
:::

The annotated data frames have a lot of redundancy in for the join variables and the `sentence` variable that we want to add to the calculated data frames. We can reduce the redundancy by using the `distinct()` function from {dplyr}. In this case we want all observations where `doc_id`, `sentence_id` and `sentence` are distinct. We then select these variables with `distinct()`, as seen in @exm-transform-merging-annotation-distinct.

::: {#exm-transform-merging-annotation-distinct}

```{r}
#| label: transform-merging-annotation-distinct

# Reduce annotated data frames to unique sentences
enntt_natives_ann_distinct <-
  enntt_natives_ann_tbl |>
  distinct(doc_id, sentence_id, sentence)

enntt_translations_ann_distinct <-
  enntt_translations_ann_tbl |>
  distinct(doc_id, sentence_id, sentence)
```

\cindex{distinct()}
:::

We now have two datasets that are ready to be joined with the recoded datasets. The next step is to join the two. We will employ a left join\index{joining datasets!left join} where the syntactic complexity data frames are on the left and the join variables will be both the `doc_id` and `sentence_id` variables. The code is seen in [Examples @exm-transform-merging-join-left-syn-comp-natives] and [-@exm-transform-merging-join-left-syn-comp-translations].

::: {#exm-transform-merging-join-left-syn-comp-natives}
```{r}
#| label: transform-merging-join-left-syn-comp-natives

# Join the native datasets
enntt_natives_transformed_tbl <-
  left_join(
    x = enntt_natives_syn_comp_tbl,
    y = enntt_natives_ann_distinct,
    by = c("doc_id", "sentence_id")
  )

# Preview
glimpse(enntt_natives_transformed_tbl)
```
:::

\pagebreak

::: {#exm-transform-merging-join-left-syn-comp-translations}
```{r}
#| label: transform-merging-join-left-syn-comp-translations

# Join the translations datasets
enntt_translations_transformed_tbl <-
  left_join(
    x = enntt_translations_syn_comp_tbl,
    y = enntt_translations_ann_distinct,
    by = c("doc_id", "sentence_id")
  )

# Preview
glimpse(enntt_translations_transformed_tbl)
```

\cindex{left_join()}\cindex{glimpse()}
:::

<!-- Concatenation -->

The two data frames now have the same columns and we are closer to our final dataset. The next step is to move toward concatenating the two datasets. Before we do that, we need to do some preparation. First, and most important, we need to add a `type` column to each dataset. This column will indicate whether the sentence is a native or a translation. The second is that our `doc_id` does not serve as a unique identifier for the sentences. Only in combination with `sentence_id` can we uniquely identify a sentence.

So our plan will be to add a `type` column to each dataset specifying the values for all the observations in the respective dataset. Then we will concatenate the two datasets. Note, if we combine them before, distinguishing the type will be more difficult. After we concatenate the two datasets, we will add a `doc_id` column that will serve as a unique identifier for the sentences and drop the `sentence_id` column. OK, that's the plan. Let's execute it in @exm-transform-merging-concatenation.

::: {#exm-transform-merging-concatenation}

```{r}
#| label: transform-merging-concatenation

# Add a type column
enntt_natives_transformed_tbl <-
  enntt_natives_transformed_tbl |>
  mutate(type = "natives")

enntt_translations_transformed_tbl <-
  enntt_translations_transformed_tbl |>
  mutate(type = "translations")

# Concatenate the datasets
enntt_transformed_tbl <-
  bind_rows(
    enntt_natives_transformed_tbl,
    enntt_translations_transformed_tbl
  )

# Overwrite the doc_id column with a unique identifier
enntt_transformed_tbl <-
  enntt_transformed_tbl |>
  mutate(doc_id = row_number()) |>
  select(doc_id, type, t_units, word_len, text = sentence)

# Preview
glimpse(enntt_transformed_tbl)
```

\cindex{mutate()}\cindex{bind_rows()}\cindex{row_number()}\cindex{select()}
:::

The output of @exm-transform-merging-concatenation now looks structurally like the idealized form in @tbl-transform-integration-idealized. We have a dataset that has the syntactic complexity measures for both the natives and the translations. We can now write this dataset to disk and document it in the data dictionary.\index{data dictionary}

## Activities {.unnumbered}

In the following activities, you will review the concept of transforming data to prepare it for analysis and working to implement these steps with R. This includes preparation and enrichment of curated datasets using normalization, tokenization, recoding, generation, and/ or integration strategies.

::: {.callout}
**{{< fa regular file-code >}} Recipe**

**What**: Transforming and documenting datasets\
**How**: Read Recipe 7, complete comprehension check, and prepare for Lab 7.\
**Why**: To work with two primary types of transformations, tokenization and joins. Tokenization is the process of recasting textual units into units of other sizes. The process of joining datasets aims to incorporate other datasets to augment or filter the dataset of interest.
:::

<!-- [ ] tmp fmt -->
\pagebreak

::: {.callout}
**{{< fa flask >}} Lab**

**What**: Dataset alchemy\
**How**: Fork, clone, and complete the steps in Lab 7.\
**Why**: To gain experience working with coding strategies for transforming datasets using tidyverse functions and regexes, practice reading from and writing to disk, and implement organizational strategies for organizing and documenting a dataset in reproducible fashion.
:::

## Summary {.unnumbered}

In this chapter we covered the process of transforming datasets. The goal is to manipulate the curated dataset to make it align better for analysis.

We covered various types of transformation procedures from text normalization to data frame integrations. In any given research project some or all of these steps will be employed ---but not necessarily in the order presented in this chapter. It is not uncommon to mix procedures as well. The etiology of the transformation is as unique as the data that you are working with.

Since you are applying techniques that have a significant factor on the shape and contents of your dataset(s) it is important to perform data checks to ensure that the transformations are working as expected. You may not catch everything, and some things may not be caught until later in the analysis process, but it is important to do as much as you can as early as you can.

In line with the reproducible research principles, it is important to write the transformed dataset to disk and to document it in the data dictionary. This is especially important if you are working with multiple datasets. Good naming conventions also come into play. Choosing descriptive names is so easily overlooked by your present self but so welcomed by your future self.