/
citesource_benchmark_testing.Rmd
145 lines (110 loc) · 7.75 KB
/
citesource_benchmark_testing.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
---
title: "Benchmark Testing"
author: ""
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Benchmark Testing}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
```{r setup, include = FALSE}
# This code chunk sets global options for all subsequent code chunks in the document using the `knitr` package in R.
knitr::opts_chunk$set(
collapse = TRUE, # Collapses output with no extra whitespace.
comment = "#>", # Uses `#>` to comment out code in output.
warning = FALSE, # Turns off warnings for all code chunks.
fig.width = 6, # Sets default figure width to 6 inches.
fig.height = 6 # Sets default figure height to 6 inches.
)
```
## About this vignette
When estimating he comprehensiveness of a search, researchers often compile a list of relevant studies and evaluate whether or not they are discovered using their search strategy. While benchmarking is an important step in testing the sensitivity of a search, this process can be very time consuming if variations of a string are being tested.
This vignette will provide an example of how CiteSource can be used to speed up the process of benchmarking especially when comparing variations of search strings or search strategies.
## 1. Installation of packages and loading libraries
Use the following code to install CiteSource. Currently, CiteSource lives on GitHub, so you may need to first install the remotes package. This vignette also uses functions from the ggplot2 and dplyr packages.
```{r, results = FALSE, message=FALSE, warning=FALSE}
#Install the remotes packages to enable installation from GitHub
#install.packages("remotes")
#library(remotes)
#Install CiteSource
#remotes::install_github("ESHackathon/CiteSource")
#Load the necessary libraries
library(CiteSource)
library(ggplot2)
library(dplyr)
```
## 2. Import files from multiple sources
Users can import multiple .ris or .bib files into CiteSource, which the user can label with source information such as database or platform. In this case we are uploading the results from six different strings, which were applied to Web of Science.
```{r}
# Import citation files from folder
citation_files <- list.files(path = "benchmark_data", pattern = "\\.ris", full.names = TRUE)
# Print list of citation files to console
citation_files
# Set the path to the directory containing the citation files
file_path <- "../vignettes/benchmark_data/"
# Create a tibble that contains metadata about the citation files
metadata_tbl <- tibble::tribble(
~files, ~cite_sources, ~cite_labels,
"Benchmarking.ris", "Benchmark", "Benchmark",
"Search1_1.ris", "search1", "search",
"Search2_1.ris", "search2", "search",
"Search2_2.ris", "search2", "search",
"Search2_3.ris", "search2", "search",
"Search2_4.ris", "search2", "search",
"Search3_1.ris", "search3", "search",
"Search3_2.ris", "search3", "search",
"Search3_3.ris", "search3", "search",
"Search3_4.ris", "search3", "search",
"Search3_5.ris", "search3", "search",
"Search4_1.ris", "search4", "search",
"Search4_2.ris", "search4", "search",
"Search4_3.ris", "search4", "search",
"Search5_1.ris", "search5", "search",
"Search5_2.ris", "search5", "search",
"Search5_3.ris", "search5", "search",
"Search5_4.ris", "search5", "search",
"Search5_5.ris", "search5", "search",
"Search5_6.ris", "search5", "search",
"Search5_7.ris", "search5", "search",
"Search5_8.ris", "search5", "search",
"Search6_1.ris", "search6", "search",
"Search6_2.ris", "search6", "search",
"Search6_3.ris", "search6", "search",
"Search6_4.ris", "search6", "search",
) %>%
# Append the file path to each file name in the 'files' column
dplyr::mutate(files = paste0(file_path, files))
# Read in citations using metadata table
citations <- read_citations(metadata = metadata_tbl)
```
## 3. Deduplication and source information
CiteSource allows users to merge duplicates while maintaining information in the cite_source metadata field. Thus, information about the origin of the records is not lost in the deduplication process. The next few steps produce the dataframes that we can use in subsequent analyses.
```{r, results = FALSE, message=FALSE, warning=FALSE}
#Deduplicate citations. This yields a dataframe of all records with duplicates merged, but the originating source information maintained in a new variable called cite_source.
unique_citations <- dedup_citations(citations)
#Count number of unique and non-unique citations from different sources and labels
n_unique <- count_unique(unique_citations)
#For each unique citation, determine which sources were present
source_comparison <- compare_sources(unique_citations, comp_type = "sources")
#Initial upload/post internal deduplication table creation
initial_counts<-record_counts(unique_citations, citations, "cite_source")
record_counts_table(initial_counts)
```
## 4. Upset plot to compare discovery of benchmarking articles
An upset plot is useful for visualizing overlap across multiple sources and provides detail about the number of shared and unique records. Using this data we'll outline a few potential uses, when looking at the discovery of benchmarking articles.
We have uploaded 55 benchmarking articles. Of these 55 articles we can see that all but 6 have been found across the six searches. We can see the number of benchmarking articles that were discovered by each string as well as the number of articles that were shared between searches.
Looking at the first column, we see that 9 benchmarking articles were found across every search. One may hypothesize that the 140 citations that follow in the second column may have a high number of relevant articles due to the fact that they were also discovered across the 6 searches. If a researcher was interested in building a larger group of benchmarking articles, they may want to review these articles first.
Looking at the plot we can see that search #5 has the largest number of results, well over 6000. Of these, 5,964 are unique to that search. We can also see that search #5 finds 3 benchmarking articles that would have otherwise not been found. While a researcher may want to ensure that they capture the highest number of benchmarking articles, the addition of ~6k articles may not be efficient when the result is only 3 benchmarking articles. Instead of including this search in their final strategy, they may consider reviewing the three articles that were found by this search and work to adjust their other searches instead.
Another decision in this case may be to drop search #4 and #6 as each of these strings do not contribute uniquely to the discovery of any benchmarking articles. While the data backs up this decision, there may also be more to consider. For example, if benchmarking articles are biased for any known reason, certain search strategies may be employed with an understanding that benchmarking data may not accurately reflect their potential contribution. (e.g. benchmarking articles were gathered from previous systematic reviews that focused on a specific geographic region and the current review is global in nature).
```{r}
#Generate a source comparison upset plot.
plot_source_overlap_upset(source_comparison, decreasing = c(TRUE, TRUE))
```
## 5. Reviewing the record table
This record table is helpful in reviewing which citations were found across each database as well as quickly checking to see which benchmarking articles were not found in the search.
```{r}
unique_citations %>%
dplyr::filter(stringr::str_detect(cite_label, "Benchmark")) %>%
record_level_table(return = "DT")
```