-
Notifications
You must be signed in to change notification settings - Fork 24
/
C-retrieval.Rmd
263 lines (221 loc) · 13.9 KB
/
C-retrieval.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
---
title: "Data Retrieval and Reuse"
date: "2017-06-15"
output:
html_document:
fig_caption: false
toc: true
toc_float:
collapsed: false
smooth_scroll: false
toc_depth: 2
vignette: >
%\VignetteIndexEntry{3. Data Retrieval and Reuse}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
This vignette shows how to download data from Dataverse using the dataverse package. We'll focus on a Dataverse repository that contains supplemental files for [Jamie Monogan](http://spia.uga.edu/faculty-member/jamie-monogan/)'s book [*Political Analysis Using R*](http://www.springer.com/gb/book/9783319234458), which is stored at Harvard University's [IQSS Dataverse Network](https://dataverse.harvard.edu/):
> Monogan, Jamie, 2015, "Political Analysis Using R: Example Code and Data, Plus Data for Practice Problems", [doi:10.7910/DVN/ARKOTI](https://doi.org/10.7910/DVN/ARKOTI), Harvard Dataverse, V1, UNF:6:+itU9hcUJ8I9E0Kqv8HWHg==
This study is persistently retrievable by a "[Digital Object Identifier (DOI)](https://www.doi.org/)": https://doi.org/10.7910/DVN/ARKOTI and the citation above (taken from the Dataverse page) includes a "[Universal Numeric Fingerprint (UNF)](http://guides.dataverse.org/en/latest/developers/unf/index.html)": `UNF:6:+itU9hcUJ8I9E0Kqv8HWHg==`, which provides a versioned, multi-file hash for the entire study, which contains 32 files.
If you don't already know what datasets and files you want to use from Dataverse, see the ["Data Search" vignette](B-search.html) for guidance on data search and discovery.
## Retrieving Dataset and File Metadata
We will download these files and examine them directly in R using the **dataverse** package. To begin, we need to loading the package and using the `get_dataset()` function to retrieve some basic metadata about the dataset:
```r
library("dataverse")
Sys.setenv("DATAVERSE_SERVER" = "dataverse.harvard.edu")
(dataset <- get_dataset("doi:10.7910/DVN/ARKOTI"))
```
```
## Dataset (75170):
## Version: 1.0, RELEASED
## Release Date: 2015-07-07T02:57:02Z
## License: CC0
## 17 Files:
## label version id contentType
## 1 alpl2013.tab 2 2692294 text/tab-separated-values
## 2 BPchap7.tab 2 2692295 text/tab-separated-values
## 3 chapter01.R 2 2692202 text/plain; charset=US-ASCII
## 4 chapter02.R 2 2692206 text/plain; charset=US-ASCII
## 5 chapter03.R 2 2692210 text/plain; charset=US-ASCII
## 6 chapter04.R 2 2692204 text/plain; charset=US-ASCII
## 7 chapter05.R 2 2692205 text/plain; charset=US-ASCII
## 8 chapter06.R 2 2692212 text/plain; charset=US-ASCII
## 9 chapter07.R 2 2692209 text/plain; charset=US-ASCII
## 10 chapter08.R 2 2692208 text/plain; charset=US-ASCII
## 11 chapter09.R 2 2692211 text/plain; charset=US-ASCII
## 12 chapter10.R 1 2692203 text/plain; charset=US-ASCII
## 13 chapter11.R 1 2692207 text/plain; charset=US-ASCII
## 14 comprehensiveJapanEnergy.tab 2 2692296 text/tab-separated-values
## 15 constructionData.tab 2 2692293 text/tab-separated-values
## 16 drugCoverage.csv 1 2692233 text/plain; charset=US-ASCII
## 17 hanmerKalkanANES.tab 2 2692290 text/tab-separated-values
## 18 hmnrghts.tab 2 2692298 text/tab-separated-values
## 19 hmnrghts.txt 1 2692238 text/plain
## 20 levant.tab 2 2692289 text/tab-separated-values
## 21 LL.csv 1 2692228 text/plain; charset=US-ASCII
## 22 moneyDem.tab 2 2692292 text/tab-separated-values
## 23 owsiakJOP2013.tab 2 2692297 text/tab-separated-values
## 24 PESenergy.csv 1 2692230 text/plain; charset=US-ASCII
## 25 pts1994.csv 1 2692229 text/plain; charset=US-ASCII
## 26 pts1995.csv 1 2692231 text/plain; charset=US-ASCII
## 27 sen113kh.ord 1 2692239 text/plain; charset=US-ASCII
## 28 SinghEJPR.tab 2 2692299 text/tab-separated-values
## 29 SinghJTP.tab 2 2692288 text/tab-separated-values
## 30 stdSingh.tab 2 2692291 text/tab-separated-values
## 31 UN.csv 1 2692232 text/plain; charset=US-ASCII
## 32 war1800.tab 2 2692300 text/tab-separated-values
```
The output prints some basic metadata and then the `str()` of the `files` data frame returned by the call. This lists all of the files in the dataset along with a considerable amount of metadata about each. We can see a quick glance at these files using:
```
dataset$files[c("filename", "contentType")]
```
This shows that there are indeed 32 files, a mix of .R code files and tab- and comma-separated data files.
You can also retrieve more extensive metadata using `dataset_metadata()`:
```r
str(dataset_metadata("doi:10.7910/DVN/ARKOTI"), 1)
```
```
## List of 2
## $ displayName: chr "Citation Metadata"
## $ fields :'data.frame': 7 obs. of 4 variables:
```
We'll focus here on the code and data files for Chapter 2 from the book.
## Retrieving Files
Let's start by grabbing the code using `get_file()` (note that this always returns a raw vector):
```r
code3 <- get_file("chapter03.R", "doi:10.7910/DVN/ARKOTI")
writeBin(code3, "chapter03.R")
```
Now we'll get the corresponding data and save it locally. For this code we need two data files:
```r
writeBin(get_file("constructionData.tab", "doi:10.7910/DVN/ARKOTI"),
"constructionData.dta")
writeBin(get_file("PESenergy.csv", "doi:10.7910/DVN/ARKOTI"),
"PESenergy.csv")
```
To confirm that the data look the way we want, we can also (perhaps alternatively) load it directly into R:
```r
constructionData <- foreign::read.dta("constructionData.dta")
str(constructionData)
PESenergy <- utils::read.table("PESenergy.csv")
str(PESenergy)
```
```
## 'data.frame': 50 obs. of 55 variables:
## $ year : int 1997 1997 1997 1997 1997 1997 1997 1997 1997 1997 ...
## $ stno : int 1 2 3 4 5 6 7 8 9 10 ...
## $ totalreg : int 329 500 314 963 2106 643 634 239 1996 880 ...
## $ totalhealth : int 300 424 263 834 1859 554 501 204 1640 732 ...
## $ raneyfolded97 : num 0.58 0.69 0.85 0.63 0.5 ...
## $ healthagenda97 : int 49 180 137 220 1409 153 324 40 408 157 ...
## $ predictedtotalig : num 51.8 99 81.8 111.2 224.1 ...
## $ supplytotalhealth : int 1168 6991 4666 9194 70014 8847 7845 1438 35363 13471 ...
## $ totalhealthsupplysq : int 136 4887 2177 8453 490196 7827 6154 207 125054 18147 ...
## $ partratetotalhealth : num 2.48 1.09 1.09 1.4 0.35 ...
## $ ighealthcare : int 29 76 51 129 247 89 133 35 356 148 ...
## $ supplydirectpatientcare : int 1137 6687 4458 8785 66960 8320 7439 1365 33793 12760 ...
## $ dpcsupplysq : int 129 4472 1987 7718 448364 6922 5534 186 114197 16282 ...
## $ partratedpc : num 1.14 0.51 0.43 0.68 0.17 ...
## $ igdpcare : int 13 34 19 60 112 40 67 12 212 74 ...
## $ supplypharmprod : int 0 174 78 229 2288 340 202 36 962 360 ...
## $ pharmsupplysq : int 0 30276 6084 52441 5234944 115600 40804 1296 925444 129600 ...
## $ partratepharmprod : num 0 10.34 19.23 5.24 2.05 ...
## $ igpharmprod : int 4 18 15 12 47 23 22 12 46 32 ...
## $ supplybusiness : int 0 51 28 93 315 55 36 14 317 78 ...
## $ businesssupplysq : int 0 2601 784 8649 99225 3025 1296 196 100489 6084 ...
## $ partratebusness : num 0 1.96 14.29 15.05 6.03 ...
## $ igbusiness : int 2 1 4 14 19 5 4 2 25 6 ...
## $ supplygovt : int 14 26 80 23 70 71 105 2 67 176 ...
## $ govsupplysq : num 0.02 0.07 0.64 0.05 0.49 ...
## $ partrategov : num 0 38.5 2.5 30.4 10 ...
## $ iggovt : int 0 10 2 7 7 1 8 0 12 2 ...
## $ supplyadvocacy : int 16 37 14 57 344 54 51 18 206 76 ...
## $ advossq : int 256 1369 196 3249 118336 2916 2601 324 42436 5776 ...
## $ partrateadvo : num 31.25 16.22 28.57 31.58 8.72 ...
## $ ig97advoc : int 5 6 4 18 30 7 9 4 26 17 ...
## $ rnmedschools : int 1 16 8 7 37 7 12 3 18 21 ...
## $ rnmedschoolssq : int 1 256 64 49 1369 49 144 9 324 441 ...
## $ rnmedschoolpartrate : num 100 0 12.5 28.57 5.41 ...
## $ rnmedschooligs : int 1 0 1 2 2 0 1 0 6 1 ...
## $ healthprofessionals : int 12890 128980 82140 122760 749620 111550 121110 22740 471270 215670 ...
## $ healthprofessionalssquared: int 16615 1663584 674698 1507002 56193014 1244340 1466763 51711 22209541 4651355 ...
## $ partrateprofessionals : num 0.03 0.01 0.01 0.01 0 ...
## $ ighealthprofessionals : int 4 7 6 16 30 13 22 5 29 16 ...
## $ predictdpcpartrate : num 1.175 0.915 1.016 0.826 0.348 ...
## $ predictdpcig : num 23.1 49.7 39.4 58.8 103.5 ...
## $ predictprofpartrate : num 0.02475 0.01383 0.01788 0.01434 0.00579 ...
## $ predictprofig : num 7.59 12.58 10.69 12.34 22.47 ...
## $ predictmedschoolparttrate : num 17.39 8.08 12.3 12.95 5.02 ...
## $ predictmedschoolig : num 0.355 1.269 0.774 0.713 2.65 ...
## $ predictadvopartrate : num 31.9 26.4 32.5 21.6 13 ...
## $ predictadvoig : num 5.96 7.98 5.76 9.83 28.53 ...
## $ predictbuspartrate : num 25.78 18.08 21.33 13.1 7.27 ...
## $ predictbusig : num 2.58 7.96 5.66 11.66 20.04 ...
## $ predictpharmpartrate : num 21.38 15.22 18.52 13.44 4.14 ...
## $ predictpharmig : num 11.3 18.1 14.4 20.1 45.1 ...
## $ predictgovpartrate : num 14.41 12.61 5.84 13.03 6.93 ...
## $ predictgovig : num 2.06 2.43 3.78 2.35 3.57 ...
## $ predicttotalpartrate : num 2.41 1.823 2.047 1.623 0.752 ...
## $ predicttotalig : num 54.2 99.2 81.9 114.8 228.3 ...
## - attr(*, "datalabel")= chr ""
## - attr(*, "time.stamp")= chr " 1 Jun 2013 16:59"
## - attr(*, "formats")= chr "%8.0g" "%8.0g" "%8.0g" "%8.0g" ...
## - attr(*, "types")= int 252 251 252 252 254 252 254 253 253 254 ...
## - attr(*, "val.labels")= chr "" "" "" "" ...
## - attr(*, "var.labels")= chr "Year" "StNo." "97 TotalReg" "97Total-Health" ...
## - attr(*, "version")= int 12
## 'data.frame': 181 obs. of 1 variable:
## $ V1: Factor w/ 181 levels "Apr-69,5,3.4,60,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,39.2",..: 31 62 47 107 1 122 92 77 16 167 ...
```
In addition to visual inspection, we can compare the UNF signatures for each dataset against what is reported by Dataverse to confirm that we received the correct files:
```r
library("UNF")
unf(constructionData)
unf(PESenergy)
dataset$files[c("label", "UNF")]
```
```
## UNF6:+4pc5114xS0ryr1sSvdX6g==
## UNF6:TD7TEMZyrX4iGTlTsUKQDg==
## label UNF
## 1 alpl2013.tab UNF:6:d9ZNXvmiPfiunSAiXRpVfg==
## 2 BPchap7.tab UNF:6:B3/HJbnzktaX5eEJA2ItiA==
## 3 chapter01.R <NA>
## 4 chapter02.R <NA>
## 5 chapter03.R <NA>
## 6 chapter04.R <NA>
## 7 chapter05.R <NA>
## 8 chapter06.R <NA>
## 9 chapter07.R <NA>
## 10 chapter08.R <NA>
## 11 chapter09.R <NA>
## 12 chapter10.R <NA>
## 13 chapter11.R <NA>
## 14 comprehensiveJapanEnergy.tab UNF:6:Vhb3oZb9m4Nk9N7s6UAHGg==
## 15 constructionData.tab UNF:6:+4pc5114xS0ryr1sSvdX6g==
## 16 drugCoverage.csv <NA>
## 17 hanmerKalkanANES.tab UNF:6:lrQrhDAXFc8lSRP9muJslw==
## 18 hmnrghts.tab UNF:6:uEg24jBA2ht0P4WeNLjI+w==
## 19 hmnrghts.txt <NA>
## 20 levant.tab UNF:6:zlgG7+JXsIZYvS383eQOvA==
## 21 LL.csv <NA>
## 22 moneyDem.tab UNF:6:7M/QM5i6IM/VUM94UJjJUQ==
## 23 owsiakJOP2013.tab UNF:6:0ZEvCFuUQms2zYD57hmwNQ==
## 24 PESenergy.csv <NA>
## 25 pts1994.csv <NA>
## 26 pts1995.csv <NA>
## 27 sen113kh.ord <NA>
## 28 SinghEJPR.tab UNF:6:iDGp9dXOl4SiR+rCBWo8Tw==
## 29 SinghJTP.tab UNF:6:lDCyZ7YQF5O++SRsxh2kGA==
## 30 stdSingh.tab UNF:6:A5gwtn5q/ewkTMpcQEQ73w==
## 31 UN.csv <NA>
## 32 war1800.tab UNF:6:jJ++mepKcv9JbJOOPLMf2Q==
```
## Reusing Files and Reproducing Analysis
To reproduce the analysis, we can simply run the code file either as a `system()` call or directly in R using `source()` (note this particular file begins with an `rm()` call so you may want to run it in a clean R session):
```R
system("Rscript chapter03.R")
source("chapter03.R")
```
Any well-produced set of analysis reproduction files, like this one, will easily run without error once the data and code are in-hand.
To archive your own reproducible analyses using Dataverse, see the ["Archiving Data" vignette](D-archiving.html).