/
README.Rmd
321 lines (262 loc) · 22.3 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r setup, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/",
out.width = "100%")
```
# `ruODK`: An R Client for the ODK Central API <img src="man/figures/ruODK.png" align="right" alt="Are you ODK?" width="120" />
<!-- badges: start -->
[](https://github.com/ropensci/software-review/issues/335)
[](https://www.repostatus.org/#active)
[](https://github.com/dbca-wa/ruODK/commits/master)
[](https://github.com/dbca-wa/ruODK/issues/)
[](https://travis-ci.org/dbca-wa/ruODK)
[](https://ci.appveyor.com/project/dbca-wa/ruODK)
[](https://codecov.io/github/dbca-wa/ruODK?branch=master)
[](https://mybinder.org/v2/gh/dbca-wa/urODK/master?urlpath=rstudio)
<!-- badges: end -->
Especially in these trying times, it is important to ask: "ruODK?"
[OpenDataKit](https://opendatakit.org/) (ODK) is a free and open-source software
for collecting, managing, and using data in resource-constrained environments.
ODK consists of a range of [software packages and apps](https://opendatakit.org/software/).
`ruODK` assumes some familiarity of its users with the ODK ecosystem and workflows.
For a detailed overview, read the extensive [ODK documentation](https://docs.opendatakit.org/)
and visit the friendly [ODK forum](https://forum.opendatakit.org/).
[ODK Central](https://docs.opendatakit.org/central-intro/) is a cloud-based
data clearinghouse for digitally captured data, replacing the older software
[ODK Aggregate](https://docs.opendatakit.org/aggregate-intro/).
ODK Central manages user accounts and permissions, stores form definitions,
and allows data collection clients like
[ODK Collect](https://docs.opendatakit.org/collect-intro/) to connect to it for
form download and submission upload.
A typical [ODK workflow](https://docs.opendatakit.org/#how-is-odk-used):
An XForm is designed e.g. in [ODK Build](https://build.opendatakit.org/),
[published to ODK Central](https://docs.opendatakit.org/central-forms/),
and downloaded onto an Android device running ODK Collect.
After data have been captured digitally using
[ODK Collect](https://docs.opendatakit.org/collect-intro/), the data are uploaded
and stored in ODK Central. The next step from there is to extract the data,
optionally upload it into another data warehouse, and then to analyse and
generate insight from it.
While data can be retrieved in bulk through the GUI, ODK Central's API provides
access to its data and functionality through both an OData and a RESTful API
with a comprehensive and interactive
[documentation](https://odkcentral.docs.apiary.io/#reference/odata-endpoints).
`ruODK` is aimed at the technically minded researcher who wishes to access and
use the data from ODK Central using the programming language R.
Benefits of using the R ecosystem in combination with ODK:
* Scalability: Both R and ODK are free and open source software. Scaling to
many users does not incur license fees.
* Ubiquity: R is known to many scientists and is widely taught at universities.
* Automation: The entire data access workflow can be automated through R scripts.
* Reproducible reporting (e.g.
[Sweave](https://support.rstudio.com/hc/en-us/articles/200552056-Using-Sweave-and-knitr),
[RMarkdown](https://rmarkdown.rstudio.com/)), interactive web apps
([Shiny](https://shiny.rstudio.com/)),
workflow scaling ([drake](https://docs.ropensci.org/drake/))
* Rstudio-as-a-Service (RaaS) at
[](https://mybinder.org/v2/gh/dbca-wa/urODK/master?urlpath=rstudio)
`ruODK`'s scope:
* To wrap all ODK Central API endpoints with a focus on **data access**.
* To provide working examples of interacting with the ODK Central API.
* To provide convenience helpers for the day to day tasks when working with
ODK Central data in R: **data munging** the ODK Central API output into tidy
R formats.
`ruODK`'s use cases:
* Smaller projects:
1. Data collection: ODK Collect
2. Data clearinghouse: ODK Central
3. Data analysis and reporting: `Rmd` (ruODK)
4. Publishing and dissemination: [`ckanr`](https://docs.ropensci.org/ckanr/),
[`CKAN`](https://ckan.org/)
* Larger projects:
1. Data collection: ODK Collect
2. Data clearinghouse: ODK Central
3. ETL pipeline into data warehouses: `Rmd` (ruODK)
4. QA: in data warehouse
5. Reporting: `Rmd`
6. Publishing and dissemination: [`ckanr`](https://docs.ropensci.org/ckanr/),
[`CKAN`](https://ckan.org/)
Out of scope:
* To wrap "management" API endpoints. ODK Central is a
[VueJS/NodeJS application](https://github.com/opendatakit/central-frontend/)
which provides a comprehensive graphical user interface for the management of
users, roles, permissions, projects, and forms.
* To provide extensive data visualisation. We show only minimal examples of data
visualisation and presentation, mainly to illustrate the example data.
Once the data is in your hands as tidy tibbles... urODK!
## A quick preview
<img src="man/figures/odata.svg" alt="ruODK screencast" width="100%" />
## Install
You can install `ruODK` from GitHub with:
```{r gh-installation, eval = FALSE}
if (!requireNamespace("remotes")) {install.packages("remotes")}
remotes::install_github("dbca-wa/ruODK", dependencies = TRUE)
```
## ODK Central
### Access to an ODK Central instance
First, we need an ODK Central instance and some data to play with!
Either ask in the [ODK forum](https://forum.opendatakit.org/) for an account in
the [ODK Central Sandbox](https://sandbox.central.opendatakit.org/),
or follow the [setup instructions](https://docs.opendatakit.org/central-intro/)
to build and deploy your very own ODK Central instance.
### ODK Central setup
The ODK Central [user manual](https://docs.opendatakit.org/central-using/)
provides up-to-date descriptions of the steps below.
* [Create a web user account](https://docs.opendatakit.org/central-users/#creating-a-web-user)
on an ODK Central instance. Your username will be an email address.
* [Create a project](https://docs.opendatakit.org/central-projects/) and give
the web user at least
[read permissions](https://docs.opendatakit.org/central-projects/#managing-project-managers).
* Create an XForm, e.g. using ODK Build, or use the
[example forms](https://github.com/dbca-wa/ruODK/tree/master/inst/extdata)
provided by `ruODK`. The `.odkbuild` versions can be loaded into
[ODK Build](https://build.opendatakit.org/), while the `.xml` versions can be
directly imported into ODK Central.
* [Publish the form](https://docs.opendatakit.org/central-forms/)
to ODK Central.
* Collect some data for this form on ODK Collect and let ODK Collect submit the
finalised forms to ODK Central.
## Configure `ruODK`
Set up `ruODK` with an OData Service URL and credentials of a read-permitted
ODK Central web user.
```{r ru_setup_rmd, echo=FALSE, eval=TRUE}
ruODK::ru_setup(
svc = "https://sandbox.central.opendatakit.org/v1/projects/14/forms/build_Flora-Quadrat-0-4_1564384341.svc",
un = Sys.getenv("ODKC_TEST_UN"),
pw = Sys.getenv("ODKC_TEST_PW")
)
```
```{r ru_setup_demo, echo=TRUE, eval=FALSE}
ruODK::ru_setup(
svc = "https://sandbox.central.opendatakit.org/v1/projects/14/forms/build_Flora-Quadrat-0-4_1564384341.svc",
un = "me@email.com",
pw = "..."
)
```
For all available detailed options to configure `ruODK`, read
[`vignette("setup", package = "ruODK")`](https://dbca-wa.github.io/ruODK/articles/setup.html).
## Use ruODK
A quick example browsing projects, forms, submissions, and accessing the data:
```{r odata_example, eval=T, message=TRUE}
library(ruODK)
# Part 1: Data discovery ------------------------------------------------------#
# List projects
proj <- ruODK::project_list()
proj %>% head() %>% knitr::kable(.)
# List forms of default project
frms <- ruODK::form_list()
frms %>% knitr::kable(.)
# Form details of default form
frmd <- ruODK::form_detail()
frmd %>% knitr::kable(.)
# Form schema
meta <- ruODK::form_schema()
meta %>% knitr::kable(.)
# Part 2: Data access ---------------------------------------------------------#
# Form tables
srv <- ruODK::odata_service_get()
srv %>% knitr::kable(.)
# Form submissions
d <- fs::path("docs/articles/attachments/media") # choose your own
tz <- "Australia/Perth" # g'day mate
data <- ruODK::odata_submission_get(verbose = TRUE, local_dir = d, tz = tz)
data %>% knitr::kable(.)
```
A more detailed walk-through with some data visualisation examples is available
in the [`vignette("odata-api", package="ruODK")`](https://dbca-wa.github.io/ruODK/articles/odata.html).
See also [`vignette("restful-api", package="ruODK")`](https://dbca-wa.github.io/ruODK/articles/api.html)
for examples using the alternative RESTful API.
## Try ruODK
[](https://mybinder.org/v2/gh/dbca-wa/urODK/master?urlpath=rstudio)
will launch a disposable, hosted RStudio instance with `ruODK` installed and
the companion package [`urODK`](https://github.com/dbca-wa/urODK) opened as
starting point for a hands-on workshop or instant demo of `ruODK` usage.
Create a new RMarkdown workbook from `ruODK` template "ODK Central via OData"
and follow the instructions within.
## Contribute
Contributions through [issues](https://github.com/dbca-wa/ruODK/issues) and PRs
are welcome!
See the [contributing guide](https://dbca-wa.github.io/ruODK/CONTRIBUTING.html)
on best practices and further readings for code contributions.
## Attribution
`ruODK` was developed, and is maintained, by Florian Mayer for the Western Australian
[Department of Biodiversity, Conservation and Attractions (DBCA)](https://www.dbca.wa.gov.au/).
The development was funded both by DBCA core funding and Offset funding through
the [North West Shelf Flatback Turtle Conservation Program](https://flatbacks.dbca.wa.gov.au/).
To cite package `ruODK` in publications use:
```{r citation}
citation("ruODK")
```
## Acknowledgements
The Department of Biodiversity, Conservation and Attractions (DBCA) respectfully
acknowledges Aboriginal people as the traditional owners of the lands and waters
it manages.
One of the Department's core missions is to conserve and protect the value of
the land to the culture and heritage of Aboriginal people.
This software was created both as a contribution to the ODK ecosystem and for
the conservation of the biodiversity of Western Australia, and in doing so,
caring for country.
## Package functionality
There are several other R packages interacting with the ODK ecosystem, and/or
[OData](https://www.odata.org/).
### Comparison of ODK related software packages (non-ODK core)
| Package | [`ruODK`](https://dbca-wa.github.io/ruODK/) | [`odkr`](https://validmeasures.org/odkr/) | [`odk`](https://cran.r-project.org/package=odk) | [`odkmeta`](https://github.com/nap2000/odkmeta) | [`koboloadeR`](https://unhcr.github.io/koboloadeR/docs/index.html) | [Pentaho Kettle tutorial](https://github.com/schemetrica/automating-data-delivery-odk-central)
|------------------------------|---------------|---------------|---------------|---------------|---------------|---------------|
| Elevator pitch | "[ckanr](https://github.com/ropensci/ckanr) for ODK Central" | "Drive ODK Briefcase through R" | "Export ODK Aggregate to SPSS" | "Export ODK Aggregate to STATA" | "Metapackage for extended ODK ecosystem" | "What ruODK does, but as GUI" |
| Last commit | [](https://github.com/dbca-wa/ruODK/commits/master) | [](https://github.com/validmeasures/odkr/commits/master) | Nov 2017 | [](https://github.com/nap2000/odkmeta/commits/master) | [](https://github.com/unhcr/koboloadeR/commits/master) | [](https://github.com/schemetrica/automating-data-delivery-odk-central/commits/master) |
| Website | [](https://github.com/dbca-wa/ruODK) [](https://dbca-wa.github.io/ruODK/) | [](https://github.com/validmeasures/odkr) [](https://validmeasures.org/odkr/) | [](https://rdrr.io/cran/odk/) | [](https://github.com/nap2000/odkmeta) | [](https://github.com/unhcr/koboloadeR) [](https://unhcr.github.io/koboloadeR/docs/index.html) | [](https://github.com/schemetrica/automating-data-delivery-odk-central) |
| Test coverage | [](https://travis-ci.org/dbca-wa/ruODK) [](https://ci.appveyor.com/project/dbca-wa/ruODK) [](https://codecov.io/github/dbca-wa/ruODK?branch=master) | [](https://codecov.io/gh/validmeasures/odkr) | ❌ | In repo | [](https://travis-ci.org/unhcr/koboloadeR) [](https://ci.appveyor.com/project/unhcr/koboloadeR) [](https://codecov.io/gh/unhcr/koboloadeR) | NA |
| Working examples | README, 3 vignettes, pkgdown, Rmd templates | README, pkgdown | CRAN PDF | README | README, 9 vignettes, shiny apps, pkgdown | Tutorial with screenshots |
| Available on CRAN | [](https://cran.r-project.org/package=ruODK) | [](https://cran.r-project.org/package=odkr) | [](https://CRAN.R-project.org/package=odk) | NA | [](https://cran.r-project.org/package=koboloadeR) | NA |
| Technologies | Tidyverse R, XForms | Base R | Base R | Stata | R metapackage, XlsForms | Pentaho Kettle GUI |
| External dependencies | None | Java, ODK Briefcase | SPSS | Stata | Java, ODK Briefcase, wraps `odkr` | [Pentaho Kettle](http://www.ibridge.be/), Java |
| Affiliation | [DBCA WA](https://www.dbca.wa.gov.au/science) / [ROpenSci prospectus](https://github.com/ropensci/software-review/issues/335) | [Valid Measues](https://github.com/validmeasures) | [Muntashir-Al-Arefin](https://stackoverflow.com/users/8875690/muntashir-al-arefin) | [ODK Central developer Matt White](https://github.com/matthew-white) | [UNHCR](https://github.com/unhcr) | [Schemetrica](https://github.com/schemetrica) |
| Covers ODK Central OData API | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Covers ODK Central REST API | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Covers ODK Central bulk export | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ |
| Covers ODK Central OpenRosa API | ❌ no need, gets all data through OData/REST API | ✅ via ODK Briefcase | ❌ | ❌ | ✅ via ODK Briefcase | ✅ |
| Data post-processing | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ |
| Data visualisation examples | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ |
In summary:
`ruODK` provides a dependency-free interface to ODK Central.
`koboloadeR` is a metapackage containing lots of ancillary packages, with some
heavy dependencies on Java and ODK Briefcase (which in turn can access ODK Central).
Although built around the XlsForm standard and paradigm, `koboloadeR` is well worth
exploring as a larger context to data wrangling in the ODK ecosystem.
Schemetrica's tutorial illustrates data ETL from ODK Central and deserves a special
mention, as it is both very recent and aimed specifically against ODK Central.
The GUI paradigm of Pentaho Kettle addresses a different audience to the scripting
paradigm of `ruODK`. It should be mentioned that Kettle's composable data
manipulation steps can be used for many other use cases apart from ODK Central.
### Comparison of OData related R packages
| Package | [`ruODK`](https://dbca-wa.github.io/ruODK/) | [`odataR`](https://github.com/HanOostdijk/odataR) | [`cbsodataR`](https://github.com/edwindj/cbsodataR) | [`OData`](https://cran.r-project.org/web/packages/OData/index.html) | [OData JDBC R tutorial](https://www.cdata.com/kb/tech/odata-jdbc-r.rst)
|------------------------------|---------------|---------------|---------------|---------------|--------------|
| Elevator pitch | "[ckanr](https://github.com/ropensci/ckanr) for ODK Central" | "OData client for https://opendata.cbs.nl (and similar)" | "OData client for https://www.cbs.nl" | "Minimal OData example" | "Minimal RJDBC example" |
| Last commit | [](https://github.com/dbca-wa/ruODK/commits/master) | [](https://github.com/HanOostdijk/odataR/commits/master) | [](https://github.com/edwindj/cbsodataR/commits/master) | Dec 2016 | ❓ |
| Website | [](https://github.com/dbca-wa/ruODK) [](https://dbca-wa.github.io/ruODK/) | [](https://github.com/HanOostdijk/odataR) | [](https://cran.r-project.org/web/packages/cbsodataR/) [](https://github.com/edwindj/cbsodataR) [](https://edwindj.github.io/cbsodataR/) | [](https://cran.r-project.org/web/packages/OData/index.html) | [](https://www.cdata.com/kb/tech/odata-jdbc-r.rst) |
| Test coverage | [](https://travis-ci.org/dbca-wa/ruODK) [](https://ci.appveyor.com/project/dbca-wa/ruODK) [](https://codecov.io/github/dbca-wa/ruODK?branch=master) | ❌ | [](https://travis-ci.org/edwindj/cbsodataR) [](https://ci.appveyor.com/project/edwindj/cbsodatar) | ❌ | ❌ |
| Targets ODK Central | ✅ | ❌ | ❌ | ❌ | ❌ |
| Works with ODK Central | ✅ | ❓ | ❓ | ❌ | ❌ |
| Data wrangling helpers for post-processing | ✅ | some | some | ❌ | ❌ |
| Actively maintained to work against ODK Central | ✅ | ❌ | ❌ | ❌ | ❌ |
| Technologies | R, httr, xml2, tidyr, purrr | R, jsonlite, tidyverse | R, tidyverse | R, XML, RJSONIO | R, RJDBC, Java |
| External dependencies | ✅ None | ✅ None | ✅ None | ✅ None | ❌ JDBC, Java |
| Available on CRAN | [](https://cran.r-project.org/package=ruODK) | [](https://cran.r-project.org/package=odataR) | [](https://CRAN.R-project.org/package=cbsodataR) | [](https://CRAN.R-project.org/package=OData) | NA |
In summary:
`ruODK` is the only R package explicitly aimed at ODK Central's OData and
RESTful API, as well as providing context and helpers around specific recurring
data wrangling tasks.
The value of OData lies in its self-descriptive nature, which allows tools to
introspect the data structures and types. Both GUI-driven tools like MS PowerBI
and `ruODK` use this introspection to assist users in wrangling their own data.
The script-based approach of `ruODK` allows to automate the data extraction,
transformation, and reporting pipeline, and therefore provide reproducible
reporting.