Provide standard, accessible data import and export

[florianm]

Source

OIM

Related issues

#43

Requirement

The solution shall provide a standardised, accessible way to import and export all data, like a RESTful or GraphQL API.

Access needs to be following standard protocols and formats, be entirely independent of both the systems it is stored in, as well as independent of the software packages it is analysed with.

Use cases

Other systems, scripts, software interact with data (read/write) in the solution through a RESTful or GraphQL API.

Example: the workbooks accessing NTP track count data or WAMTRAM tagging data access data programmatically without human intervention.

The workbook turtle tracks accesses the data prepared by the above mentioned workbooks through a RESTful API from one central location without human intervention or any manual steps.

Implementation

WAStD has both a RESTful API and an experimental GraphQL API, documented in the API webpages themselves, as well as in the WAStD User Manual chapter on the API.

A dedicated R package, wastdr, provides convenience wrappers around the API endpoints and is used in production to upload, download, parse, summarise and visualise WAStD data. wastdr provides both built-in and online documentation.

A second dedicated R package, ruODK, was written to wrap the API of the electronic data capture clearinghouse server, ODK Central. ruODK has passed peer review and was accepted as part of the prestigious rOpenSci community of software packages.

The entire data ETL, QA and reporting data pipeline is implemented in its own R package, etlTurtleNesting. It has internal documentation only at this point. This package uses ruODK and wastdr, it contains all mapping between the data structures of data capture forms and WAStD, and all skip logic.
The process runs as a single automated build using the R package for workflow automation, drake.

This ecosystem of APIs, documentation, and software packages provides worked examples both for future users and external developers seeking to re-use parts of this system.