Skip to content

traitecoevo/datastorr

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
R
 
 
 
 
man
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

datastorr

Simple data retrieval and versioning using GitHub

This project is described in a paper by Daniel Falster, Rich FitzJohn, Matt Pennell, and Will Cornwell. Below we describe the motivation and general idea. Please see the paper for full details.

The problem

Over the last several years, there has been an increasing recognition that data is a first-class scientific product and a tremendous about of repositories and platforms have been developed to facilitate the storage, sharing, and re-use of data. However we think there is still an important gap in this ecosystem: platforms for data sharing offer limited functions for distributing and interacting with evolving datasets - those that continue to grow with time as more records are added, errors fixed, and new data structures are created. This is particularly the case for small to medium sized datasets that a typical scientific lab, or collection of labs, might produce.

In addition to enabling data creators to maintain and share a living dataset, ideally, such an infrastructure would allow enable data users to:

  • Cache downloads, including across R sessions, to make things faster and to work offline
  • Keep track of which versions are downloaded and available remotely
  • Access multiple versions of the data at once; this would be especially helpful if trying to understand why results have changed with the version of the data.

How datastorr helps

This package can be used in two ways:

  1. Use data stored elsewhere in R efficiently (e.g., work with csv files that are too large to comfortably fit in git).
  2. Create another lightweight package designed to allow easy access to your data.

For both of these use-cases, datastorr will store your data using GitHub releases which do not clog up your repository but allow up to 2GB files to be stored (future versions may support things like figshare).

datastorr is concerned about a simple versioning scheme for your data. If you do not imagine the version changing that should not matter. But if you work with data that changes (and everyone does eventually) this approach should make it easy to update files.

From the point of view of a user, using your data could be as simple as:

d <- datastorr::datastorr("richfitz/datastorr.example")

(see below for details, how this works, and what it is doing).

End user interface

See here for the aim from the point of view for an end user.

They would install your package (which contains no data so is nice and light and can be uploaded to CRAN).

devtools::install_github("richfitz/datastorr.example")

The user can see what versions they have locally

datastorr.example::mydata_versions()

and can see what versions are present on GitHub:

datastorr.example::mydata_versions(local=FALSE) # remote

To download the most recent dataset:

d <- datastorr.example::mydata()

Subsequent calls (even across R sessions) are cached so that the mydata() function is fast enough you can use it in place of the data.

To get a particular version:

d <- datastorr.example::mydata("0.0.1")

Downloads are cached across sessions using rappdirs.

Package developer process

The simplest way is to run the (hidden) function datastorr:::autogenerate, as

datastorr:::autogenerate(repo="richfitz/datastorr.example", read="readRDS", name="mydata")

which will print to the screen a bunch of code to add do your package. There will be a vignette explaining this more fully soon. A file generated in this way can be seen here.

Once set up, new releases can be made by running, within your package directory:

datastorr.example::mydata_release("description of release", "path/to/file")

provided you have your GITHUB_TOKEN environment variable set appropriatey. See the vignette for more details.

Installation

devtools::install_github("ropenscilabs/datastorr")

License

MIT + file LICENSE © Rich FitzJohn.

About

Simple data versioning and distribution

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published