Branch: master
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
571 lines (413 sloc) 39.6 KB

Preston*: a biodiversity dataset tracker

Build Status standard-readme compliant DOI

*Named after Frank W. Preston (1896-1989) and the Prestonian shortfall, one of the "Seven Shortfalls that Beset Large-Scale Knowledge of Biodiversity" as described by Hortal et al. 2015.

usage / install / use cases / architecture

Preston is an open-source software system that keeps track of biodiversity datasets. Scientists can use Preston to keep a local, uniquely identifiable, versioned copy of all or parts of GBIF-indexed datasets. Institutions can use Preston to check that their collections are indexed and available through iDigBio. Biodiversity informatics researchers can use Preston to perform metadata analysis. And finally, archivists can distribute archives across the world.

Preston uses the PROV and PAV ontologies to model the actors, activities and entities involved in discovery, access and change control of digital biodiversity datasets. In addition, Preston uses content-addressed storage and SHA256 hashes to uniquely identify and store content. A hexastore-like index is used to navigate a local graph of biodiversity datasets. Preston is designed to work offline and can be cloned, copied and moved across storage media with existing tools and infrastructures like rsync, dropbox, the internet archive or thumbdrives. In addition to versioned copies of uniquely identifiable original ABCD-A, DWC-A and EML files, Preston also keeps track of the GBIF, iDigBio and BioCASe registries to help retain the relationships between the institutions to keep a detailed record of provenance.

The process diagram below shows how Preston starts crawls to download copies of biodiversity registries and their datasets. A detailed log of the crawl activities are recorded to describe what data was discovered and how. This activity log is referred to as the history of a biodiversity dataset graph. The numbers indicate the sequence of events. Click on the image to enlarge.

The figure above shows how Preston starts (1) a crawl activity. This crawl activity then accesses (2) a registry to save (3,4) a snapshot (or version) of it. Now, datasets referenced in this registry version are accessed, downloaded and saved (6,7,8). After all this, the crawl activity saves the log that contains its activities (1-8) as a version of a biodiversity dataset and linked to previous versions (see figure below). This log can be used to retrace the steps of the crawl activity to reconstruct the relationships between the registries, datasets as well as their respective content signatures or content hashes. Actual instances of crawl activities contains multiple registries (e.g., GBIF, iDigBio) and potentially thousands of datasets.

If you haven't yet tried Preston, please see the Installation section. Please also see a template repository and use cases for examples. If you are interested in learning how Preston works, please visit the architecture page.

Table of Contents


Preston was designed with the unix philosophy in mind: a simple tool with a specific focus that works well with others. For Preston, the focus is keeping track of biodiversity archives available through registries like GBIF, iDigBio and BioCASe. The functionality is currently available through a command line tool.

Command Line Tool

The command line tool provides four commands: update, ls, get and history. In short, the commands are used to track and access DwC-A, EMLs and various registries. The output of the tools is nquads or tsv. Both output formats are structured in "columns" to form a three term sentence per line. In a way, this output is telling you the story of your local biodiversity data graph in terms of simple sentences. This line-by-line format helps to re-use existing text processing tools like awk, sed, cut, etc. Also, tab-separated-values output plays well with spreadsheet applications and R.

The examples below assume that you've created a shortcut preston to java -jar preston.jar (see installation).


The update command updates your local biodiversity dataset graph using remote resources. By default, Preston uses GBIF, iDigBio and BioCASe to retrieve associated registries and data archives. The output is statements, expressed in nquads (or nquad-like tsv). An in depth discussion of rdf, nquads and related topics are beyond the current scope. However, with a little patience, you can probably figure out what Preston is trying to communicate.

For instance:

$ preston update
<> <> <> .
<> <> <> .
<> <> "Preston is a software program that finds, archives and provides access to biodiversity datasets."@en .
<0b472626-1ef2-4c84-ab8f-9e455f7b6bb6> <> <> .
<0b472626-1ef2-4c84-ab8f-9e455f7b6bb6> <> "A crawl event that discovers biodiversity archives."@en .
<0b472626-1ef2-4c84-ab8f-9e455f7b6bb6> <> "2018-09-05T04:42:40.108Z"^^<> .
<0b472626-1ef2-4c84-ab8f-9e455f7b6bb6> <> <> .
<0659a54f-b713-4f86-a917-5be166a14110> <> <> .

tells us that there's a software program called "Preston" that started a crawl on 2018-09-05 . A little farther down, you'll see things like:

<> <> <> .
<> <> "Provides a registry of Darwin Core archives, and EML descriptors."@en .
<> <> <> .
<> <> "application/json" .
<> <> <hash://sha256/5d1bb4f3a5a9da63fc76efc4d7b4a7debbec954bfd056544225c294fff679b4c> .

which says that GBIF, an organization created a registry that has a version at hash://sha256/5d1bb4f3a5a9da63fc76efc4d7b4a7debbec954bfd056544225c294fff679b4c . This weird looking url is a content-addressed hash. Rather than describing where things are (e.g.,, content-addressed hashes describe what they contain.

If you don't want to download the entire biodiversity dataset graph (~60GB) onto your computer, you can also use GBIF's dataset registry search api as a starting point. For instance, if you run preston update ";type=OCCURRENCE", you only get occurence datasets that GBIF suggests are related to the Amazon. If you track these suggested datasets, you might see something like:

<> <> "application/dwca" .
<hash://sha256/5cba2f513fee9e1811fe023d54e074df2d562b4169b801f15abacd772e7528f8> <> "2018-09-05T05:11:33.592Z"^^<> .
<hash://sha256/5cba2f513fee9e1811fe023d54e074df2d562b4169b801f15abacd772e7528f8> <> <21de25a8-927f-49a1-99be-725f1f506232> .
<> <> <hash://sha256/5cba2f513fee9e1811fe023d54e074df2d562b4169b801f15abacd772e7528f8> .

which tells us that a darwin core archive was found and a copy of it was made on 2018-09-05. The copy, or version, has a content hash of hash://sha256/5cba2f513fee9e1811fe023d54e074df2d562b4169b801f15abacd772e7528f8 . Incidentally, you can reach this same exact dataset at web-accessible preston archive. With this, we established that on 2018-09-05 a specific web addressed produced a specific content. On the next update run, Preston will download the content again. If the content is the same as before, nothing happens. If the content changed, a new version will be created associated with the same address, establishing a versioning of the content produced by the web address. This is addressed in a statement like <some hash> <.../previousVersion> <some previous hash>.

So, in a nutshell, the update process produces a detailed record of which resources are downloaded, what they look like and were they came from. You can retrieve the record of a successful run by using ls.


ls print the results of the previous updates. An update always refers to a previous update, so that a complete history can be printed / replayed of all past updates. So, the ls commands lists your (local) copy of the biodiversity dataset graph.


get retrieves a specific node in the biodiversity dataset graph. This can be a darwin core archive, EML file but also a copy of the iDigBio publisher registry. For instance, if you'd like to retrieve the node with DwC-A content, get the file and list the content using unzip and access the references in the taxa.txt file.

$ preston get hash://sha256/5cba2f513fee9e1811fe023d54e074df2d562b4169b801f15abacd772e7528f8 > 
$ unzip -l 
  Length      Date    Time    Name
---------  ---------- -----   ----
    11694  2016-01-03 13:36   meta.xml
     4664  2016-01-03 13:36   eml.xml
     5533  2017-06-20 02:39   taxa.txt
      284  2017-06-20 02:39   occurrences.txt
    16978  2017-06-20 02:39   description.txt
       54  2017-06-20 02:39   distribution.txt
    48439  2017-06-20 02:39   media.txt
     9280  2017-06-20 02:39   references.txt
       33  2017-06-20 02:39   vernaculars.txt
---------                     -------
    96959                     9 files
$ unzip -p taxa.txt | cut -f16

The implication of using content addressed storage is that if the hash is the same, you are guaranteed that the content is the same. So, you can reproduce the exact same results above if you have a file with the same content hash.


History helps to list your local content versions associated with a web address. Because the internet today might not be the internet of yesterday, and because publishers update their content for various reasons, Preston helps you keep track of the different versions retrieved from a particular location. Just like the Internet Archive's Way Back Machine keeps track of web page content, Preston help you keep track of the datasets that you are interested in.

To inspect the history you can type:

$ preston history
<0659a54f-b713-4f86-a917-5be166a14110> <> <hash://sha256/c253a5311a20c2fc082bf9bac87a1ec5eb6e4e51ff936e7be20c29c8e77dee55> .
<hash://sha256/b83cf099449dae3f633af618b19d05013953e7a1d7d97bc5ac01afd7bd9abe5d> <> "2018-09-04T20:48:35.096Z" .
<hash://sha256/b83cf099449dae3f633af618b19d05013953e7a1d7d97bc5ac01afd7bd9abe5d> <> <hash://sha256/c253a5311a20c2fc082bf9bac87a1ec5eb6e4e51ff936e7be20c29c8e77dee55> .
<hash://sha256/7efdea9263e57605d2d2d8b79ccd26a55743123d0c974140c72c8c1cfc679b93> <> "2018-09-04T23:14:22.292Z" .
<hash://sha256/7efdea9263e57605d2d2d8b79ccd26a55743123d0c974140c72c8c1cfc679b93> <> <hash://sha256/b83cf099449dae3f633af618b19d05013953e7a1d7d97bc5ac01afd7bd9abe5d> .

By default, the history command shows the versions of your local biodiversity dataset graph as a whole. A list of versions associated with the sequence of updates. If you'd like to know what the UUID 0659a54f-b713-4f86-a917-5be166a14110 is described as, you can use ls and filter by the UUID:

$ preston ls | grep 0659a54f-b713-4f86-a917-5be166a14110
<0659a54f-b713-4f86-a917-5be166a14110> <> <> .
<0659a54f-b713-4f86-a917-5be166a14110> <> "A biodiversity dataset graph archive."@en .

So, the UUID ending on 4110 is describe as "A biodiversity dataset graph archive". This UUID is the same across all Preston updates, so in a way we are help to create different versions of the same "a biodiversity dataset graph". Good to know right?


Preston stores versioned copies of biodiversity dataset graphs and their associated datasets in the data/ directory. The copyTo command moves the locally available biodiversity dataset graphs and their data to another location.

$ preston copyTo /home/someuser/target/data
indexing... done.
copying... [0.1]%
Copied [279636] datasets from [/home/someuser/source/data] to [/home/someuser/target/data] in [543] minutes.


The check command takes the locally versions of the dataset graph and verifies that the associated datasets are available locally. In addition, the content hash (e.g., hash://sha256/...) for each local dataset graph and dataset is re-computed to verify that the content is still consistent with the content hash signatures recorded previously. The check command produces tab-separated values with four columns. The first column is the content hash of the file being checked, the second contains OK/FAIL to record the success of the check, the third gives a reason for check outcome and the fourth contains the total number of bytes of the local file associated with the hash.

$ preston check
hash://sha256/3eff98d4b66368fd8d1f8fa1af6a057774d8a407a4771490beeb9e7add76f362  OK     MATCHING_HASH   89931
hash://sha256/184886cc6ae4490a49a70b6fd9a3e1dfafce433fc8e3d022c89e0b75ea3cda0b  OK     MATCHING_HASH   210344

Use Cases

In the previous section the commands update, ls, get and history were introduced. Now, some use cases are covered to show how to combine these basic commands to make for useful operations. This is by no means an exhaustive list of all the potential uses, but instead is just to provide some inspiration on how to get the most out of preston.

Mining Citations

The Ecological Metadata Language (EML) files contain citations, and your biodiversity dataset graph contains EML files. To extract all citations you can do:

# first make a list of all the emls
preston ls | grep application/eml | cut -f1 > emls.txt
# then 
preston ls -l tsv | grep -f emls.tsv | grep "Version" | grep hash | cut -f3 | preston get | grep citation | sed 's/<[^>]*>//g' > citations.txt
head citations.txt
            HW Jackson C, Ochieng J, Musila S, Navarro R, Kanga E (2018): A Checklist of the Mammals of Arabuko-Sokoke Forest, Kenya, 2018. v1.0. A Rocha Kenya. Dataset/Checklist.;v=1.0
            Adda M., Sanou L., Bori H., 2018. Specimens d&apos;herbier de la flore du Niger. Données d&apos;occurrence publiées dans le cadre du Prjet BID Régional. CNSF-Niger  					
                Michel.C., 2000. Arbres,arbustes et lianes des zones sèches d&apos;Afrique de l&apos;Ouest.3ème édition.Quae.MNHN.573p.
            Hendrickson D A, Cohen A, Casarez M (2018): Ichthyology. v1.3. University of Texas at Austin, Biodiversity Collections. Dataset/Occurrence.;v=1.3
            Urrutia N S (2014): Caracterización Florística de un Área Degradada por Actividad Minera en la Costa Caucana. v2.0. Instituto de Investigaciones Ambientales del Pacifico John Von Neumann (IIAP). Dataset/Occurrence.

So, now we have a way to attribute each and every dataset individually.


Preston creates a "data" folder that stores the biodiversity datasets and associated information. For archiving, you can take this "data" folder, copy it and move it somewhere safe. You can also use tools like git-annex, rsync, or use distributed storage systems like the Interplanetary File System (ipfs) or Dat.


For instance, assuming that a preston data directory exists on a serverA which has ssh and rsync installed on it, you can keep a local copy in sync by running the following command on your local server:

$ rsync -Pavz preston@someserver:~/preston-archive/data /home/someuser/preston-archive/

where someserver is the remote server you'd like to sync with and /home/someuser/preston-archive is the place on your own server you'd like to store the rsync-ed Preston archive.

On a consumer internet connection with bandwidth < 10Mb/s, an initial sync with a remote trans-atlantic server with a 67GB preston archive took about 3 days. After the initial sync, only files that you don't have yet are included. For instance, if no new files are added to the remote preston archive, a sync take a few minutes instead of hours or days.

Note that ssh and rsync comes with frequently used linux distributions like Ubuntu v18.04 by default).

The Internet Archive, a 501(c)(3) non-profit, is building a digital library of Internet sites and other cultural artifacts in digital form. Like a paper library, we provide free access to researchers, historians, scholars, the print disabled, and the general public. Our mission is to provide Universal Access to All Knowledge.

One of the services of the Internet Archive is the Wayback Machine . If your Preston archive is Web Accessible, you can use the Wayback Machine to make snapshots of your cached datasets. The bash scripts below can be used to do so.

# Register all preston urls with
set -x


function register_with_internet_archive {
  zcat $1 | grep "hash:\/\/sha256" | sort | uniq | sed -e "s/hash:\/\/sha256/${domain}/g" | tee domain_urls.txt | sed -e 's/^/https:\/\/\/wayback\/available?url=/g' | xargs --no-run-if-empty -L1 curl -s | jq --raw-output ".archived_snapshots.closest | select(.available == true) | .url" | sort | uniq > domain_url_available_snapshots.txt

  cat domain_url_available_snapshots.txt | sed -e "s/^.*${domain}//g" | sed -e "s/^/${domain}/g" > domain_urls_archived.txt
  diff --changed-group-format='%>' --unchanged-group-format='' domain_urls_archived.txt domain_urls.txt > domain_urls_to_be_archived.txt 
  cat domain_urls_to_be_archived.txt | sed -e "s/^/https:\/\/\/save\//g" | tee domain_urls_save_request.txt | xargs --no-run-if-empty -L1 curl -s 

/usr/local/bin/preston ls -l tsv | grep Version | head -n13 | cut -f1,3 | tr '\t' '\n' | grep -v "${domain}/\.well-known/genid" | sort | uniq | gzip > url_uniq.tsv.gz

register_with_internet_archive url_uniq.tsv.gz

In the script above, a list of urls is extracted and registered with if they haven't already.

Another way to submit content to the Internet Archive is using their s3-like interface via the Internet Archive Command-line Interface. Assuming that your preston archive is stored in /home/preston/preston-archive, your Internet Archive project id is preston-archive and that the commandline tool ia is configured properly, you can upload all the data using:

find /home/preston/preston-archive | grep -v "tmp" | grep "data.*/data$" | sed 's/.*preston-archive\///g' | xargs -L1 bash -c 'echo upload preston-archive /home/preston/preston-archive/$0 --remote-name=$0' | tee uploaded_req.txt | xargs -L1 ia  

Web Access

If you'd like to make your Preston archive accessible via http/https by using a nginx webserver, you can use a following address mapping to your nginx configuration:

location ~ "/\.well-known/genid/" {
		return 302;

location ~ "^/([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{58})$" {
            	try_files /preston/$1/$2/$uri =404;

The first location block redirects any URIs describing skolemized blank nodes to the appropriate w3c documentation on the topic. The second location block configures the server to attempt to retrieve a static file with a 64 hexadecimal sha256 hash from the appropriate data file in preston archive directory on the web server.


Similary, for Caddy, add the following to your Caddyfile:

redir 302 {
  if {path} starts_with /.well-known/genid/

rewrite {
  r ^/([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{58})$
  to data/{1}/{2}/{path}

Where you can replace data/ with the relative location of the local preston archive data directory.

With this, you can access your Preston archive remotely using the URLs like https://someserver/[sha256 content hash] . So, if you'd like to dereference (or download) hash://sha256/1846abf2b9623697cf9b2212e019bc1f6dc4a20da51b3b5629bfb964dc808c02 , you can now point your http client or browser at https://someserver/1846abf2b9623697cf9b2212e019bc1f6dc4a20da51b3b5629bfb964dc808c02 . Note that you do not need any other software than the (standard) nginx webserver, because you are serving the content as static files from the file system of your server.

Data Access Monitor

By running update periodically and checking for "blank", or "missing" nodes (see blank skolemization), you can make a list of the dataset providers that went offline or are not responding.

The example below was created on 2018-09-05 using biodiversity dataset graph with hash hash://sha256/7efdea9263e57605d2d2d8b79ccd26a55743123d0c974140c72c8c1cfc679b93.

$ preston ls -l tsv | grep "/.well-known/genid/" | grep "Version" | cut -f1,3 | tr '\t' '\n' | grep -v "/.well-known/genid/" | grep -v "hash" | sort | uniq -c | sort -nr | head -n10

Compare Versions

Keeping track of changes across a diverse consortium of data publishers is necessary for reproducible workflows and reliable results. As datasets change, Preston can help you give insights into what changed exactly. For instance, the GBIF dataset registry changes as datasets are added, updated or deprecated. Below is an example of two version of the endpoint, one from 2018-09-03 and the other from 2018-09-04. Using jq and diff in combination with preston get and preston history gives us a way to check and see what changed.

$ preston ls | grep
<> <> <hash://sha256/184886cc6ae4490a49a70b6fd9a3e1dfafce433fc8e3d022c89e0b75ea3cda0b> .
<> <> <hash://sha256/1846abf2b9623697cf9b2212e019bc1f6dc4a20da51b3b5629bfb964dc808c02> .
$ preston get hash://sha256/184886cc6ae4490a49a70b6fd9a3e1dfafce433fc8e3d022c89e0b75ea3cda0b | jq . > one.json
$ preston get hash://sha256/1846abf2b9623697cf9b2212e019bc1f6dc4a20da51b3b5629bfb964dc808c02 | jq . > two.json
$ diff one.json two.json
<         "text": "Ali P A, Maddison W P, Zahid M, Butt A (2017). New chrysilline and aelurilline jumping spiders from Pakistan (Araneae, Salticidae). taxonomic treatments database. Checklist dataset accessed via on 2018-08-31."
>         "text": "Ali P A, Maddison W P, Zahid M, Butt A (2017). New chrysilline and aelurilline jumping spiders from Pakistan (Araneae, Salticidae). taxonomic treatments database. Checklist dataset accessed via on 2018-09-03."

Generating Citations

Preston provides both a date and a content-based identifier for the datasets that you are using and the biodiversity dataset graph as a whole. Also, it produces the information in a format that is machine readable. This enables the automated generation of citations, for human or machine consumption, as evidenced by the reference to a particular version of the biodiversity dataset graph in the previous section.

So now, instead of a citation to a dataset like:

Levatich T, Padilla F (2017). EOD - eBird Observation Dataset. Cornell Lab of Ornithology. Occurrence dataset accessed via on 2018-09-11.

A citation might look something like:

Levatich T, Padilla F (2017). EOD - eBird Observation Dataset. Cornell Lab of Ornithology. Occurrence dataset hash://sha256/29d30b566f924355a383b13cd48c3aa239d42cba0a55f4ccfc2930289b88b43c accessed at at 2018-09-02 with provenance hash://sha256/b83cf099449dae3f633af618b19d05013953e7a1d7d97bc5ac01afd7bd9abe5d .

The latter citation tells you exactly what file was used and where it came from. The former tells you that some eBird dataset was accessed via GBIF on a specific date and leaves it up to the reader to figure out exactly which dataset was used.

Finding copies with is a project by Ben Trask, the same person who suggested to use hash uris to represent content hashes (e.g., hash://sha256/...). The hash archive keeps track of what content specific urls created using content hashes. To make the hash archive update the hash associated with a url, you can send a http get request in the form of[some url] . For example, to register a url that is known to host an DwC-A at, you can click on , or using curl like


On successful completion of the request, returns something like:

    "url": "",
    "timestamp": 1537269631,
    "status": 200,
    "type": "text/xml;charset=utf-8",
    "length": 3905,
    "hashes": [

This response indicates that the hash archive has independently downloaded the EML url, and calculated various content hashes. Now, you should be able to do to , and see the history of content that this particular url has produced.

In short, hash-archive provides a way to check whether content produced by a url has changed. Also, it provides a way to lookup which urls are associated with a unique content hash.

The example (also see related gist) below shows how Preston was used to register biodiversity source urls as well as Preston web-accessible urls via (see Web Access).

# Register all preston urls with
# Please replace "deeplinker\.bio" instances below with you own escaped hostname of your Preston instance.

# see on how to install preston

preston ls -l tsv | grep Version | cut -f1,3 | tr '\t' '\n' | grep -v "deeplinker\.bio/\.well-known/genid" | sort | uniq | sed -e 's/hash:\/\/sha256/https:\/\/' | sed -e 's/^/https:\/\/\/api\/enqueue\//g' | xargs -L1 curl 

If all web-accessible Preston instances would periodically register their content like this, could serve as a way to lookup backup for the an archive that you got from some no longer active archive url.

Tracking an GBIF IPT

GBIF's Integrated Publishing Toolkit (IPT) helps to publish and register biodiversity datasets with GBIF. IPT provide a RSS feeds that lists publicly available collections/datasets. Using this RSS feed, Preston can track datasets of individual IPTs, such as GBIF Norway's IPT at . You can find the RSS link at the bottom of the home page of the ipt. GBIF Norway's RSS feed is . Now, you can update/track the IPT using Preston by running:

preston update 

By running this periodically, you can keep track of dataset changes and retain historic datasets in your Preston archive.


Preston needs Java 8+.



Preston is a stand-alone java application, packaged in a jarfile. You can build you own (see building) or download a prebuilt jar at releases.

On linux (and Mac) you can install Preston by running:

sudo sh -c '(echo "#!/usr/bin/env sh" && curl -L > /usr/local/bin/preston && chmod +x /usr/local/bin/preston' && preston version

On successful installation, execute preston version on the commandline should print the version of preston. . Alternatively, you can download the jar manually and run preston by using commands like java -jar preston.jar version.


If you'd like to run Preston inside a docker container so that you don't have to worry about installing/conflicting java dependencies use:

  1. download the image wget
  2. load the image sudo docker load --input preston.image.tar
  3. run a container, mapping a host volume onto the containers /data folder sudo docker run -v [some absolute host dir]/data:/data bio.guoda/preston:0.0.8
  4. generation of preston updating / crawl messages like:
<> <> <> .
<> <> <> .
<> <> "Preston is a software program that finds, archives and provides access to biodiversity datasets."@en .
<a4accddb-bf8a-477f-aa6f-413281c8d650> <> <> .

running periodically

If you'd like to run Preston as a service to periodically update, you can use a systemd service combined with a systemd timer, or perhaps some jenkins job like those at . Both have advantages. The following example focuses on systemd.

Assuming that you have some standalone installation of preston running, you might be inspired by the following examples preston.service and preston.timer. The .service file defined how to run the update, while the .timer file defines how to run that update.

To use, copy preston.service and preston.timer into /etc/systemd/system/ on your debian/ubuntu server.

Example of preston.service :

Description=Preston tracks biodiversity datasets.

ExecStart=/usr/local/bin/preston update

Example of preston.timer :

Description=Run Preston



After installing the systemd service and timer, apply changes by sudo systemctl daemon-reload and enable timer using sudo systemctl enable preston.timer and sudo systemctl start preston.timer to start the timer.

See systemd for more information.


Please use maven version 3.3+.

  • Clone this repository
  • Run tests using mvn test (optional).
  • Run mvn package -DskipTests to build (standalone) jar
  • Copy preston/target/preston-[version]-jar-with-dependencies.jar to [some dir]/preston.jar


Maven, Gradle, SBT

Preston is made available through a maven repository.

To include preston in your project, add the following sections to your pom.xml (or equivalent for sbt, gradle etc):




Usage: <main class> [command] [command options]
    ls      list biodiversity dataset graph
      Usage: ls [options]
          -l, --log
            log format
            Default: nquads
            Possible Values: [tsv, nquads, dots]

    get      get biodiversity node(s)
      Usage: get node id (e.g., [hash://sha256/8ed311...])

    update      update biodiversity dataset graph
      Usage: update [options] content URLs to update. If specified, the seeds
            will not be used.
          -i, --incremental
            resume unfinished update
            Default: false
          -l, --log
            log format
            Default: nquads
            Possible Values: [tsv, nquads, dots]
          -u, --seed-uris
            starting points for graph discovery. Only active when no content
            urls are provided.
            Default: [,,]

    history      show history of biodiversity dataset graph
      Usage: history [options] biodiversity resource locator
          -l, --log
            log format
            Default: nquads
            Possible Values: [tsv, nquads, dots]

    version      show version
      Usage: version


Feel free to join in. All welcome. Open an issue!