add support for DataOne as content registry

[jhpoelen]

dataone.org allows for searching indexed content by their content id .

@cboettig already implemented a way to retrieve content from dataone given a content id (or content hash)

https://github.com/cboettig/contentid/blob/master/R/dataone_registry.R

Suggest to implement support for adding dataone as a Preston remote, similar to already existing support for zenodo and software heritage.

[jhpoelen]

example request for hash://md5/e27c99a7f701dab97b7d09c467acf468

$ curl 'https://cn.dataone.org/cn/v2/query/solr/?q=checksum:e27c99a7f701dab97b7d09c467acf468&fl=identifier,size,formatId,checksum,checksumAlgorithm,replicaMN,dataUrl&rows=10&wt=json' | jq . 
{
  "responseHeader": {
    "status": 0,
    "QTime": 2,
    "params": {
      "q": "checksum:e27c99a7f701dab97b7d09c467acf468",
      "fl": "identifier,size,formatId,checksum,checksumAlgorithm,replicaMN,dataUrl",
      "rows": "10",
      "wt": "json"
    }
  },
  "response": {
    "numFound": 4,
    "start": 0,
    "docs": [
      {
        "identifier": "ess-dive-0462dff585f94f8-20180716T160600643874",
        "formatId": "text/plain",
        "size": 11036,
        "checksum": "e27c99a7f701dab97b7d09c467acf468",
        "checksumAlgorithm": "MD5",
        "replicaMN": [
          "urn:node:ESS_DIVE",
          "urn:node:UIC",
          "urn:node:KNB"
        ],
        "dataUrl": "https://cn.dataone.org/cn/v2/resolve/ess-dive-0462dff585f94f8-20180716T160600643874"
      },
      {
        "identifier": "ess-dive-642bb2e5bd48452-20180716T160320693400",
        "formatId": "text/plain",
        "size": 11036,
        "checksum": "e27c99a7f701dab97b7d09c467acf468",
        "checksumAlgorithm": "MD5",
        "replicaMN": [
          "urn:node:ESS_DIVE",
          "urn:node:UIC",
          "urn:node:KNB"
        ],
        "dataUrl": "https://cn.dataone.org/cn/v2/resolve/ess-dive-642bb2e5bd48452-20180716T160320693400"
      },
      {
        "identifier": "ess-dive-f8d52384ece0ba9-20180716T154049822990",
        "formatId": "text/plain",
        "size": 11036,
        "checksum": "e27c99a7f701dab97b7d09c467acf468",
        "checksumAlgorithm": "MD5",
        "replicaMN": [
          "urn:node:ESS_DIVE",
          "urn:node:KNB",
          "urn:node:UIC",
          "urn:node:mnORC1"
        ],
        "dataUrl": "https://cn.dataone.org/cn/v2/resolve/ess-dive-f8d52384ece0ba9-20180716T154049822990"
      },
      {
        "identifier": "ess-dive-457358fdc81d3a5-20180726T203952542",
        "formatId": "application/octet-stream",
        "size": 11036,
        "checksum": "e27c99a7f701dab97b7d09c467acf468",
        "checksumAlgorithm": "MD5",
        "replicaMN": [
          "urn:node:KNB",
          "urn:node:ESS_DIVE",
          "urn:node:UIC",
          "urn:node:mnORC1"
        ],
        "dataUrl": "https://cn.dataone.org/cn/v2/resolve/ess-dive-457358fdc81d3a5-20180726T203952542"
      }
    ]
  }
}

with

$ curl -L "https://cn.dataone.org/cn/v2/resolve/ess-dive-0462dff585f94f8-20180716T160600643874" | md5sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1614    0  1614    0     0   4458      0 --:--:-- --:--:-- --:--:--  4458
100 11036    0 11036    0     0  14040      0 --:--:-- --:--:-- --:--:-- 10.5M
e27c99a7f701dab97b7d09c467acf468  -

note the redirect.

[jhpoelen]

After implementing a first pass at supporting DataOne as a Preston remote, the following results where obtained:

$ preston cat --algo md5 --remote https://dataone.org hash://md5/e27c99a7f701dab97b7d09c467acf468 
*******************************************************************************
*** Historical CO2 Record from the Vostok Ice Core                          ***
***                                                                         ***
*** Source: J.M. Barnola                                                    ***
***         D. Raynaud                                                      ***
***         C. Lorius                                                       ***
***         Laboratoire de Glaciologie et de Geophysique de l'Environnement ***
***         38402 Saint Martin d'Heres Cedex, France                        ***
***                                                                         ***
***         N. I. Barkov                                                    ***
***         Arctic and Antarctic Research Institute                         ***
***         Beringa Street 38                                               ***
***         St. Petersburg 199226, Russia                                   ***
***                                                                         ***
*** January 2003                                                             ***
*******************************************************************************
                Mean
       Age of   age of    CO2
Depth  the ice  the air concentration
 (m)   (yr BP)  (yr BP)  (ppmv)

149.1	5679	2342	284.7
173.1	6828	3634	272.8
...
3301.4	417638	415434	286.9
3304.4	419328	417160	277.6

[jhpoelen]

@mbjones does DataOne support searching by sha256 hashes as well as md5 ?

[jhpoelen]

btw - note that zenodo also knows hash://md5/e27c99a7f701dab97b7d09c467acf468

$ preston cat --algo md5 --remote https://zenodo.org hash://md5/e27c99a7f701dab97b7d09c467acf468 
[https://zenodo.org/api/r...dab97b7d09c467acf468%22] 3 kB at 0.02 MB/s completed in < 1 minute
[https://zenodo.org/api/f...3dc2/vostok.icecore.co2] 100.0% of 10 kB at 1.17 MB/s completed in < 1 minute
*******************************************************************************
*** Historical CO2 Record from the Vostok Ice Core                          ***
***                                                                         ***
*** Source: J.M. Barnola                                                    ***
***         D. Raynaud                                                      ***
***         C. Lorius                                                       ***
***         Laboratoire de Glaciologie et de Geophysique de l'Environnement ***
***         38402 Saint Martin d'Heres Cedex, France                        ***
***                                                                         ***
***         N. I. Barkov                                                    ***
***         Arctic and Antarctic Research Institute                         ***
***         Beringa Street 38                                               ***
***         St. Petersburg 199226, Russia                                   ***
***                                                                         ***
*** January 2003                                                             ***
*******************************************************************************
                Mean
       Age of   age of    CO2
Depth  the ice  the air concentration
 (m)   (yr BP)  (yr BP)  (ppmv)

149.1	5679	2342	284.7
173.1	6828	3634	272.8
...
3301.4	417638	415434	286.9
3304.4	419328	417160	277.6

[jhpoelen]

and . . . https://softwareheritage.org knows about it too . . . but only when the sha256 content id is known. Unfortunately, as far as I can tell appears that neither Zenodo nor DataOne support sha256. . . yet. @slint @mbjones

$ preston cat --remote https://softwareheritage.org hash://sha256/9412325831dab22aeebdd674b6eb53ba6b7bdd04bb99a4dbb21ddff646287e37
[https://archive.software...dbb21ddff646287e37/raw/] 100.0% of 10 kB at 0.53 MB/s completed in < 1 minute
*******************************************************************************
*** Historical CO2 Record from the Vostok Ice Core                          ***
***                                                                         ***
*** Source: J.M. Barnola                                                    ***
***         D. Raynaud                                                      ***
***         C. Lorius                                                       ***
***         Laboratoire de Glaciologie et de Geophysique de l'Environnement ***
***         38402 Saint Martin d'Heres Cedex, France                        ***
***                                                                         ***
***         N. I. Barkov                                                    ***
***         Arctic and Antarctic Research Institute                         ***
***         Beringa Street 38                                               ***
***         St. Petersburg 199226, Russia                                   ***
***                                                                         ***
*** January 2003                                                             ***
*******************************************************************************
                Mean
       Age of   age of    CO2
Depth  the ice  the air concentration
 (m)   (yr BP)  (yr BP)  (ppmv)

149.1	5679	2342	284.7
...
3301.4	417638	415434	286.9
3304.4	419328	417160	277.6

[jhpoelen]

Also, after upgrading the https://linker.bio proxy to include both zenodo and dataone remotes (e.g., preston s --algo md5 --remotes https://zenodo.org,https://dataone.org), you can now say:

$ preston cat --algo md5 --remote https://linker.bio hash://md5/e27c99a7f701dab97b7d09c467acf468
[https://linker.bio/hash:...701dab97b7d09c467acf468] 100.0% of 10 kB at 0.75 MB/s completed in < 1 minute
*******************************************************************************
*** Historical CO2 Record from the Vostok Ice Core                          ***
***                                                                         ***
*** Source: J.M. Barnola                                                    ***
***         D. Raynaud                                                      ***
***         C. Lorius                                                       ***
***         Laboratoire de Glaciologie et de Geophysique de l'Environnement ***
***         38402 Saint Martin d'Heres Cedex, France                        ***
***                                                                         ***
***         N. I. Barkov                                                    ***
***         Arctic and Antarctic Research Institute                         ***
***         Beringa Street 38                                               ***
***         St. Petersburg 199226, Russia                                   ***
***                                                                         ***
*** January 2003                                                             ***
*******************************************************************************
                Mean
       Age of   age of    CO2
Depth  the ice  the air concentration
 (m)   (yr BP)  (yr BP)  (ppmv)

149.1	5679	2342	284.7
173.1	6828	3634	272.8
...
3304.4	419328	417160	277.6

[mbjones]

Thanks @jhpoelen !

[jhpoelen]

In addition, you can now use:

https://linker.bio/hash://md5/e27c99a7f701dab97b7d09c467acf468

to retrieve content with specified md5 hash. https://linker.bio is currently configured as a caching proxy for various repositories, including zenodo.org and dataone.org . Note that the cache is temporary and is not archived (yet ;) )

For instance,

curl --silent "https://linker.bio/hash://md5/e27c99a7f701dab97b7d09c467acf468"\
 | head

produced:

*******************************************************************************
*** Historical CO2 Record from the Vostok Ice Core                          ***
***                                                                         ***
*** Source: J.M. Barnola                                                    ***
***         D. Raynaud                                                      ***
***         C. Lorius                                                       ***
***         Laboratoire de Glaciologie et de Geophysique de l'Environnement ***
***         38402 Saint Martin d'Heres Cedex, France                        ***
***                                                                         ***
***         N. I. Barkov                                                    ***

[mbjones]

Awesome @jhpoelen . I tried it with a recent dataset we published to DataONE, and linker.bio already had it:

curl -s -o ice_202303.nc https://linker.bio/hash://md5/14a12438fe3f97776e6739914d51c62e

How are you updating the checksums from DataONE, and which algorithms are you supporting?

On a related note, we are in the process of revamping our system to be able to store multiple checksums for objects in the network (and not just the one submitted by the authoritative repo), which should improve coverage across the network without having to download and calculate all of the individual checksums. It would be good to chat about how to expose that metadata to linker.bio and similar services.

[mbjones]

@jhpoelen I just noticed your earlier questions about whether DataONE supports searching by SHA-256 -- sorry I missed that before. DataONE indexes (and audits) whichever checksum is provided by the authoritative repository for each data object, and many of them use MD5 (just from historical implementation I think). If they provide SHA-256, then it will be discoverable using the same mechanism of a SOLR query that you showed above. For example, this works today:

curl -s -o GTH83_Sonar_Final_ADC.csv https://linker.bio/hash://sha256/bd2f8004d746be0b6e2abe08e7e21474bfd5ccd855734fe971a8631de1e2bf39

As mentioned in my last comment, we're in process of adding code to systematically calculate a broader set of checksums (MD5, SHA-1, SHA-256, SHA-512, and open to suggestions) and make those accessible to the search and discovery system. I also think a new API method to retrieve an object based on it's checksum would be straightforward (e.g., hypothetically something like https://api.dataone.org/objects/hash://sha256/bd2f8004d746be0b6e2abe08e7e21474bfd5ccd855734fe971a8631de1e2bf39 would be pretty tractable, but it would have to choose which replica in the object to redirect someone to when there are multiple).