How to download data objects from a Solr search result?

[robyngit]

This question came form a DataONE community member recently:

We are trying to download DataONE datasets using dataUrl returned from search results but when making a get request to the dataUrl, we always obtain an EML metadata file or a request timeout... How can I download dataset files (csv, tsv, ...) from a result item?

[robyngit]

The dataUrl field defines the "URL that can be used to resolve the location of the object given its PID," where object can refer to either a metadata document (like EML), a resource map, or a data object (like a CSV). The DataONE Solr index is populated from metadata (e.g. EML) and resource maps, not the content of data files. Most common queries to our index search for text that is contained within the metadata document, and so the associated dataUrl in the search result gives the URL to the metadata object, not the associated data files.

For example, the query title:*jellyfish* means: "find all documents where the title element contains the term 'jellyfish'". Only metadata objects have a title element, so only objects with the format type "metadata" are returned. You can see for yourself that data objects do not have a populated title field using the query formatType:DATA AND title:*, which gives 0 results.

To find the URL for the data files that a particular metadata object documents, we can add another field to our results list: documents. The documents field "lists all PIDs that this object describes." So we search the metadata for our search term, then get the list of IDs for the files that the metadata describes. We can then do another query with these IDs to get the dataUrls, or we can construct the data URL ourselves. Below is an example.

Example

Goal: Find the URLs for all data objects that have something to do with jellyfish

The query: text:*jellyfish* AND documents:* AND -obsoletedBy:*

• text:*jellyfish* - means find all the metadata where any text contains the term jellyfish (the text field is only populated for objects with the metadata format type)
• documents:* - means "only include metadata that has associated data files", i.e. only metadata that documents something. (Sometimes metadata can exist without the data files being available)
• -obsoletedBy:* - this is added on so that we only get the most recent version of a metadata object in our results, and exclude all older versions. It means, "only include metadata objects that have not yet been obsoleted by a newer metadata object"

The fields to return: title,documents

• title - The title of the data package, as indicated in the metadata object
• documents - The list of IDs for any data objects that this metadata object documents

URL: https://cn.dataone.org/cn/v2/query/solr/?q=text:*jellyfish* AND documents:* AND -obsoletedBy:*&fl=title,documents

Example result:

<doc>
  <str name="title">Jellyfish and forage fish catch data, Bering Sea, 2004-2012</str>
  <arr name="documents">
    <str>45b9581c-ef1a-4aed-9028-b5069fe5d5ee</str>
    <str>28c5a386-ff2f-4e24-9264-fe04c76b7a88</str>
  </arr>
</doc>

The result shows that the "Jellyfish and forage fish catch" data package comprises two data files with the IDs 45b9581c-ef1a-4aed-9028-b5069fe5d5ee and 28c5a386-ff2f-4e24-9264-fe04c76b7a88

Find the URL for one of the data files

We can now use the ID of each data file to find more information about it, including the URL, using another Solr query.

For example, the query... q=id:"28c5a386-ff2f-4e24-9264-fe04c76b7a88"&fl=dataUrl,formatType,formatId,size ... gives the result:

<doc>
  <str name="formatId">text/csv</str>
  <str name="formatType">DATA</str>
  <long name="size">67463</long>
  <str name="dataUrl">https://cn.dataone.org/cn/v2/resolve/28c5a386-ff2f-4e24-9264-fe04c76b7a88</str>
</doc>

Showing that this object is a 67KB CSV data file which can be downloaded at https://cn.dataone.org/cn/v2/resolve/28c5a386-ff2f-4e24-9264-fe04c76b7a88.

Construct the URL instead

If we just want to download the data right away without doing a second query, then we can construct the dataUrl ourselves, by attaching the base URL https://cn.dataone.org/cn/v2/resolve/ to the ID of the object, 28c5a386-ff2f-4e24-9264-fe04c76b7a88

More information:

• Detailed documentation about the DataONE Solr index
• List of DataONE Solr index fields
• The resolve method in the DataONE API
• Searching DataONE's Solr index with R