Skip to content

Using REST API

Stace Beaulieu edited this page May 21, 2019 · 51 revisions

What does "REST API" mean?

A REST API provides a set of consistent URLs that, when fetched, return data in machine-readable form that may be used by application code.

Note: The NES-LTER REST API is currently experimental and subject to change. Not all data is currently available. The number of data types will increase as we continue to develop the backend code and flesh out the API. During the development phase, the server will sometimes be taken down for software updates and so will be briefly unavailable. In addition, there may be bugs and errors encountered even when making correct requests.

Overview

The REST API provides a set of URLs from which your code can directly access conveniently-formatted NES-LTER datasets. To do this, construct a URL according to the instructions below, and use one of the provided code snippets to fetch the data.

Several data types are currently provided per cruise (list cruises available):

  • CTD casts and metadata related to them
  • Underway data
  • Station metadata
  • Event logs
  • Nutrients
  • Chlorophyll

In addition to fetching the data directly into your code, you can access the URL with your browser and download the data. The downloaded file will have a filename that distinguishes it from other similar files.

When you request data from a URL, there are two reasons why you might get a 404 "not found" response. One is if your URL is not correctly formatted according to the REST URL syntax; the other is that the requested data is not available.

URL syntax

URLs begin with a standard URL prefix followed by a data type indicator followed by the cruise ID followed by parameters including a final filename. For example, to fetch CTD cast 8 from en608, use the following URL:

https://nes-lter-data.whoi.edu/api/ctd/en608/cast_8.csv

Example URLs for all data types are given below.

CTD URLs

CTD metadata provides a list of casts for a given cruise along with metadata about each cast (e.g., time, lat/lon, nearest station). For example:

https://nes-lter-data.whoi.edu/api/ctd/en617/metadata.csv

CTD bottle data provides a list of niskins along with data from the CTD as it has been provided by the SeaBird CTD processing software. For example:

https://nes-lter-data.whoi.edu/api/ctd/en608/bottles.csv

CTD bottle summary data provides a concise summary of when and where each bottle was fired. For example:

https://nes-lter-data.whoi.edu/api/ctd/en627/bottle_summary.csv

CTD profiles for each cast are available at URLs like:

https://nes-lter-data.whoi.edu/api/ctd/en617/cast_12.csv

Underway URL pattern

Underway 1-minute data is provided per-cruise. Use a URL based on the following pattern:

https://nes-lter-data.whoi.edu/api/underway/ar31a.csv

Station metadata URL pattern

Station metadata briefly describes each station along with pertinent information like lat/lon. Use the following URL pattern:

https://nes-lter-data.whoi.edu/api/stations/en617.csv

Station metadata is provided per-cruise because the stations may change from cruise to cruise.

Event log URLs

Event logs are available per-cruise. The event logs are corrected to align with the other products, including correction of timestamps and positions.

For example:

https://nes-lter-data.whoi.edu/api/events/en627.csv

Nutrients and Chlorophyll URLs

Use the pattern as above for underway/stations/events, but replace with nut or chl.

Code snippets

The following code snippets show how to read NES-LTER data into your preferred language using the REST API. To use each example, change the URL to the URL for whichever dataset you want.

If the URL pattern is incorrect or no data is available, the server will respond with HTTP 404, "not found".

MATLAB

To read data from one of these URLs from MATLAB, use the following two lines of code.

options = weboptions('ContentType', 'table');
mytable = webread('https://nes-lter-data.whoi.edu/api/ctd/en608/cast_7.csv', options);

The result is a MATLAB table.

Python

Use Pandas read_csv:

import pandas as pd

pd.read_csv('https://nes-lter-data.whoi.edu/api/ctd/en608/cast_7.csv')

R

In R, use read.csv:

read.csv('https://nes-lter-data.whoi.edu/api/ctd/en608/cast_7.csv')

Parsing dates and times

When possible, dates and times are provided in ISO 8601 format in the UTC timezone. Using those fields typically requires additional parsing, and default rules for parsing dates and times in tools such as Excel and MATLAB will often fail or produce unexpected results because of limitations of those tools. For some datatypes, date and/or time are provided in separate columns in various other formats (e.g., seconds since the beginning of a CTD cast).

Here are some code snippets for parsing ISO 8601 formatted timestamps:

MATLAB

iso8601format = 'yyyy-mm-dd hh:MM:ss'
mydatenum = datenum('2019-02-22 12:34:56+00:00', iso8601format)

Python

Use Pandas to_datetime:

import pandas as pd

mydatetime = pd.to_datetime('2019-02-22 12:34:56+00:00', utc=True)

R

iso8601format <- "%Y-%m-%d %H:%M:%S"
mydatetime <- strptime("2019-02-22 12:34:56+00:00", iso8601format, tz="UTC")

Excel

The following formula will convert an ISO 8601 timestamp to an Excel timestamp value, see this answer for details.

=DATEVALUE(MID(A2,1,10))+TIMEVALUE(MID(A2,12,8))

For developers

REST endpoints respond with JSON if no extension is specified or if the extension provided is "json". The JSON format is the one provided by Pandas to_json.

Some JSON-only endpoints are/will be included. For example this endpoint returns a list of CTD cast numbers for en608:

https://nes-lter-data.whoi.edu/api/ctd/en608/casts

And this endpoint returns a list of cruises:

https://nes-lter-data.whoi.edu/api/cruises

Clone this wiki locally
You can’t perform that action at this time.