Skip to content

Latest commit

 

History

18 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

EMODnet Biology web service documentation

EMODnet Biology brings together marine biodiversity data from a large network of European and international data providers. This page explains how to access these data in a machine-readable way: through OGC web services, through R packages, and as parquet files in the EDITO data lake.

Where to start

  • EMODnet catalogue: every EMODnet Biology dataset is described by a metadata record in the EMODnet catalogue. This is the best entry point to find datasets. Each record gives the title, abstract, citation, licence and access links of a dataset, together with its dataset identifier (datasetid), which you can use to filter the web services described below. Record links follow the pattern https://emodnet.ec.europa.eu/geonetwork/srv/eng/catalog.search#/metadata/6d617269-6e65-696e-666f-<datasetid padded with zeros to 12 digits>, for example dataset 4659.
  • EMODnet Viewer: the EMODnet Viewer lets you explore, filter and download EMODnet Biology occurrence data on a map.
  • EMODnet Viewer
  • Web services: the OGC web services (WFS, WMS) give direct access to the same data from scripts and GIS software. They are explained in detail on this page. An overview of all EMODnet Biology service endpoints is available in the EMODnet web service documentation.
  • EDITO data lake: the complete occurrence data are also published as parquet files in the European Digital Twin of the Ocean (EDITO).
  • EurOBIS: EMODnet Biology occurrence data are integrated and served through EurOBIS, hosted by the Flanders Marine Institute (VLIZ). The EurOBIS Download Toolbox offers the same data with an interactive selection interface.

EMODnet Biology occurrence data (WFS)

The EMODnet Biology occurrence data are available as a Web Feature Service (WFS) following the Open Geospatial Consortium (OGC) specifications. A WFS request returns geographical features (vector geometries with their attributes). The base link for a WFS request to EMODnet Biology is:

https://geo.vliz.be/geoserver/Dataportal/ows?

You do not need to write requests by hand. The EurOBIS Download Toolbox works as an interactive builder for WFS requests: at the last step of the selection, the URL of the WFS request can be copied to the clipboard.

The technical details of the service, including the available layers and output formats, can be retrieved with a GetCapabilities request:

https://geo.vliz.be/geoserver/Dataportal/ows?service=wfs&version=1.1.0&request=GetCapabilities


Building a WFS request

A WFS request is composed of four parts, joined with &:

  1. the base URL
  2. the data format
  3. filtering options (optional)
  4. the output format

1. The base URL

https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&

All examples on this page use WFS version 1.0.0, where the number of returned records is limited with maxFeatures. Requests for the Basic and Full Occurrence Data should always contain at least one filter, for example a dataset or a taxon (see Filtering options). Without a filter, these requests can take too long to complete, even with maxFeatures. One request returns at most 1,000,000 records. For larger downloads, split the request (for example by dataset) or use the parquet files in EDITO.

2. The data format

The data format is selected with the typeName parameter:

typeName=Dataportal:<data_format>

EMODnet Biology offers the following data formats. The terms returned by each format are listed in the Data format section of that page.

Data format typeName
Basic Occurrence Data eurobis-obisenv_basic
Full Occurrence Data eurobis-obisenv_full
Full Occurrence Data and Parameters eurobis-obisenv
DNA derived data eurobis-dna (see Genomics data)
DNA derived data without parameters eurobis-dna_full (see Genomics data)
  • Basic Occurrence Data

    The Basic Occurrence Data contain everything needed for spatial and temporal analyses of taxa: which taxon was found (scientificname and aphiaid), when (datecollected) and where (decimallongitude and decimallatitude in WGS84, EPSG:4326), together with a dataset identifier (datasetid). The Basic Occurrence Data have no geometry column: a Shapefile cannot be created and KML files contain no positions. Use the Full Occurrence Data for these output formats.

    This request returns the first 50 basic occurrence records of the dataset Monitoring of birds in the Voordelta (datasetid 4659):

    https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-obisenv_basic&viewParams=datasetid:4659&maxFeatures=50&outputformat=csv
    

    Open this request

  • Full Occurrence Data

    The Full Occurrence Data contain all fields of the Basic Occurrence Data plus additional information that helps to interpret them, such as the institute that collected the data, the sampling methodology, the exact time and location, and their uncertainty.

    This request returns the first 50 full occurrence records of the dataset Monitoring of birds in the Voordelta (datasetid 4659):

    https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-obisenv_full&viewParams=datasetid:4659&maxFeatures=50&outputformat=csv
    

    Open this request

  • Full Occurrence Data and Parameters

    The Full Occurrence Data and Parameters add all measurements or facts associated with the occurrence or the sampling event (for example abundance, biomass, sampling gear or environmental variables). Each measurement is returned as a separate row, so one occurrence can appear on several rows.

    This request returns the first 50 records of the dataset Monitoring of birds in the Voordelta (datasetid 4659):

    https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-obisenv&viewParams=datasetid:4659&maxFeatures=50&outputformat=csv
    

    Open this request

3. Filtering options

Filters are passed with the viewParams parameter. Each filter has the form name:value, and several filters are combined with a semicolon:

viewParams=<filter_1>;<filter_2>

The following filters are available:

Filter Syntax Example
Dataset datasetid:<datasetid> datasetid:4659
Taxon (WoRMS AphiaID) aphiaid:<AphiaID> aphiaid:137138
Several taxa aphiaid:<AphiaID>\,<AphiaID> aphiaid:141433\,140733
Absence records includeAbsences:1 see Absence data
Marine region (MRGID) bounds:geoid && ARRAY[<MRGID>] bounds:geoid && ARRAY[3293]

The dataset, taxon and geography filters link EMODnet Biology to other services, which are described at the end of this page.

The where: parameter used in older examples is still accepted, but the dedicated filters in the table above are recommended wherever they are available.

Filter names are not checked. A misspelled or unknown filter is ignored without an error message, and the request then returns data without that filter.

  • Dataset

    Every dataset has a unique datasetid, which is shown in its record in the EMODnet catalogue. Behind the catalogue, the metadata are managed in the Integrated Marine Information System (IMIS) of the Flanders Marine Institute (VLIZ).

    The data format examples above already use the dataset filter. This request returns the first 50 full occurrence records of the dataset LifeWatch observatory data: permanent Cetacean passive acoustic sensor network in the Belgian part of the North Sea (datasetid 5531) as a CSV file:

    https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-obisenv_full&viewParams=datasetid:5531&maxFeatures=50&outputformat=csv
    

    Open this request

  • Taxonomy: World Register of Marine Species (WoRMS)

    EMODnet Biology data are linked to the World Register of Marine Species (WoRMS) through the AphiaID. The output contains both the AphiaID of the name as provided in the source dataset (aphiaid) and the AphiaID of the currently accepted name (aphiaidaccepted).

    This request returns the full occurrence data of the bivalve Ensis ensis (AphiaID 140733) as a CSV file:

    https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-obisenv_full&viewParams=aphiaid:140733&outputformat=csv
    

    Open this request

    The taxon filter works on the accepted name in WoRMS. It also returns records that were published under a synonym, and records of all lower taxa. For example, aphiaid:137043 (the genus Larus) returns records of Larus argentatus, Larus fuscus, Larus marinus and other gull species.

    Several taxa can be requested at once by separating the AphiaIDs with an escaped comma (\,, or %5C, in a URL). This request returns the full occurrence data and parameters of the bivalves Abra alba (AphiaID 141433) and Ensis ensis (AphiaID 140733):

    https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-obisenv&viewParams=aphiaid:141433\,140733&outputformat=csv
    

    Open this request

  • Geography: Marine Regions

    EMODnet Biology data can be queried for standardised areas using the MRGID of MarineRegions.org, a unique and persistent identifier for geographic objects. The easiest way to build such a request is to select the area in the EurOBIS Download Toolbox and copy the generated WFS request. You can currently select Exclusive Economic Zones (EEZ), IHO Sea Areas, Marine Ecoregions of the World (MEOW), Marine Regions and Territorial Seas.

    The area filter contains spaces and &&. In a URL, write the spaces as %20 and && as %26%26, otherwise the & characters split the request.

    This request returns the first 50 basic occurrence records in the Belgian Exclusive Economic Zone (MRGID 3293):

    https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-obisenv_basic&viewParams=bounds:geoid && ARRAY[3293]&maxFeatures=50&outputformat=csv
    

    Open this request

  • Combining filters

    Filters are combined with a semicolon. This request returns the Herring gull records (AphiaID 137138) from the dataset Monitoring of birds in the Voordelta (datasetid 4659) as JSON:

    https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-obisenv_full&viewParams=datasetid:4659;aphiaid:137138&outputformat=application/json
    

    Open this request

4. The output format

EMODnet Biology data are available in several output formats, set at the end of the request:

outputFormat=<output_format>
Output format Value Example
CSV csv first 50 records
GeoJSON application/json first 50 records
Shapefile (zipped) SHAPE-ZIP first 50 records
KML application/vnd.google-earth.kml+xml first 50 records

The complete list of output formats for each layer is returned by the GetCapabilities request.


Absence data

Many monitoring and survey datasets record not only where a taxon was found, but also where it was looked for and not found. These absence records are published in EMODnet Biology with the Darwin Core term occurrenceStatus, which has the value present or absent.

To avoid that existing scripts silently count absences as presences, absence records are not returned by default. You can include them by adding includeAbsences:1 to the viewParams (the default is includeAbsences:0). This works for all occurrence data formats (eurobis-obisenv_basic, eurobis-obisenv_full, eurobis-obisenv, eurobis-dna and eurobis-dna_full). The output then contains the column occurrenceStatus.

The EMODnet Viewer offers the same choice in the layer filter Presence/absence records: only presences (default), only absences, or presences and absences.

A good example is the dataset LifeWatch observatory data: permanent Cetacean passive acoustic sensor network in the Belgian part of the North Sea (datasetid 5531), in which a network of acoustic sensors registers whether harbour porpoises (Phocoena phocoena) are detected. This request returns the first 50 presence and absence records of this dataset:

https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-obisenv_basic&viewParams=datasetid:5531;includeAbsences:1&maxFeatures=50&outputformat=csv

Open this request

To get only the absence records, add a CQL filter on occurrenceStatus (written occurrenceStatus%3D%27absent%27 in a URL):

https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-obisenv_basic&viewParams=datasetid:5531;includeAbsences:1&CQL_FILTER=occurrenceStatus='absent'&maxFeatures=50&outputformat=csv

Open this request

Please keep in mind:

  • The absences are observed absences as reported by the data provider: the taxon was looked for during a sampling event but not detected. Modelled pseudo-absences are not included.
  • An absence is only meaningful together with the sampling context. The Full Occurrence Data and Parameters (eurobis-obisenv) contain the sampling protocol, effort and other event information that is needed to interpret them.

Genomics data (DNA derived occurrences)

EMODnet Biology also publishes occurrences based on DNA. Two types are distinguished:

  • DNA derived occurrences, where the DNA sequence is the only evidence of the occurrence, for example in metabarcoding studies.
  • Enriched occurrences, where DNA adds evidence to a specimen or an observation that was identified in another way.

These data follow the Darwin Core DNA derived data extension. Guidance on how to format them is available in Publishing DNA-derived data through biodiversity data platforms.

The DNA data are served in the layer eurobis-dna. Next to the full occurrence data and parameters, it contains the DNA specific fields:

Field Content
dnaderived true for DNA derived occurrences, false for enriched occurrences
dna_sequence the DNA sequence (for example of the ASV or OTU)
target_gene, target_subfragment the targeted gene or marker and subfragment
seq_meth sequencing method
otu_class_appr, otu_db approach and reference database used to cluster and classify the sequences
sop standard operating procedures used
dna_key, dna_value all other terms of the DNA derived data extension (for example PCR primers), one term per row
associatedsequences reference to the raw sequence data in a sequence archive, such as the European Nucleotide Archive

Because the additional DNA terms (dna_key, dna_value) and the measurements (parameter, parameter_value) are returned in a long format, one occurrence appears on several rows. Use occurrenceid to group the rows of one occurrence.

The layer eurobis-dna_full contains the same DNA fields, but no measurements (parameter columns) and fewer occurrence fields (for example no associatedsequences). It therefore returns fewer rows per occurrence.

This request returns the first 50 records of the dataset ARMS-MBON data on long-term monitoring of hard-bottom communities: COI results from 2018-2020 (datasetid 8357):

https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-dna&viewParams=datasetid:8357&maxFeatures=50&outputformat=csv

Open this request


EMODnet Biology data as parquet files in EDITO

The complete EMODnet Biology occurrence data are also published in the data lake of the European Digital Twin of the Ocean (EDITO) as Apache Parquet files. Parquet is a column-based file format: tools such as DuckDB or Arrow read only the columns and rows that a query needs, directly over the internet, so you can work with the full EMODnet Biology data without downloading them first or building WFS requests.

The data are split into three files:

File Content
occurrence occurrence records
measurement_or_fact measurements or facts linked to the occurrences
dna DNA derived data linked to the occurrences

The files measurement_or_fact and dna refer to the occurrence records through the column occurrence_ref, which corresponds to the column id in the occurrence file. All three files also contain the dataset identifier IMISDatasetId.

Unlike the web services, the occurrence file contains both presence and absence records. Use the column occurrenceStatus to select them.

The files are listed in the EDITO catalogue item EMODnet Biology occurrences (collection emodnet-occurrence_data), where each file can also be opened in the EDITO Data Explorer. They can be browsed on a map in the EDITO viewer.

The files are refreshed after each data publication and their names contain the date of the release. It is therefore better to read the current file location from the catalogue item than to copy a file link into your scripts, as in the R example below.

library(jsonlite)
library(DBI)
library(duckdb)

# Get the current location of the occurrence file from the EDITO catalogue
item <- fromJSON("https://api.dive.edito.eu/data/collections/emodnet-occurrence_data/items/9f72de0c-e45a-53e3-9bdc-9f1d0a15618f")
occurrence_url <- item$assets$parquet_occurrence$href

con <- dbConnect(duckdb())
dbExecute(con, "INSTALL httpfs")
dbExecute(con, "LOAD httpfs")

# List the available columns
dbGetQuery(con, sprintf("DESCRIBE SELECT * FROM read_parquet('%s')", occurrence_url))

# Read selected columns of the Herring gull presence records
herring_gull <- dbGetQuery(con, sprintf(
  "SELECT occurrenceID, IMISDatasetId, eventDate, decimalLongitude, decimalLatitude
   FROM read_parquet('%s')
   WHERE scientificName = 'Larus argentatus' AND occurrenceStatus = 'present'",
  occurrence_url
))

dbDisconnect(con, shutdown = TRUE)

The links to the other two files are available in the same way through item$assets$parquet_measurement_or_fact$href and item$assets$parquet_dna$href.


EMODnet Biology summary and data product services

Next to the occurrence data, the EMODnet Biology web services can return the number of occurrences for a query, the occurrences summarised in geospatial grids, and the data products developed by the EMODnet Biology community.

  • Number of occurrences

    The layer Dataportal:eurobis-obisenv_count returns the total number of records for a query. This number includes the absence records, also when includeAbsences:1 is not set. This request returns the number of records in the dataset Monitoring of birds in the Voordelta (datasetid 4659):

    https://geo.vliz.be/geoserver/Dataportal/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=Dataportal:eurobis-obisenv_count&viewParams=datasetid:4659&outputformat=application/json
    

    Open this request

  • Occurrences summarised in a geospatial grid

    The number of occurrences is also available per grid cell, at four grid resolutions, and as a layer of individual points. Select the layer with typeName:

    Grid typeName
    1 degree Dataportal:eurobis_grid_1d-obisenv
    30 minutes Dataportal:eurobis_grid_30m-obisenv
    15 minutes Dataportal:eurobis_grid_15m-obisenv
    6 minutes Dataportal:eurobis_grid_6m-obisenv
    Points Dataportal:eurobis_points-obisenv

    This request returns the number of Herring gull records (AphiaID 137138) in a 30 minutes grid as KML:

    https://geo.vliz.be/geoserver/wfs/ows?service=WFS&version=1.1.0&request=GetFeature&typeName=Dataportal:eurobis_grid_30m-obisenv&viewParams=aphiaid:137138&outputFormat=application/vnd.google-earth.kml+xml
    

    Open this request

  • Data products

    Data products developed by EMODnet Biology are described in the EMODnet catalogue and are available through OGC web services. For example, the following Web Map Service (WMS) requests show the OOPS Copepod gridded abundances in 10-year and 1-year bins:

    Gridded data products are also available on the EMODnet ERDDAP server.


Other marine data systems connected to EMODnet Biology

EMODnet Biology follows the FAIR principles. Its data formats improve interoperability by linking to controlled vocabularies and registers such as the World Register of Marine Species (WoRMS), MarineRegions.org and the vocabularies of the British Oceanographic Data Centre (BODC). Dataset metadata are managed in the Integrated Marine Information System (IMIS) of the Flanders Marine Institute (VLIZ), where information on datasets, people, organisations and publications is stored.

The web services of these systems can be combined with the EMODnet Biology services.


R clients for EMODnet Biology web services

EMODnet Biology data can be accessed in R with the following packages:

Practical examples that combine these packages are available in the EMODnet Biology Geospatial R Tutorials.

Some applications built with EMODnet Biology data in R Shiny:

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

2 watching

Forks

Releases

Packages

Used by

Contributors