Data queries

Xavier Sosnovsky edited this page Dec 22, 2017 · 11 revisions

Overview

All the data disseminated in a repository compliant with the SDMX RESTful web services can be retrieved using the following syntax:

protocol://wsEntryPoint/resource/flowRef/key/providerRef?startPeriod=value&endPeriod=value&updatedAfter=value&firstNObservations=value&lastNObservations=value&detail=value&includeHistory=value

  • protocol: RESTful web services are typically available over http and https.
  • wsEntryPoint: The web service entry point indicates the root of the web service.
  • resource: The resource for data queries is data. The resource for metadata queries is metadata.

Path parameters

The flowRef path parameter: Defining the dataflow reference

The flowRef parameter represents a reference to the dataflow describing the data that needs to be returned. It is mandatory.

The syntax is the identifier of the agency maintaining the dataflow, followed by the identifier of the dataflow, followed by the dataflow version, separated by a comma: AGENCY_ID,FLOW_ID,VERSION

If the parameter contains only one of these 3 elements, it is considered to be the identifier of the dataflow. The value for the identifier of the agency maintaining the dataflow will default to all, while the value for the dataflow version will default to latest.

If the string contains only two of these 3 elements, they are considered to be the identifier of the agency maintaining the dataflow and the identifier of the dataflow. The value for the dataflow version will default to latest.

In order to see all dataflows available in the target web service, a metadata query for all dataflows can be performed:

http://ws-entry-point/dataflow

The key path parameter: Defining the dimension values of the data to be returned

As explained below, the combination of dimensions allows statistical data to be uniquely identified. Such a combination is known as a series key in SDMX and this is what is needed in the key parameter. This parameter is optional and defaults to all when left empty.

Let's say for example that exchange rates can be uniquely identified by the frequency at which they are measured (e.g.: on a daily basis - code D), the currency being measured (e.g.: US dollar - code USD), the currency against which a currency is being measured (e.g.: Euro - code EUR), the type of exchange rates (Foreign exchange reference rates - code SP00) and the series variation (such as average or standardised measure for given frequency, code A). In order to build the series key, you need to take the value for each of the dimensions (in the order in which the dimensions are defined in the DSD) and join them with a dot. The series key for the example above therefore becomes: D.USD.EUR.SP00.A

Wildcarding is supported by omitting the value for the dimension to be wildcarded. For example, the following series key can be used to retrieve the data for all daily currencies against the euro: D..EUR.SP00.A

The OR operator is supported using the + character. For example, the following key can be used to retrieve the exchange rates against the euro for both the US dollar and the Japanese Yen: D.USD+JPY.EUR.SP00.A

You can of course combine wildcarding and the OR operator. For example, the following key can be used to retrieve daily or monthly exchange rates of any currency against the euro: D+M..EUR.SP00.A

The providerRef path parameter: Defining who supplies the data

This parameter allows you to define who supplies the data you are interested in. The parameter is optional and defaults to all when left empty.

Query parameters

The startPeriod and endPeriod query parameters: Defining a date range

It is possible to define a date range for which observations should be returned by using the startPeriod and/or endPeriod parameters. The values should be given according to the syntax defined in ISO 8601 or as SDMX reporting periods. The values will vary depending on the frequency. The supported formats are:

  • YYYY for annual data (e.g.: 2013).
  • YYYY-S[1-2] for semi-annual data (e.g.: 2013-S1).
  • YYYY-Q[1-4] for quarterly data (e.g.: 2013-Q1).
  • YYYY-MM for monthly data (e.g.: 2013-01).
  • YYYY-W[01-53] for weekly data (e.g.: 2013-W01).
  • YYYY-MM-DD for daily and business data (e.g.: 2013-01-01).

In case data of varying frequencies are requested and the values set for the start and end periods are not in the format of the most granular frequency, it is necessary, for the service provider, to interpret the values of start and end periods using the most granular frequency. For example, in case daily, monthly and annual data are requested, and the values for the start and end periods are 2016 and 2017 respectively, these values should be interpreted by the service provider, as being equivalent to 2016-01-01 and 2017-12-31 respectively.

The updatedAfter query parameter: Retrieving deltas

By supplying a percent-encoded timestamp to the updatedAfter parameter, it is possible to only retrieve the latest version of changed values in the database since a certain point in time (so-called updates and revisions or deltas).

The response to such a query could include one or more dataset(s) representing:

  1. The observations that have been added since the last time the query was performed.
  2. The observations that have been revised since the last time the query was performed.
  3. The observations that have been deleted since the last time the query was performed.

An example of the above can be seen when querying the ECB's web services and asking for the deltas using the following parameters:

  • updatedAfter=2014-11-01T00%3A00%3A00%2B01%3A00 the percent-encoded representation of 2014-11-01T00:00:00+01:00 as the point in time of our last retrieval of data

In this example, cases 1 & 3 can be observed in the following response message:

<?xml version="1.0" encoding="UTF-8"?>
<message:GenericData xmlns:message="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message" xmlns:common="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/common" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:generic="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/data/generic" xsi:schemaLocation="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message http://sdw-wsrest.ecb.europa.eu:80/vocabulary/sdmx/2_1/SDMXMessage.xsd http://www.sdmx.org/resources/sdmxml/schemas/v2_1/common http://sdw-wsrest.ecb.europa.eu:80/vocabulary/sdmx/2_1/SDMXCommon.xsd http://www.sdmx.org/resources/sdmxml/schemas/v2_1/data/generic http://sdw-wsrest.ecb.europa.eu:80/vocabulary/sdmx/2_1/SDMXDataGeneric.xsd">
    <message:Header>
        <message:ID>388d1c9a-d187-4f6a-8792-e117cf34047f</message:ID>
        <message:Test>false</message:Test>
        <message:Prepared>2016-12-20T16:19:56.398+01:00</message:Prepared>
        <message:Sender id="ECB"/>
        <message:Structure structureID="ECB_RTD1" dimensionAtObservation="TIME_PERIOD">
            <common:Structure>
                <URN>urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=ECB:ECB_RTD1(1.0)</URN>
            </common:Structure>
        </message:Structure>
    </message:Header>
    <message:DataSet action="Replace" validFromDate="2016-12-20T16:19:56.398+01:00" structureRef="ECB_RTD1">[...]</message:DataSet>
    <message:DataSet action="Delete" validToDate="2016-12-20T16:19:56.440+01:00" structureRef="ECB_RTD1">[...]</message:DataSet>
</message:GenericData>

In the response the action attribute of the Dataset element is of importance whereas the validFromDate is just there for information purposes.

Developers who update their local databases should make use of the updatedAfter parameter as it is likely to significantly improve performance. Instead of systematically downloading data that may not have changed, you would only receive the consolidated changes to be made in your database since the last time your client performed the same query.

The detail query parameter: Defining the amount of details

Using the detail parameter, it is possible to specify the desired amount of information to be returned by the web service. Possible options are:

  • full: The data - series and observations - and the attributes will be returned. This is the default.
  • dataonly: The attributes will be excluded from the returned message.
  • serieskeysonly: Only the series, but without the attributes and the observations, will be returned. This can be useful for performance reasons, to return the series that match a certain query, without returning the actual data.
  • nodata: The series, including the attributes, will be returned, but the observations will not be returned.

The firstNObservations and lastNObservations query parameters: Defining the number of observations to be returned

Using the firstNObservations and/or lastNObservations parameters, it is possible to specify the maximum number of observations to be returned for each of the matching series, starting from the first observation (firstNObservations) or counting back from the most recent observation (lastNObservations). This can be useful for building an overview page, for example, where, for each indicator, you only display 2 values (the current one and the previous one).

Using the dimensionAtObservation query parameter: Defining the way the data should be organized in the returned message

Using the dimensionAtObservation parameter, you can define the way the data should be organized in the returned message. Possible options are:

  • TIME_PERIOD: This will return a timeseries view of the data. This is the default value.
  • AllDimensions: This will return a flat view of the data, with no groupings.
  • The ID of any other dimension: This will return a cross-sectional view of the data.

Using the includeHistory query parameter: Retrieving how a time series evolved over time

Using the includeHistory parameter, you can instruct the web service to return previous versions of the matching data. This is useful to see how the data have evolved over time, i.e. when new data have been released or when data have been revised or deleted. Possible options are:

  • false: Only the version currently in production will be returned. This is the default.
  • true: The version currently in production, as well as all previous versions, will be returned.

Such a query would give you the history and not the deltas between that point in time and today.

Here is an example response message:

<?xml version="1.0" encoding="UTF-8"?>
<message:GenericData xmlns:message="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message" xmlns:common="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/common" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:generic="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/data/generic" xsi:schemaLocation="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message http://sdw-wsrest.ecb.europa.eu:80/vocabulary/sdmx/2_1/SDMXMessage.xsd http://www.sdmx.org/resources/sdmxml/schemas/v2_1/common http://sdw-wsrest.ecb.europa.eu:80/vocabulary/sdmx/2_1/SDMXCommon.xsd http://www.sdmx.org/resources/sdmxml/schemas/v2_1/data/generic http://sdw-wsrest.ecb.europa.eu:80/vocabulary/sdmx/2_1/SDMXDataGeneric.xsd">
    <message:Header>
        <message:ID>a2c99026-00c8-4f9a-a3d4-1891f89bd2b0</message:ID>
        <message:Test>false</message:Test>
        <message:Prepared>2016-12-20T17:16:51.578+01:00</message:Prepared>
        <message:Sender id="ECB"/>
        <message:Structure structureID="ECB_RTD1" dimensionAtObservation="TIME_PERIOD">
            <common:Structure>
                <URN>urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=ECB:ECB_RTD1(1.0)</URN>
            </common:Structure>
        </message:Structure>
    </message:Header>
    <message:DataSet action="Replace" validFromDate="2014-12-03T15:30:00.000+01:00" structureRef="ECB_RTD1">[...]</message:DataSet>
    <message:DataSet action="Replace" validFromDate="2015-06-02T15:30:00.000+02:00" structureRef="ECB_RTD1">[...]</message:DataSet>
    <message:DataSet action="Replace" validFromDate="2015-10-21T15:30:00.000+02:00" structureRef="ECB_RTD1">[...]</message:DataSet>
    <message:DataSet action="Replace" validFromDate="2016-06-01T15:30:00.000+02:00" structureRef="ECB_RTD1">[...]</message:DataSet>
    <message:DataSet action="Delete" validToDate="2014-11-05T15:30:00.000+01:00" structureRef="ECB_RTD1">[...]</message:DataSet>
</message:GenericData>

In the response the action & the validFromDate attributes of the Dataset element are both equally important. While the action attribute indicates what needs to be done, the validFromDate attribute allows defining the validity periods of the reported data.

Examples

Note: The examples on this page use the test services provided by the ECB and are therefore not suitable for production use. To retrieve data and metadata from the production instance, please use the following resource instead.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.