Allow slicing of data messages

[dosse]

Today, the SDMX Rest ws standard foresees one way for restricting number of observations through the firstNObservations and lastNObservations URL parameters. But those apply individually to each "series" and not to the whole data message, and are thus not useful when the client needs to restrict the message size, or for chunking the request into smaller pieces that are better manageable by the client.

A nice solution would be provided by the HTTP protocol that includes a way to request only a specific range of data. The related RFC 7233 standard says:

The "Range" header field on a GET request modifies the method semantics to request transfer of only one or more subranges of the selected representation data, rather than the entire selected representation data.

It works in the following way:

1. The server indicates with the HTTP header field "Accept-Ranges" that it supports range requests for the target resource, and which types of range units, e.g. bytes.
   Accept-Ranges: bytes

2. The client uses the "Range" header field in the GET request to indicate the requested subrange, e.g. for the first 1000 bytes (with 0-based indexes):
   Range: bytes=0-999
   And for the next 1000 bytes:
   Range: bytes=1000-1999

3. The server, if the range request is admissible, returns the 206 (Partial Content) status code that indicates that the server is successfully fulfilling a range request for the target resource by transferring one or more parts of the selected representation that correspond to the satisfiable ranges found in the request's "Range" header field. It also returns a "Content-Range" header field, describing what range of the selected representation is enclosed, e.g.
   Content-Range: bytes 0-999/1234
   Note: 1234 is the total number of bytes for the complete resource, or
   Content-Range: bytes 0-999/*
   Note: * is indicating that the total number of bytes for the complete resource is unknown by the server, or
   Content-Range: bytes 0-566/567
   Alternatively, the server generates a 416 (Range Not Satisfiable) response and a "Content-Range" header field with an unsatisfied-range value "*", e.g.
   Content-Range: bytes */1234

Also see here and here, for examples of requests for multiple sub-ranges.

However

Since the bytes unit is inappropriate for our purposes, in .Stat Suite we have tested the HTTP Range header approach with another range unit: values. While this works fine in most environments, some users have reported issues in AWS hosting scenarios. Indeed, some cloud hosting services do not allow using range units that have not been registered with IANA,

We therefore decided to use a non-standard/proprietary X-Range HTTP header instead of Range. This worked in all environments. The specification we have used is as follow:

1. The server indicates with the HTTP header field "Accept-Ranges" that it supports range requests for the target resource and the range unit values.:
   Accept-Ranges: values

2. The client uses the "X-Range" header field in the GET request to indicate the subrange of the selected representation data, e.g. for the first 1000 SDMX observation values:
   X-Range: values=0-999
   And for the next 1000 values:
   X-Range: values=1000-1999

3. The server, if the range request is admissible, returns the 206 (Partial Content) status code that indicates that the server is successfully fulfilling a range request for the target resource by transferring one or more parts of the selected representation that correspond to the satisfiable ranges found in the request's "X-Range" header field. It also returns a "Content-Range" header field, describing what range of the selected representation is enclosed, e.g.
   Content-Range: values 0-999/1234
   Note: 1234 is the total number of values for the complete resource, or
   Content-Range: values 0-999/*
   Note: * is indicating that the total number of values for the complete resource is unknown by the server, or
   Content-Range: values 0-566/567
   Alternatively, the server generates a 416 (Range Not Satisfiable) response and a "Content-Range" header field with an unsatisfied-range value "*", e.g.
   Content-Range: values */1234

Note

This ticket should be address together with ticket Allow ordering the data query results by dimensions, since slicing/pagination makes only really sense if the order of the observations is deterministic.

[sosna]

Received via the SDMX mailbox and related to this issue:

"There is my view a feature that would be nice to have for avoiding possible timeout and improving the responsiveness when a client is querying a large amount of data. It would be nice to allow a retrieval by blocks. To do so, it would require two additional query string parameters: LimitNbKeys and StartSearchAfterKey. The LimitNbKeys will define the size of the block and the StartSearchAfterKey will allow to start a search after a defined key for retrieving the next block."

[agent96]

What is the benefit of StartSearchAfterKey vs using a numerical limit & offset (or range) - a numerical range can translate directly to SQL LIMIT and OFFSET.

[nicharr]

The Range header unit values is unfortunately not supported by AWS CloudFront:

If a client sends a request with a Range header using the Range unit values to an API deployed behind a CloudFront instance, CloudFront will drop the Range header as CloudFront interprets values as an invalid Range unit.

AWS has advised, in a support ticket, that the Range header is dropped because the Range unit is not bytes. Citing that bytes is the only unit value defined in the http1.1 standard https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.12

This lack of support for the Range unit values from AWS CloudFront is of concern as my people / organisations would likely want to run SDMX APIs behind a CloudFront instance.

[sosna]

This might be useful for item schemes too (cf. initial request from @buhaiovos)

[sosna]

Re. item schemes, there are some rather large codelists (with up to 1 million codes for example). See related discussion.

[dosse]

@nicharr I updated the description accordingly. The new proposal is compatible with AWS CloudFront.

[egreising]

What about using parameters in the RESTful URL, like StartObs=nnnn&NumObs=nnnn? That would work in any environment, wouldn't it?
I would suggest using "Observations" or "Cases" and not "Values" since with multiple measures we'll have several "values" per "observation" or "case".

[sosna]

Fixed by #185