properties for language used in structure and possibly data SDMX JSON messages

[tasosc]

In this ticket it is is proposed to have new optional properties for indicating the message default language and any alternative language used for specific text.

SDMX JSON Messages contain only one name/description per nameable and annotation but it does not contain:

1. The default language used for all text in the document could be used in cases where Content-Language information is not available or contains multiple languages. This property could for example be added under header.
   e.g.
   "header" : {..., "defaultLanguage" : "fr", ... }
2. When a text was not available in the language from (1) then another (optional) property could be added at nameable objects (DSD, Concept, Code) and/or annotation. This property could either indicate that no text exists because it was not available or the alternative language that was used. E.g.

"name" : "Frequency", alternativeLanguage: { "name" : "en"}
or
"name" : "FREQ Stub", alternativeLanguage: { "missingText" : "name"}

To avoid using such property in all codes of a possibly big codelist the property could also be set once :

"name" : "Frequency", alternativeLanguage: { "name" : "en", "child" : "de" }, "values": [ { "id": "M", "name": "monatlich" } ]

[OECD-BBAZART]

As another option I would suggest like in HTML a cascading approach:
Always use a language attribute in the header section. The language defined will be inherited by all other elements in the document.

If there is content that is in a different language that the one declared in the header, then we can use a language attribute on the elements themselves.

Else if there is multiple languages available except the default one we can suggest the other available languages in a new attribute 'otherLang' and display the prefered language if exists as priority (this is against REST rule: we have to decide).

cf. attached an example.

Language proposal based on step3.txt

[dr-leo]

Hello,

by way of an introduction, I am not involved in the SDMX TWG, but over
the past two years or so I've implemented an SDMX 2.1 Client in
Python:
http://pandasdmx.readthedocs.io/en/master/

Recently I have added a Reader for data Messages in JSON. It is not
part of the current release, but it is in the repo on github. Based on
my admittedly narrow perspective I would like to comment on this
proposal as follows; I am intentionally somewhat provocative:

It is certainly useful to think about international strings, to Quote
the term from the SDMX Information model specs. That said, I am
unaware of a JSON implementation of These strings. SDMXML message
headers do not contain any Information on the Default language.
Rather, individual string entities specify one or more languages.
While the proposed Addition to the JSON specification is probably
compliant with the Information Model specs. However, in my experience
adding ever more nice-to-haves to the JSON file Format complicates
things for implementers in that it blurs the hitherto clean Separation
of logic (model) and representation (SDMXML or SDMXJSON).

In my humble view, the JSON Format already departs to some extent from
the Information Model. Two examples:

1. Unlike in SDMXML, Datasets in JSON have no 'structured_by'
   Attribute containing the DSD ID. Rather, there is a collection of
   links which may or may not contain a link to the DSD in some Format.
   In SDMXML things are cleaner as the 'structured_by' Attribute is both
   contained in the Header and the dataset. It can thus be processed
   easily at Information-model-Level. The JSON representation departs
   from this Approach thus requiring a fair amount of dedicated code in
   the gray Zone between model and implementation (JSON Format).
2. Datasets in JSON lack a 'dim_at_obs' Attribute. In SDMXML this is
   contained in the dataset and arguably the Header. The JSON Format, in
   turn, requires implementers to count the dimensions in the structure
   section. This is entirely possible, but requires a completely
   different Approach even at model Level. So the header's dim_at_obs
   Attribute must be concocted by the Reader in order to Keep the model
   intact where it relies on this Attribute.

While the JSON Format is clearly a step in the right direction as it
is simpler more efficient, I am concerned that proposals as this,
however nice it might be, may gradually diminish the SDMX Standard's
clarity, consistency and elegance, thereby hampering its Adoption and
making implementations error-prone. Eventually, without updating the
Information model in parallel, things could get worse rather than
better.

Leo

On 27/06/2016, Benoit BAZART notifications@github.com wrote:

As another option I would suggest like in HTML a cascading approach:
Always use a language attribute in the header section. The language defined
will be inherited by all other elements in the document.

If there is content that is in a different language that the one declared in
the header, then we can use a language attribute on the elements
themselves.

Else if there is multiple languages available except the default one we can
suggest the other available languages in a new attribute 'otherLang' and
display the prefered language if exists as priority (this is against REST
rule: we have to decide).

cf. attached an example.

Language proposal based on
step3.txt

---

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#40 (comment)

[tasosc]

@OECD-BBAZART In my opinion what you suggest cascading like html is fine. Although I am afraid I don't understand the following sentence:

(this is against REST rule: we have to decide).

@dr-leo I am also an SDMX developer. The problem is that knowing what language was used is useful in many use cases. And what I am afraid is that if it is not added in the standard, it is quite possible that custom solutions will popup like the many workarounds to SDMX v2.0 or the custom parameters and content type in SDMX REST.

Another option would be to include a "lang" JSON attribute to the links/link object:

{
        "href": "http://www.myorg.org/ws/data/ECB_ICP1/M.PT.N.071100.4.INX",
        "rel": "request",
        "title": "Link to the url that returns this response",
        "type": "application/json",
        "lang" : "fr"
}

[dr-leo]

I see the Need for Standardisation. My only Point is that it should be
done so as to

1. remain as Close as possible to the SDMXML implementation of the
   Information model
2. avoid adding any Features to a particular implementation that are
   not contained in the Information model
3. give priority to extending the Information model rather than adding
   all kinds of nice-to-haves to a particular implementation.

That said, I realise that there might be an incentive at TWG Level to
do lots of things proposed here merely because altering the
Information model is not technical thus requiring a more tedious
process.

However, we all know that API's are build on the model, not on any
particular implementation. If implementations grow ever more complex -
and the links section contained in sdmxjson but not in sdmxml is
another dubious example - I am afraid you may be creating new Problems
by solving one.

Leo

On 29/06/2016, tasosc notifications@github.com wrote:

@OECD-BBAZART In my opinion what you suggest cascading like html is fine.
Although I am afraid I don't understand the following sentence:

(this is against REST rule: we have to decide).

@dr-leo I am also an SDMX developer. The problem is that knowing what
language was used is useful in many use cases. And what I am afraid is that
if it is not added in the standard, it is quite possible that custom
solutions will popup like the many workarounds to SDMX v2.0 or the custom
parameters and content type in SDMX REST.

Another option would be to include a "lang" JSON attribute to the links/link
object:

{
        "href":
"http://www.myorg.org/ws/data/ECB_ICP1/M.PT.N.071100.4.INX",
        "rel": "request",
        "title": "Link to the url that returns this response",
        "type": "application/json",
        "lang" : "fr"
}

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#40 (comment)

[OECD-BBAZART]

@tasosc about:

(this is against REST rule: we have to decide).

I just wanted to say: if a content is not available in a particular language then a 404 should be sent but this should not be handle at this level but by the service itself.

[sosna]

Just as background info, the SDMX-JSON format was meant to be used along with an SDMX RESTful API. This is why any feature offered by the RESTful API or HTTP in general has not been duplicated in the SDMX-JSON spec.

For languages for example:

• HTTP offers Content-Negotiation to indicate the language you are interested in. This is done via the HTTP Accept-Language request header.
• The server then responds with the message that contains a Content-Language Response header or with an HTTP 406 status code, in case the requested language is not supported.
• Whenever the resource is available in other languages, the idea was to use the link property along with the alternate relationship. The language could be handled via the hreflang property of links, but, I just checked the last version of the spec, and this is not defined there. Would that help you? If so, I'll send a change request...

Regarding the comment from @dr-leo about the lack of support for dim_at_obs or structured_by: This is mandatory in XML because, if you do not know the metadata, you cannot parse SDMX-ML messages in the compact format or its successor (the structure specific format). However, the idea with SDMX-JSON is that the data message should contain the metadata necessary to parse and interpret the data message (via the structure object). This tells you which dimensions and attributes are attached where and what are their expected values. In that context, it would for example be allowed in SDMX-JSON to attach a dimension which is the same for the entire dataset directly at the dataset level (I'm not aware any implementers supports this, because it does not play nicely with streaming web services, but it is possible). This being said, it's still recommended to point to the full metadata info using links objects.

I hope this clarifies a bit the design decisions.

[dr-leo]

Thanks for clarifying this.

I think it should be possible to fully process sdmxjson messges after
a careful read of the specs. Thus I would definitely prefer having
everything in there.

On links objects in sdmxjson: I haven't checked recently. But I am
unaware of this being defined in the Information Model. And here is
what concerns me: the model does define a 'structured_by' Attribute
for datasets. But it is missing in sdmxjson. I do recognize that an
sdmxjson data message has a link to the structure. But this does not
help at all: The 'structured_by' Attribute is the ID, and it is the
Job of the app sitting on top to assemble the link to get it. In the
brave new json world, the link is contained in the json jsg, the ID is
clearly contained in the link. My Point is that the app sitting on top
of the model must care about whether the DataSet object Comes from a
json or sdmxml file. So the model does no longer serve as an
abstraction layer. Would you agree on this? Am I missing anything?

On dim_at_obs: The structure object in json is great. Yet, my comment
above applies here, too: While in an sdmxml msg, dim_at_obs could be
used to easily distinguish between flat and other data sets. The
structure object does not tell you succinctly whether the data is
flat. You may conclude flatness from the Absence of dimensions at
series or dataset Level. But this is cumbersome, and your Reader has
to make up the good old dim_at_obs Attribute itself. This is
cumbersome and may be error-prone as it relies on certain assumptions:
What if the Server Returns a message with no Series object within the
Dimensions section or it does return it with Zero Content? - I am
assuming here that either is allowed...

One final source of confusion: The sdmxjson specs do not specify
Groups between the dataset and series Level, the Information model
does, and so does sdmxml. Again, I may be missing something. But I
would very much favor as much consistency between sdmxml and sdmxjson.

Leo

On 01/07/2016, Xavier Sosnovsky notifications@github.com wrote:

Just as background info, the SDMX-JSON format was meant to be used along
with an SDMX RESTful API. This is why any feature offered by the RESTful API
or HTTP in general has not been duplicated in the SDMX-JSON spec.

For languages for example:

• HTTP offers Content-Negotiation to indicate the language you are
  interested in. This is done via the HTTP Accept-Language request header.
• The server then responds with the message that contains a Content-Language
  Response header or with an HTTP 406 status code, in case the requested
  language is not supported.
• Whenever the resource is available in other languages, the idea was to use
  the link property along with the alternate relationship. The language
  could be handled via the hreflang property of links, but, I just checked
  the last version of the spec, and this is not defined there. Would that help
  you? If so, I'll send a change request...

Regarding the comment from @dr-leo about the lack of support for
dim_at_obs or structured_by: This is mandatory in XML because, if you do
not know the metadata, you cannot parse SDMX-ML messages in the compact
format or its successor (the structure specific format). However, the idea
with SDMX-JSON is that the data message should contain the metadata
necessary to parse and interpret the data message (via the structure
object). This tells you which dimensions and attributes are attached where
and what are their expected values. In that context, it would for example be
allowed in SDMX-JSON to attach a dimension which is the same for the entire
dataset directly at the dataset level (I'm not aware any implementers
supports this, because it does not play nicely with streaming web services,
but it is possible). This being said, it's still recommended to point to the
full metadata info using links objects.

I hope this clarifies a bit the design decisions.

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#40 (comment)

[sosna]

@dr-leo, you raised a lof of interesting points. I'll try to tackle some of them below.

• Re. the lack of links in the model: Yes, that's the case and this is currently being discussed. The idea would be to have linkeable artefacts, i.e. objects which can hold an array of link objects.
• Re. structured_by: I'm not sure to understand the issue. The model defines a structuredBy and a describedBy that can be used to reference a DSD or a dataflow respectively. structureBy is mandatory. In SDMX-ML, this translates into a Structure element in the header, which must contain a valid reference to DSD (via another Structure element) or a dataflow (via a StructureUsage element) or a provision agreement (via a ProvisionAgreement element). The choice of which element to use is left to the implementer because all of them allow you to reach the DSD anyway. In SDMX-JSON, the necessary metadata are already embedded in the data message but there is the option of adding links to DSDs, dataflows and/or provision agreements via the links array. So, yes, the readers for various formats need to know how to do things in various formats but, unless I overlooked something, the logic is the same.
• Re. dim_at_obs: there is an observationDimension defined in the model. In SDMX-ML, it translates into a dimensionAtObservation attribute of the Structure element in the header. SDMX-JSON is meant to be very close to the RESTful API and the assumption is that, as the data message is organised the way the client asked in its query (time series, cross-sectional or flat) there is no need to repeat the information. What we found important was to indicate where dimensions can be found (dataset, obs or series) and this is what the SDMX-JSON structure object is about. If you need to know if the dataset is flat, you can simply check the existence or the length of the dataset series property (either are allowed indeed). For our own purpose, we found this to be OK (and the reports of early implementers go in the same direction). However, if you believe this is an issue, please open an issue and we might still be able to propose alternatives before the official release of version 1.0.
• Re. groups: SDMX-JSON is very close to the RESTful API, as mentioned. The RESTful APIs we are aware of are streaming apis (i.e. they start returning results asap instead of keeping the entire result set in memory). Use of groups make streaming impossible (in SDMX-ML for example, you first have to write all the groups before you can start writing the series elements). That's why it was not added to the SDMX-JSON spec. Again, if you believe this is an important issue, please open an issue in Github and we will review it :).

[dosse]

Added hreflang name to link object (see commit 259a66c) to allow indicating availability of the same resource in alternative languages. This change doesn't include a new feature to allow for and indicate varying languages within the same message.

[dosse]

The SDMX-TWG decided:

The first best language match according to the user’s preferred language choices in the http Accept-Language header (or if that is not available than according to the system's default language order) is to be used for each localisable message element. The message does however not indicate the returned language per localisable element. In case that there is no such language match for a particular localisable element, it is optional to:

• return the element in a system-default language or alternatively to not return the element
• indicate available alternative languages for the element's maintainable artefact through links to underlying localised resources

It is recommended to indicate all languages used anywhere in the message for localised elements through http Content-Language response header (languages of the intended audience) and/or through a “content-language” property in the meta tag. The main language used can be indicated through the “lang” property in the meta tag.

The documentation, schema and examples have been updated.

[dosse]

As only one default language is not sufficient especially for the case of (submitting) structure messages, but a consistent approach for all message types is envisaged, the SDMX-TWG TF3 on formats proposes to add another property allowing for multiple languages.

[dosse]

Both static best-match language texts and dynamic multi-language objects are now implemented. For optimising message sizes and avoid unnecessary duplication, a new rest parameter driving the usage of one of the text options would be useful.