YAML Streams and JSON Sequences

[ioggstream]

Question

I've been pointed to JSON Sequences https://datatracker.ietf.org/doc/html/rfc7464

Maybe this can be a related work for converting multi-document YAML Streams to JSON-LD.

WDYT?

@gkellogg @VladimirAlexiev

[anatoly-scherbakov]

Thank you for pointing this out. This approach is also known as:

• NDJSON
• Which is forked from JSON Lines, often abbreviated as JSONL.

It seems that a conversion of a JSON Lines stream to a YAML stream, and vice versa, is straightforward.

I have tested yq. I cannot seem to be able to convert a JSON Sequence file into a YAML Stream using that tool. Documented it here: mikefarah/yq#1279

The reverse direction is functional though:

$ cat sample.yaml 
foo: bar
---
baz: kaboom

$ yq -o=json -I=0 . sample.yaml 
{"foo":"bar"}
{"baz":"kaboom"}

So, at least partially, JSON Sequence to YAML Stream mapping is kind of supported in the ecosystem already. Would be interesting to draw a comparison matrix for other tools and libraries.

[ioggstream]

Edited: as @TallTed points, it's application/json-seq

@anatoly-scherbakov could you please integrate the examples above showing the 0x1E and 0x0A characters when they are present? I have not understood whether the output is a compliant JSON Sequence (mediatype application/seq+json application/json-seq) or not.

Producing a seq+json might need to define a new mediatye (e.g. ld+json-seq :DDD). This is another interesting thread. Personally, I don't know if ++ media type are ok or not, not I am advocating their use.

[TallTed]

Regarding media types with multiple plus signs, see Draft RFC Media Types with Multiple Suffixes. They're not yet officially permitted nor sanctioned by IANA nor IETF, but at least one such media type registration is pending.

[gkellogg]

Thank you for pointing this out. This approach is also known as:

• NDJSON
• Which is forked from JSON Lines, often abbreviated as JSONL.

Note that NDJSON/JSONL doesn't make use of the RS character, as required in the RFC. In practice, I've seen neither, and they as yet bear no relationship with JSON-LD.

So, at least partially, JSON Sequence to YAML Stream mapping is kind of supported in the ecosystem already. Would be interesting to draw a comparison matrix for other tools and libraries.

They both seem generally related to the concept of a YAML Stream, but differ from the treatment of multiple JSON script tags in HTML.

@anatoly-scherbakov could you please integrate the examples above showing the 0x1E and 0x0A characters when they are present? I have not understood whether the output is a compliant JSON Sequence (mediatype application/seq+json) or not.

Producing a seq+json might need to define a new mediatye (e.g. ld+seq+json :DDD). This is another interesting thread. Personally, I don't know if ++ media type are ok or not, not I am advocating their use.

Actually application/json+seq, not seq+json, which makes sense to me. But, again, NDJSON doesn't strictly seem to conform to the RFC.

As @TallTed notes, there is a proposal for multiple + media types (for Verifiable Credentials, I think), and I think there is general support for this, but wheels move slowly.

If we constrain ourself to the JSON-LD internal representation, we don't have a target for a document stream. Of course, we could introduce such by extension, but it doesn't really seem to relate to the -LD use case, without also extending into some notion of multiple graphs, which aren't treated as a Dataset.

[TallTed]

It turns out that JavaScript Object Notation (JSON) Text Sequences a/k/a JSON Text Sequences (but, so far as I can tell, not properly known as JSON Sequences) have the media type application/json-seq -- no plus sign; not a sub-type of JSON (as would be suggested by application/seq+json) nor of some nonexistent "Sequence" (as would be suggested by application/json+seq); and not requiring multiple + at all.

[anatoly-scherbakov]

Thank you for pointing out the difference between the spec and NDJSON. Indeed, they use different separator characters.

I have encountered usages of NDJSON (say, you can process that with jq) and support for that format in commercial systems (say in https://data.world, which is by the way RDF oriented and supports SPARQL).

I haven't seen usages for JSON Sequences RFC before.

[ioggstream]

@anatoly-scherbakov if you could wrote a one pager presenting the various JSON seq alternatives on the market i will ask to mediatype folks if standardizing a different format for JSON is advisable.

This is clearly unrelated to this spec and this issue should be probably addressed in yaml media type document.

Wdyt?

[anatoly-scherbakov]

@ioggstream I started to write a little memo about it but found this page in Wikipedia which, it seems, about does the job. Does it?

[ioggstream]

@anatoly-scherbakov I think we can say

This spec does not mandate a way to convert multi-document YAML Streams to a specific format such as json-seq.
The implementer is then free to convert a multi-document YAML stream to multiple, separate JSON texts,
to a single json-seq file, or to some other custom multi-document JSON format

WDYT?

[gkellogg]

@anatoly-scherbakov I think we can say

This spec does not mandate a way to convert multi-document YAML Streams to a specific format such as json-seq.
The implementer is then free to convert a multi-document YAML stream to multiple, separate JSON texts,
to a single json-seq file, or to some other custom multi-document JSON format

WDYT?

If we say this, then we can't really have any tests involving multi-document streams, which is fine. If the concept of multi-document streams is important, it would be for JSON-LD as well, and should probably be taken up there. However, I don't really see how it fits with our data model, which already has the concept of named graphs which is a closely related way of dealing with this in the RDF world.

[anatoly-scherbakov]

Yes, certainly.

[ioggstream]

... named graph ...

Can you clarify the relation between a YAML-LD document and a named graph? iiuc a YAML stream can contain multiple documents, and each one can contain multiple named graphs.

[gkellogg]

Named Graphs are often used to describe the content of some particular RDF source, particularly when used in SPARQL. That is why I said that the closet analog to multiple files from the RDF world is probably named graphs. Generally, each separate document would be considered an "RDF Source", and the only system that I can think of that deals with more than one RDF Source is SPARQL.

For example:

PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX dc: <http://purl.org/dc/elements/1.1/>

SELECT ?who ?g ?mbox
FROM <http://example.org/dft.ttl>
FROM NAMED <http://example.org/alice>
FROM NAMED <http://example.org/bob>
WHERE
{
   ?g dc:publisher ?who .
   GRAPH ?g { ?x foaf:mbox ?mbox }
}

Where http://example.org/alice and http://example.org/bob represent different endpoints/RDF Sources. (Example taken from 13.2.3 Combining FROM and FROM NAMED in SPARQL 1.1 Query.

In this view, each document in a YAML-LD stream would be a different RDF Source, although the analogy breaks down as IIUC there is no way to name separate documents in a YAML stream, but we might define a confention, if naming individual documents is deemed important.

[ioggstream]

each document in a YAML-LD stream would be a different RDF Source

How many graphs does the following yaml document from https://w3c.github.io/json-ld-syntax/#example-referencing-named-graphs-using-an-id-map-with-none contain?

---
- "@id": http://example.org/foaf-graph
  http://www.w3.org/ns/prov#generatedAtTime:
    - "@value": 2012-04-09T00:00:00
      "@type": http://www.w3.org/2001/XMLSchema#dateTime
  http://example.org/graphMap:
    - "@graph":
        - "@id": http://manu.sporny.org/about#manu
          "@type":
            - http://xmlns.com/foaf/0.1/Person
          http://xmlns.com/foaf/0.1/name:
            - "@value": Manu Sporny
          http://xmlns.com/foaf/0.1/knows:
            - "@id": https://greggkellogg.net/foaf#me
    - "@graph":
        - "@id": https://greggkellogg.net/foaf#me
          "@type":
            - http://xmlns.com/foaf/0.1/Person
          http://xmlns.com/foaf/0.1/name:
            - "@value": Gregg Kellogg
          http://xmlns.com/foaf/0.1/knows:
            - "@id": http://manu.sporny.org/about#manu  
...