Input stream as source

[dachafra]

issue: not possible to describe an input stream as a source

suggestion: add support for describing input streams. This will facilitate the usage of RML in transformation pipelines.

[marioscrock]

Naive solution proposed here RMLio/rmlmapper-java#34

However, my suggestion is to "shift" the problem to the RML processor allowing the user to define a custom logic to process an rml:source of the logical sources defined in the mapping file and build an abstraction to let the processor access the records. For example, in the case of the RMLMapper, defining a custom AccessFactory and/or Access. As an example, in Chimera we implemented an alternative AccessFactory that can access InputStreams but can also bypass the rml:source defined and apply the mappings to the entire message body https://github.com/cefriel/chimera/tree/master/chimera-rml/src/main/java/com/cefriel/chimera/rml).

Generally speaking, I think it is important to let users define mappings that are not bound to specific sources (e.g. to a specific file name).

[dachafra]

@DylanVanAssche is this already allowed with the LogicalSource spec?

[pmaria]

FYI: CARML supports this with an extension of rml:source https://github.com/carml/carml#input-stream-extension

[DylanVanAssche]

Spec is now available at https://github.com/kg-construct/rml-target-source-spec.
Currently, only Targets are defined there, but can be easily extended to Sources as well.

FYI: CARML supports this with an extension of rml:source https://github.com/carml/carml#input-stream-extension

Great! This would be allowed in the Sources & Targets because CARML's extension is just a different part for rml:source and rmlt:target. It was so intended to support these things in the spec through DataIO spec: https://rml.io/specs/dataio.
The RMLStreamer also have similar extensions: https://github.com/RMLio/RMLStreamer#processing-a-stream

My suggestion: we need to discuss to have an extension to something like the DataIO spec to define how a stream should look like in the mapping rules.

[DylanVanAssche]

Relevant ontology: https://github.com/streamreasoning/vois

Which stream sources do we want to support?

• Kafka
• MQTT
• WebSocket
• CoAP
• Server Sent Events
• TCP
• ...

Characteristics to consider:

• Windows for joins
• Specifying the content (JSON, XML, etc.) inside the stream
• Restarts?
• ...

Please comment here which one definitely need to be supported of the sources & characteristics.

[DylanVanAssche]

I have been investigating this further and I think we can just 'extend' the Web of Things description :)
Web of Things provides a way to extend their specification with flavor specific things with Web of Things Binding Templates: https://www.w3.org/TR/wot-binding-templates.

They have already binding templates for:

• HTTP
• CoAP
• MQTT streams

My suggestion is to re-use Web of Things descriptions and define binding templates for other streams we may want as listed above. This way, we don't need to define this vocabulary ourselves, we re-use standardized specifications and are open for the future.

[pmaria]

Could you give an example what this would look like for a simple input stream/pipe of data passed at runtime?

[DylanVanAssche]

@pmaria

There are currently no binding templates for a pipe, but I don't see any requirements here that are specific to a pipe compared to the others, so we don't need a binding template.

We only need a proper URL with a schema defining a pipe such as tcp://, I don't know a scheme definition for pipes though.
However, pipes are just files so named pipes are easily described as file:///path/to/named_pipe:

<#TriplesMap> a rr:TriplesMap;
  rml:logicalSource [ a rml:LogicalSource;
    rml:source [ a td:Thing;
      td:hasPropertyAffordance [ a td:PropertyAffordance;
          td:hasForm [
            # URL and content type
            hctl:hasTarget "file:///path/to/named_pipe";
            hctl:forContentType "application/json";
            # Read only
            hctl:hasOperationType td:readproperty;
          ];
       ];
    ];
    rml:referenceFormulation ql:JSONPath;
    rml:iterator "$.station.[*]";
  ];

  # Subject, Predicate, Object Maps

At least this works fine under Linux where everything is a file, I'm curious how these things are handled on Windows & Mac.

[pmaria]

That honestly looks quite complex only to specify the location of a source. Don't you think?
I mean hctl:forContentType and hctl:hasOperationType don't seem necessary for our case. That's already 2/3 of the description.

However, pipes are just files so named pipes are easily described as file:///path/to/named_pipe

Right, usually you would use an input stream for file IO, but it can also be used as an intermediary stream from one internally running process to another, in which case we're not dealing with files.
But it could be an different type of object as well, e.g. a compiled JSON object.

So it would indeed be good to have some way of saying that the user will provide the source to the engine running the mapping as @marioscrock suggested in #2 (comment)
We would still need to define a way to specify that.

[DylanVanAssche]

That honestly looks quite complex only to specify the location of a source. Don't you think?
I mean hctl:forContentType and hctl:hasOperationType don't seem necessary for our case. That's already 2/3 of the description.

Hmmmmm you're right.
I copied them from the WoT paper, but in the end, you don't really need them since hctl:forContentType is not needed since we have rml:referenceFormulation for it. hctl:hasOperationType is also not really necessary.
I checked the code in the RMLMapper and these are not even used 🙈
Let's drop them.

WoT also allows security to be integrated there, but is not applicable here.

but it can also be used as an intermediary stream from one internally running process to another, in which case we're not dealing with files.

Ah you mean this Java InputStream thing...
If we have a scheme, it would work:

<#TriplesMap> a rr:TriplesMap;
  rml:logicalSource [ a rml:LogicalSource;
    rml:source [ a td:Thing;
      td:hasPropertyAffordance [ a td:PropertyAffordance;
          td:hasForm [
            hctl:hasTarget "inputstream://$NAME";
          ];
       ];
    ];
    rml:referenceFormulation ql:JSONPath;
    rml:iterator "$.station.[*]";
  ];

  # Subject, Predicate, Object Maps

If we want something 'simpler' we would need to define an ontology first (if we cannot re-use something existing).
I rather don't want to maintain that TBH.
However, rml:source is fully open especially for this. If you define a custom ontology like you did in CARML, you can just put that here. Problem with that ontology is that it only support InputStreams.
I would welcome a better approach that covers more than just InputStreams and uses standardized ontologies as much as possible. I picked WoT here because WoT is currently in already for Web APIs & some streams and it seems 'easy' to support more types of streams thanks to the URL scheme, but that's my opinion. Open for suggestions!