Discuss whether importing ts-reaktive-marshal would be a good fit

[jypma]

The marshalling part of ts-reaktive (small presentation and demos here) provides a DSL to describe bi-directional marshalling of custom data types to and from XML or JSON. A short example:

String NS = "http://test.com/namespace/api";

XMLProtocol<Date> date(QName tagName) { ... } // shows composability

XMLProtocol<Person> proto = 
  tag(qname(NS, "person"),
    attribute(qname("firstName")),
    attribute(qname("lastName")),
    arrayList(
      tag(qname(NS, "address"), 
        body
      )
    ),
    date(qname("birthday")),
    Person::new, /* ... */
  );

The above XMLProtocol can then be used to create an akka Flow that (with proper back-pressure) takes XMLEvent in and Person out, or, Person in and XMLEvent out.

The result would read and write something like this:

<person xmlns="http://test.com/namespace/api" firstName="Bob" lastName="Jones">
  <address>Hello</address>
  <address>World</address>
  <birthday>1970-01-01</birthday>
</person>

An equivalent DSL exists for JSON, but using actual JSON primitives instead.

Combinators like arrayList, option, etc. are shared between the two DSL.

I think it's a nice fit for alpakka, since it allows us to describe higher-level XML API's in a streaming fashion. For example, we could streamingly list an S3 bucket. Note that akka-http's current streaming marshalling capabilities depend on specific "framing" use cases. This proposed library can split stream elements at (almost) arbitrary locations.

It'd need a Scala API in addition to the Java one of course, but that's small potatoes.

What do you think?

[johanandren]

I recently created akka/akka#21826, sort of related to this

[jypma]

Oh yes, don't mind if I plug the lib there as well :)

[drewhk]

Looks interesting. I will probably take a look.

[jypma]

@ktoso mentioned that it may be a better fit to have ts-reaktive-marshal augment/replace the JSON framing support in akka-http, rather than have it in alpakka.

[jrudolph]

@jypma we finally looked at ts-reaktive again during last week's team meeting. Sorry for delaying this for so long.

We think it is indeed a good fit for alpakka. We looked at the code a bit which looks well-structured so it shouldn't be too hard to extract.

The scope would be the ts-reaktive-marshal and ts-reaktive-marshal-http modules. I'll try to summarize the functionality (please correct if I'm wrong):

• Java DSLs for specifying marshalling formats from / to XML and JSON
• Streaming parsers that allow parsing XML and JSON data using the above marshallers
• Glue code to make it easy to use with akka-http

Here are a few remarks:

• Dependencies:
  • The module should not have any dependencies to ts-reaktive. As you already extracted it into an extra module in https://github.com/Tradeshift/ts-reaktive/tree/master/ts-reaktive-marshal, that doesn't seem to be a big problem.
  • There seem to be multiple backends for JSON parsing (actson, jackson) in the original code. Would it stay that way?
• Let's put all of the code into a single module that has a provided dependency on akka-http. This way you can either use it with or without akka-http without needing an extra module.
• The usage of javaslang internally is fine (though binary compatibility might become a problem if javaslang starts to become commonly used in incompatible versions). Not exposing javaslang in the API might simplify changing internals later on (e.g. partial reimplementation in Scala).
• A Scala API for the marshaller would be nice-to-have but not strictly required for the first version.

WDYT?

[ktoso]

Adding context from gitter chat on last Friday which Johannes missed:

Jan Ypma @jypma Mar 23 11:30
I was a bit worried "requiring" a certain version... but actually, now I think of it, the bindings for their collections possibly need to change together with the version anyways.
OK, let's go for
implementation depends on Javaslang
plain Java collection bindings separate from javaslang bindings (can be in the same JAR though, javaslang is already on board)
scala bindings in a separate JAR
?
however, the internal API would then still hava Javaslang signatures. In other words, if one would want to add support for e.g. YAML or sth, you'd be coding against Javaslang classes.

Patrik Nordwall @patriknw Mar 23 11:34
sounds good, but why scala bindings in separate jar (scala is also already a dependency)?

Jan Ypma @jypma Mar 23 11:40
it isn't. It only is for the layer that binds the marshalling to akka streams. The basic marshalling doesn't depend on scala (or akka). The thing can conceivably be used with e.g. RxJava

---

@ktoso is this acceptable as far as scala marshalling DSL? https://github.com/Tradeshift/ts-reaktive/pull/56/files#diff-87884016caa105133b216b64cd5088faR17 If so, I'll park that PR and start working on a first PR onto alpakka, as tentatively talked about with Patrik.

[ktoso]

And just as a sanity check: I think Johannes's summary contains everything we discussed.
So the parsing is one thing, and the nice glue code for HTTP is another :)