Normalised results output

[AltGr]

Catala has an interpreter and a bunch of different backends. It is important through comprehensive testing to ensure that they all correctly implement the same semantics, and always return the same results.

For that, we need the results to be easily comparable, i.e. all backends should be able to return them in the exact same format, which is not the case at the moment.
For example, the interpreter would display:

The ocaml implementation, which embeds no printer at the moment, just shows:

Scope A executed successfully.

The Python implementation is only intended as a lib and won't execute anything, so without a dedicated wrapper it will exit without any output.

There already has been efforts in the OCaml runtime to provide JSON, which is used by the web-app and visualisation tools. But we need something more uniform.

JSON seems like the obvious choice (it's simple, unambiguous, human-readable if reformatted, and most importantly, universally interoperable) so the proposal is to provide uniform JSON output for all backends. The format should order struct fields as defined, use only strings for values (no numbers), and have no extra spacing or newlines. Since we don't have strings, there should be no escaping issues.

The interpreter can of course keep its current output by default ; other backends should provide an entry-point consisting of a very thin wrapper around scopes that don't require input, and prints their outputs as JSON to stdout.

With this, it'll be possible to have cross-testing of all backends and ensuring the uniformity of their outputs.

[AltGr]

Some more insights related to JSON output — but not to the tests actually:

We want to (and already do) output reusable traces of programs when compiled with --trace ; we already have a printer for them based on JSON that is used for the French benefits visualisation interface. And these traces do include embedded runtime values, which we already have a printer for in the OCaml / JS backend!

So the question could actually be, do we want to extend this trace printing to the other backends, and do we want to make it more accessible or not (e.g. to print computation results) ?
See also my suggestion of using ATD to have a shared definition of this format across backends.

[pierregoutagny]

I think it could be quite useful to me to have a normalised output format. However, because I generate input structures as well as output structures, I'm wondering if it would be possible for runtimes to also take input structures as input. Do you think it would be easier for runtimes to take input JSONs or for me to output valid surface language code ?

[AltGr]

Definitely we don't want to include a parser in the runtime, which would be much more complex, and doesn't fit with the use-cases we target: catala programs are expected to be run natively as libraries, rather than providing APIs.

Of course, for final needs, one can easily build an API server around the core computation done by Catala; but that should be left to the application as it certainly will need to be specialised for the use-case. We might provide tools for that, but it shouldn't be part of the runtime.

We are actually discussing, for the same reasons, the relevance of JSON output (although most of this discussion was offline), and were leaning against it, but the need for traces might affect the decision.

[AltGr]

I think #585 supersedes this, so closing now.