Debugging and getting intermediate result and metadata

[m-mohr]

A big topic during the hackathon was debugging of our process graphs and getting information of intermediate result and other meta data. For example, after a filter one might want to know what data is left. GEE allows to add print() calls to get information about a variable with results between different steps. How could something like that be made available in openEO?
Related to Open-EO/openeo-python-client#24

[m-mohr]

One idea would be to define some processes like print/debug/get_information/... that simply passes through the data (input = output), but sends the requested (debugging) information to the subscriptions, maybe in a separate "channel".

[edzer]

Couldn't this be (optional) under /job/status, like /job/job_id/status/logs or so?

When in synchronous mode, you'd probably expect that this is printed to your screen?

[m-mohr]

@edzer
I think it could be part of the subscriptions API as described above for batch jobs. Then you get the information just-in-time and don't need to request the status over and over again. It would exclude synchronous calls though and web services (on demand processing) is also still to be considered. Web services though should probably only run on process graphs that are already "validated".

In synchronous mode, how would you print it to screen? It needs to be transferred via HTTP to the user, but the processed result (i.e. often a binary image) is already in the body so there is no good way to transfer this data over to the client. You could send some kind of multipart response, similar to file uploads in browsers, but this would make the endpoint overly complex compared to how simple its response is now.

We (Hackathon participants) haven't found a solution yet and there's probably no simple solution for it. That needs some more serious thoughts.

[m-mohr]

We have a draft for this using the Subscriptions API. The topic openeo.output can be used to send intermediate results and metadata and the topic openeo.debug can be used for debugging purposes. For debugging there's actually a concrete structure to send information, but for output we still need to define how specifically intermediate results can be sent and which structure/schema to use. Needs to be solved in v0.5 based on feedback in v0.4.