Make plots reading and rendering API friendly

[Suor]

Context: this will help using it in viewer.

This things need to be implemented somewhat public:

• separate data and template extraction
• separate raw data fetching and post-processing (add rev and some json magic)
• provide a way to make a spec from raw data and template

The last point is culmination. The viewer scenario of using it is:

• parse repo, save raw plot data and templates (as text) to DB.
• on user request fetch data for selected commits and pass it to some template (postprocessing happens somewhere inbetween or within a template part)

Ideally one won't need post-processing or if that is not possible it might be separated into postread and prerender parts. A thing to discuss.

This is not p0, since I'll be handling it in some dirty way for now myself, but still important.

NOTE: this issue is scoped to plot data reading and plot rendering, plot data file discovery is discussed separately in #3807.

[Suor]

@pared what do you think?

[pared]

@Suor
It makes sense to me, so we kind of want to detach data loading from data processing and from template handling, and "join" those elements in the latest moment possible, so that viewer can have more control over the plot creation process?

[Suor]

@pared yes, I want this to be separate to be able to store some raw data, probably in normalized intermediate format, and then finish the processing later.

The thing is that parsing a repo and generating plots are completely separate for me. Plots are generated on user request and so the chosen commits can't be known beforehand.

[pared]

@Suor that makes total sense

[Suor]

Another thing we should consider - stop treating templates as strings, these are JSONs or structured data. Now their serialization is baked in, so I need to encode data, parse and reencode to serve it the way I want. Otherwise I need to serve it as string with all those unneeded spaces and newlines.

[pared]

@Suor that is a good point, but there was a reason why we are not treating them as JSON:
It is possible now to create custom template that is not JSON (for example HTML page filled with placeholders for data) and still be able to work with that kind of template.

I agree that JSON would be much more convenient to work with. We could actually get rid of anchors, because Vega template structure is quite constant, so data will always be under son["data"]["values"] and so on...

I think we could use @dmpetrov input on this one.