Support for custom chart rendering engines

[rclement]

As proposed by @20after4:

• It would be nice if datasette-dashboards would support custom chart rendering engines
• Using Datasette plugin hooks would allow other plugins to implement rendering engines

Let's discuss about the possibilities here, as Datasette Dashboards should keep some rendering engines built-in allowing the "one-stop shop" experience for new users.

[rclement]

Currently supported rendering engines:

• vega: Uses Vega-Lite
• markdown: Uses Datasette Render Markdown

Ideally, the following rendering engines should be added to datasette-dashboards:

• leaflet: Uses Leaflet.js to render custom maps
• generic: Uses Vega-Lite + Markdown + Leaflet behind the scenes to provide generic components such as Bars, Lines, Scatter, Maps, Text, etc. more in the lines of BI tools such as Metabase. I would very much like it for it to be the default configuration providing the basic "one-stop shop" experience.

[20after4]

One thing I'm immediately tempted to add is custom filter widgets - it'd be awesome if there was a dropdown select widget that can be populated by an arbitrary query to some datasette table or view.

The other thing I'd like to be able to add is a rendering engine that provides more freedom, perhaps just a jinja template or some arbitrary javascript module that returns the rendered content. One use-case is to build some custom navigation links (dynamically generated) that lives within the dashboard.

[20after4]

* Using [Datasette plugin hooks](https://docs.datasette.io/en/stable/plugin_hooks.html) would allow other plugins to implement rendering engines

So how does a plugin add it's own hooks for other plugins to implement? I didn't see that in the docs you linked.

[rclement]

One thing I'm immediately tempted to add is custom filter widgets - it'd be awesome if there was a dropdown select widget that can be populated by an arbitrary query to some datasette table or view.

I thought about it a few months ago when implementing the filtering feature: it should be doable! If your are up for it, you can propose a PR!

The other thing I'd like to be able to add is a rendering engine that provides more freedom, perhaps just a jinja template or some arbitrary javascript module that returns the rendered content. One use-case is to build some custom navigation links (dynamically generated) that lives within the dashboard.

Could you try drafting here how such feature would be used? As a starting point, you can try imagining how to specify such things in the metadata.yml configuration.

So how does a plugin add it's own hooks for other plugins to implement? I didn't see that in the docs you linked.

No idea, I don't know yet if that's even possible. I just formalized the issue directly based on your original comment.

[20after4]

Here's my proof of concept and it appears to work. I'll have to clean up the code a bit and write tests. If you think this is on the right track I can try to get it into shape and submit a PR:

Example hook:

@hookimpl
async def render_custom_dashboard_chart(chart_display):
    return "<h3>test <b>1</b> 2 3</h3>"

Example metadata with custom query filter:

plugins:
  datasette-dashboards:
    project-metrics:
      title: Data³ - workflow metrics
      description: Metrics about projects, tasks and workflows
      layout:
      - [ custom-chart-display, custom-chart-display]
      - [project-events, column-metrics]
      - [project-tasks-state, project-tasks-state]
      - [task-states,task-states ]
      filters:
        project:
          name: Project
          type: select
          query: select phid as key, name as label from Project
        task:
          name: Task ID
          type: text
        date_start:
          name: Date Start
          type: date
          default: '2021-01-01'
        date_end:
          name: Date End
          type: date
      charts:
        custom-chart-display:
          title: test custom chart
          db: metrics
          query: ''
          library: custom

Screenshot of the dashboard:

9d

The POC:

https://gitlab.wikimedia.org/twentyafterfour/datasette-dashboards/-/commit/bf87331e1ec2e461ceff321870ba2564a7a582

[20after4]

One thing I'm immediately tempted to add is custom filter widgets - it'd be awesome if there was a dropdown select widget that can be populated by an arbitrary query to some datasette table or view.

I thought about it a few months ago when implementing the filtering feature: it should be doable! If your are up for it, you can propose a PR!

The other thing I'd like to be able to add is a rendering engine that provides more freedom, perhaps just a jinja template or some arbitrary javascript module that returns the rendered content. One use-case is to build some custom navigation links (dynamically generated) that lives within the dashboard.

Could you try drafting here how such feature would be used? As a starting point, you can try imagining how to specify such things in the metadata.yml configuration.

From most practical to most speculative:

1. The simplest use-case would be adding some dynamically generated navigation links to a dashboard panel.
2. The more complex use case would be adding plotly or d3 charts to the dashboard.
3. possibly an interactive chart editor so that the user can edit the query from right within the dashboard and see the chart update in real time.

Mock metadata.yaml:

Hypothetical and overly simplified jinja2 engine config that contains a query and a snippet of template markup to render some links:

charts:
        custom-chart-display:
          title: Navigation links
          db: metrics
          query: 'select href, label from links'
          library: jinja2
          display:
             <ul>{% or href,label in query.result %}<li><a href="{{ href }}">{{ label }}</a><li> {% endfor %}</ul>

---

So how does a plugin add it's own hooks for other plugins to implement? I didn't see that in the docs you linked.

No idea, I don't know yet if that's even possible. I just formalized the issue directly based on your original comment.

Turns out it's pretty easy to add hooks via pluggy.

[rclement]

Thanks for your POC @20after4 ! Some nice experimentations and results right there, glad to see this in action!

As I can see from your POC, there are at least 3 new distinct features at hand:

• A new filter type select based on the result of an SQL query
• A new chart library type named custom
• A new chart library type named jinja2 (wouldn't it be redundant with the custom chart type?)

In order to improve the reviewing process, would you mind split these independent features into separate PRs?
That would be awesome!

[20after4]

Thanks for your POC @20after4 ! Some nice experimentations and results right there, glad to see this in action!

As I can see from your POC, there are at least 3 new distinct features at hand:

* A new filter type `select` based on the result of an SQL query

* A new chart library type named `custom`

* A new chart library type named `jinja2` (wouldn't it be redundant with the `custom` chart type?)

In order to improve the reviewing process, would you mind split these independent features into separate PRs? That would be awesome!

Thanks for the feedback and thanks for being open to collaborating on this!

I definitely don't mind splitting it up. Also not terribly attached to the implementation. I'm not sure whether a pluggy hook is the best way to achieve what I did there. It may be possible with the existing hooks and some cleaver jinja template overrides, or something else entirely, but this is what I came up with in a very short time experimenting. I'll see if I can clean it up more and I should be able to submit separate PRs in the next couple of days or so.

[20after4]

I apologize for not getting back to this sooner than now.

My project kind of outgrew the framework provided by Datasette Dashboads, however, it's still heavily influenced by your design.
There probably isn't much code that I can contribute back simply because I moved most of the code into the client, leaving datasette mostly unmodified.

@rclement: You can see the result at https://data.releng.team/dev/-/ddd/dashboard/project-metrics if you're interested.

[rclement]

@20after4 thanks for your feedback, I understand.

I'll leave this issue open as having support for custom chart rendering in datasette-dashboards would still be nice have!

[lukasdei]

Would you consider adding support for custom Vega charts? I mocked up a renderCustomVegaChart function and minimal metadata file that mostly works (there's a bug where the grid height continuously grows whenever the window resizes. Not sure if it's an issue w/ the Vega content or a CSS quirk. Would you happen to know?). The function would need to be called in the dashboard_chart.html and dashboard_view.html templates if you test it.

vega_function.txt
metadata.txt

[rclement]

@lukasdei There is already full support for custom Vega charts using library: vega in chart metadata.
What is missing with the current implementation? I do not seem to understand the bug you are referring to. Please open a dedicated issue with a reproducible snippet for me to investigate.

For instance, most of the charts in the demo dashboard are implemented using custom Vega definitions:
https://github.com/rclement/datasette-dashboards/blob/master/demo/metadata.yml

As a rule of thumb: if you can implement your graph in the Vega Editor, you can implement it in Datasette Dashboards

[lukasdei]

@rclement - There is currently support for the Vega-Lite spec, but not for the Vega spec. Vega supports everything that vega-lite does, as vega-lite is compiled into vega code. However, vega-lite does not support everything that vega does. It would be useful to specify a custom Vega spec rather than just a Vega-lite spec.

[rclement]

@lukasdei Alright, got it. I opened a dedicated issue, do not hesitate to post some more snippets of Vega charts there: #63