Queries from Grafana

[DrRuhe]

I thought it could be nice to allow trustfall queries to be executed from Grafana. So, I jumped into the rabbit hole and drafted a simple architecture that could support that.

flowchart LR

    subgraph Grafana[Grafana Process]
        UI -- Utilizes --> Plugin[Grafana-Trustfall Plugin]
    end
    Plugin -- sends queries --> Webserver

    subgraph Server
        Webserver -- executes queries --> Trustfall
    end

Loading

Basically two things are required to enable this use case. A Grafana-Plugin that enables trustfall Datasources and a way to expose trustfall adapters via HTTP.

• The Grafana plugin would connect to the server and send the queries from Grafana. The plugin could also provide a fancy query editor, but that's for a later version. In theory, it is also possible to bundle the query engine in the Grafana-Plugin using wasm, but this would require users to replace the plugins every time the adapter changes.

• A standardized template webserver that is generic over the exposed adapter could be a nice fit for exposing Adapters over HTTP (possibly implement HttpServiceFactory for an Adapter, so an adapter can be easily exposed as a HTTP route via actix-web).

I might implement this myself, depending on the free time available.
I wanted to put it out there, so others can provide further ideas and so that I don't forget these thoughts.

[obi1kenobi]

This looks awesome! I'm not familiar with Grafana plugins, but if there's anything I can do on the Trustfall end to make this easier for you to build, I'm all ears!

[obi1kenobi]

You can make a rustdoc file by running cargo rustdoc -- -w json -Zunstable-options in a lib crate, and then grabbing the rustdoc JSON file from inside the target/doc directory.

I am working on making it possible to run cross-adapter queries! I have a prototype running locally but the API is hard to get right and not everything is implemented yet, so it'll be a bit longer before I can publish that.

Great to know about the Arc vs Rc on Adapter being problematic. In a future version I might make Arc<Adapter> the default, or make it possible to support both. I think there's a way to generalize over Arc and Rc as well, but I need to look up the blog post that showed how to do that.

In the meantime, an unfortunate but probably working hack might be to make an Arc<Rc<Adapter>> 😅

[DrRuhe]

After our session I got stuck on actix-web not wanting to parse my requests. After being busy with other stuff for a few days, I looked at it again today. I got it to work under axum and now I can send queries to a webserver and it responds with the answers. With that, I also noticed, that trustfall::execute_query is a library function that returns anyhow::Result. It should define distinct error variants (perhaps utilizing thiserror) to allow proper error handling.

As the Grafana Integration only really makes sense once queries can utilize multiple adapters I will wait with the implementation until that's possible.

[obi1kenobi]

trustfall::execute_query is just a façade in front of the underlying trustfall_core APIs: https://github.com/obi1kenobi/trustfall/blob/main/trustfall/src/lib.rs#L69-L87

Many changes to thiserror-like error types are breaking changes, and I didn't want every incremental improvement there to be a breaking change. This is why the trustfall façade layer erases the error type info. The underlying trustfall_core implementations use thiserror error types like you suggest.

If you're okay with somewhat more common breaking changes, replace trustfall::execute_query with analogous direct use of the underlying trustfall_core functionality.

As the Grafana Integration only really makes sense once queries can utilize multiple adapters I will wait with the implementation until that's possible.

Can you give me an example of "queries utilizing multiple adapters" here? I feel like could mean one of several things, some of which are already possible today and I'd hate to keep you waiting if I don't have to :)

[obi1kenobi]

I'm about to merge PR #286 which makes query execution use Arc<Adapter> instead of Rc<Adapter>. Hopefully that will simplify this use case! I'd love feedback on whether it worked, or if there's something else that is needed too.

It's a major breaking change so if I didn't fully fix the problem, I'd love to group as many of the breaking fixes I need into one release so as not to annoy users needlessly with too many consecutive breaking changes.

[obi1kenobi]

I just released Trustfall v0.5.0 with the Arc<Adapter> change! Thanks for pointing out the problem that needed to be fixed there.