Don't block onMetricsPanelRefresh() when streaming data

[seanlaff]

When a datasource provides an observable for a panel to watch, all onMetricsPanelRefresh() behavior is sidestepped. See:

grafana/public/app/features/panel/metrics_panel_ctrl.ts

Lines 73 to 76 in 68ae17e

	// // ignore if we have data stream
	if (this.dataStream) {
	return;
	}

I'm guessing this was done because we don't wan't live panels to refresh when someone has a dashboard refresh interval- however we do want the panel to refresh if queries, time ranges, or report variables change. Additionally, if a user hits the manual refresh button, we should respect that.

In my datasource I was able to work around some of this behavior by nulling out the queryCtrl's datastream object before calling refresh, but I'm not sure if theres workarounds for the other cases without making the change in Grafana itself.

Here was my partial workaround in my datasource
https://github.com/seanlaff/simple-streaming-datasource/blob/33052e32ddb8bc6850cf123a520c3d888a4d4e27/src/query_ctrl.js#L27

[torkelo]

The streaming feature is still in POC phase and we have yet to figure out the right way to handle things like this. I would guess is very dependant on what kind of dashboard & panels you have along side your live data source panel.

[seanlaff]

@torkelo Agreed. I wasn't sure of all of the design considerations that went into the current implementation. I'll highlight in my repo that this all hinges on unspecified behavior at the moment. If there's anything else related to streaming that I could pitch in with, let me know 👍

[torkelo]

Given the response to your talk I would like to make this a higher prio than it was over the next few months.

And feedback from an actual implementation that uses the live streaming capability is very valuable!

We have yet to add streaming support for react panels: https://github.com/grafana/grafana/blob/master/public/app/features/dashboard/dashgrid/DataPanel.tsx#L121

[markusr]

Maybe it helps to look at netdata.

Netdata is capable of streaming several thousands of values in real-time (actually 1s granularity) to the browser, see demos.

[ritchiecarroll]

I would like to add a streaming Grafana plugin interface (data source?) for the Streaming Telemetry Transport Protocol (STTP) - on it's way to becoming IEEE 2664:

http://sttp.stream

I may tinker with what @seanlaff has put together as a POC.

[ryantxu]

@ritchiecarroll -- if there is a simple way to setup an example server, I would happily help flush this out. In 6.2+ the streaming structure is significantly improved, but we are looking for the best standard ways to expose it. See also: https://github.com/lapnap/grafana-tail-datasource/blob/master/src/FileWorker.ts#L79

[ritchiecarroll]

I will work on setting up a test server.

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[seanlaff]

Grafana 6.4 and up have streaming as a first class citizen (see the grafana-test-datasource/loki in the grafana repo