Live data stops being showed

[wauterstoon]

I have set up a Grafana local server v8.4.5 which is connected through an MQTT broker plugin for grafana which can be found here: GitHub - grafana/mqtt-datasource: MQTT Datasource for Grafana allows streaming data from any MQTT broker running either locally or remotely. 1 Succesfully installed that and works fine. The only problem i face is that my realtime data stops being showed after some minutes. I taught it was the auto refresh option but it wasn’t. Once i refresh the browser my live data is back and will be shown for another minute or two. I would like to have realtime data or data with 5-10 seconds delay.Does someone has any suggestions? Couldn’t find any thing on the forum.In the MQTT explorer application if see my realtime data so its very weird why it stops

.

[Drakynn]

You may want to post your log from the time period where the graph goes from properly displaying the data to stopping.

[santi-wg]

I have the same problem
sorry for my english
When I use "1 data source" with 1 topic "no problem" ,
if "1 data source" with "2 topic" have a problem
and if I use 2 data source with 2 topic no problem

Thanks
Sorry my english

[bmedici]

Same here: data limited to "last 1m", when display graph flickers and stops. Eventually, it will start again streaming live data a few seconds/minutes later.
Any idea ?

[EricZeiberg]

Also having this issue, any updates?

[ridomin]

I also see this behavior with a single topic

[gpinkos]

Readings are being sent through the broker about once every 3 seconds. Grafana displays 5 or 6 readings (approx. 15-20 seconds) and then goes blank. I confirmed that the readings are still being sent by the broker (they are visible in Node-RED and other apps).

Same behavior for the Stat panel:

[VincentPapernest]

Hi everyone ! I'm currently getting the same issue... Do you have some updates about it ?

[gdxas]

We also experiencing the same issue in 9.4.1, looks like its fixed in 9.4.3

[Koivader]

I’m using 9.4.3 and I still experience the issue

[gdxas]

How long does it take for you before the issue occur for you?
For me it happened pretty fast in 9.4.1, but I have been running it for an hour+ without any issues in 9.4.3 now.

[Koivader]

In my testing it only worked for about a minute. I don't know it it is a feature or a bug, but it looses its data if the timescale is changed. But sometimes changing the from eg 15min to 5 min made it work again.
For completeness, I'm using mosquitto in docker as broker with no TLS and no authentication.

[gdxas]

OK. Here is the settins I use to make it work.

For the dashboard I run in now-5m with no refresh.
I set up all the widgets with MQTT, and then I restart the grafana service.
After that it seams to run stable.

But if I change query options it will some times stop working again until i restart grafana service.

MQTT plugin is still in beta I see.

[Koivader]

Thanks for the tip with the restart. Now it's working more reliable

[K3das]

I've been getting this issue on 9.4.3, and for some reason it thinks that I'm no longer subscribed to it after ~15s, even though I am?

logger=live.runstream t=2023-03-21T01:54:15.291216774Z level=debug msg="Stop stream since no active subscribers" channel=1/ds/OzkQYZf4z/1s/TestData/timeElapsed path=1s/TestData/timeElapsed
logger=live.runstream t=2023-03-21T01:54:15.291510139Z level=debug msg="Stream cleanly finished" path=1s/TestData/timeElapsed
logger=plugin.grafana-mqtt-datasource t=2023-03-21T01:54:15.291597254Z level=debug msg="stopped streaming (context canceled)" path=1s/TestData/timeElapsed
logger=plugin.grafana-mqtt-datasource t=2023-03-21T01:54:15.291625607Z level=debug msg="Unsubscribing from MQTT topic" topic=TestData/timeElapsed

Despite it thinking that I'm not subscribed it continues sending blank updates ("data":{"values":[[],[]]}) every second until the client actually sends an unsubscribe message

[gpinkos]

I am still on OSS v9.4.3. Per the suggestion from @gdxas, I have set the time selector to 5 minutes with no refresh, and the MQTT streaming data is now working for more than 20 minutes (even though only 5 minutes are displayed on the graph).

[jasuade]

Could be something as in here the broker close old connection because the client id is same?

[peter9091]

I have the same problem with Grafana 9.4.7 on Raspberry.
I started mosquitto_sub in another window, so i can see that the messages are still published every second, but MQTT Source stops after receiving a few messages, sometimes after a few minutes.

In my case it's not because of the same client ID, I checked the logs and there is no "already connected" in mosquitto.log, that appears when I try to connect to mosquitto by mosquitto_sub using the same client ID as grafana MQTT source.

This seems to be a bug of this source.

[chefscho]

Hi,

after some more investigation my initial problem (not receiving any palyoads) was caused by the topic I was trying to subscribe to.
Publishing - for testing purposes - on a topic without leading "/" is working. So far the stream is working - no outages until now. Will keep you guys updated.

[PiotrMarzecVSG]

That was my initial assumption as well but it turned out to be not true. I can have just one client publishing one small topic, without any "/" signs, running and then stopping at random points.

I tried different panel types - it doesn't matter.
I also tried increasing the freqency of data publishing to 0.5 sec - did not help.

Nothing is being added to GrafanaLabs\grafana\data\log when data stops.

However, looks like it is somehow related to the selected time range - both on the dashboard and Data Source Explore functionality. Whenever data stops, I change the time range and data is being visible and updated again. Refreshing the dashboard, refreshing the browser, restarting the broker and cancelling/run query (in Data Source Explore) do not help but changing time range always helps - still temporarily though.

[Neumi]

OK. Here is the settins I use to make it work.

For the dashboard I run in now-5m with no refresh. I set up all the widgets with MQTT, and then I restart the grafana service. After that it seams to run stable.

But if I change query options it will some times stop working again until i restart grafana service.

MQTT plugin is still in beta I see.

I can confirm that this method works for me as well!

[wemod123]

However, looks like it is somehow related to the selected time range - both on the dashboard and Data Source Explore functionality. Whenever data stops, I change the time range and data is being visible and updated again. Refreshing the dashboard, refreshing the browser, restarting the broker and cancelling/run query (in Data Source Explore) do not help but changing time range always helps - still temporarily though.

Exactly what I am experiencing now

[mike006322]

I'm experiencing the same issue.

MQTT logs say that Grafana MQTT client unsubscribed from the topic.

Only unsubscribe invocations in source code are in pkg/stream.go on lines 27 and 35.

[isarantidis]

The plugin registers requests for MQTT topic data in a map with key format "<TIME_INTERVAL>/<MQTT_TOPIC>". This is derived from the Grafana Live channel e.g. ds/be19d38d-2126-430f-95b1-d2eebd9ef710/500ms/MQTT_TOPIC_A. Some times due to events coming from the UI (not sure which the full set of those cases is), the panels subscribe to a different Grafana Live channel, where the interval is different e.g. ds/be19d38d-2126-430f-95b1-d2eebd9ef710/1000ms/MQTT_TOPIC_A.

The problem is that the plugin's backend code as it is, will insert into a map the new "requests for data" with key 1000ms/MQTT_TOPIC_A but then it will remove from the map the previous entry 500ms/MQTT_TOPIC_A and cancel the subscription to MQTT_TOPIC_A. Therefore the subscribers with key 1000ms/MQTT_TOPIC_A will not receive any data and therefore neither will the panels.

I have modified the backend code in my repository fork, in order to not cancel the subscription to MQTT topic if there is at least on other "request for data" from it. I built the plugin and the problem is solved for me at least.

[lzaugg]

Thanks @isarantidis for analysing and providing a fix for (parts of) this issue. Works much more reliable on our end with the patch. Hope someone will review the PR #76 soon.

[JohanScheepers]

Also exercising stability issues.

Grafana v10.0

[goerdi]

10.1 the same....

[effgee]

Still happening Grafana v10.0.2-cloud
Haven't tried implementing isarantidis's fix, but not sure how I'd do that with the cloud version anyway

[KhurramFHassan]

I have the same problem on grafana version 10.0.3 running on Ubuntu on a raspberry pi 4. Can someone please post instructions on compiling the mqtt plugin incorporating @isarantidis patch?

[patsuhiko]

Same here, activated two queries then the data will sometimes show up with the correct values and then the stat visu will display "No data", but the data is there when i check it with MQTT explorer.

[toddtreece]

v1.0.0-beta.3 has been released with the fix from #84

[stenfalp]

I have the same problem with v1.0.0-beta.3. It only works with 5mins no refresh, but on the same dashboard I have visualizations that require 24h of data, so it is not a solution for me.
(Grafana 10.1.2 on RPi 3B+)

[GalamexMc]

Has anyone been able to find a solution to this issue for Grafana Cloud? Thanks.

[frankvandenhurk]

I have the same problem with v1.0.0-beta.3. It only works with 5mins no refresh, but on the same dashboard I have visualizations that require 24h of data, so it is not a solution for me. (Grafana 10.1.2 on RPi 3B+)

EDIT: It seems that it can be solved to set "Relative time" (under Query options) to a low value (in my case 1 minute). With the added benefit of the "live data" experience, since the Stat visualisation showes the values from the last minute in a very nice way (my source updates about every 2 seconds).

@stenfalp could you test?