Add JSON field support #55

toddtreece · 2022-11-18T04:38:06Z

buffer in chunks and push chunks on the query interval
add JSON field type support
update form layouts

ryantxu · 2022-11-18T10:28:55Z

go.mod

-go 1.16
+go 1.19
+
+replace github.com/grafana/grafana-plugin-sdk-go => /home/todd/grafana-plugin-sdk-go


^^^. remove before commit

removed in 88e5496

ryantxu · 2022-11-18T10:40:50Z

src/plugin.json

@@ -37,7 +37,7 @@
    "updated": "%TODAY%"
  },
  "dependencies": {
-    "grafanaDependency": ">=8.0.0",
+    "grafanaDependency": ">=9.2.0",


any reason to bump all the way to 9.2? with 8.5, i think all the relevant APIs are the same

SecretInput is missing pre 9.1.x, so i set the dependency to 9.1.0 in 5105e8d. i was thinking that the first official release should target a recent 9.x version since we don't have a lot of bandwidth to support 8.x issues. thoughts?

sounds reasonable -- especially for cloud

pkg/plugin/datasource.go

ryantxu · 2022-11-18T10:43:23Z

pkg/plugin/datasource.go

 	return &MQTTDatasource{
 		Client:        client,
-		channelPrefix: fmt.Sprintf("ds/%s/", uid),
+		channelPrefix: path.Join("ds", uid),


does this always use /? on windows does it use \

path/filepath is OS aware, but path.Join always uses /

ryantxu · 2022-11-18T10:46:48Z

This looks good -- What do you think about copying:
https://github.com/grafana/grafana/blob/main/pkg/services/live/pipeline/json_to_frame.go#L102

to this plugin and using it directly? Things got stalled with this plugin since I had hoped to use that in grafana server, but I don't see us getting that super production ready anytime soon

toddtreece · 2022-11-22T02:24:45Z

pkg/mqtt/framer.go

+	fieldMap map[string]int
+}
+
+func (df *framer) next() error {


@ryantxu i attempted to modify the framer you linked to fit this use case. is this kind of what you were thinking?

toddtreece · 2022-11-22T02:27:07Z

.bra.toml

@@ -0,0 +1,22 @@
+# default configuration created by the `mage watch` command.


mage watch was added in grafana/grafana-plugin-sdk-go#563

NiklasCi · 2022-11-23T14:39:53Z

Hey. First of all thanks for implementing this.
I tried out your PR and can give some feedback:

The connection is very unstable. Tested with two brokers on different machines but the stream is failing alot:

INFO [11-23|14:56:13] MQTT Connecting                          logger=plugin.grafana-mqtt-datasource
ERROR[11-23|14:56:28] Error running stream, re-establishing    logger=live.runstream path=15s/test/json error="error running stream: rpc error: code = Unavailable desc = error reading from server: EOF" wait=400ms
ERROR[11-23|14:56:28] Error running stream, re-establishing    logger=live.runstream path=15s/test/json error="plugin unavailable" wait=0s
ERROR[11-23|14:56:28] Error running stream, re-establishing    logger=live.runstream path=15s/test/json error="plugin unavailable" wait=0s
ERROR[11-23|14:56:28] Error running stream, re-establishing    logger=live.runstream path=15s/test/json error="plugin unavailable" wait=200ms
ERROR[11-23|14:56:28] Error running stream, re-establishing    logger=live.runstream path=15s/test/json error="plugin unavailable" wait=400ms

Got also errors where the length of the fields has a missmatch

ERROR[11-23|14:54:19] failed to send data frame                logger=plugin.grafana-mqtt-datasource error="frame has different field lengths, field 0 is len 14 but field 1 is len 7" path=15s/test/json

Support JSON time values (RF3339)

case jsoniter.StringValue:
    v := df.iterator.ReadString()
    time, err := time.Parse(time.RFC3339, v)
    if err != nil {
        df.addValue(data.FieldTypeNullableString, &v)
    } else {
	df.addValue(data.FieldTypeNullableTime, &time)
    }

I don't know what exactly triggers the two errors above, but with main i don't have these problems as it seems. Maybe i do something wrong ?

My test value:

{
    "timestamp": "2022-11-08T09:05:50.989644528Z",
    "available": true,
    "ownership": "",
    "attached": true,
    "state": "attach",
    "P": -65440,
    "Q": -1930,
    "PSet": -65229.046875,
    "QSet": -1985.50830078125,
    "PLim": [
        100000,
        100000,
        100000,
        100000
    ]
}

toddtreece · 2022-11-23T15:03:35Z

The connection is very unstable. Tested with two brokers on different machines but the stream is failing alot:

are there any other errors in your logs? it kind of seems like the plugin might be panicking and restarting

errors where the length of the fields has a missmatch

@NiklasCi thanks for testing! i'll follow up with a fix and test case for the field length issue.

Support JSON time values (RF3339)

this one could be nice to support in a future version if others request it, but for now i think it's covered by the convert field type transformation:

NiklasCi · 2022-11-23T15:15:18Z

are there any other errors in your logs? it kind of seems like the plugin might be panicking and restarting

Ok tested back. Only the field lenght issue is reproducible. So the first issue could be on my end.

this one could be nice to support in a future version if others request it, but for now i think it's covered by the convert field type transformation:

Ah didn't know about the transformer. So my suggestion is not needed.

toddtreece · 2022-11-23T15:17:28Z

@NiklasCi great, thanks for checking!

toddtreece · 2022-11-23T21:31:10Z

@NiklasCi i think i have resolved the issues with field length. let me know if that works for you

NiklasCi · 2022-11-24T08:24:56Z

@NiklasCi i think i have resolved the issues with field length. let me know if that works for you

Works like a charm now. Even with big json files and arrays containing objects.
Thanks alot.

toddtreece added 2 commits November 17, 2022 23:32

update UI to 9.2.0

e69279b

stream on interval

c4b9548

toddtreece requested a review from ryantxu November 18, 2022 04:38

ryantxu reviewed Nov 18, 2022

View reviewed changes

pkg/plugin/datasource.go Show resolved Hide resolved

ryantxu reviewed Nov 18, 2022

View reviewed changes

toddtreece added 3 commits November 21, 2022 11:24

remove replace in go.mod

88e5496

add JSON framer

177036f

set dependencies to 9.1.0

5105e8d

toddtreece requested a review from ryantxu November 22, 2022 02:22

toddtreece commented Nov 22, 2022

View reviewed changes

toddtreece requested a review from a team November 22, 2022 12:48

toddtreece self-assigned this Nov 22, 2022

update framer to handle dynamic types and sparse values

af9758a

toddtreece force-pushed the toddtreece/nested-json branch from 3927d27 to af9758a Compare November 23, 2022 20:54

toddtreece added 2 commits November 23, 2022 16:23

update README

8f43989

fix lint err

5f3ea06

nmarrs added the enhancement New feature or request label Nov 29, 2022

ryantxu approved these changes Nov 30, 2022

View reviewed changes

toddtreece merged commit c11488f into main Nov 30, 2022

toddtreece deleted the toddtreece/nested-json branch November 30, 2022 19:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add JSON field support #55

Add JSON field support #55

toddtreece commented Nov 18, 2022

ryantxu Nov 18, 2022

toddtreece Nov 22, 2022

ryantxu Nov 18, 2022 •

edited

toddtreece Nov 22, 2022

ryantxu Nov 30, 2022

ryantxu Nov 18, 2022

toddtreece Nov 21, 2022

ryantxu commented Nov 18, 2022

toddtreece Nov 22, 2022

toddtreece Nov 22, 2022

NiklasCi commented Nov 23, 2022

toddtreece commented Nov 23, 2022

NiklasCi commented Nov 23, 2022

toddtreece commented Nov 23, 2022

toddtreece commented Nov 23, 2022

NiklasCi commented Nov 24, 2022

		@@ -0,0 +1,22 @@
		# default configuration created by the `mage watch` command.

Add JSON field support #55

Add JSON field support #55

Conversation

toddtreece commented Nov 18, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryantxu Nov 18, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryantxu commented Nov 18, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NiklasCi commented Nov 23, 2022

toddtreece commented Nov 23, 2022

NiklasCi commented Nov 23, 2022

toddtreece commented Nov 23, 2022

toddtreece commented Nov 23, 2022

NiklasCi commented Nov 24, 2022

ryantxu Nov 18, 2022 •

edited