Failed to fetch query data: Data not showing for query type 'get property value aggregates' and 'get property value history'

[egheie]

Different issues with grafana-iot-sitewise-datasource when using get property value aggregates and get property value history query types. Really like the ability to zoom out and quickly get an overview of the data for a longer timeframe, and zoom in to relevant time intervals. Sometimes not all the data is being visualized, and other times nothing is showing for unknown reasons, since data is available through other APIs, usch as boto3 or directly from AWS CLI.

I understand it is a lot of information below, so feel free to reach out if something does not make sense or you have any other questions.

Thank you!

What happened:
Different versions of grafana-iot-sitewise-datasource and with query type get property value aggregates and get property value history does not show any data, or all the data when data is availble and can be obtained through either AWS CLI or boto3

What you expected to happen:
Expect the available data in the selected timeframe to be displayed in the panels.

How to reproduce it (as minimally and precisely as possible):

There are different issues, but below is a more detailed description of the different cases, including screenshots and descriptions on how to replicate.

Anything else we need to know?:

Environment:

• Grafana version: Different ones
• Plugin version: 1.4.0 - 1.6.0

Longer description of the problem we are facing

A document to state issues with the different versions of IoT SiteWise datasource, with different test cases.

General comments

• Version 1.6.0 are not stable at all, and continuously show error messgae "an error occured within the plugin". While having the same timeframe, could get data, some data or no data just by pressing the refresh dashboard a few times. Seem to have greater difficulty when there is more than 250 data points in the query.
• Seems like expression limits the number of data points, and usually stops at 250 (which suscipscously is the same number as the maxResults in the API.
• Quickly checked version 1.4.0, which had similar if not same results as 1.4.1.
• Aggregates seem to break if the first query would result in an empty response but with a nextToken for values with a more recent timestamp.
• The number of total data points and in the first response/batch from query is dependent on the data source and timeframe. The source for data used in the tests was created once a minute, and therefore the timeframe/duration and number of was often the same.

Test 1 with data points on both side of desired timeframe

For this test, ensure that there exists data points before and after time interval. Meaning it should be a subset like the figure shows below:

Test case 1.1 using GetAssetPropertyAggregates

Test 1.1.1 - Grafana IoT SiteWise datasource version 1.4.1

Test 1.1.1.1 - single query

Aggregate resolution 'AUTO' in Grafana or '1m' in API, and duration of 250 minutes (250 points)

Checks	Grafana	Boto3	AWS CLI
Total # points	250	250	250
# points 1st batch	250(?)	100	250
nextToken	NA	NA	NA
success?	Yes	Yes	Yes

Aggregate resolution 'AUTO' in Grafana or '1m' in API, and duration of 251 minutes (251 points)

Checks	Grafana	Boto3	AWS CLI
Total # points	251	251	251
# points 1st batch	250(?)	100	251
nextToken	Yes	Yes	NA
success?	Yes	Yes	Yes

Test 1.1.1.2 - single query + expression

Add an expression (e.g. Math operation, $A+1)
Adding an expression limits the number of values in the response.

Aggregate resolution 'AUTO' in Grafana or '1m' in API, and duration of 311 minutes (311 points)

Checks	Grafana	Boto3	AWS CLI
Total # points	250	311	311
# points 1st batch	250(?)	100	311
nextToken	Yes	Yes	NA
success?	No	Yes	Yes

Description if not succesful: stops after 250 data points, and graph does not show any more.

Test 1.1.2 - Grafana IoT SiteWise datasource version 1.5.0

Test 1.1.2.1 - single query

Usually works, but depending on the timeframe (Get different number of results by selecting '<' or '>' next to the timeframe). Sometimes it breaks even though property history value indicates there is data.

Test 1.1.2.1 - single query + expression

Same as previous version. Stops after 250 data points, and graph does not show any more.

Figure illustrates the problem. Panel on top is 'Stat' with 'Count' calculation. Panel on bottom is timeseries with a math expression ($A-1000)

Test 1.1.3 - Grafana IoT SiteWise datasource version 1.5.1

Test 1.1.3.1 - single query

Same as version 1.5.0

Test 1.1.3.2 - single query + expression

Same as before, breaks at 250.

Test case 1.2 using GetAssetPropertyValueHistory

Test 1.2.1 - Grafana IoT SiteWise datasource version 1.4.1

Get Property Value History and duration of 250 and 251 minutes for our case (1 test point a minute maxResults from API is 250)

Test 1.2.1.1 - single query

Duration of 250 minutes (250 points)

Checks	Grafana	Boto3	AWS CLI
Total # points	250	250	250
# points 1st batch	250(?)	100	250
nextToken	No	Yes	NA
success?	Yes	Yes	Yes

Duration of 251 minutes (251 points)

Checks	Grafana	Boto3	AWS CLI
Total # points	251	251	251
# points 1st batch	250(?)	100	251
nextToken	Yes	yes	NA
success?	Yes	Yes	Yes

Test 1.2.1.1 - single query + expression

Checks	Grafana	Boto3	AWS CLI
Total # points	250	311	311
# points 1st batch	250(?)	100	311
nextToken	Yes	Yes	NA
success?	No	Yes	Yes

Description if not succesful: stops after 250 data points, and graph does not show any more. See figure below, same response was showing for the aggregate test above.

From Grafana:

From Boto3:

Test 1.2.2 - Grafana IoT SiteWise datasource version 1.5.0

Test 1.2.2.1 - single query

Usually works, but depending on the timeframe (Get different number of results by selecting '<' or '>' next to the timeframe). Sometimes it breaks even though property history value indicates there is data. Seems to be more unstable close to areas where datapoints are starting. (E.g. test case 2, where it shows 'No data' until a certain point when data points seem to be covering the majority of the timeframe.)

Test 1.2.2.1 - single query + expression

Same as previous version. Stops after 250 data points, and graph does not show any more. See similar problem in test 1.1.2.1

Test 1.2.3 - Grafana IoT SiteWise datasource version 1.5.1

Test 1.2.3.1 - single query

Same as previous versions.

Test 1.2.3.2 - single query + expressions

Breaks and show 'No Data'
Works when changing 'Format' to time series, but this is new compared to earlier versions. Does still break after 250 values.

Test 2 with available data points at the end desired timeframe

For this test, ensure that there exists data points from around half of the timeframe and later. Should also be more than 250 data points availble in the frame (more than maxResults requirement for the aws sitewise API). Meaning it should be a subset like the figure shows below:

Test case 2.1 using GetAssetPropertyAggregates

Test 2.1.1 - Grafana IoT SiteWise datasource version 1.4.1

Test 2.1.1.1 - single query

Aggregate resolution 'AUTO' in Grafana or '1m' in API, and duration of 630 minutes

Checks	Grafana	Boto3	AWS CLI
Total # points	No Data	260	260
# points 1st batch	No Data	0	260
nextToken	NA	Yes	NA
success?	No	Yes	Yes

Description of failure: showing "No data", even though we know there exists data. Could it be because the first query gives no values in response, and somehow it does not check if there is a nextToken for values later in the timeframe?

Test 2.1.2 - Grafana IoT SiteWise datasource version 1.5.0

Same result for version 1.5.0. See comment in Test 1.2.2.1 - single query

Test 2.1.3 - Grafana IoT SiteWise datasource version 1.5.1

Same result as earlier. Show 'No Data'

Test case 2.2 using GetAssetPropertyValueHistory

Test 2.2.1 - Grafana IoT SiteWise datasource version 1.4.1

Get Property Value History and duration of more than 250 minutes for our case (1 test point a minute maxResults from API is 250)

Test 2.2.1.1 - single query

Duration of 250 minutes

Checks	Grafana	Boto3	AWS CLI
Total # points	269	269	269
# points 1st batch	250(?)	100	269
nextToken	Yes	Yes	NA
success?	Yes	Yes	Yes

Test 2.2.1.1 - single query + expression

Checks	Grafana	Boto3	AWS CLI
Total # points	250	269	269
# points 1st batch	250(?)	100	269
nextToken	Yes	Yes	NA
success?	No	Yes	Yes

Description if not succesful: stops after 250 data points, and graph does not show any more. Show difference of response just by adding a math expression to the query ($A+1).

Test 2.2.2 - Grafana IoT SiteWise datasource version 1.5.0

Same result for this version

Test 2.2.3 - Grafana IoT SiteWise datasource version 1.5.1

Same result for this version

Test 3 with available data points at the beginning desired timeframe

Test 3 with Grafana IoT SiteWise datasource

Version 1.4.1, 1.5.0, 1.5.1

This test case results in the same as Test 1, having data points throughout the timeframe as wellas before and after. Works, but limits the datapoints when adding expressions for both aggregates and raw data (history value).

[tracy-french]

Hi! :)

What's the status of this item? I'm reaching out on behalf of the customer who reported this issue to get an update. Thank you!

[kevinwcyu]

Hi, this is currently in progress.

[RubenEmpresa]

Hi! :)

We are currently working on a project for a customer and we are experiencing the same issue. When doing query that includes math operation only returns 250 values.

When doing the query without math it returns all the values (even if it is more than 250).

Here you can see the query without math:

Here the query with the math:

[kevinwcyu]

Hi, this should be fixed in v1.9.2

[egheie]

Hi Kevin,

Thanks for the update. It helped us to be able to show data that earlier we were not able to do, but seems to not completely solve the problem.

I tested the new version (1.9.2) with Grafana version 10.0.0.

With my current setup, I get the following response:

By downloading the data and looking at it, there are duplicates of most of the values. Without knowing, I suspect it is the paginated response that is duplicated.

Using Boto3 - get_asset_property_aggregates I get the expected value of 600 data points.

As the screenshot shows above, the expression fails when using the value aggregates (top right corner), only showing 1 data point. By simply taking the expression query away, the result would be the same as top left corner (as expected).

Similarly, the expression does not get all the values for value history either. Again, I suspect that this is something with pagination/next token, not being able to show all the result.

Please let me know if anything is unclear or you require more information.

[kevinwcyu]

Hi @egheie, thanks for trying it out and providing an update!

duplicates of most of the values
To clarify is this shown in the screenshot you have with the 1080 points? Should that be 600?

Would it be possible for you to provide the queries you have for each of the panels? Screenshots of the query editor with any private info redacted would work.

[egheie]

Hi @kevinwcyu, no problem.

Here are the screenshots, starting bottom left, going clockwise. Note: every quadrant will have the same query, just the panel is different. (Either Time series, or Stat with Calculation = Count).

When doing this, I saw that the bottom left Stat was set to Get property value aggregates and not Get property value history. See new screenshot at the bottom, sorry for that.

Screenshots of query editor:
Bottom left

Top left

Top right

Bottom right

Updated screenshot:

[kevinwcyu]

Hi @egheie, I've only been able to recreate the property aggregate issue with duplicate values. I've moved that to a new issue #200 to be able to track the work and make sure it doesn't get lost.

For the queries with expressions, I tried with different time ranges and resolutions, but haven't been able to recreate it. I'm using the demo data that is available to be set up in the AWS Sitewise Console to try to reproduce this. Do you think you'd be able to try to reproduce it with that data?

[egheie]

Hi @kevinwcyu, thanks for your support and that you follow up the duplicated values in a new issue.

I could reproduce it with the demo data. See gif below for the result. So you had no issues with the expressions and both time series (query + expression) cover the entire expected time range?

For all panels, the following was used:
Asset used: Demo Turbine Asset 1
Property: Wind Speed

[egheie]

Another short gif to show how it changes with the expression being disabled or not and by removing the expression completely.

There is also something a bit funky happening with the name in the legend in the bottom right corner when adding an expression. Raised a similar issue in #198, but worth mentioning here as well.

[kevinwcyu]

@egheie, thanks for your patience on this. I was finally able to reproduce the issue and I've opened a ticket for it here #202. I've moved the ticket to be prioritized in our next sprint.

[egheie]

Hi @kevinwcyu, thanks for the update and appreciate the issue is being looked at in the next sprint.