Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CloudWatch GetMetricData support #11624

Merged
merged 5 commits into from Jul 22, 2018

Conversation

Projects
None yet
@mtanda
Copy link
Collaborator

commented Apr 17, 2018

Fixes #11487

I introduce two fields, "Id" and "Expression".

These fields are pass as MetricDataQuery, and other parameters are filled by existing fields.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDataQuery.html

If user uses templating expansion in dimension value, the Id is populated by templating value.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html

The populated Id is useful for calculating sum by SUM(METRICS("<Id>")).

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 17, 2018

Waiting for aws/aws-sdk-go#1892 fix release.

@mtanda mtanda force-pushed the mtanda:cw_get_metric_data branch 2 times, most recently from 2a35655 to 0b7eee0 Apr 24, 2018

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 24, 2018

aws/aws-sdk-go#1892 is released and applied to this PR.
Ready for merge.

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 24, 2018

@bergquist Would you check this changes?

@mtanda mtanda force-pushed the mtanda:cw_get_metric_data branch 2 times, most recently from 1a5bb8f to 9364c8e May 8, 2018

@daniellee daniellee self-requested a review May 21, 2018

@daniellee daniellee self-assigned this May 21, 2018

@daniellee
Copy link
Member

left a comment

Hi @mtanda how do you use this feature? Could you give some examples of how to use math in the expression.

We will also need to add some documentation for this feature (I can help!)

@@ -31,6 +31,13 @@
</div>
</div>

<div class="gf-form-inline">
<div class="gf-form">
<label class="gf-form-label query-keyword width-8">Expresion</label>

This comment has been minimized.

Copy link
@daniellee

daniellee May 25, 2018

Member

Spelling - should be Expression

@mtanda mtanda force-pushed the mtanda:cw_get_metric_data branch from 9364c8e to e37893e May 28, 2018

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented May 28, 2018

At first, this feature has limitation, it doesn't support multiple statistics in one target.
User should specify only one statistics (e.g. Average).
I improve query editor in last commit, if user specify multiple statistics, extra field is disappeared.

Unless user set extra field, Id and Expression, the result is same as current one.

Id and Expression is explained in following docs.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html

User need to set Id for each time series, and then use it in Expression.

By using this feature, user can calculate like used / total = used rate.
To use with templating feature, generate unique id for each expanded target.
https://github.com/grafana/grafana/pull/11624/files#diff-5be6ea1dd05a4db39f000439bbca4f6aR397
User can simply using aggregate function like "AVG" with expanded target.

If user don't want to show the intermediate result, user can hide it by clicking eye icon in query editor, but it still available in expression.


I provide two examples.

Example 1: calculating DynamoDB capacity usage rate
calculate ConsumedReadCapacityUnits per second usage.
image

divide ConsumedReadCapacityUnits per second usage by ProvisionedReadCapacityUnits, get used rate.
image

Example 2: calculating Average of CPU Utilization of All EC2 instance
image

@mtanda mtanda force-pushed the mtanda:cw_get_metric_data branch from e37893e to e229d87 May 28, 2018

<input type="text " class="gf-form-input " ng-model="target.id " spellcheck='false' ng-model-onblur ng-change="onChange() ">
</div>
<div class="gf-form max-width-30 ">
<label class="gf-form-label query-keyword width-7 ">Expresion</label>

This comment has been minimized.

Copy link
@gcacace

gcacace May 28, 2018

Typo here: expresion

This comment has been minimized.

Copy link
@mtanda

mtanda May 29, 2018

Author Collaborator

I forgot to fix it. Thanks for pointing out.

@mtanda mtanda force-pushed the mtanda:cw_get_metric_data branch from e229d87 to fb0cd5a May 29, 2018

@daniellee

This comment has been minimized.

Copy link
Member

commented Jun 21, 2018

@mtanda I tested now and had some problems:

  • If there are template variables in the query then the query id gets a unique uid added to the end: https://github.com/mtanda/grafana/blob/fb0cd5a29d84165d261f800cbcd9df7a21478c1b/public/app/plugins/datasource/cloudwatch/datasource.ts#L397

    So this query does not work as the query id for the first query gets set to network_packets_inaS1iMGU4YTQ0Nw00:
    image

    If I change the expression to network_packets_inaS1iMGU4YTQ0Nw00/2 then it works or if I take out all the template variables from the query then it works.

  • If the second query does not have an id then it does not work and the error message does not help much:

    InvalidParameter: 2 validation error(s) found.
    - minimum field size of 1, GetMetricStatisticsInput.MetricName.
    - minimum field size of 1, GetMetricStatisticsInput.Namespace.
    

@mtanda mtanda force-pushed the mtanda:cw_get_metric_data branch from fb0cd5a to 4f2fe91 Jun 22, 2018

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 22, 2018

@daniellee Thank you for feedback!

So this query does not work as the query id for the first query gets set to network_packets_inaS1iMGU4YTQ0Nw00:

For this issue, I add check the variable is multi or not.
If variable is not multi, network_packets_in / 2 should be valid expression, so don't generate unique id, user can use network_packets_in in expression.
If variable is multi, datasource should generate unique id, and user need to use expression like AVG(METRICS("network_packets_in")) / 2. This is expected behavior, because user set variable as multi.

If the second query does not have an id then it does not work and the error message does not help much:

Improve error message. It is little tricky, so will need documentation...

@mdgbayly

This comment has been minimized.

Copy link

commented Jul 7, 2018

@mtanda Thanks for working on this - this is something we would really like to see supported by Grafana. We've had to switch back to Cloudwatch for some dashboards due to inability to use math expressions with ConsumedReadCapacityUnits.

Regarding using this feature for DynamoDB capacity utilizations, do you know if it will be possible in Grafana to utilize the current period in the calculation.

E.g. above you show calculating used_per_sec as used / 60 (i realize you are setting min period to 60).
In Cloudwatch math expressions you can include the period in the expression.
used / PERIOD(used).

Does the Grafana support allow using any Cloudwatch Math functions in an expression?

@mtanda mtanda force-pushed the mtanda:cw_get_metric_data branch from e89826e to 7722045 Jul 9, 2018

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 9, 2018

@mdgbayly

Does the Grafana support allow using any Cloudwatch Math functions in an expression?

Yes, you can use all Math functions.

To calculate used_per_sec, your example is correct!
used / PERIOD(used).

@daniellee

This comment has been minimized.

Copy link
Member

commented Jul 12, 2018

@mtanda I’m on vacation this week (and last week). Will merge this on Monday after testing again.

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 12, 2018

@daniellee No problem. I know it. Thanks for informing me :-)

@daniellee daniellee added this to the 5.3 milestone Jul 22, 2018

@daniellee

This comment has been minimized.

Copy link
Member

commented Jul 22, 2018

Merging now. There is one more slightly strange thing that some people might get stuck on - that id's have to start with a lowercase letter. From the docs:

You can change the value of Id. It can include numbers, letters, and underscore, and must start with a lowercase letter

Not sure what to do about it though. Maybe should an info popover for the id field. What do you think Mtanda?

@daniellee daniellee merged commit 5129ea8 into grafana:master Jul 22, 2018

8 checks passed

ci/circleci: build-all Your tests passed on CircleCI!
Details
ci/circleci: codespell Your tests passed on CircleCI!
Details
ci/circleci: gometalinter Your tests passed on CircleCI!
Details
ci/circleci: mysql-integration-test Your tests passed on CircleCI!
Details
ci/circleci: postgres-integration-test Your tests passed on CircleCI!
Details
ci/circleci: test-backend Your tests passed on CircleCI!
Details
ci/circleci: test-frontend Your tests passed on CircleCI!
Details
license/cla Contributor License Agreement is signed.
Details
@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 23, 2018

@daniellee Thanks for merging!
I didn't notice id's limitation. I'll add popover or input validator.

@kadern0

This comment has been minimized.

Copy link

commented Sep 11, 2018

Hi @mtanda,
great new feature! I'm trying to use it but unfortunately if I enter any value in the Id field all datapoints disappear.
Any ideas?

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 11, 2018

Hi @kadern0,

At first, you or Grafana running instance should have privilege to call new cloudwatch:GetMetricData API.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html

Please check your IAM policy.
If IAM policy is correct and Grafana report error, please share error message.

@rickbowden

This comment has been minimized.

Copy link

commented Sep 28, 2018

@mtanda
Thanks so much for working on this. I've spent quite some time attempting to make aggregated panels using CloudWatch datasource, this will make it so much better.
Thanks again.

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Oct 9, 2018

Hi @gipeshka,

Would you capture your query editor and share with me?
And if you have a time, would you try for simple EC2 metrics (e.g. CPUUtilization)?

@gipeshka

This comment has been minimized.

Copy link

commented Oct 9, 2018

Hi @mtanda.

Thank you for a quick reply. I was able to solve the issue. With the new api apparently one should pay more attention to the naming (first I used "-" symbol in the id, grafana didn't complain, however I got no data) and to general query structure. Maybe some additional validation with more explicit errors would help the case.

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Oct 10, 2018

@gipeshka Sorry, Id input validation has bug.
I fix it in #13583 .

@gipeshka

This comment has been minimized.

Copy link

commented Oct 10, 2018

@mtanda another quick question, does grafana support alerts on the calculated metrics/metrics with Id?
When I try to test such an alert grafana always returns no_data.

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Oct 10, 2018

@gipeshka Now I understand Grafana alerting model...
Currently, alerting doesn't work.

When alerting expression evaluated, only one expression is passed to datasource.
exp A (id=a): CPUUtlization
exp B (id=b): AVG(a)

And then, set alert rule for exp B, only AVG(a) is passed to datasource.

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Oct 10, 2018

Graphite datasource can handle such nested query alert.

I'll try to fix metric math alerting.
#9917

@tphambgs

This comment has been minimized.

Copy link

commented Mar 21, 2019

Hi @mtanda,

I'm trying to use repeat math expression for a multi value dimension:

image

In the screenshot above, $tables is a list of DynamoDB tables. The query generates a unique used ID for each table but doesn't repeat this ID in the expression.

Is there a use-case where we wouldn't want to repeat this ID?

@mayukuse24

This comment has been minimized.

Copy link

commented Mar 23, 2019

At first, this feature has limitation, it doesn't support multiple statistics in one target.
User should specify only one statistics (e.g. Average).
I improve query editor in last commit, if user specify multiple statistics, extra field is disappeared.

Unless user set extra field, Id and Expression, the result is same as current one.

Id and Expression is explained in following docs.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html

User need to set Id for each time series, and then use it in Expression.

By using this feature, user can calculate like used / total = used rate.
To use with templating feature, generate unique id for each expanded target.
https://github.com/grafana/grafana/pull/11624/files#diff-5be6ea1dd05a4db39f000439bbca4f6aR397
User can simply using aggregate function like "AVG" with expanded target.

If user don't want to show the intermediate result, user can hide it by clicking eye icon in query editor, but it still available in expression.

I provide two examples.

Example 1: calculating DynamoDB capacity usage rate
calculate ConsumedReadCapacityUnits per second usage.
image

divide ConsumedReadCapacityUnits per second usage by ProvisionedReadCapacityUnits, get used rate.
image

Example 2: calculating Average of CPU Utilization of All EC2 instance
image

Hi @mtanda

Will the support for multiple statistics be added anytime soon or is it in progress?

@adithyamenon

This comment has been minimized.

Copy link

commented Apr 2, 2019

@daniellee Thank you for feedback!

So this query does not work as the query id for the first query gets set to network_packets_inaS1iMGU4YTQ0Nw00:

For this issue, I add check the variable is multi or not.
If variable is not multi, network_packets_in / 2 should be valid expression, so don't generate unique id, user can use network_packets_in in expression.
If variable is multi, datasource should generate unique id, and user need to use expression like AVG(METRICS("network_packets_in")) / 2. This is expected behavior, because user set variable as multi.

If the second query does not have an id then it does not work and the error message does not help much:

Improve error message. It is little tricky, so will need documentation...

I don't think this serves the purpose entirely.

For example, in my case, I am trying to plot the 'ConsumedWriteCapacityUnits' from DynamoDB keeping the list of tables as variable - '$tables'. In this case, it wouldn't make sense if I get the average of 'ConsumedWriteCapacityUnits' of the selected multiple DynamoDB tables. I would essentially want individual values for the multiple tables selected without the generated unique id as mentioned by @daniellee

Am I right here or am I missing something? @mtanda

I am currently using Grafana Version 5.4.2 (commit: d812109, branch: HEAD).

Screenshot 2019-04-02 at 1 12 31 PM

@mtanda

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 4, 2019

I misunderstand about some part of GetMetricData API.
This will fix issue.
#16383

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.