Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add statistic option to cloudwatch metricset config #12840

Merged
merged 10 commits into from Jul 25, 2019
Merged

Add statistic option to cloudwatch metricset config #12840

merged 10 commits into from Jul 25, 2019

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Jul 9, 2019

This PR is to add statistic as an optional parameter in cloudwatch metricset configuration. If there is no statistic given, then the default statistic methods will be applied: Average, Sum, Maximum, Minimum and SampleCount.

New cloudwatch configuration looks like:

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/EC2
      name: ["CPUUtilization", "DiskWriteOps"]
      tags.resource_type_filter: ec2:instance
      statistic: ["Maximum", "Minimum"]
    - namespace: AWS/EBS
      statistic: ["Average"]
    - namespace: AWS/ELB
      tags.resource_type_filter: elasticloadbalancing

How to test it

  • make sure in the aws account that you will use for testing has EC2, EBS and ELB created with tags on some of them
  • pull this PR and run mage update; mage build; ./metricbeat modules enable aws to build metricbeat and enable aws module
  • edit ./modules.d/aws.yml to only keep the part for cloudwatch metricset
  • start metricbeat if you are using environment variables for aws credentials: AWS_ACCESS_KEY_ID=<access-key-id> AWS_SECRET_ACCESS_KEY=<secret-access-key> AWS_SESSION_TOKEN=<session-token> AWS_REGION=us-east-1 ./metricbeat -e -c metricbeat.yml -d "publish"
  • or add credential_profile_name: test-mb into ./modules.d/aws.yml to use credential profile and starts metricbeat with ./metricbeat -e
  • check in Kibana to see if metrics are collected from:
    all EC2 instances with only CPUUtilization.max, DiskWriteOps.max, CPUUtilization.min and DiskWriteOps.min
    all EBS and ELB metrics
    EC2 and ELB with tags
    no tags for EBS
    EBS metrics with only .avg statistic
    ELB metrics with all 5 different statistics

closes #12370
closes #12933

@kaiyan-sheng kaiyan-sheng self-assigned this Jul 9, 2019
@kaiyan-sheng kaiyan-sheng added Team:Integrations Label for the Integrations team Metricbeat Metricbeat labels Jul 10, 2019
@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review July 15, 2019 18:23
@kaiyan-sheng kaiyan-sheng requested review from a team as code owners July 15, 2019 18:23
Copy link
Contributor

@exekias exekias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great! It will need a changelog entry (don't forget to add an entry in Breaking Changes section too)

x-pack/metricbeat/module/aws/_meta/config.yml Outdated Show resolved Hide resolved
Copy link
Contributor

@sayden sayden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks ok. I'll be nice to add some comments in some parts of the code so that it's easier to understand it in the future

@andresrc andresrc added the v7.4.0 label Aug 7, 2019
@kaiyan-sheng kaiyan-sheng added the test-plan Add this PR to be manual test plan label Aug 12, 2019
@sorantis
Copy link
Contributor

Nice work!
@kaiyan-sheng the CloudWatch documentation for statistics also mentions pNN.NN - The value of the specified percentile. Is it also included in support?

@kaiyan-sheng
Copy link
Contributor Author

Nice work!
@kaiyan-sheng the CloudWatch documentation for statistics also mentions pNN.NN - The value of the specified percentile. Is it also included in support?

Thanks! Yep it should support pNN.NN as well!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Metricbeat Metricbeat review Team:Integrations Label for the Integrations team test-plan Add this PR to be manual test plan v7.4.0
Projects
None yet
5 participants