Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat] Create ebs metricset using light module #13167

Merged
merged 5 commits into from Aug 8, 2019
Merged

[Metricbeat] Create ebs metricset using light module #13167

merged 5 commits into from Aug 8, 2019

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Aug 2, 2019

This PR is to use #12465 to create ebs metricset for aws. ebs metricset collects metrics from aws cloudwatch with namespace AWS/EBS using aws module cloudwatch metricset.

Amazon Elastic Block Store (Amazon EBS) sends data points to CloudWatch for several metrics. Amazon EBS General Purpose SSD (gp2), Throughput Optimized HDD (st1) , Cold HDD (sc1), and Magnetic (standard) volumes automatically send five-minute metrics to CloudWatch. Provisioned IOPS SSD (io1) volumes automatically send one-minute metrics to CloudWatch. Data is only reported to CloudWatch when the volume is attached to an instance. Please see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cloudwatch_ebs.html for more details on each metric.

For ebs metricset, default collection period is set to 5min with Average/Sum as statistic method.
Overview dashboard:
Screen Shot 2019-08-07 at 7 33 31 PM

closes #11699

How to test this:

  1. Make sure the AWS environment you are using for testing has EBS, which is a part of EC2. So to check if your environment has volumes, go to EC2 service and then click Volues on the left side under ELASTIC BLOCK STORE: https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Volumes:sort=desc:createTime
  2. Make sure AWS credentials are correct and valid in ~/.aws/credentials under profile test-mb
  3. Download Metricbeat and enable aws module: ./metricbeat modules enable aws
  4. Change aws.yml to enable ebs metricset:
- module: aws
  period: 300s
  credential_profile_name: test-mb
  metricsets:
    - ebs
  1. Check Kibana dashboard or Discover for ebs data

@kaiyan-sheng kaiyan-sheng self-assigned this Aug 2, 2019
@kaiyan-sheng kaiyan-sheng added Team:Integrations Label for the Integrations team [zube]: In Progress in progress Pull request is currently in progress. Metricbeat Metricbeat v7.4.0 labels Aug 2, 2019
@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review August 5, 2019 19:08
@kaiyan-sheng kaiyan-sheng requested review from a team as code owners August 5, 2019 19:08
@kaiyan-sheng kaiyan-sheng added [zube]: In Review review and removed [zube]: In Progress in progress Pull request is currently in progress. labels Aug 5, 2019
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

@kaiyan-sheng
Copy link
Contributor Author

jenkins, test this please

@kaiyan-sheng kaiyan-sheng merged commit 5ad66d3 into elastic:master Aug 8, 2019
@kaiyan-sheng kaiyan-sheng deleted the add_ebs branch August 8, 2019 18:24
sorantis pushed a commit to sorantis/beats that referenced this pull request Aug 9, 2019
* Add ebs metricset
* Change some metrics to use Sum instead of Average
@kaiyan-sheng kaiyan-sheng added the test-plan Add this PR to be manual test plan label Aug 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Metricbeat Metricbeat review Team:Integrations Label for the Integrations team test-plan Add this PR to be manual test plan v7.4.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Metricbeat] Adding AWS EBS Metricset
3 participants