[Metricbeat] Update cloudwatch metricset mappings

[kaiyan-sheng]

This PR is to continue working on #14653 to update mappings in cloudwatch metrics and all the metricsets that are created using light module based on cloudwatch metricset.

Mapping change:
aws.cloudwatch.metrics.BurstBalance.avg -> aws.ebs.metrics.BurstBalance.avg
aws.cloudwatch.dimensions.VolumeId -> aws.dimensions.VolumeId

TODO - fix dashboards:

• Fix ebs dashboard
• Fix elb dashboard
• Fix sns dashboard
• Fix aws overview dashboard
• Fix usage dashboard
• Fix billing dashboard

TODO - regenerate data.json files in metricsets:

• Regenerate data.json in cloudwatch
• Regenerate data.json in ebs
• Regenerate data.json in elb
• Regenerate data.json in usage
• Regenerate data.json in billing
• Regenerate data.json in sns

How to test this

Make sure the AWS account you use for testing has multiple different services. Enalbe aws metricbeat module and modify aws.yml with:

- module: aws
  period: 1m
  metricsets:
    - usage
- module: aws
  period: 12h
  metricsets:
    - billing
- module: aws
  period: 5m
  metricsets:
    - ebs
    - elb
    - sns

• Make sure ebs, elb, usage, billing, sns metricsets are reporting metrics
• Make sure dashboards from the metricsets above are still working as well with the new mapping

[kaiyan-sheng]

Tried to change mapping to aws.*.*.* instead of aws.*.metrics.*.* didn't work. When trying with cloudwatch metricset to collect from AWS/EC2 namespace and the same time running ec2 metricset with config:

- module: aws
  period: 5m
  metricsets:
    - cloudwatch
  credential_profile_name: elastic-beats
  metrics:
    - namespace: AWS/EC2
      name: ["CPUUtilization", "DiskWriteOps"]
      tags.resource_type_filter: ec2:instance
      statistic: ["Average"]
- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  metricsets:
    - ec2

I got error:

2019-12-20T14:38:57.997-0700    WARN    elasticsearch/client.go:516     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf776db6996b7df8, ext:2793788450, loc:(*time.Location)(0x95278e0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"444e60bd-5a6f-4695-867f-f340e10bc9d9", "hostname":"KaiyanMacBookPro", "id":"23fd6ee9-2255-43f3-9565-78a051337c04", "type":"metricbeat", "version":"8.0.0"}, "aws":common.MapStr{"ec2":common.MapStr{"cpu":common.MapStr{"credit_balance":56.963139, "credit_usage":0.006812, "surplus_credit_balance":0, "surplus_credits_charged":0, "total":common.MapStr{"pct":0.1333880080189532}}, "diskio":common.MapStr{"read":common.MapStr{"bytes":0, "bytes_per_sec":0, "count":0, "count_per_sec":0}, "write":common.MapStr{"bytes":0, "bytes_per_sec":0, "count":0, "count_per_sec":0}}, "instance":common.MapStr{"core":common.MapStr{"count":1}, "image":common.MapStr{"id":"ami-00068cd7555f543d5"}, "monitoring":common.MapStr{"state":"disabled"}, "private":common.MapStr{"dns_name":"ip-172-31-81-156.ec2.internal", "ip":"172.31.81.156"}, "public":common.MapStr{"dns_name":"ec2-3-83-255-51.compute-1.amazonaws.com", "ip":"3.83.255.51"}, "state":common.MapStr{"code":16, "name":"running"}, "threads_per_core":1}, "network":common.MapStr{"in":common.MapStr{"bytes":815.8, "bytes_per_sec":2.719333333333333, "packets":8.4, "packets_per_sec":0.028}, "out":common.MapStr{"bytes":651.6, "bytes_per_sec":2.172, "packets":9.2, "packets_per_sec":0.030666666666666665}}, "status":common.MapStr{"check_failed":0, "check_failed_instance":0, "check_failed_system":0}}, "tags":common.MapStr{"created-by":"ks", "service":"mysql"}}, "cloud":common.MapStr{"account":common.MapStr{"id":"428152502467", "name":"elastic-beats"}, "availability_zone":"us-east-1b", "instance":common.MapStr{"id":"i-05381e495aa0f95ef"}, "machine":common.MapStr{"type":"t2.micro"}, "provider":"aws", "region":"us-east-1"}, "ecs":common.MapStr{"version":"1.2.0"}, "event":common.MapStr{"dataset":"aws.ec2", "duration":22486295401, "module":"aws"}, "host":common.MapStr{"architecture":"x86_64", "hostname":"KaiyanMacBookPro", "id":"9C7FAB7B-29D1-5926-8E84-158A9CA3E25D", "name":"KaiyanMacBookPro", "os":common.MapStr{"build":"17G9016", "family":"darwin", "kernel":"17.7.0", "name":"Mac OS X", "platform":"darwin", "version":"10.13.6"}}, "metricset":common.MapStr{"name":"ec2", "period":300000}, "service":common.MapStr{"type":"aws"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [aws.ec2.network.in] of type [double] in document with id 'S7c_JW8BCg_wxAud1ac3'. Preview of field's value: '{packets_per_sec=0.028, bytes_per_sec=2.719333333333333, bytes=815.8, packets=8.4}'","caused_by":{"type":"json_parse_exception","reason":"Current token (START_OBJECT) not numeric, can not use numeric value accessors\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@16c0f5f3; line: 1, column: 186]"}}

If we use aws.*.*.* mapping, I guess anytime we run cloudwatch metricset before ec2 metricset, this error will happen. @exekias What do you think?

[exekias]

Thank you for taking this! I think metrics look now much better, the change should make them easier to discover 🎉

[kaiyan-sheng]

@exekias I updated all the data.json files and dashboards. This is ready for review. Thanks!

[exekias]

Could you please add a message into the breaking changes section of the CHANGELOG?

Also I guess existing PRs using cloudwatch will need to rebase

[kaiyan-sheng]

Could you please add a message into the breaking changes section of the CHANGELOG?

Also I guess existing PRs using cloudwatch will need to rebase

@exekias The current changelog entry is already under breaking changes - metricbeat. Yes, two PRs for adding new aws metricset are waiting for this PR to get in first.

[exekias]

Ouch I'm blind 🙈, then 👍 to