Fail to index AWS events with fields with one more level in their name paths

[jsoriano]

Some events collected by the cloudwatch metricset fail to be indexed.

Errors found in logs:

2020-01-28T19:37:07.710Z	WARN	elasticsearch/client.go:517	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf844130ce47b20d, ext:3698387180, loc:(*time.Location)(0x7eb1ba0)}, Meta:null, Fields:{"agent":{"ephemeral_id":"f9321cda-c194-4250-981d-897638e99888","hostname":"stack-7-6-0-test","id":"74fe85f9-65e1-4ac3-8561-e5450bc4e923","type":"metricbeat","version":"7.6.0"},"aws":{"cloudwatch":{"namespace":"AWS/Firehose"},"dimensions":{"DeliveryStreamName":"phone-home-staging-xpack"},"firehose":{"metrics":{"BytesPerSecondLimit":{"avg":5242880.000000},"DeliveryToS3":{"Bytes":{"avg":11465.000000},"DataFreshness":{"avg":61.000000},"Records":{"avg":1.000000},"Success":{"avg":1.000000}},"IncomingBytes":{"avg":11465.000000},"IncomingPutRequests":{"avg":1.000000},"IncomingRecords":{"avg":1.000000},"KMSKeyAccessDenied":{"avg":0.000000},"KMSKeyDisabled":{"avg":0.000000},"KMSKeyInvalidState":{"avg":0.000000},"KMSKeyNotFound":{"avg":0.000000},"PutRecordBatch":{"Bytes":{"avg":11465.000000},"Latency":{"avg":9.000000},"Records":{"avg":1.000000},"Requests":{"avg":1.000000}},"PutRequestsPerSecondLimit":{"avg":2000.000000},"RecordsPerSecondLimit":{"avg":5000.000000},"ThrottledRecords":{"avg":0.000000}}}},"cloud":{"account":{"id":"627959692251","name":"elastic-test"},"provider":"aws","region":"us-west-2"},"ecs":{"version":"1.4.0"},"event":{"dataset":"aws.cloudwatch","duration":28562828950,"module":"aws"},"host":{"architecture":"x86_64","containerized":false,"hostname":"stack-7-6-0-test","id":"45661bd1e51648e38783c568c541c5ac","name":"stack-7-6-0-test","os":{"codename":"bionic","family":"debian","kernel":"4.15.0-66-generic","name":"Ubuntu","platform":"ubuntu","version":"18.04.3 LTS (Bionic Beaver)"}},"metricset":{"name":"cloudwatch","period":300000},"service":{"type":"aws"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [aws.firehose.metrics.DeliveryToS3.DataFreshness] of type [double] in document with id '2zao7W8BuJq_FrJ2QkUq'. Preview of field's value: '{avg=61}'","caused_by":{"type":"json_parse_exception","reason":"Current token (START_OBJECT) not numeric, can not use numeric value accessors\n at [Source: org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper@72aee4fc; line: 1, column: 860]"}}

2020-01-28T19:55:14.388Z	WARN	elasticsearch/client.go:517	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf8442403a94c3b9, ext:3624232310, loc:(*time.Location)(0x7eb1ba0)}, Meta:null, Fields:{"agent":{"ephemeral_id":"ad5bfc43-fc04-4e1f-b739-f34460673918","hostname":"stack-7-6-0-test","id":"74fe85f9-65e1-4ac3-8561-e5450bc4e923","type":"metricbeat","version":"7.6.0"},"aws":{"cloudwatch":{"namespace":"AWS/Firehose"},"dimensions":{"DeliveryStreamName":"phone-home-staging-xpack"},"firehose":{"metrics":{"BytesPerSecondLimit":{"avg":5242880.000000},"DeliveryToS3":{"Bytes":{"avg":23512.000000},"DataFreshness":{"avg":61.000000},"Records":{"avg":1.000000},"Success":{"avg":1.000000}},"IncomingBytes":{"avg":23512.000000},"IncomingPutRequests":{"avg":1.000000},"IncomingRecords":{"avg":1.000000},"KMSKeyAccessDenied":{"avg":0.000000},"KMSKeyDisabled":{"avg":0.000000},"KMSKeyInvalidState":{"avg":0.000000},"KMSKeyNotFound":{"avg":0.000000},"PutRecordBatch":{"Bytes":{"avg":23512.000000},"Latency":{"avg":10.000000},"Records":{"avg":1.000000},"Requests":{"avg":1.000000}},"PutRequestsPerSecondLimit":{"avg":2000.000000},"RecordsPerSecondLimit":{"avg":5000.000000},"ThrottledRecords":{"avg":0.000000}}}},"cloud":{"account":{"id":"627959692251","name":"elastic-test"},"provider":"aws","region":"us-west-2"},"ecs":{"version":"1.4.0"},"event":{"dataset":"aws.cloudwatch","duration":32357121069,"module":"aws"},"host":{"architecture":"x86_64","containerized":false,"hostname":"stack-7-6-0-test","id":"45661bd1e51648e38783c568c541c5ac","name":"stack-7-6-0-test","os":{"codename":"bionic","family":"debian","kernel":"4.15.0-66-generic","name":"Ubuntu","platform":"ubuntu","version":"18.04.3 LTS (Bionic Beaver)"}},"metricset":{"name":"cloudwatch","period":300000},"service":{"type":"aws"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [aws.firehose.metrics.PutRecordBatch.Records] of type [double] in document with id 'KQC47W8BlbxP6epG4o0l'. Preview of field's value: '{avg=1}'","caused_by":{"type":"json_parse_exception","reason":"Current token (START_OBJECT) not numeric, can not use numeric value accessors\n at [Source: org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper@1e8b10fb; line: 1, column: 942]"}}

They complain about fields with one more level in their paths, like:

• aws.firehose.metrics.DeliveryToS3.DataFreshness.avg
• aws.firehose.metrics.PutRecordBatch.Records.avg

Other working fields are like aws.firehose.metrics.BytesPerSecondLimit.avg.

The failing ones don't match with the expected field definitions for metrics:

    - name: aws
      type: group
      description: >
      fields:
        ...
        - name: '*.metrics.*.*'
          type: object
          object_type: double
          object_type_mapping_type: "*"
          description: >
            Metrics that returned from Cloudwatch API query.

For confirmed bugs, please report:

• Version: 7.6 (probably other versions are affected too)
• Steps to Reproduce: It can be reproduced with the test instructions in Allow namespace: "*" for cloudwatch metricset #14971, seen when capturing firehose metrics using the cloudwatch metricset.

[kaiyan-sheng]

Thanks @jsoriano for catching this bug. I think your idea of dedot metric names is better. "AWS/Firehose" is the first namespace that I see with dot in metric names. But there might be more namespaces with the same problem or even more than one level in the metric name path.

With automatic dedot for metric names, new metric names from cloudwatch metricset would be:
aws.firehose.metrics.DeliveryToS3_DataFreshness.avg
aws.firehose.metrics.PutRecordBatch_Records.avg

WDYT?

[jsoriano]

@kaiyan-sheng dedotting sound good yes