Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AWS] Collect metrics from linked accounts #35540

Merged
merged 11 commits into from May 26, 2023
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Expand Up @@ -318,6 +318,7 @@ automatic splitting at root level, if root level element is an array. {pull}3415
- Handle duplicated TYPE line for prometheus metrics {issue}18813[18813] {pull}33865[33865]
- Add GCP Carbon Footprint metricbeat data {pull}34820[34820]
- Add event loop utilization metric to Kibana module {pull}35020[35020]
- Support collecting metrics from both the monitoring account and linked accounts from AWS CloudWatch. {pull}35540[35540]

*Osquerybeat*

Expand Down
20 changes: 10 additions & 10 deletions NOTICE.txt
Expand Up @@ -4932,11 +4932,11 @@ Contents of probable licence file $GOMODCACHE/github.com/aws/aws-lambda-go@v1.13

--------------------------------------------------------------------------------
Dependency : github.com/aws/aws-sdk-go-v2
Version: v1.16.6
Version: v1.18.0
Licence type (autodetected): Apache-2.0
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2@v1.16.6/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2@v1.18.0/LICENSE.txt:


Apache License
Expand Down Expand Up @@ -5992,11 +5992,11 @@ Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2/servi

--------------------------------------------------------------------------------
Dependency : github.com/aws/aws-sdk-go-v2/service/cloudwatch
Version: v1.18.2
Version: v1.26.0
Licence type (autodetected): Apache-2.0
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2/service/cloudwatch@v1.18.2/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2/service/cloudwatch@v1.26.0/LICENSE.txt:


Apache License
Expand Down Expand Up @@ -8748,11 +8748,11 @@ Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2/servi

--------------------------------------------------------------------------------
Dependency : github.com/aws/smithy-go
Version: v1.12.0
Version: v1.13.5
Licence type (autodetected): Apache-2.0
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/aws/smithy-go@v1.12.0/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/aws/smithy-go@v1.13.5/LICENSE:


Apache License
Expand Down Expand Up @@ -31861,11 +31861,11 @@ Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2/featu

--------------------------------------------------------------------------------
Dependency : github.com/aws/aws-sdk-go-v2/internal/configsources
Version: v1.1.13
Version: v1.1.33
Licence type (autodetected): Apache-2.0
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2/internal/configsources@v1.1.13/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2/internal/configsources@v1.1.33/LICENSE.txt:


Apache License
Expand Down Expand Up @@ -32073,11 +32073,11 @@ Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2/inter

--------------------------------------------------------------------------------
Dependency : github.com/aws/aws-sdk-go-v2/internal/endpoints/v2
Version: v2.4.7
Version: v2.4.27
Licence type (autodetected): Apache-2.0
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.4.7/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.4.27/LICENSE.txt:


Apache License
Expand Down
10 changes: 5 additions & 5 deletions go.mod
Expand Up @@ -28,10 +28,10 @@ require (
github.com/apoydence/eachers v0.0.0-20181020210610-23942921fe77 // indirect
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5
github.com/aws/aws-lambda-go v1.13.3
github.com/aws/aws-sdk-go-v2 v1.16.6
github.com/aws/aws-sdk-go-v2 v1.18.0
github.com/aws/aws-sdk-go-v2/config v1.15.12
github.com/aws/aws-sdk-go-v2/credentials v1.12.7
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.18.2
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.26.0
github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.15.5
github.com/aws/aws-sdk-go-v2/service/costexplorer v1.18.4
github.com/aws/aws-sdk-go-v2/service/ec2 v1.36.1
Expand Down Expand Up @@ -192,7 +192,7 @@ require (
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.17
github.com/aws/aws-sdk-go-v2/service/cloudformation v1.20.4
github.com/aws/aws-sdk-go-v2/service/kinesis v1.15.8
github.com/aws/smithy-go v1.12.0
github.com/aws/smithy-go v1.13.5
github.com/awslabs/kinesis-aggregation/go/v2 v2.0.0-20220623125934-28468a6701b5
github.com/elastic/bayeux v1.0.5
github.com/elastic/elastic-agent-autodiscover v0.5.0
Expand Down Expand Up @@ -243,8 +243,8 @@ require (
github.com/aws/aws-sdk-go v1.38.60 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.3 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.7 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.13 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.7 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.33 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.27 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.14 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.3 // indirect
Expand Down
16 changes: 10 additions & 6 deletions go.sum
Expand Up @@ -270,8 +270,9 @@ github.com/aws/aws-sdk-go v1.38.60/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2z
github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZwxzkQq9wy+g=
github.com/aws/aws-sdk-go-v2 v1.9.0/go.mod h1:cK/D0BBs0b/oWPIcX/Z/obahJK1TT7IPVjy53i/mX/4=
github.com/aws/aws-sdk-go-v2 v1.16.3/go.mod h1:ytwTPBG6fXTZLxxeeCCWj2/EMYp/xDUgX+OET6TLNNU=
github.com/aws/aws-sdk-go-v2 v1.16.6 h1:kzafGZYwkwVgLZ2zEX7P+vTwLli6uIMXF8aGjunN6UI=
github.com/aws/aws-sdk-go-v2 v1.16.6/go.mod h1:6CpKuLXg2w7If3ABZCl/qZ6rEgwtjZTn4eAf4RcEyuw=
github.com/aws/aws-sdk-go-v2 v1.18.0 h1:882kkTpSFhdgYRKVZ/VCgf7sd0ru57p2JCxz4/oN5RY=
github.com/aws/aws-sdk-go-v2 v1.18.0/go.mod h1:uzbQtefpm44goOPmdKyAlXSNcwlRgF3ePWVW6EtJvvw=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.3 h1:S/ZBwevQkr7gv5YxONYpGQxlMFFYSRfz3RMcjsC9Qhk=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.3/go.mod h1:gNsR5CaXKmQSSzrmGxmwmct/r+ZBfbxorAuXYsj/M5Y=
github.com/aws/aws-sdk-go-v2/config v1.15.12 h1:D4mdf0cOSmZRgJe0DDOd1Qm6tkwHJ7r5i1lz0asa+AA=
Expand All @@ -283,19 +284,21 @@ github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.7/go.mod h1:81k6q0UUZj6AdQZ1
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.17 h1:9Y+OvoIvC8KocGNqbbBNDvMu0zsIgzKg3r+ZllSuH5Y=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.17/go.mod h1:z/7g6Z78jPG0l3HeShseUWzA+aBJDK4Mu5DkKkYdIW0=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.10/go.mod h1:F+EZtuIwjlv35kRJPyBGcsA4f7bnSoz15zOQ2lJq1Z4=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.13 h1:WuQ1yGs3TMJgxpGVLspcsU/5q1omSA0SG6Cu0yZ4jkM=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.13/go.mod h1:wLLesU+LdMZDM3U0PP9vZXJW39zmD/7L4nY2pSrYZ/g=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.33 h1:kG5eQilShqmJbv11XL1VpyDbaEJzWxd4zRiCG30GSn4=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.33/go.mod h1:7i0PF1ME/2eUPFcjkVIwq+DOygHEoK92t5cDqNgYbIw=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.4/go.mod h1:8glyUqVIM4AmeenIsPo0oVh3+NUwnsQml2OFupfQW+0=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.7 h1:mCeDDYeDXp3loo/xKi7nkx34eeh7q3n1mUBtzptsj8c=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.7/go.mod h1:93Uot80ddyVzSl//xEJreNKMhxntr71WtR3v/A1cRYk=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.27 h1:vFQlirhuM8lLlpI7imKOMsjdQLuN9CPi+k44F/OFVsk=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.27/go.mod h1:UrHnn3QV/d0pBZ6QBAEQcqFLf8FAzLmoUfPVIueOvoM=
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.14 h1:bJv4Y9QOiW0GZPStgLgpGrpdfRDSR3XM4V4M3YCQRZo=
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.14/go.mod h1:R1HF8ZDdcRFfAGF+13En4LSHi2IrrNuPQCaxgWCeGyY=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.4 h1:wusoY1MJ9JNrPoX3n4kxY4MTIUivCiXvTYQbYh59yxs=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.4/go.mod h1:cHTMyJVEXRUZ25f8V+pq6CAwoYARarJRFGf3XH4eIxE=
github.com/aws/aws-sdk-go-v2/service/cloudformation v1.20.4 h1:faP794ma9ZY/24XAV8cm/lkQzRFSg3zBHCi5Nc8+CaM=
github.com/aws/aws-sdk-go-v2/service/cloudformation v1.20.4/go.mod h1:ybjChNDMfPtc7f8ILTb+ov6CpE/KtAae9fD8HHtYfzU=
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.18.2 h1:UecVUEdx3xe80X+guNwR2hN1d4L4v4qX2SHLYd/4cIk=
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.18.2/go.mod h1:UhlwJ2aI2pzirWzrcsoqvbv7Yao3dgB0AK+yyAWESkA=
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.26.0 h1:sSzrsKQULJmPtmu6By4wR6g0701nGqonssKOy35uOd0=
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.26.0/go.mod h1:t5mizLPjCYafXoHCXOHJU7z4OvLbY70Echvb1ciBTV4=
github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.15.5 h1:aPK8IBVKeozo/pNGshT8xOJ2V3Y7ykOM49QcY0vhUSM=
github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.15.5/go.mod h1:ErjxucZaraVbYm66xxub00qmGBw7md2RFqy6624KbR8=
github.com/aws/aws-sdk-go-v2/service/costexplorer v1.18.4 h1:jbfG3cbq1kiK1/OAfUh4zf1ADtAU8KoeOPfF94S96pU=
Expand Down Expand Up @@ -334,8 +337,9 @@ github.com/aws/aws-sdk-go-v2/service/sts v1.16.8 h1:GLGfpqX+1bmjNvUJkwB1ZaDpNFXQ
github.com/aws/aws-sdk-go-v2/service/sts v1.16.8/go.mod h1:50YdFq1WIuxA0AGrygvYGucnNYrG24WYzu5fNp7lMgY=
github.com/aws/smithy-go v1.8.0/go.mod h1:SObp3lf9smib00L/v3U2eAKG8FyQ7iLrJnQiAmR5n+E=
github.com/aws/smithy-go v1.11.2/go.mod h1:3xHYmszWVx2c0kIwQeEVf9uSm4fYZt67FBJnwub1bgM=
github.com/aws/smithy-go v1.12.0 h1:gXpeZel/jPoWQ7OEmLIgCUnhkFftqNfwWUwAHSlp1v0=
github.com/aws/smithy-go v1.12.0/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
github.com/aws/smithy-go v1.13.5 h1:hgz0X/DX0dGqTYpGALqXJoRKRj5oQ7150i5FdTePzO8=
github.com/aws/smithy-go v1.13.5/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
github.com/awslabs/goformation/v3 v3.1.0/go.mod h1:hQ5RXo3GNm2laHWKizDzU5DsDy+yNcenSca2UxN0850=
github.com/awslabs/goformation/v4 v4.1.0 h1:JRxIW0IjhYpYDrIZOTJGMu2azXKI+OK5dP56ubpywGU=
github.com/awslabs/goformation/v4 v4.1.0/go.mod h1:MBDN7u1lMNDoehbFuO4uPvgwPeolTMA2TzX1yO6KlxI=
Expand Down
6 changes: 5 additions & 1 deletion metricbeat/docs/modules/aws.asciidoc
Expand Up @@ -34,6 +34,10 @@ Users can either use `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and/or
`AWS_SESSION_TOKEN`, or use shared AWS credentials file.
Please see <<aws-credentials-config,AWS credentials options>> for more details.

If you use AWS CloudWatch cross-account observability, credentials of the monitoring account
should be used here and Metricbeat will collect all metrics from both the monitoring
account and the linked source accounts.

* *regions*

This module also accepts optional configuration `regions` to specify which
Expand All @@ -51,7 +55,7 @@ latency amount.

* *data_granularity*

AWS CloudWatch allows to define the granularity of the returned datapoints, by setting "Period" while querying metrics.
AWS CloudWatch allows to define the granularity of the returned data points, by setting "Period" while querying metrics.
Please see https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDataQuery.html[MetricDataQuery parameters] for more information.

By default, metricbeat will query CloudWatch setting "Period" to Metricbeat collection period. If you wish to set a custom value for "Period", please specify a `data_granularity` parameter.
Expand Down
6 changes: 5 additions & 1 deletion x-pack/metricbeat/module/aws/_meta/docs.asciidoc
Expand Up @@ -22,6 +22,10 @@ Users can either use `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and/or
`AWS_SESSION_TOKEN`, or use shared AWS credentials file.
Please see <<aws-credentials-config,AWS credentials options>> for more details.

If you use AWS CloudWatch cross-account observability, credentials of the monitoring account
should be used here and Metricbeat will collect all metrics from both the monitoring
account and the linked source accounts.

* *regions*

This module also accepts optional configuration `regions` to specify which
Expand All @@ -39,7 +43,7 @@ latency amount.

* *data_granularity*

AWS CloudWatch allows to define the granularity of the returned datapoints, by setting "Period" while querying metrics.
AWS CloudWatch allows to define the granularity of the returned data points, by setting "Period" while querying metrics.
Please see https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDataQuery.html[MetricDataQuery parameters] for more information.

By default, metricbeat will query CloudWatch setting "Period" to Metricbeat collection period. If you wish to set a custom value for "Period", please specify a `data_granularity` parameter.
Expand Down
38 changes: 26 additions & 12 deletions x-pack/metricbeat/module/aws/billing/billing.go
Expand Up @@ -42,7 +42,11 @@ var (
"RESERVATION_ID",
}

dateLayout = "2006-01-02"
dateLayout = "2006-01-02"
accountIdIdx = 0
accountLabelIdx = 1
metricDataValueIdx = 2
dimensionStartIdx = 3
)

// init registers the MetricSet with the central registry as soon as the program
Expand All @@ -65,7 +69,7 @@ type MetricSet struct {
CostExplorerConfig CostExplorerConfig `config:"cost_explorer_config"`
tommyers-elastic marked this conversation as resolved.
Show resolved Hide resolved
}

// Config holds a configuration specific for billing metricset.
// CostExplorerConfig holds a configuration specific for billing metricset.
type CostExplorerConfig struct {
GroupByDimensionKeys []string `config:"group_by_dimension_keys"`
GroupByTagKeys []string `config:"group_by_tag_keys"`
Expand Down Expand Up @@ -195,11 +199,16 @@ func (m *MetricSet) getCloudWatchBillingMetrics(
}
for valI, metricDataResultValue := range output.Values {
labels := strings.Split(*output.Label, labelSeparator)
event := mb.Event{}
if labels[accountIdIdx] != "" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

under what circumstances is this empty when we pass includeLinkedAccounts?

event = aws.InitEvent("", labels[accountLabelIdx], labels[accountIdIdx], output.Timestamps[valI])
} else {
event = aws.InitEvent("", m.AccountName, m.AccountID, output.Timestamps[valI])
}

event := aws.InitEvent("", m.AccountName, m.AccountID, output.Timestamps[valI])
_, _ = event.MetricSetFields.Put(labels[0], metricDataResultValue)
_, _ = event.MetricSetFields.Put(labels[metricDataValueIdx], metricDataResultValue)

i := 1
i := dimensionStartIdx
for i < len(labels)-1 {
_, _ = event.MetricSetFields.Put(labels[i], labels[i+1])
i += 2
Expand Down Expand Up @@ -338,7 +347,7 @@ func (m *MetricSet) addCostMetrics(metrics map[string]costexplorertypes.MetricVa
return event
}

func constructMetricQueries(listMetricsOutput []types.Metric, dataGranularity time.Duration) []types.MetricDataQuery {
func constructMetricQueries(listMetricsOutput []aws.MetricWithID, dataGranularity time.Duration) []types.MetricDataQuery {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, not required for this PR

We should rename constructMetricQueries and createMetricDataQuery with something clearer (e.g createQueries vs createSingleQuery)

var metricDataQueries []types.MetricDataQuery
metricDataQueryEmpty := types.MetricDataQuery{}
for i, listMetric := range listMetricsOutput {
Expand All @@ -351,27 +360,32 @@ func constructMetricQueries(listMetricsOutput []types.Metric, dataGranularity ti
return metricDataQueries
}

func createMetricDataQuery(metric types.Metric, index int, dataGranularity time.Duration) types.MetricDataQuery {
func createMetricDataQuery(metric aws.MetricWithID, index int, dataGranularity time.Duration) types.MetricDataQuery {
statistic := "Maximum"
dataGranularityInSeconds := int32(dataGranularity.Seconds())
id := metricsetName + strconv.Itoa(index)
metricDims := metric.Dimensions
metricName := *metric.MetricName
metricDims := metric.Metric.Dimensions
metricName := *metric.Metric.MetricName

label := metricName + labelSeparator
label := metric.AccountID + labelSeparator + "${PROP('AccountLabel')}" + labelSeparator + metricName + labelSeparator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit not related to this PR - it would be better if we could avoid the manual work of keeping this label in sync with the index constants used to parse it. one way would just be to wrap the label creation and parsing in a type; at least that way the indexing concerns don't bleed out into the code. let's not change it now, just a thought though.

for _, dim := range metricDims {
label += *dim.Name + labelSeparator + *dim.Value + labelSeparator
}

return types.MetricDataQuery{
metricDataQuery := types.MetricDataQuery{
Id: &id,
MetricStat: &types.MetricStat{
Period: &dataGranularityInSeconds,
Stat: &statistic,
Metric: &metric,
Metric: &metric.Metric,
},
Label: &label,
}

if metric.AccountID != "" {
metricDataQuery.AccountId = &metric.AccountID
}
return metricDataQuery
}

func getStartDateEndDate(period time.Duration) (string, string) {
Expand Down