aws - add credential caching for aws assume role sessions (#37788)

Add caching so that AWS `AssumeRole` session credentials are not requested for every single request. Sessions are valid for 15m by default but without caching that does not matter. This will speed up requests for users of `role_arn` by removing the overhead of most STS (session token service) calls and stop users from hitting rate-limiting issues with the STS.

Fixes #37787

(cherry picked from commit a6e5b04)

CHANGELOG.next.asciidoc

@@ -51,6 +51,9 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 - Upgraded apache arrow library used in x-pack/libbeat/reader/parquet from v11 to v12.0.1 in order to fix cross-compilation issues {pull}35640[35640]
 - Fix panic when MaxRetryInterval is specified, but RetryInterval is not {pull}35820[35820]
 - Support build of projects outside of beats directory {pull}36126[36126]
+- Support Elastic Agent control protocol chunking support {pull}37343[37343]
+- Upgrade elastic-agent-libs to v0.7.5. Removes obsolete "Treating the CommonName field on X.509 certificates as a host name..." deprecation warning for 8.0. {pull}37755[37755]
+- aws: Add credential caching for `AssumeRole` session tokens. {issue}37787[37787]
 
 *Auditbeat*
 

x-pack/libbeat/common/aws/credentials.go

@@ -10,6 +10,7 @@ import (
 	"fmt"
 	"net/http"
 	"net/url"
+	"time"
 
 	"github.com/aws/aws-sdk-go-v2/service/sts"
 
@@ -44,6 +45,13 @@ type ConfigAWS struct {
 	FIPSEnabled          bool              `config:"fips_enabled"`
 	TLS                  *tlscommon.Config `config:"ssl" yaml:"ssl,omitempty" json:"ssl,omitempty"`
 	DefaultRegion        string            `config:"default_region"`
+
+	// The duration of the role session. Defaults to 15m when not set.
+	AssumeRoleDuration time.Duration `config:"assume_role.duration"`
+
+	// AssumeRoleExpiryWindow will allow the credentials to trigger refreshing prior to the credentials
+	// actually expiring. If expiry_window is less than or equal to zero, the setting is ignored.
+	AssumeRoleExpiryWindow time.Duration `config:"assume_role.expiry_window"`
 }
 
 // InitializeAWSConfig function creates the awssdk.Config object from the provided config
@@ -154,8 +162,15 @@ func addAssumeRoleProviderToAwsConfig(config ConfigAWS, awsConfig *awssdk.Config
 		if config.ExternalID != "" {
 			aro.ExternalID = awssdk.String(config.ExternalID)
 		}
+		if config.AssumeRoleDuration > 0 {
+			aro.Duration = config.AssumeRoleDuration
+		}
+	})
+	awsConfig.Credentials = awssdk.NewCredentialsCache(stsCredProvider, func(options *awssdk.CredentialsCacheOptions) {
+		if config.AssumeRoleExpiryWindow > 0 {
+			options.ExpiryWindow = config.AssumeRoleExpiryWindow
+		}
 	})
-	awsConfig.Credentials = stsCredProvider
 }
 
 // addStaticCredentialsProviderToAwsConfig adds a static credentials provider to the current AWS config by using the keys stored in Beats config

x-pack/libbeat/docs/aws-credentials-config.asciidoc

@@ -15,6 +15,9 @@ To configure AWS credentials, either put the credentials into the {beatname_uc}
 * *fips_enabled*: Enabling this option instructs {beatname_uc} to use the FIPS endpoint of a service. All services used by {beatname_uc} are FIPS compatible except for `tagging` but only certain regions are FIPS compatible. See https://aws.amazon.com/compliance/fips/ or the appropriate service page, https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html, for a full list of FIPS endpoints and regions.
 * *ssl*: This specifies SSL/TLS configuration. If the ssl section is missing, the host's CAs are used for HTTPS connections. See <<configuration-ssl>> for more information.
 * *default_region*: Default region to query if no other region is set. Most AWS services offer a regional endpoint that can be used to make requests. Some services, such as IAM, do not support regions. If a region is not provided by any other way (environment variable, credential or instance profile), the value set here will be used.
+* *assume_role.duration*: The duration of the requested assume role session. Defaults to 15m when not set. AWS allows a maximum session duration between 1h and 12h depending on your maximum session duration policies.
+* *assume_role.expiry_window*: The expiry_window will allow refreshing the session prior to its expiration.
+  This is beneficial to prevent expiring tokens from causing requests to fail with an ExpiredTokenException.
 
 [float]
 ==== Supported Formats