Unable to get Vector to use the FIPS AWS endpoints

[alex-rowe]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

The AWS Rust SDK can the ability to use the AWS FIPS service endpoints if you supply a config file with use_fips_endpoint=true or an environment variable of AWS_USE_FIPS_ENDPOINT=true. For example: https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/aws-config/src/default_provider/use_fips.rs#L26

By providing these options to Vector either through the AWS_CONFIG_FILE environment variable, directly in a config file, systemd unit file overrides and via running it on the command line, I can't get Vector to read this file and use these options.

Our use case is for the AWS Kinesis Stream sink using an Assumed Role. The calls to AWS STS to assume the role, and the calls to Kinesis must use the FIPS endpoints as shown here https://aws.amazon.com/compliance/fips/

Configuration

Test command (also tested via systemd as below):
AWS_USE_FIPS_ENDPOINT=true VECTOR_LOG=debug vector --config=/etc/vector/vector.toml > vector.log 2>&1

# cat /etc/systemd/system/vector.service.d/override.conf
# Also attempted to set the AWS_USE_FIPS_ENDPOINT environment variable here
[Service]
Environment="AWS_CONFIG_FILE=/var/lib/vector/.aws/config"

# cat /var/lib/vector/.aws/config
[default]
use_fips_endpoint = true

# cat /root/.aws/config
[default]
use_fips_endpoint = true

Snippet from /etc/vector/vector.toml
[sinks.to_kinesis]
  type = "aws_kinesis_streams"
  inputs = ["logs"]
  region = "us-west-2"
  stream_name = "logs"
  batch.max_bytes = 5000000
  batch.max_events = 500
  buffer.type = "memory"
  buffer.max_events = 160000
  buffer.when_full = "drop_newest"

  [sinks.to_kinesis.encoding]
  codec = "json"

  [sinks.to_kinesis.healthcheck]
  enabled = false

  [sinks.to_kinesis.auth]
  assume_role = "our arn role"
  external_id = "external-id"

Version

vector 0.32.1 (x86_64-unknown-linux-gnu 9965884 2023-08-21 14:52:38.330227446)

Debug Output

2023-08-24T23:10:57.664997Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-08-24T23:10:57.665109Z  INFO vector::app: Log level is enabled. level="debug"
2023-08-24T23:10:57.665122Z  WARN vector::app: DEPRECATED The openssl legacy provider provides algorithms and key sizes no longer recommended for use. Set `--openssl-legacy-provider=false` or `VECTOR_OPENSSL_LEGACY_PROVIDER=false` to disable. See https://vector.dev/highlights/2023-08-15-0-32-0-upgrade-guide/#legacy-openssl for details.
2023-08-24T23:10:57.665758Z  INFO vector::app: Loaded openssl provider. provider="legacy"
2023-08-24T23:10:57.665799Z  INFO vector::app: Loaded openssl provider. provider="default"
2023-08-24T23:10:57.666386Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-08-24T23:10:57.668290Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2023-08-24T23:10:57.674868Z DEBUG vector::topology::builder: Building new source. component=syslog_in
2023-08-24T23:10:57.675257Z DEBUG vector::topology::builder: Building new transform. component=syslog_logs
2023-08-24T23:10:57.679391Z DEBUG vector::topology::builder: Building new sink. component=to_file
2023-08-24T23:10:57.680762Z DEBUG vector::topology::builder: Building new sink. component=to_kinesis
2023-08-24T23:10:57.730037Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}: aws_config::fs_util: loaded home directory src="HOME"
2023-08-24T23:10:57.730085Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Config)}:aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2023-08-24T23:10:57.730117Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Config)}:aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2023-08-24T23:10:57.730169Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Config)}:aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=35
2023-08-24T23:10:57.730191Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2023-08-24T23:10:57.730202Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2023-08-24T23:10:57.730217Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-08-24T23:10:57.730228Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2023-08-24T23:10:57.872317Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}: vector_core::tls::settings: Fetching system root certs.
2023-08-24T23:10:57.909204Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}: vector_core::tls::settings: Fetching system root certs.
2023-08-24T23:10:57.949259Z  INFO vector::topology::running: Running healthchecks.
2023-08-24T23:10:57.949305Z DEBUG vector::topology::running: Connecting changed/added component(s).
2023-08-24T23:10:57.949374Z  INFO vector::topology::builder: Healthcheck disabled.
2023-08-24T23:10:57.949427Z DEBUG vector::topology::running: Configuring outputs for source. component=syslog_in
2023-08-24T23:10:57.949435Z  INFO vector::topology::builder: Healthcheck passed.
2023-08-24T23:10:57.949440Z DEBUG vector::topology::running: Configuring output for component. component=syslog_in output_id=None
2023-08-24T23:10:57.949452Z DEBUG vector::topology::running: Configuring outputs for transform. component=syslog_logs
2023-08-24T23:10:57.949460Z DEBUG vector::topology::running: Configuring output for component. component=syslog_logs output_id=None
2023-08-24T23:10:57.949471Z DEBUG vector::topology::running: Connecting inputs for transform. component=syslog_logs
2023-08-24T23:10:57.949484Z DEBUG vector::topology::running: Adding component input to fanout. component=syslog_logs fanout_id=syslog_in
2023-08-24T23:10:57.949498Z DEBUG vector::topology::running: Connecting inputs for sink. component=to_kinesis
2023-08-24T23:10:57.949512Z DEBUG vector::topology::running: Adding component input to fanout. component=to_kinesis fanout_id=syslog_logs
2023-08-24T23:10:57.949522Z DEBUG vector::topology::running: Connecting inputs for sink. component=to_file
2023-08-24T23:10:57.949532Z DEBUG vector::topology::running: Adding component input to fanout. component=to_file fanout_id=syslog_logs
2023-08-24T23:10:57.949551Z DEBUG vector::topology::running: Spawning new source. key=syslog_in
2023-08-24T23:10:57.949583Z DEBUG vector::topology::running: Registered new allocation group. component_kind="source" component_type="file" component_id="syslog_in" group_id="2"
2023-08-24T23:10:57.949614Z DEBUG vector::topology::running: Spawning new transform. key=syslog_logs
2023-08-24T23:10:57.949636Z DEBUG vector::topology::running: Registered new allocation group. component_kind="transform" component_type="remap" component_id="syslog_logs" group_id="3"
2023-08-24T23:10:57.949636Z DEBUG source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}: vector::topology::builder: Source pump supervisor starting.
2023-08-24T23:10:57.949662Z DEBUG vector::topology::running: Registered new allocation group. component_kind="sink" component_type="aws_kinesis_streams" component_id="to_kinesis" group_id="4"
2023-08-24T23:10:57.949679Z DEBUG source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}: vector::topology::builder: Source pump starting.
2023-08-24T23:10:57.949682Z DEBUG vector::topology::running: Registered new allocation group. component_kind="sink" component_type="file" component_id="to_file" group_id="5"
2023-08-24T23:10:57.949707Z DEBUG source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}: vector::topology::builder: Source starting.
2023-08-24T23:10:57.949735Z  INFO source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}: vector::sources::file: Starting file server. include=["/var/log/syslog"] exclude=[]
2023-08-24T23:10:57.949766Z  INFO vector: Vector has started. debug="false" version="0.32.1" arch="x86_64" revision="9965884 2023-08-21 14:52:38.330227446"
2023-08-24T23:10:57.949787Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-08-24T23:10:57.949872Z DEBUG transform{component_kind="transform" component_id=syslog_logs component_type=remap component_name=syslog_logs}: vector::topology::builder: Synchronous transform starting.
2023-08-24T23:10:57.949894Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}: vector::topology::builder: Sink starting.
2023-08-24T23:10:57.949993Z DEBUG sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector::topology::builder: Sink starting.
2023-08-24T23:10:57.950055Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}: vector::utilization: utilization=0.4031854493462015
2023-08-24T23:10:57.950323Z  INFO source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}:file_server: file_source::checkpointer: Loaded checkpoint data.
2023-08-24T23:10:57.950509Z  INFO source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/var/log/syslog file_position=696653
2023-08-24T23:10:57.952486Z DEBUG sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector::utilization: utilization=0.003860596633269875
2023-08-24T23:10:57.954606Z ERROR sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector::internal_events::file: Unable to open the file. path=b"/tmp/vector-2023-08-24.log" error=Permission denied (os error 13) error_code=failed_opening_file error_type="io_failed" stage="sending" internal_log_rate_limit=true
2023-08-24T23:10:57.954652Z ERROR sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Unable to open the file." internal_log_rate_limit=true
2023-08-24T23:10:57.954801Z ERROR sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector::internal_events::file: Internal log [Unable to openthe file.] is being suppressed to avoid flooding.
2023-08-24T23:10:57.954820Z ERROR sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.
2023-08-24T23:10:58.954031Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=2
2023-08-24T23:10:58.955479Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role: aws_config::sts::assume_role: retrieving assumed credentials
2023-08-24T23:10:58.955562Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=the credential provider was not enabled: environment variable not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "environment variable not set" }))
2023-08-24T23:10:58.955593Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::fs_util: loaded home directory src="HOME"
2023-08-24T23:10:58.955604Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2023-08-24T23:10:58.955620Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2023-08-24T23:10:58.955651Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=35
2023-08-24T23:10:58.955662Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2023-08-24T23:10:58.955669Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2023-08-24T23:10:58.955678Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-08-24T23:10:58.955685Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2023-08-24T23:10:58.955711Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profilecontext=the credential provider was not enabled: profile `default` did not contain credential information (CredentialsNotLoaded(CredentialsNotLoaded { source: ProfileDidNotContainCredentials { profile: "default" } }))
2023-08-24T23:10:58.955730Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=WebIdentityToken context=the credential provider was not enabled: $AWS_WEB_IDENTITY_TOKEN_FILE was not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "$AWS_WEB_IDENTITY_TOKEN_FILE was not set" }))
2023-08-24T23:10:58.955746Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=EcsContainer context=the credential provider was not enabled: ECS provider not configured (CredentialsNotLoaded(CredentialsNotLoaded { source: "ECS provider not configured" }))
2023-08-24T23:10:58.955755Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loading credentials from IMDS
2023-08-24T23:10:58.955884Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:async_map_request{name="attach_imds_token"}:send_operation{operation="get-token" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-08-24T23:10:58.956297Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:async_map_request{name="attach_imds_token"}:send_operation{operation="get-token" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2023-08-24T23:10:58.956496Z DEBUG hyper::proto::h1::io: flushed 242 bytes
2023-08-24T23:10:58.956916Z DEBUG hyper::proto::h1::io: parsed 6 headers
2023-08-24T23:10:58.956928Z DEBUG hyper::proto::h1::conn: incoming body is content-length (56 bytes)
2023-08-24T23:10:58.956945Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-08-24T23:10:58.957075Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-08-24T23:10:58.957293Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2023-08-24T23:10:58.957380Z DEBUG hyper::proto::h1::io: flushed 307 bytes
2023-08-24T23:10:58.957663Z DEBUG hyper::proto::h1::io: parsed 8 headers
2023-08-24T23:10:58.957810Z DEBUG hyper::proto::h1::conn: incoming body is content-length (26 bytes)
2023-08-24T23:10:58.957820Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-08-24T23:10:58.957896Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loaded profile profile=dummy-profile
2023-08-24T23:10:58.957972Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-08-24T23:10:58.958100Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2023-08-24T23:10:58.958181Z DEBUG hyper::proto::h1::io: flushed 333 bytes
2023-08-24T23:10:58.958414Z DEBUG hyper::proto::h1::io: parsed 8 headers
2023-08-24T23:10:58.958421Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1570 bytes)
2023-08-24T23:10:58.958433Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-08-24T23:10:58.958550Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=Ec2InstanceMetadata
2023-08-24T23:10:58.958565Z  INFO sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: credentials cache miss occurred; retrieved new AWS credentials (took 3.012906ms)
2023-08-24T23:10:58.958740Z DEBUG hyper::client::connect::dns: resolving host="sts.us-west-2.amazonaws.com"
2023-08-24T23:10:58.960819Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:dispatch: hyper::client::connect::http: connecting to 54.240.248.204:443
2023-08-24T23:10:58.961759Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:dispatch: hyper::client::connect::http: connected to 54.240.248.204:443
2023-08-24T23:10:58.965424Z DEBUG hyper::proto::h1::io: flushed 2113 bytes
2023-08-24T23:10:58.986612Z DEBUG source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}:file_server: file_source::file_server: event_throughput=168.000/sec bytes_throughput=21.331k/sec ratios={"discovery": 4.567172e-5, "other": 0.00013248506, "reading": 0.0016011929, "sending": 0.0019266325, "sleeping": 0.99629456}
2023-08-24T23:10:58.991154Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-08-24T23:10:58.991171Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1526 bytes)
2023-08-24T23:10:58.991199Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-08-24T23:10:58.991245Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:dispatch: hyper::client::pool: pooling idle connection for ("https", sts.us-west-2.amazonaws.com)
2023-08-24T23:10:58.991382Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role: aws_config::sts::assume_role: obtained assumed credentials access_key_id=Some(Some("key"))
2023-08-24T23:10:58.991398Z  INFO sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: credentials cache miss occurred; retrieved new AWS credentials (took 35.926074ms)
2023-08-24T23:10:58.991532Z DEBUG hyper::client::connect::dns: resolving host="kinesis.us-west-2.amazonaws.com"
2023-08-24T23:10:58.992188Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:dispatch: hyper::client::connect::http: connecting to 34.223.45.90:443
2023-08-24T23:10:58.993889Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:dispatch: hyper::client::connect::http: connected to 34.223.45.90:443
2023-08-24T23:10:58.997971Z DEBUG hyper::proto::h1::io: flushed 2012 bytes
2023-08-24T23:10:59.025925Z DEBUG hyper::proto::h1::io: parsed 5 headers
2023-08-24T23:10:59.025947Z DEBUG hyper::proto::h1::conn: incoming body is content-length (169 bytes)
2023-08-24T23:10:59.025968Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-08-24T23:10:59.026012Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:dispatch: hyper::client::pool: pooling idle connection for ("https", kinesis.us-west-2.amazonaws.com)
2023-08-24T23:10:59.026282Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=2}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: loaded credentials from cache
2023-08-24T23:10:59.026397Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=2}:request:send_operation{operation="PutRecords" service="kinesis"}:dispatch: hyper::client::pool: reuse idle connection for ("https", kinesis.us-west-2.amazonaws.com)

Example Data

CloudTrail event logs for the AssumeRole event, showing connection to the non-FIPS endpoint

    "tlsDetails": {
        "tlsVersion": "TLSv1.2",
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "sts.us-west-2.amazonaws.com"
    }

Additional Context

This isn't related to making all the encryption in Vector FIPS compliant, like in this PR #18261 but only for using the FIPS endpoints as provided by the AWS SDK.

References

No response

[jszwedko]

Thanks for reporting this @alex-rowe ! I was able to confirm it.

I opened #18390 to have Vector honor the FIPS configuration for service access (AWS Kinesis Streams in your case).

I actually can't figure out how to have the SDK use the FIPS endpoints for fetching credentials via STS though. I opened a discussion upstream to get thoughts: awslabs/aws-sdk-rust#880 . This might be a missing feature from the Rust SDK.

[alex-rowe]

Thanks @jszwedko for taking a look at this.

In the Rust SDK, for the sts endpoints configs, they have this for setting use_fips from a param

https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/sts/src/config/endpoint/internals.rs#L19

And from here to use regional endpoints for STS and FIPS if enabled and the region supports is

https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/sts/src/config/endpoint/internals.rs#L315

And here for the STS builder config

https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/sts/src/config.rs#L604

Is the issue that the main AWS SDK setting for use_fips isn't being automatically passed through to the STS builder, but is automatically for Kinesis Streams?

[jszwedko]

Hi @alex-rowe !

Thanks for taking a look. In this case, we aren't accessing the STS service directly and so aren't building an STS SDK config. Instead it is accessed through the AssumeRoleProvider (https://docs.rs/aws-config/latest/aws_config/sts/struct.AssumeRoleProvider.html).

I'm not seeing a way to have that provider use the FIPS endpoints. The Rust SDK maintainers have historically been pretty responsive so I'm hoping they chime in on the GitHub Discussion I opened soon.

[jszwedko]

It looks like this was indeed a bug in the Rust SDK. They opened awslabs/aws-sdk-rust#882 to track it.

[alex-rowe]

@jszwedko we've worked with AWS to get the patch resolved in the Rust SDK and it's in the next branch waiting for the 0.57 release I believe.

Once that release comes out, what is the process/timeline for it to be included in Vector?

[alex-rowe]

It's available now https://github.com/awslabs/aws-sdk-rust/releases/tag/release-2023-11-02

🐛⚠️🎉 (awslabs/aws-sdk-rust#882, smithy-lang/smithy-rs#3007) STS and SSO-based credential providers will now respect both use_fips and use_dual_stack when those settings are configured in a user's environment or profile.

Would it be best to create a new issue to track the SDK update, or would that be included in this to make it fully use the FIPS endpoints?

[alex-rowe]

@jszwedko I can see the next version of Vector was released and it's on aws-config 0.54.1, can we get it bumped to 0.57.1 for the next release?

I also saw this comment on a previous dependabot pr #17510 (comment) and that referenced SDK issue is also now closed.

Upgrading the AWS SDK or supporting crates is blocked until the following issue is resolved
awslabs/aws-sdk-rust#749

[jszwedko]

Hey! Yes, we plan to try to update the AWS SDK by the end of this year now that the upstream issue we had forked to address has been fixed. This means it probably won't make it into the next release, but probably v0.36.0 in January unless we run into many issues upgrading (we anticipate some non-trivial efforts to be involved to manage breaking changes in the SDK). We'd also be happy to see a PR for this if someone wants to take a crack at this sooner.

[cetanu]

Immediately upon trying to do this myself (using aws-config 0.57.1), I noticed that native TLS was removed in 0.56.0. The migration path recommends using rustls instead, and I can't tell if it's FIPS compliant or not

[jszwedko]

Immediately upon trying to do this myself (using aws-config 0.57.1), I noticed that native TLS was removed in 0.56.0. The migration path recommends using rustls instead, and I can't tell if it's FIPS compliant or not

I don't believe rustls is FIPS compliant since it depends on ring for the cryptography, which is not FIPS certified as far as I know. rustls/rustls#521 is tracking the ability to plug in other cryptographic back-ends for rustls.

However, it looks like, with native-tls's removal, the AWS crates added the ability to plug in a custom connector the HTTP client which would allow plugging in one that uses native-tls instead. See https://awslabs.github.io/smithy-rs/design/transport/connector.html

[jszwedko]

Note that Vector statically compiles in OpenSSL so linking with a local OpenSSL installation, including the FIPS module, currently requires building Vector yourself. #16535 is tracking having a generally available "FIPS Compliant" build.

[alex-rowe]

For our use case, we don't need the crypto in Vector to be running on FIPS validated modules, only able to talk to FIPS endpoints like the -fips ones provided by AWS.

[alex-rowe]

Separately, if it's going to be difficult to move to 0.57.1 due to the SSL changes, can an additional config parameter for Kinesis Streams be made available for auth.endpoint where we can configure the STS endpoint ourselves?

I see there is an endpoint parameter to use, but I would expect that to just be for the Kinesis endpoint

[cetanu]

I gave the AWS smithy migration guide a second read and noticed that it speaks of a custom connector
https://awslabs.github.io/smithy-rs/design/transport/connector.html

Here's the example repository
https://github.com/smithy-lang/smithy-rs/blob/main/examples/pokemon-service-tls/tests/common/mod.rs#L75

Maybe we can do something like this...

[alex-rowe]

This would be easy if we could override the endpoint using environment variables provided by AWS such as AWS_ENDPOINT_URL_STS but that's not supported either awslabs/aws-sdk-rust#932

Could adding https://docs.rs/aws-config/latest/aws_config/endpoint/index.html to the AssumeRoleProviderBuilder using https://docs.rs/aws-config/latest/aws_config/sts/struct.AssumeRoleProviderBuilder.html#method.configure be an option.

Similar to how auth.region and auth.external_id are used now?

[alex-rowe]

Unfortunately, it looks like the AssumeRoleProvider option won't work either.

awslabs/aws-sdk-rust#921 and
smithy-lang/smithy-rs#3014

They were both part of the same 0.57.1 release that fixed FIPS for STS, so moving to that would be a requirement for those methods, unsure about the separate custom connector approach.

[alex-rowe]

Cross posting this PR #19312 which is also doing the AWS crate updates to the new GA release of the SDK it looks like