Not able to use FIPS in AWS non-GovCloud

[tunguyen9889]

Expected behavior:

• Teleport should be able to run with FIPS in AWS non-GovCloud

Current behavior:

• We're running some self-hosted Teleport clusters in AWS and has FIPS-enabled in some environments (us-west-2, ca-central-1). Below is our current architect of a Teleport cluster:

• In each Teleport cluster, we have a S3 bucket and 2 DynamoDB tables (one for cluster state, the other for audit events) as Storage backend.

• After the change implemented in [v14] Disable AWS IMDSv1 fallback and enforce use of FIPS endpoints #34433, Teleport is enforced all FIPS endpoints if in FIPS-mode. That's causing the issue because Dynamo Stream is only have FIPS endpoints in AWS GovCloud (see https://aws.amazon.com/compliance/fips/) and Teleport is reporting this error message:

{"caller":"dynamo/shards.go:60","component":"dynamodb","level":"error","message":"Poll streams returned with error: RequestError: send request failed\ncaused by: Post \"https://streams.dynamodb-fips.ca-central-1.amazonaws.com/\": dial tcp: lookup streams.dynamodb-fips.ca-central-1.amazonaws.com on 169.254.20.10:53: no such host.","timestamp":"2023-11-20T18:41:40Z"}

• We want the ability to fallback to non-FIPS endpoint for DynamoDB Stream. Previous version 1.14.3 is working fine and doesn't have any error message like above.

Bug details:

• Teleport version: 14.1.5
• Recreation steps: Run Teleport with FIPS-mode in AWS with S3 bucket and DynamoDB tables as backends.
• Debug logs:

{"caller":"dynamo/shards.go:60","component":"dynamodb","level":"error","message":"Poll streams returned with error: RequestError: send request failed\ncaused by: Post \"https://streams.dynamodb-fips.ca-central-1.amazonaws.com/\": dial tcp: lookup streams.dynamodb-fips.ca-central-1.amazonaws.com on 169.254.20.10:53: no such host.","timestamp":"2023-11-20T18:41:40Z"}

[tunguyen9889]

Hi @reedloden, thanks for working on the PR! But I'm a bit confusing why it was reverted after merged? Does it mean this issue will not be fixed and Teleport will enforce DynamoDB stream using FIPS endpoint?

[reedloden]

Hi @reedloden, thanks for working on the PR! But I'm a bit confusing why it was reverted after merged? Does it mean this issue will not be fixed and Teleport will enforce DynamoDB stream using FIPS endpoint?

No worries. Our release process can be a bit confusing. The original change and the follow-up fix are on master (what will become Teleport 15 in January 2024). Instead of fixing the issue you discovered on Teleport 14, we decided to revert the original change on that release branch just in case there are additional regressions. So, the next release of Teleport 14.x will still have this issue resolved.

Thanks again for reporting it!

[tunguyen9889]

Awesome! Thanks a lot for your quick fix, @reedloden! I'm waiting for next release to test it on our clusters.

[tunguyen9889]

Confirmed version 14.2.1 fixed the issue.