IAM authentication for MySQL and Redis

[edwardsb]

Goal

User story
As a Security Engineer,
I want Fleet to use Amazon's IAM authentication for MySQL and Redis
so that I can use a form of authentication that's stronger than basic auth (username + password).

Roadmap item

None.

Original requests

• Redis: AWS IAM authentication #20527

Context

• Product Designer: @noahtalerman

• Engineer: @sharon-fdm

• @noahtalerman: PR from @titanous (community member): Add AWS IAM auth for RDS MySQL/MariaDB and ElastiCache Redis/Valkey #31075

Changes

Product

• Fleet server configuration changes: [Fleet server config] IAM authentication for MySQL and Redis #32442
• UI changes: No changes
• CLI (fleetctl) usage changes: No changes
• YAML changes: No changes
• REST API changes: No changes
• Fleet's agent (fleetd) changes: No changes
• GitOps mode UI changes: No changes
• GitOps generation changes: No changes
• Activity changes: No changes
• Permissions changes: No changes
• Changes to paid features or tiers: No changes
• My device and fleetdm.com/better changes: No changes
• Other reference documentation changes: No changes
• First draft of test plan added
• Once shipped, requester has been notified: Redis: AWS IAM authentication #20527 (comment)
• Once shipped, dogfooding issue has been filed: https://github.com/fleetdm/confidential/issues/12581#issue-3502907800

Engineering

• Test plan is finalized
• Contributor API changes: no
• Feature guide changes: TODO
• Database schema migrations: no
• Load testing: no
• Load testing/osquery-perf improvements: no <-- List, or link a subtask for, any osquery-perf or load test environment changes required to comprehensively load test this story if load testing is needed. -->

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

• Requires load testing: Probably not (Developer can override)
• Risk level: Med
• Risk description: Medium because this changes access to DB/Redis with a potential for a breach in data protection. (theft or manipulation.) However, if using the AWS libraries properly, this risk is not high.

Test setup

Note that for ease-of-testing, a couple of tools have been provided:

• tools/mysql-tests/rds/deploy-test-env.sh -- this sets up an RDS instance and a couple of IAM users (one with direct access permissions, one that requires a role to be assumed). The output contains everything you need to point Fleet at the instance. For an end-to-end test you'll need to run the db migrations first!
• tools/redis-tests/elasticache/deploy-test-env.sh -- this does the same for Elasticache. Elasticache is not meant to be publicly available, but the test script opens a publicly-available EC2 with SSH, so we can test by first opening a tunnel:

ssh -i <path to SSH Key from setup script output> -N -L 7000:<ElastiCache Endpoint from setup script output>:6379 ec2-user@<EC2 Public IP from setup script output>`

for example:

ssh -i ./fleet-test-key-1756242722.pem -N -L 7000:fleet-iam-test-cache-ddngoy.serverless.use1.cache.amazonaws.com:6379 ec2-user@34.202.160.51

and then pointing Fleet at it using the following settings:

• redis​_address: localhost:7000
• redis​_use​_tls: true
• redis​_tls​_server​_name: <server address from setup script output>

After testing is complete, make sure to tear down both the mysql and redis environments using:

• tools/mysql-tests/rds/cleanup-test-env.sh
• tools/redis-tests/elasticache/cleanup-test-env.sh

Test plan

Make sure to go through the list and consider all events that might be related to this story, so we catch edge cases earlier.

• Standup a Fleet instance (can be local or in AWS; either way you'll want to follow the test setup steps above unless you want to manually set up reachable RDS/Elasticache instances).
• For MySQL, specify an Amazon Relational Database Service (RDS) endpoint for mysql_address. Don't specify mysql_password nor mysql_password_path is specified, but provide mysql_region. Verify that Identity and Access Management (IAM) authentication is used.
• For MySQL, do the above but also specify mysql_sts_assume_role_arn with and without a mysql_sts_external_id. Verify that MySQL works in both scenarios by saving some queries.
• For Redis, specify an AWS ElastiCache endpoint for redis_address. Don't specify redis_password, but specify redis_cache_name and redis_region. Verify that Identity and Access Management (IAM) authentication is used.
• For Redis, do the above but also specify redis_sts_assume_role_arn with and without a redis_sts_external_id. Verify that Redis works in both scenarios by running some live queries.
• For MySQL and Redis, verify that if username and password are set then basic auth is used, regardless of other config. @noahtalerman currently it's set up so to only use IAM if no password is provided and other AWS-specific config (like region is set). We can flip this if we want, and use the presence of region config to mean "use IAM and ignore username/password".

Testing notes

Confirmation

1. Engineer: Added comment to user story confirming successful completion of test plan.
2. QA: Added comment to user story confirming successful completion of test plan.

[edwardsb]

{"mysql":"could not connect to db: x509: certificate signed by unknown authority, sleeping 3s","ts":"2022-02-16T13:56:07.203299221Z"}

[ChefAustin]

FWIW: Would love to see this come to fruition!

[noahtalerman]

Hey @edwardsb I closed this request because it looks like a duplicate of the following request:

• Redis: AWS IAM authentication #20527

Please let me know if I'm wrong!

[fleet-release]

In cloud's embrace, Fleet,
RDS IAM access blooms,
Ease for users meets.

[titanous]

@noahtalerman This is not a duplicate, #20527 is about Redis, and this is about MySQL (RDS).