message-rustler

lambda para consumir dados de uma fila SQS e atualizar um registro no banco

Architecture

flowchart LR
    SQS[("SQS queue")] -->|batch| Lambda["Lambda (Rust)<br/>handle_batch"]
    Lambda -->|parse + whitelist.validate| Whitelist["whitelist.yaml<br/>(embedded, cold start)"]
    Whitelist -->|table/column OK| Query["build_update_sql<br/>(bind params)"]
    Query -->|sqlx| DB[("Aurora MySQL<br/>via RDS Proxy, IAM auth")]
    Lambda -->|batchItemFailures| SQS
    SQS -->|maxReceiveCount exceeded| DLQ[("Dead-letter queue")]

Loading

Commands

make help lists every command below with a one-line description. The sections that follow show the same commands with more context.

Testing locally

Requires Rust (edition 2021, developed against 1.95 — install via rustup if you don't have it):

cargo test

All tests are unit tests (whitelist validation, message parsing, query building, repository via mock, auth token signing, handler) — no database, no AWS credentials, no network access needed. 17 tests as of this writing.

Full CI-equivalent check before committing:

cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo test

cargo-lambda and zig (below) are only needed to build/deploy the real Lambda binary, not to run the test suite.

Build

Requires cargo-lambda and Zig (used by cargo-lambda for cross-compiling):

curl -fsSL https://cargo-lambda.info/install.sh | sh
pip3 install ziglang   # or: npm install -g @ziglang/cli

If zig isn't a binary on your PATH after the pip install, add a shim (~/.local/bin/zig → exec python3 -m ziglang "$@") and make sure ~/.local/bin is on PATH.

cargo lambda build --release

Produces the target/lambda/message-rustler/bootstrap binary. Terraform deploys a zip, so package it manually before applying:

cd target/lambda/message-rustler && zip bootstrap.zip bootstrap && cd -

Deploy

cd terraform
terraform init
terraform apply \
  -var="aws_region=<region>" \
  -var="rds_proxy_name=<existing-rds-proxy-name>" \
  -var="rds_proxy_resource_id=<prx-xxxxxxxxxxxx>" \
  -var="db_name=<database-name>" \
  -var="db_user=<iam-auth-db-user>"

Run everything locally

Simulates the full flow end to end — a message landing on SQS, the Lambda picking it up, the row getting updated in MySQL, and a failing message ending up in the DLQ — without touching AWS.

What it uses:

• MySQL (docker-compose) seeded with an addresses table (local/init-db.sql) — id, street, city, state, zip_code, already added to config/whitelist.yaml.
• LocalStack (docker-compose), SQS only — real SQS semantics (redrive policy included), no AWS account needed.
• cargo lambda watch — runs the actual Lambda binary as a local HTTP server, same code that ships to AWS.
• local/poll-and-invoke.sh — stands in for the real SQS→Lambda event source mapping: long-polls the LocalStack queue, invokes the running Lambda for each batch, deletes only the messages it reports as successful. Messages it reports as failed stay in the queue and get redriven to the DLQ by LocalStack after 3 receives — same mechanism as production.
• sqs-admin (docker-compose) — web UI at localhost:3999 to watch both queues: in-flight message counts, in-flight vs. visible, and the DLQ filling up as retries happen.

Prerequisites beyond the Build section above: docker compose, aws CLI, jq.

1. Start the infra

docker compose up -d

Waits are built into the healthchecks; docker compose ps should show both services healthy within ~30s.

2. Create the queue + DLQ on LocalStack

./local/setup-localstack.sh

3. Run the Lambda locally

In one terminal, pointed at the docker-compose MySQL instead of RDS Proxy — DB_PASSWORD is what switches src/main.rs from IAM auth to plain password auth (see the comment at src/main.rs:56); it's never set in the Terraform-managed deploy, so this path never runs in production:

DB_HOST=127.0.0.1 DB_PORT=3306 DB_NAME=app DB_USER=app_user DB_PASSWORD=app_password \
  cargo lambda watch

4. Run the bridge script

In a second terminal:

./local/poll-and-invoke.sh

5. Publish a message

In a third terminal — update an address (id: 1 is seeded):

aws --endpoint-url http://localhost:4566 sqs send-message \
  --queue-url http://localhost:4566/000000000000/message-rustler-queue \
  --message-body '{"table":"addresses","id":1,"fields":{"city":"Rio de Janeiro","state":"RJ"}}'

The bridge script logs [OK] <id> -> processed, deleted. Confirm the update landed:

docker compose exec mysql mysql -uapp_user -papp_password app -e "SELECT * FROM addresses WHERE id = 1;"

6. Force a DLQ message

Send something the whitelist rejects (unknown table, so it never reaches MySQL) — it'll retry 3 times, then land in the DLQ:

aws --endpoint-url http://localhost:4566 sqs send-message \
  --queue-url http://localhost:4566/000000000000/message-rustler-queue \
  --message-body '{"table":"not_a_real_table","id":1,"fields":{"city":"Nowhere"}}'

Watch the bridge script log [FAIL] up to 3 times, then check the DLQ:

aws --endpoint-url http://localhost:4566 sqs receive-message \
  --queue-url http://localhost:4566/000000000000/message-rustler-dlq

Tear down / reset:

docker compose down -v   # -v drops the MySQL volume, so the next `up` reseeds it

Message format

The Lambda expects SQS message bodies shaped like:

{
  "table": "orders",
  "id": 123,
  "fields": {
    "status": "shipped",
    "updated_at": "2026-08-14T12:00:00Z"
  }
}

Only tables/columns listed in config/whitelist.yaml are accepted; changing the whitelist requires a rebuild and redeploy.