Skip to content

Add an Athena workgroup with a bytes-scanned limit #21

Description

@hughgrigg

Problem

Athena bills per byte scanned, with a 10MB minimum per query. A query that names no partition, or names one wrongly, scans the whole dataset and the first sign is the bill. Nothing currently bounds that. Query results also accumulate in S3 indefinitely unless something expires them.

Desired behaviour

A workgroup with a per-query bytes-scanned cutoff, and a results bucket whose objects expire. An accidental full scan then fails at the point it is run, naming the limit, instead of arriving at the end of the month.

Acceptance criteria

  • A workgroup with BytesScannedCutoffPerQuery set, and the chosen value justified.
  • A results bucket with a lifecycle expiry.
  • Workgroup configuration enforced, so a client cannot override the result location or the limit.
  • Deployed into a simulated account and read back in a test.
  • A docs/ page covering what the limit is, why, and how to raise it deliberately.

Implementation notes

This is its own issue rather than a footnote on the Glue table because a guardrail left until later is one that never gets retrofitted. It is also the cheapest possible insurance against the one failure mode this project cares most about, which is a cost that arrives without anyone deciding to spend it.

Yulin has simulated Athena since 1.20.8, and from 1.20.9 it measures bytes scanned from the objects a query actually reads. The cutoff can therefore be proven to refuse a query rather than merely to have been configured, which is the difference between testing this guardrail and testing that somebody typed a number.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions