Problem
Athena bills per byte scanned, with a 10MB minimum per query. A query that names no partition, or names one wrongly, scans the whole dataset and the first sign is the bill. Nothing currently bounds that. Query results also accumulate in S3 indefinitely unless something expires them.
Desired behaviour
A workgroup with a per-query bytes-scanned cutoff, and a results bucket whose objects expire. An accidental full scan then fails at the point it is run, naming the limit, instead of arriving at the end of the month.
Acceptance criteria
Implementation notes
This is its own issue rather than a footnote on the Glue table because a guardrail left until later is one that never gets retrofitted. It is also the cheapest possible insurance against the one failure mode this project cares most about, which is a cost that arrives without anyone deciding to spend it.
Yulin has simulated Athena since 1.20.8, and from 1.20.9 it measures bytes scanned from the objects a query actually reads. The cutoff can therefore be proven to refuse a query rather than merely to have been configured, which is the difference between testing this guardrail and testing that somebody typed a number.
Problem
Athena bills per byte scanned, with a 10MB minimum per query. A query that names no partition, or names one wrongly, scans the whole dataset and the first sign is the bill. Nothing currently bounds that. Query results also accumulate in S3 indefinitely unless something expires them.
Desired behaviour
A workgroup with a per-query bytes-scanned cutoff, and a results bucket whose objects expire. An accidental full scan then fails at the point it is run, naming the limit, instead of arriving at the end of the month.
Acceptance criteria
BytesScannedCutoffPerQueryset, and the chosen value justified.docs/page covering what the limit is, why, and how to raise it deliberately.Implementation notes
This is its own issue rather than a footnote on the Glue table because a guardrail left until later is one that never gets retrofitted. It is also the cheapest possible insurance against the one failure mode this project cares most about, which is a cost that arrives without anyone deciding to spend it.
Yulin has simulated Athena since 1.20.8, and from 1.20.9 it measures bytes scanned from the objects a query actually reads. The cutoff can therefore be proven to refuse a query rather than merely to have been configured, which is the difference between testing this guardrail and testing that somebody typed a number.