Skip to content

feat(execution): add validate mode and a row guardrail - #116

Merged
kannandreams merged 1 commit into
mainfrom
feat/validate-mode
Aug 30, 2026
Merged

feat(execution): add validate mode and a row guardrail#116
kannandreams merged 1 commit into
mainfrom
feat/validate-mode

Conversation

@kannandreams

Copy link
Copy Markdown
Collaborator

§1 of the scale and exposure plan. Two bounds on what a chart query returns, plus the architectural ceiling they imply.

Nothing bounded the input, and not by accident

Altair carries a 5,000-row cap, so this looked like it was already handled. It is not: altair/utils/save.py:283 runs data_transformers.disable_max_rows() while saving, because vl-convert needs the data inlined — and saving is exactly the path glyf renders through. Confirmed directly: at 20k rows chart.to_dict() raises MaxRowsError while chart.save() happily writes a 5 MB SVG.

Measured on the real render path, one chart:

rows SVG PNG time
1,000 0.26 MB 129 KB 0.5s
10,000 2.55 MB 214 KB 0.3s
100,000 25.7 MB 644 KB 2.3s

SVG spends a DOM node per mark; PNG rasterises. embed_charts: true is the default and inlines that SVG into the dashboard page, so the page is the 25 MB.

glyf build --validate

Each query runs wrapped in limit 0 — columns, no rows. glyf checks every column bound by VISUALISE is present, writes the compiled SQL, and stops: no images, no data files, no dashboards, no export. execution.mode: validate does the same from glyf.yml.

On examples/simple_dbt with a clean output directory it writes five compiled SQL files and nothing else, in 0.47s against 0.87s for a full render — and that gap is on four rows of demo data, where fetching and drawing cost almost nothing.

Nothing is drawn deliberately. A chart rendered from a sample looks exactly like a real one and would be reviewed, or published, as if it were. The CLI says what it skipped rather than printing a green tick that reads like a build.

execution.max_rows

Unset by default — no existing project changes behaviour. When set, a query returning more fails:

visualisations/revenue.ggsql returned more than 3 rows. Aggregate the query or
raise execution.max_rows; glyf will not draw a chart from part of a result.

It never truncates. A chart drawn from an arbitrary slice of a result is indistinguishable from a correct one, which is the worst way for this to fail.

Both bounds are applied in SQL via wrap_row_limit (the original becomes a subquery, so a trailing order by cannot swallow the limit), so the warehouse sends less over the wire. The docs state plainly what that does not do: LIMIT bounds transfer and render time, not what the warehouse scans — an aggregate computes in full whatever follows it.

Ceiling recorded

ARCHITECTURE.md gains one Design Decisions entry: glyf is a compile-time renderer, not a data engine; a chart is a picture of an aggregate, and the SELECT is where aggregation belongs. A line chart is ~2,000 pixels wide, so a million points is 500 marks per pixel column — invisible to the reader, paid for by everyone. Rendering large results faster is noted as a separate question, and is next up as a spike.

Checks

247 passed, up from 225 — 15 for validate mode and the guardrail (including CLI coverage), 7 for the config keys.

Beyond the suite: validated the example project on a clean target (five compiled/*.sql, no charts/, no data/), confirmed max_rows: 3 fails and max_rows: 100 renders normally, and confirmed a full build still produces dashboards and the site unchanged.

Nothing bounded what a chart query returned. Altair carries a 5,000-row
cap, but altair/utils/save.py disables it while saving because vl-convert
needs the data inlined -- which is the path glyf renders through, so the
cap never applied here. Measured on the real path, one chart at 1k rows
is a 0.26 MB SVG, at 10k 2.55 MB, at 100k 25.7 MB; `embed_charts` inlines
that markup, so the dashboard page *is* that size.

`glyf build --validate` runs each query wrapped in `limit 0`, checks that
every column bound by VISUALISE is present, writes the compiled SQL and
stops -- no images, no data files, no dashboards, no export. That is what
CI actually wants to know: whether the SQL still runs and still binds the
columns the chart draws, not what the numbers are this morning. On the
example project it writes five compiled SQL files and nothing else.

Nothing is drawn on purpose. A chart rendered from a sample looks exactly
like a real one, and would be reviewed, or published, as if it were.

`execution.max_rows` bounds a normal build and fails when a query exceeds
it, naming the chart and saying to aggregate. It never truncates: a chart
drawn from an arbitrary slice of a result is indistinguishable from a
correct one. Unset by default, so no existing project changes behaviour.

Both bounds are applied in SQL so the warehouse sends less over the wire,
and both the docs and the commit say plainly what that does not do --
`LIMIT` bounds transfer and render time, not what the warehouse scans.

ARCHITECTURE.md records the ceiling this implies: glyf is a compile-time
renderer, not a data engine, and a chart is a picture of an aggregate.
@kannandreams
kannandreams merged commit 1e9a118 into main Aug 30, 2026
6 checks passed
@kannandreams
kannandreams deleted the feat/validate-mode branch August 30, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant