Repository navigation
Pre-flight SQL query syntax validation and partition guard in Common SQL provider #76373
Unanswered
meel-ayush
asked this question in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hey everyone,
Wanted to share an idea and see what people think before putting together any code.
Right now in
providers/common/sql,SQLExecuteQueryOperatorsends whatever SQL string it receives straight to the database hook. If there's a syntax typo or a query accidentally misses a partition filter on a large table, the task only fails after opening the connection and waking up the warehouse (which on Snowflake/BigQuery takes time and costs money).While static SQL files are usually caught in CI using linters like
sqlfluff, dynamic queries (where filters or table names are injected via Jinja/XComs) only get rendered at task runtime and slip through.I was looking at
sqlglotrecently (pure Python, MIT license, zero C dependencies). Since Jinja templates are already rendered by the timeexecute()runs, we could optionally run a fast client-side check (< 2ms) before creating the DB connection:ds) are present in theWHEREclause to prevent accidental full-table scans.To avoid bloating
common.sql, this could be an optional extra (likeapache-airflow-providers-common-sql[validator]). Ifsqlglotisn't installed or the check isn't enabled on the operator, behavior stays 100% unchanged.A couple of open questions:
common.sql, or is this something users should just handle in custom wrapper operators?sqlglotacross Airflow's supported dialects (Snowflake, BigQuery, Postgres, etc.)?Looking forward to hear thoughts from maintainers and anyone running heavy warehouse workloads.
All reactions