Add `sqlglot[c]` (`sqlglotc`) support for faster parsing

[fresioAS]

Background

SQLMesh currently depends on plain sqlglot~=30.8.0 with no extras. The sqlglot project
ships an optional [c] extra (sqlglotc) that compiles the full sqlglot core ahead-of-time
for significantly faster SQL parsing, generation, and optimization.

---

Compatibility concern: extend_sqlglot() and _core_cache

SQLMesh's extend_sqlglot() (in sqlmesh/core/dialect.py) patches every registered
Tokenizer subclass by mutating VAR_SINGLE_TOKENS to support the @ macro prefix:

for tokenizer in tokenizers:  
    tokenizer.VAR_SINGLE_TOKENS.update(SQLMESH_MACRO_PREFIX)

With sqlglotc, the tokenizer's _init_core() reads VAR_SINGLE_TOKENS at construction
time and passes it to a TokenizerCore instance. The result is stored in a
ThreadLocalCache (a threading.local subclass) keyed by tokenizer type, accessible as
Tokenizer._core_cache.cache.

The risk: if any thread calls _init_core() before extend_sqlglot() runs, that
thread's cached TokenizerCore will not see the updated VAR_SINGLE_TOKENS.

Concrete remediation — if call ordering cannot be guaranteed, clear the per-thread
cache after patching:

for tokenizer in tokenizers:  
    tokenizer.VAR_SINGLE_TOKENS.update(SQLMESH_MACRO_PREFIX)  
    if hasattr(tokenizer, "_core_cache") and hasattr(tokenizer._core_cache, "cache"):  
        tokenizer._core_cache.cache.clear()

This ensures any thread that already populated its cache will re-initialize TokenizerCore
with the patched attributes on next use.

---

Python version constraint

SQLMesh supports Python >= 3.9. sqlglotc requires Python >= 3.10. Users on Python 3.9
cannot install sqlglotc and would fall back to pure Python.

The dependency change must account for this — either via a conditional marker:

"sqlglot[c]~=30.8.0; python_version >= '3.10'",  
"sqlglot~=30.8.0; python_version < '3.10'",

or by dropping Python 3.9 support.

---

Tasks

• Benchmark parse/transpile throughput with and without sqlglotc on representative
  SQLMesh workloads (large model files, many-model projects). Gains are expected across
  parsing, generation, and optimization — not just tokenization.
• Verify extend_sqlglot() ordering: confirm extend_sqlglot() always runs before
  any thread triggers _init_core(). If ordering cannot be guaranteed, add
  _core_cache.cache.clear() calls on all patched tokenizer subclasses after mutating
  VAR_SINGLE_TOKENS.
• Run the full test suite with sqlglotc installed, paying particular attention to:
  Macro parsing (@var, @def, etc.)
  The Jinja rendering path in sqlmesh/core/renderer.py that calls dialect.tokenize()
  and dialect.parser().parse()
• Handle Python 3.9: decide between conditional dependency markers or dropping 3.9
  support.
• Add a CI job that explicitly installs sqlglot[c] on Python >= 3.10 to prevent
  future regressions.

[fresioAS]

Investigated with LLM

The main blocker is not tokenizer caching. SQLMesh defines 16 interpreted subclasses of SQLGlot expression classes, which cannot currently be instantiated with sqlglot[c]:

TypeError: interpreted classes cannot inherit from compiled

The likely solution is upstream support in SQLGlot for interpreted subclasses, potentially by applying @mypyc_attr(allow_interpreted_subclasses=True) to the relevant expression base classes.

[tobymao]

no planned support for this. allow_interpreted_subclasses greatly slows down mypyc.