feat(mypyc): Enable incremental compilation, deprecate Python 3.9#7574
Merged
feat(mypyc): Enable incremental compilation, deprecate Python 3.9#7574
Conversation
Switch sqlglotc's mypyc build to separate=True so each compiled module
gets its own shared lib + shim. Clean builds are roughly the same
wall-clock; incremental rebuilds after a one-line edit drop from a full
~110s monolithic rebuild to a few seconds once the cache is warm.
Python 3.9 is dropped from `sqlglot[c]` because sqlglot-mypy 1.20+ (the
build dep needed for the separate=True codegen fixes) only ships wheels
for 3.10+. Pure-Python sqlglot still supports 3.9; on 3.9
`pip install sqlglot[c]` is a no-op via an environment marker, and
`make install-devc` short-circuits with a clear "requires Python 3.10+"
message.
- sqlglotc/pyproject.toml: requires-python >= 3.10, build dep pinned
to sqlglot-mypy >= 1.20.0.post4 (which carries the cached-SCC fix
that makes the wheel-build path actually package shared libs).
- setup.py: [c] / [rs] extras gated on python_version >= '3.10'; on
3.9 fall back to upstream `mypy` for type checking instead of
sqlglot-mypy.
- Makefile: install-devc / install-devc-release skip the build on 3.9.
sqlglot-side tweaks for the new build:
- sqlglot/__init__.py: drop the legacy *__mypyc*.so bootstrap
preloader. It was written for the old monolithic build where a
single hash-named .so sat at the package root; under separate=True
per-module shared libs sit next to their .py siblings and resolve
through the shim.
- sqlglot/optimizer/__init__.py: swap eager top-level re-exports for
PEP 562 `__getattr__`. Under separate=True's cross-group init
ordering, the previous eager `from sqlglot.optimizer.optimizer
import ...` could fire while sqlglot/__init__.py was still
mid-bootstrap and trip a circular `ImportError` on
`from sqlglot import Schema, exp`. Lazy resolution defers the
cycle past sqlglot's init. Concurrent first-access lookups are
serialised with an RLock, mirroring sqlglot/dialects/__init__.py.
VaggelisD
force-pushed
the
feat/mypyc-incremental-compilation
branch
from
a348e6a to
61d1586
Compare
Contributor
SQLGlot Integration Test ResultsComparing:
By Dialect
Overallmain: 113234 total, 112044 passed (pass rate: 98.9%), sqlglot version: sqlglot:feat/mypyc-incremental-compilation: 101035 total, 101035 passed (pass rate: 100.0%), sqlglot version: Transitions: Dialect pair changes: 0 previous results not found, 3 current results not found ✅ 37 test(s) passed |
georgesittas
approved these changes
Apr 28, 2026
Collaborator
|
@VaggelisD did you check for memory leaks here? |
Collaborator
Author
|
@georgesittas The PR was fine when I first tested it but curiosity got the best of me and I checked again, looks like there's a tiny leak in the order of 5 Investigating why that is |
Collaborator
Author
|
Answered internally as well but leaving it here for reference: I think this was just a fluke caused by When we had the first Considering this a non issue for now, might need to decrease |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Switch
sqlglotc's mypyc build toseparate=True, plus the small sqlglot-side tweaks needed to make it work end-to-end. Also deprecates Python 3.9 forsqlglot[c](pure-Pythonsqlglotstill supports 3.9).The
separate=Trueflag gives each compiled module its own shared lib + shim, so mypyc only has to regenerate / recompile the modules whose source actually changed.Changes
sqlglot[c]build:sqlglotc/setup.py: passseparate=Truetomypycify().sqlglotc/pyproject.toml: bumprequires-pythonto>= 3.10, pin build depsqlglot-mypy >= 1.20.0.post4. The post4 release carries a cached-SCC codegen fix that makes pip's two-pass wheel build actually package the shared libs (without it, the secondmypycifypass returnedExtension(sources=[])and produced an installable but broken wheel).setup.py: gate the[c]/[rs]extras onpython_version >= '3.10'. On 3.9,pip install sqlglot[c]is a no-op and you get pure-Python sqlglot. Dev extras fall back to upstreammypyfor type checking on 3.9.Makefile:install-devc/install-devc-releaseskip the sqlglotc build on 3.9 with a clear message.sqlglot package:
sqlglot/__init__.py: drop the legacy*__mypyc*.sobootstrap preloader. It was written for the old monolithic build where a single hash-named.sosat at the package root; underseparate=Truethe per-module shared libs (e.g.errors__mypyc.so) live next to their.pysiblings and resolve through Python's normal import machinery via the shim.sqlglot/optimizer/__init__.py: swap the eager top-level re-exports for PEP 562__getattr__. Underseparate=True's cross-group init ordering, the previous eagerfrom sqlglot.optimizer.optimizer import ...could fire whilesqlglot/__init__.pywas still mid-bootstrap and trip a circularImportErroronfrom sqlglot import Schema, exp. Lazy resolution defers the cycle past sqlglot's init. Concurrent first-access lookups are serialised with anRLock, mirroringsqlglot/dialects/__init__.py.Replaces #7558 (squashed history).