Cache DbApiHook.inspector to avoid creating N engines#62594
Merged
kaxil merged 1 commit intoapache:mainfrom Feb 27, 2026
Merged
Cache DbApiHook.inspector to avoid creating N engines#62594kaxil merged 1 commit intoapache:mainfrom
DbApiHook.inspector to avoid creating N engines#62594kaxil merged 1 commit intoapache:mainfrom
Conversation
DbApiHook.inspector was a @Property that called get_sqlalchemy_engine() on every access. Each call creates a new SQLAlchemy engine with its own connection pool. When inspecting multiple tables (e.g., GenericTransfer, SQLColumnCheckOperator), this created N engines instead of reusing one. Change @Property to @cached_property so the inspector (and its underlying engine) is created once per hook instance. The hook is scoped to a single task execution, so the cache lifetime is appropriate. Update the .pyi stub to match.
AkshayArali
pushed a commit
to AkshayArali/airflow_630
that referenced
this pull request
Feb 28, 2026
DbApiHook.inspector was a @Property that called get_sqlalchemy_engine() on every access. Each call creates a new SQLAlchemy engine with its own connection pool. When inspecting multiple tables (e.g., GenericTransfer, SQLColumnCheckOperator), this created N engines instead of reusing one. Change @Property to @cached_property so the inspector (and its underlying engine) is created once per hook instance. The hook is scoped to a single task execution, so the cache lifetime is appropriate. Update the .pyi stub to match.
81 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DbApiHook.inspectorwas a@propertythat calledget_sqlalchemy_engine()on every access.get_sqlalchemy_engine()callscreate_engine()each time, opening a new connection pool.When inspecting multiple tables (e.g.,
GenericTransfer,SQLColumnCheckOperator, or the newLLMSQLQueryOperator), this created N SQLAlchemy engines instead of reusing one. With defaultpool_size=5, inspecting 10 tables could open up to 50 database connections.Changed
@property→@cached_property, matching the existing pattern used bydialect_nameand
dialecton the same class. The hook is scoped to a single task execution, so the cachelifetime is appropriate. Updated the
.pyistub to match.Was generative AI tooling used to co-author this PR?