Fix(duckdb): Only SET connector_config values on cursor init if they are different#4981
Merged
Fix(duckdb): Only SET connector_config values on cursor init if they are different#4981
Conversation
c0dd637 to
1bf06cf
Compare
themisvaltinos
approved these changes
Jul 17, 2025
sqlmesh/core/config/connection.py
Outdated
| option_names, | ||
| ) | ||
|
|
||
| existing_values = {r[0]: r[1] for r in cursor.fetchall()} |
Contributor
There was a problem hiding this comment.
Suggested change
| existing_values = {r[0]: r[1] for r in cursor.fetchall()} | |
| existing_values = {field: setting for field, setting in cursor.fetchall()} |
nit: maybe something more explicit like this so that it's more clear when going through the code
Collaborator
Author
There was a problem hiding this comment.
thanks, it didnt even occur to me to unpack the tuple like that. will update
sqlmesh/core/config/connection.py
Outdated
| raise ConfigError(f"Failed to set connector config {field} to {setting}: {e}") | ||
| if self.connector_config: | ||
| option_names = list(self.connector_config) | ||
| in_part = ",".join(["?" for _ in range(len(option_names))]) |
Contributor
There was a problem hiding this comment.
Suggested change
| in_part = ",".join(["?" for _ in range(len(option_names))]) | |
| in_part = ",".join("?" for _ in option_names) |
nit: this should achieve the same
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prior to this, a duckdb connection config like:
would result in a
SETcall for each item inconnector_configevery time a new cursor was initialized.This isnt a problem for many properties, but ones like
temp_directoryhave side effects. If DuckDB has had to spill data to disk (and created files intemp_directory), if you try toSET temp_directory='...', even to the same value, it will throw an error:This PR changes the cursor init to read the existing values and only make
SETcalls if the configured values differ from what is already set