Fixing bug #18746. Adding a new kwargs parameter to DbClient #18873
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Fixing issue #18746.
DuckDb can only have one write connection at a time, but multiple read_only connection. Right now we always open duckdb in write mode which can cause exceptions when multiple ops try to access the file at the same time (specially on a slow server).
The goal is to be able to open duckdb in read_only whenever possible and only open it in write mode when we need to.
I added a new kwargs parameter to DbClient connect method and a new way to pass a lambda to DbIOManager to calculate this kwargs dynamically based in the context (either reading or writing data).
How I Tested These Changes
This is my first PR to dagster and the contributing guidelines does not seems to be made for a windows dev, so unfortunately I did not test the code. Sorry in advance.