Loader parallelism strategies #1457
Conversation
✅ Deploy Preview for dlt-hub-docs canceled.
|
|
@rudolfix which one is the destination that requires sequentials jobs? |
| @@ -30,6 +30,8 @@ | |||
| # insert_values - insert SQL statements | |||
| # sql - any sql statement | |||
| TLoaderFileFormat = Literal["jsonl", "typed-jsonl", "insert_values", "parquet", "csv"] | |||
| TLoaderParallelismStrategy = Literal["parallel", "table_sequential", "sequential"] | |||
There was a problem hiding this comment.
hmm I think we are using dashes not underscores in other places? surely the above both are present :)
dlt/load/utils.py
Outdated
| return file_names | ||
|
|
||
| # destination can overwrite ps | ||
| parallelism_strategy = capabilities.loader_parallelism_strategy or config.parallelism_strategy |
There was a problem hiding this comment.
config should always has last say. by default config should be None - no strategy should be forced
dlt/load/utils.py
Outdated
|
|
||
| # we must ensure there only is one job per table | ||
| if parallelism_strategy == "table_sequential": | ||
| filtered_jobs: List[str] = [] |
There was a problem hiding this comment.
this looks good. but you could also use group_by by table name, maybe the code will be simpler?
There was a problem hiding this comment.
i made a one-liner out of it, not sure if it is easier to read now, but it is more pythonic i guess :)
There was a problem hiding this comment.
Turns out you need to sort by the key you group by for the groupby to work as expected..
|
@rudolfix I have added an explanation of the new config values to the custom destination page. I briefly thought about adding the parallelization strategy setting to the performance docs page, but I am actually not sure it is useful there, for now I believe we only need this in the custom destination. |
* remove underscore from settingsvalue * simplify job selection function * let config always override destination settings
sh-rp
force-pushed
the
feat/1456-loader-parallelism
branch
from
0ce64ba
to
6942393
Compare
|
Also I don't know what is going on with this failing duckdb test, this does not happen locally for me and I also don't see any way my changes could influence this.. |
Description