Apply default batchSize based on Loader type #738
Labels
core
Developer Experience
Resolving this issue should improve development experience for the library users.
Milestone
In order to improve DX, Flow should try to detect if the current BatchSize is suitable for a given Loader.
In this example, batchSize is equal to 1. This means that Flow will try to insert rows into the db, one by one.
This can be easily changed by putting
batchSize(1_000)
just abovewrite
, but it also requires from developer some knowledge about how loaders work internally.What we can do, is use Optimizer in order to detect current batchSize when Loaders are added, and whenever we notice that batchSize wasn't set, we can automatically apply one.
The exact numbers should be predefined, I think we can start from 1k for each of the following:
For the file-based loaders, this is irrelevant, as most of them are writing rows one by one.
The text was updated successfully, but these errors were encountered: