Support parallel reads from a single source.

[cgardens]

Tell us about the problem you're trying to solve

Not to be confused with this issue: #4081, here we are trying to make it possible to replicate multiple streams for a single source at the same time. i.e. if a source has two tables, both of those tables might be replicated in paralell.

[hpias]

One should only ever split the source load to multiple connections to have different sync schedules for different streams, never because of parallelism.
This should probably be a setting that is connection based, or even schedule based. Workers, Sources and destinations shouldnt care about it. Strictly orchestration/scheduler implementaion reaponsibility. Eg split configured streams in number of configured ques (connection parallelism/ max parallel streams), assign to worker. Or even better, Que only a single stream x parallel workers at once to account for the difference in stream loads to achieve round robin if the cost of instantiation of a single stream worker is not too high.
The scheduling optimizer could be further optimized by either using stats or changing the source spec to include estimated row counts or similar useful metrics for the stream weighting factor.
Normalization should be optionally postponed if it affects workers (not sure how it is implemented) for successful EL streams. The idea is to be able to scale in the k8s pool once EL has completed as DBT is destination workload.

[toandm]

This will be a great addition to Airbyte. I currently sync hundreds of table from a MySQL source. It takes ages to load them all in.

[blake-enyart]

@cgardens are there any updates on progress on this issue? Is this coming in tandem with the per-stream sync update or afterwards?

[luancaarvalho]

Any news ?

[Jordonkopp]

Any updates on this feature request?

[ggam]

This is much needed as the workaround is to maintain multiple connections, splitting streams between them, which increases maintenance effort.

[samsipe]

+1 on this. Would be a huge help!

[mcivorsteiner]

+1 on this, this is big issue for us currently