Skip to content

Add dynamic splitting to JdbcIO.readWithPartitions #21544

@damccorm

Description

@damccorm

Now, the JDBC IO is basically a DoFn executed with a {}ParDo{}. So, it means that parallelism is "limited" and executed on one executor. ReadWithPartitions does some preliminary partitioning of the data, but any skew in data range or workload will create an unbalanced workload.

 

Imported from Jira BEAM-14161. Original Jira may contain additional context.
Reported by: pabloem.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions