[Feature Request]: Yaml JdbcIO partitioning

[damccorm]

What would you like to happen?

Right now, Yaml's JdbcIO doesn't support reading with partitioning. To support this, we should do the following:

1. Extend ReadRows to accept partitioning -

   beam/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java

   Line 730 in d6a5e73

   public abstract static class ReadRows extends PTransform<PBegin, PCollection<Row>> {

2. Add a test here - https://github.com/apache/beam/blob/d6a5e730ad95973c762ef8f6ea4958592aa1cb6b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java
3. Extend JdbcReadSchemaTransformProvider to accept this as a parameter - https://github.com/Polber/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcReadSchemaTransformProvider.java
4. Update the mapping here -

   beam/sdks/python/apache_beam/yaml/standard_io.yaml

   Line 195 in d6a5e73

   - type: renaming

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[shubham-j-sde]

.take-issue

[svetakvsundhar]

Hey @shubham-j-sde , i am actively working on this. but If you would like to, I am happy to review the PR.

[shubham-j-sde]

I am new to Apache Beam, have completed setup and read beginning docs and I wud love to take this. Thank you so much Svetak for your understanding. I'll tag you as I publish PR.

[svetakvsundhar]

Thanks, @shubham-j-sde . This is a somewhat high priority request, so please do reach out should you need help.

[claudevdm]

ReadRows and ReadWithPartitions have incompatible parameters:

1. ReadRows allows user to give a query and statementPreparator
2. ReadWithPartition only allows a table. I think this can techically be a subquery that evaluates as a table

It could get messy to add partition support to ReadRows for this reason, so I think the simplest way forward here is:

1. Do not make any changes to JdbcIO
2. Complete tasks (3) and (4) from this issue summary
3. Add a condition in https://github.com/Polber/beam/blob/7c86bf3103fec1d3a6a3a3cbe6ace8a14a85b723/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcReadSchemaTransformProvider.java#L110 where we use ReadRows if there is no partition column argument, otherwise use ReadWithPartition.withRowOutput similar to what is done here https://github.com/Polber/beam/blob/7c86bf3103fec1d3a6a3a3cbe6ace8a14a85b723/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcSchemaIOProvider.java#L128
4. Throw a validation error if the user specifys a query AND partition parameters, since reading with partitions only supports table argument

@damccorm does this sound good?

[damccorm]

Yeah, this sounds good to me - thanks Claude!