[BEAM-12790] Add projection pushdown to JDBC SchemaIO. #15373
Conversation
|
R: @apilloud |
| if (i > 0) { | ||
| query.append(", "); | ||
| } | ||
| query.append(fieldsAccessed.get(i).getFieldName()); |
There was a problem hiding this comment.
What happens when the field name is FROM (or another reserved keyword)?
| } | ||
| } | ||
| query.append(" FROM "); | ||
| query.append(getLocation()); |
There was a problem hiding this comment.
This is an existing bug, but what happens when the table name is WHERE?
| return String.format("SELECT * FROM %s", getLocation()); | ||
| } | ||
|
|
||
| // Build query from field access descriptor. |
There was a problem hiding this comment.
JdbcUtil.generateStatement does something similar for INSERT INTO. There is also a generateWriteStatement method below in this file. They all seem to have the same bugs. It would be nice if there was some code reuse across these.
|
@apilloud Escaping strings is difficult, since JDBC has to work with various incompatible SQL implementations. The answers to this Stack Overflow question suggest that we can learn the database's quoting syntax via the JDBC Connection. The good news is that we can access the Connection through the PreparedStatement in JdbcIO's StatementPreparator. The bad news is there's no way to change the query, aside from setting parameters, which cannot be used for column and table names. beam/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java Lines 1168 to 1173 in c4e0b4a So to get this to work I think we would have to add a method to JdbcIO that takes a functional interface that takes a Connection and returns a fresh PreparedStatement. |
Codecov Report
@@ Coverage Diff @@
## master #15373 +/- ##
==========================================
+ Coverage 83.69% 83.76% +0.07%
==========================================
Files 440 442 +2
Lines 59950 60050 +100
==========================================
+ Hits 50173 50302 +129
+ Misses 9777 9748 -29
Continue to review full report at Codecov.
|
|
Closing this since string escaping has unresolved blockers. |
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunnercompliance status (on master branch)Examples testing status on various runners
Post-Commit SDK/Transform Integration Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.