-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-12790] Add projection pushdown to JDBC SchemaIO. #15373
Conversation
R: @apilloud |
if (i > 0) { | ||
query.append(", "); | ||
} | ||
query.append(fieldsAccessed.get(i).getFieldName()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens when the field name is FROM
(or another reserved keyword)?
} | ||
} | ||
query.append(" FROM "); | ||
query.append(getLocation()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an existing bug, but what happens when the table name is WHERE
?
return String.format("SELECT * FROM %s", getLocation()); | ||
} | ||
|
||
// Build query from field access descriptor. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
JdbcUtil.generateStatement does something similar for INSERT INTO
. There is also a generateWriteStatement method below in this file. They all seem to have the same bugs. It would be nice if there was some code reuse across these.
@apilloud Escaping strings is difficult, since JDBC has to work with various incompatible SQL implementations. The answers to this Stack Overflow question suggest that we can learn the database's quoting syntax via the JDBC Connection. The good news is that we can access the Connection through the PreparedStatement in JdbcIO's StatementPreparator. The bad news is there's no way to change the query, aside from setting parameters, which cannot be used for column and table names. beam/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java Lines 1168 to 1173 in c4e0b4a
So to get this to work I think we would have to add a method to JdbcIO that takes a functional interface that takes a Connection and returns a fresh PreparedStatement. |
Codecov Report
@@ Coverage Diff @@
## master #15373 +/- ##
==========================================
+ Coverage 83.69% 83.76% +0.07%
==========================================
Files 440 442 +2
Lines 59950 60050 +100
==========================================
+ Hits 50173 50302 +129
+ Misses 9777 9748 -29
Continue to review full report at Codecov.
|
Closing this since string escaping has unresolved blockers. |
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username
).[BEAM-XXX] Fixes bug in ApproximateQuantiles
, where you replaceBEAM-XXX
with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.md
with noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner
compliance status (on master branch)Examples testing status on various runners
Post-Commit SDK/Transform Integration Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.