Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PHOENIX-5197: Use extraOptions to set the configuration for Spark Workers #2

Closed
wants to merge 1 commit into from
Closed

PHOENIX-5197: Use extraOptions to set the configuration for Spark Workers #2

wants to merge 1 commit into from

Conversation

ChinmaySKulkarni
Copy link
Contributor

No description provided.

@twdsilva
Copy link
Contributor

twdsilva commented Apr 3, 2019

Is there a way to add a test to verify that if you pass in a property in via the options in the datasource it will be used?

@ChinmaySKulkarni
Copy link
Contributor Author

A couple of problems that we will run into:

  1. DataSourceOptions converts all keys in the options map to lowercase. When creating a reader or writer, we create a DataSourceOptions object which is passed in all options. See createReader and createWriter. This is a problem for passing in a lot of Phoenix configurations like phoenix.query.timeoutMs which are case-sensitive.
    -> We can potentially get past this issue by serializing all configs against a single key in the options map and passing that to the workers.

  2. I can look into setting some small values for upsert batch size or query timeouts from the driver to see if the workers honor these values.
    FYI @twdsilva

@twdsilva
Copy link
Contributor

twdsilva commented Apr 8, 2019

@ChinmaySKulkarni After talking offline, there isn't a clean way to test this, so its fine to not include a test.

Regarding DataSourceOptions converting all the keys to lower case, we should come up with a serialization format to store the key value pairs as a value so that we can preserve the case.

twdsilva
twdsilva reviewed Apr 8, 2019
View reviewed changes
...nix-spark/src/main/java/org/apache/phoenix/spark/datasource/v2/writer/PhoenixDataWriter.java
@@ -51,7 +53,7 @@ public PhoenixDataWriter(PhoenixDataSourceWriteOptions options) {
String scn = options.getScn();
String tenantId = options.getTenantId();
String zkUrl = options.getZkUrl();
Properties overridingProps = new Properties();
Properties overridingProps = options.getOverriddenProps();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can options.getOverriddenProps return null?

...src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixInputPartitionReader.java
@@ -77,14 +80,15 @@ private QueryPlan getQueryPlan() throws SQLException {
String scn = options.getScn();
String tenantId = options.getTenantId();
String zkUrl = options.getZkUrl();
Properties overridingProps = new Properties();
Properties overridingProps = options.getOverriddenProps();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think options.getOverriddenProps can never be null here, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally they can never be null, but added a null-check to avoid any problems in the future. Same for the write path.

@ChinmaySKulkarni
Copy link
Contributor Author

@twdsilva Please review now. I have added a unit test and separate logging properties for IT, src and test directories as well.

twdsilva
twdsilva reviewed Apr 8, 2019
View reviewed changes
Copy link
Contributor

@twdsilva twdsilva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM, I just had one minor comment.

...c/main/java/org/apache/phoenix/spark/datasource/v2/writer/PhoenixDataSourceWriteOptions.java Outdated Show resolved Hide resolved
@ChinmaySKulkarni
Copy link
Contributor Author

Thanks for the review @twdsilva. I will commit this today.

@ChinmaySKulkarni ChinmaySKulkarni deleted the PHOENIX-5197 branch April 10, 2019 01:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@twdsilva twdsilva twdsilva left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ChinmaySKulkarni @twdsilva