Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Spark 3.0 #6488

Closed
tedyu opened this issue Nov 26, 2020 · 2 comments
Closed

Support Spark 3.0 #6488

tedyu opened this issue Nov 26, 2020 · 2 comments
Assignees
@tedyu
Labels
area/tools kind/new-feature This is a request for a completely new feature

Comments

@tedyu
Copy link
Contributor

tedyu commented Nov 26, 2020

This issue covers the following aspects:

  • upgrade of spark-cassandra-connector to b3.0 branch upstream and port over YB patches from 2.4.x releases
  • create validation tests for spark-cassandra-connector within Yugabyte DB codebase

The above tasks would be done in tandem iteratively.
@tedyu
Copy link
Contributor Author

tedyu commented Nov 30, 2020

cql4-4.txt

@rkarthik007 rkarthik007 added kind/new-feature This is a request for a completely new feature area/tools labels Dec 1, 2020
tedyu added a commit that referenced this issue Dec 22, 2020
@tedyu
[#6488] YCQL: Add unit tests for Spark 3.0 driver
a650ec1 
Summary:
This change adds spark tests which exercises Spark 3.0 driver.

The tests are based on AppBase.java which is migrated from yb-sample-apps.
This migration is due to the fact that Spark 3.0 driver uses the 4.x (Y)CQL driver
which changes the user-facing API of the driver in a backwards-incompatible way.
Therefore, instead of refactoring yb-sample-apps (or existing CQL test framework) for CQL 4.x,
we are just creating a dedicated CQL 4.x project and using that to test the Spark 3.0 driver.

Test Plan: mvn -Dtest=org.yb.loadtest.TestCassandraSparkWordCount test

Reviewers: nikhil, mihnea

Reviewed By: mihnea

Subscribers: yql

Differential Revision: https://phabricator.dev.yugabyte.com/D10001
@m-iancu
Copy link
Contributor

m-iancu commented Jan 6, 2021

Implemented in the Spark connector repo by: yugabyte/spark-cassandra-connector@837dbaf and yugabyte/spark-cassandra-connector@857dd4e.
YB tests added in a650ec1.

@m-iancu m-iancu closed this as completed Jan 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tedyu tedyu

Labels
area/tools kind/new-feature This is a request for a completely new feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tedyu @m-iancu @rkarthik007