Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Multiple fixes to make continuous mode work #432

Merged
merged 14 commits into from
Dec 21, 2020

Conversation

jiangmichaellll
Copy link
Contributor

This brings the continuous mode working with non-uber jar solution. Uber jar still won't work.

Mainly:

  1. Lower scala version to 2.11 since it's what's provided in spark 2.4.7 in classpath
  2. Fixes the conversion from java types into spark types
  3. Added maven shade plugin for uber jar generation

@jiangmichaellll jiangmichaellll requested review from a team as code owners December 17, 2020 01:11
@product-auto-label product-auto-label bot added the api: pubsublite Issues related to the googleapis/java-pubsublite API. label Dec 17, 2020
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Dec 17, 2020
@codecov
Copy link

codecov bot commented Dec 17, 2020

Codecov Report

Merging #432 (ffd15b3) into master (34d8d02) will increase coverage by 0.07%.
The diff coverage is 83.87%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #432      +/-   ##
============================================
+ Coverage     71.15%   71.22%   +0.07%     
- Complexity      902      905       +3     
============================================
  Files           167      167              
  Lines          4770     4789      +19     
  Branches        240      242       +2     
============================================
+ Hits           3394     3411      +17     
- Misses         1238     1240       +2     
  Partials        138      138              
Impacted Files Coverage Δ Complexity Δ
...cloud/pubsublite/spark/PslCredentialsProvider.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...m/google/cloud/pubsublite/spark/PslDataSource.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...m/google/cloud/pubsublite/spark/PslSparkUtils.java 93.33% <100.00%> (+2.63%) 13.00 <5.00> (+4.00)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34d8d02...7813ffb. Read the comment docs.

@jiangmichaellll jiangmichaellll merged commit 9df4ccf into master Dec 21, 2020
@jiangmichaellll jiangmichaellll deleted the jiangmichael-spark-continuous-mode-tuning branch December 21, 2020 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: pubsublite Issues related to the googleapis/java-pubsublite API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants