Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: BlockingPullSubscriber in Spark Continuous Processing #408

Merged
merged 35 commits into from
Dec 14, 2020

Conversation

jiangmichaellll
Copy link
Contributor

Adds a BlockingPullSubscriber that's mostly similar as BufferingPullSubscriber other than 1. it only pull once and "ack" one each time 2. blocking pull. This addresses the issue where BufferingPullSubscriber will be unbounded. Also replaced the some logic inside InputPartitionReader so it's more simple.

@jiangmichaellll jiangmichaellll requested a review from a team as a code owner December 8, 2020 03:12
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Dec 8, 2020
@product-auto-label product-auto-label bot added the api: pubsublite Issues related to the googleapis/java-pubsublite API. label Dec 8, 2020
@codecov
Copy link

codecov bot commented Dec 8, 2020

Codecov Report

Merging #408 (596ae1a) into master (9346507) will increase coverage by 0.18%.
The diff coverage is 83.05%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #408      +/-   ##
============================================
+ Coverage     71.41%   71.60%   +0.18%     
- Complexity      866      877      +11     
============================================
  Files           163      164       +1     
  Lines          4566     4613      +47     
  Branches        227      233       +6     
============================================
+ Hits           3261     3303      +42     
- Misses         1178     1179       +1     
- Partials        127      131       +4     
Impacted Files Coverage Δ Complexity Δ
...le/cloud/pubsublite/internal/CloseableMonitor.java 100.00% <ø> (ø) 3.00 <0.00> (ø)
.../pubsublite/spark/PslContinuousInputPartition.java 0.00% <ø> (ø) 0.00 <0.00> (ø)
...blite/spark/PslContinuousInputPartitionReader.java 64.28% <57.14%> (-1.34%) 5.00 <2.00> (-1.00)
...ubsublite/internal/BlockingPullSubscriberImpl.java 86.53% <86.53%> (ø) 11.00 <11.00> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9346507...596ae1a. Read the comment docs.

@jiangmichaellll jiangmichaellll changed the base branch from master to jiangmichael-status-fix December 11, 2020 21:32
Base automatically changed from jiangmichael-status-fix to master December 11, 2020 21:34
@dpcollins-google dpcollins-google changed the title BlockingPullSubscriber in Spark Continuous Processing feat: BlockingPullSubscriber in Spark Continuous Processing Dec 14, 2020
@jiangmichaellll jiangmichaellll merged commit 36684c4 into master Dec 14, 2020
@jiangmichaellll jiangmichaellll deleted the jiangmichael-spark-flow-control branch December 14, 2020 23:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: pubsublite Issues related to the googleapis/java-pubsublite API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants