Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pubsublite): routing publisher implementation #3277

Merged
merged 8 commits into from Dec 2, 2020

Conversation

@tmdiep
Copy link
Contributor

@tmdiep tmdiep commented Nov 25, 2020

partitionCountWatcher polls for updates to the number of topic partitions.

routingPublisher routes messages to multiple singlePartitionPublishers and handles the topic partition count increasing. Decreases will terminate the routingPublisher.

The initial set of partition publishers must successfully start before the routingPublisher accepts published messages. However, when new partition publishers are added later, messages may be routed to them while they are starting up. The messages are queued in the publishMessageBatcher and sent to the server when the Publish stream connects.

The entire routingPublisher tree terminates if the partitionCountWatcher or any of the singlePartitionPublishers terminate.

Handles increasing topic partition count.
@tmdiep
Copy link
Contributor Author

@tmdiep tmdiep commented Nov 25, 2020

WANT_LGTM=@manuelmenzella-google, @palmere-google
Other reviewers optional, but comments welcome.

If anyone feels strongly about waiting for the new partition publishers to successfully start before routing messages to them, feel free to let me know. It can be done, but introduces some complexity, so I went with this. If any partition publisher fails to start up, the entire routing publisher will be terminated anyway.

@tmdiep tmdiep requested a review from dpcollins-google Nov 25, 2020
@tmdiep tmdiep force-pushed the routing_publisher branch from 8a2e7b9 to c2dea0a Nov 28, 2020
@tmdiep tmdiep force-pushed the routing_publisher branch from b2ae47b to a2fd97e Dec 2, 2020
@tmdiep tmdiep merged commit 88e5466 into googleapis:master Dec 2, 2020
3 checks passed
@tmdiep tmdiep deleted the routing_publisher branch Dec 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants