Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]Hope ptubes can support storage layer partition #7

Open
myksl opened this issue May 20, 2022 · 0 comments
Open

[Feature]Hope ptubes can support storage layer partition #7

myksl opened this issue May 20, 2022 · 0 comments
Labels
feature new features

Comments

@myksl
Copy link
Collaborator

myksl commented May 20, 2022

Describe the feature

At present, the concept of sharding is a logical concept, which only exists in the Client for parallel consumption. As a result, each Reader needs to pull the full binlog, and there is a visible performance upper limit for pulling and storing (cannot scale linearly)

  1. From logical sharding to physical sharding, the concept of partition is introduced, each Reader does not need to pull all binlogs, reducing storage and processing pressure, and at the same time supporting parallel processing of data by Reader (each Reader processes a part of the partition), improving ptubes Scalability
  2. The data of each partition can be subscribed separately to reduce the data backtracking when the service is duplicated
  3. Each partition has a separate writing thread, effectively breaking through the performance bottleneck of single-threaded sequential writing
@myksl myksl added the feature new features label May 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature new features
Projects
None yet
Development

No branches or pull requests

1 participant