Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelize startup sequence topics initialization #3

Closed
1 task done
markpapadakis opened this issue Jun 30, 2016 · 2 comments
Closed
1 task done

Parallelize startup sequence topics initialization #3

markpapadakis opened this issue Jun 30, 2016 · 2 comments
Assignees
@markpapadakis
Labels
Client enhancement

Comments

@markpapadakis
Copy link
Member

markpapadakis commented Jun 30, 2016 •
edited

  • Even though all we really need during startup is walk topics and partitions directories and open log files and indices - which is pretty fast, for setups of 100s of topics and 1000s of partitions, it should be beneficial to parallelize initialization of topics/partitions via std::async() jobs. It should be pretty trivial.
@markpapadakis markpapadakis self-assigned this Jun 30, 2016
@markpapadakis
Copy link
Member Author

See also #14 for a discussion on FDs caching.

markpapadakis added a commit that referenced this issue Jul 12, 2016
@markpapadakis
Will now initialize all topics in parallel during startup. As a resul…
62106d6 
…t of that, for 3.5k partitions, Tank now needs about 0.250s, down from 2.25s. See #3
@markpapadakis
Copy link
Member Author

Now parallelizing startup-sequence. See 62106d6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@markpapadakis markpapadakis

Labels
Client enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@markpapadakis