Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve assignement of kafka source pipelines on several indexers #4630

Open
fmassot opened this issue Feb 26, 2024 · 2 comments
Open

Improve assignement of kafka source pipelines on several indexers #4630

fmassot opened this issue Feb 26, 2024 · 2 comments
Assignees
Labels
bug Something isn't working enhancement New feature or request

Comments

@fmassot
Copy link
Contributor

fmassot commented Feb 26, 2024

A user reported the following issue (I checked the cluster state):

  • He has around 50 Kafka sources. The biggest Kafka source has 4 desired pipelines, most of which have one.
  • He has 10 indexers, each with 4-6 pipelines. 1 indexer has all the pipelines of the biggest Kafka source.

This is bad because, as you can imagine, only 1 indexer is responsible for indexing the biggest Kafka source. And all other indexers are doing less work.

@fmassot fmassot added bug Something isn't working enhancement New feature or request labels Feb 26, 2024
@fmassot fmassot changed the title Spread kafka source pipelines on several indexers Improve assignement of kafka source pipelines on several indexers Feb 26, 2024
@fulmicoton
Copy link
Contributor

fulmicoton commented Feb 26, 2024

so the merge are messing things up?
Should we consider this a duplicate of #4470

@fulmicoton fulmicoton self-assigned this Feb 26, 2024
@fulmicoton
Copy link
Contributor

Replied offline by @fmassot : no the problem is an unbalanced load on the different nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants