New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Overlord incremental select worker strategy #14934
Overlord incremental select worker strategy #14934
Conversation
// Assumes the host format is based on k8s statefulset FQDN - "druid-middlemanager-<numeral>.druid-middlemanager..." | ||
String[] parts = host.split("\\."); | ||
String numeralPart = parts[0].substring(parts[0].lastIndexOf("-") + 1); | ||
return Integer.parseInt(numeralPart); |
Check notice
Code scanning / CodeQL
Missing catch of NumberFormatException Note
thank you for your contribution. can you elaborate a bit on following
what is not consistent? |
For statefulset to scale down we need to take into consideration that:
But with existing |
2dd0e11
to
f079a0b
Compare
This pull request has been marked as stale due to 60 days of inactivity. |
This pull request/issue has been closed due to lack of activity. If you think that |
Description
I was trying to use existing
fillCapacity
select worker strategy in an effort to implement MiddleManager autoscaling, but since MM instances that I use are created by k8s statefulsets it wasn't consistent enough. Proposed strategy fills capacity based on worker's hostname, with assumption that host is k8s FQDN - eg.druid-middlemanager-0.druid-middlemanager.druid-test.svc.cluster.local:8091
.Similar problem was discussed in #8801.
I'm not sure if this feature is desirable and this is my first java admission, so I'll take every bit of critique here. I can add tests if it will appear that this feature might be useful and we want to add it to codebase.
Key changed/added classes in this PR
FillIncrementallyWorkerSelectStrategy
FillIncrementallyWithAffinityWorkerSelectStrategy
FillIncrementallyWithCategorySpecWorkerSelectStrategy
This PR has: