-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add lane enforcement for joinish queries #9563
Conversation
Adding this PR for 0.18 since the laning feature is first introduced in it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code changes LGTM, left a couple of nit comments. I'm wondering whether we need some a bit detailed doc about how laning and prioritization work.
@@ -462,7 +462,8 @@ public QuerySwappingQueryRunner(QueryRunner<T> baseRunner, Query<T> query, Query | |||
throw new ISE("Unexpected query received"); | |||
} | |||
|
|||
return baseRunner.run(queryPlus.withQuery(newQuery), responseContext); | |||
Sequence<T> result = baseRunner.run(queryPlus.withQuery(newQuery), responseContext); | |||
return result; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unnecessary change?
{ | ||
Set<SegmentServerSelector> segmentServerSelectors = new HashSet<>(); | ||
for (Segment s : segments) { | ||
segmentServerSelectors.add(new SegmentServerSelector(null, s.getId().toDescriptor())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would you add @Nullable
to the constructor parameter of SegmentServerSelector
and getServer()
?
This sounds nice, I will try to add an entry to the cluster tuning docs about this stuff before release (i guess that would be the most appropriate place) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, LGTM
* add lane enforcement for joinish queries * oops * style * review stuffs
Description
This PR partially squares up the joins work of #9503, #9533, and #9545 with the query laning added in #9407, by extending lane enforcement to queries that run with the
LocalQuerySegmentWalker
ofClientQuerySegmentWalker
, fixing a bug that queries that only use theLocalQuerySegmentWalker
are not counted at all. This PR puts things into an interim state so that these queries can fall under lane enforcement, but it has a flaw that it will sequentially acquire and release lanes for each time theClientQuerySegmentWalker
invokes either its local or cluster walker to run a part of a query rather than acquiring all necessary lanes up front and holding through the duration of the query. This is not really an ideal state, since it opens the possibility to fail midway through if a query loses a lane to another query, but it is better than not counting them, and so worth doing for the short term I think.Ideally, we could bring up the call to
QueryScheduler.run
intoClientQuerySegmentWalker
or even the http resources, but this isn't currently possible without a fairly heavy refactor of how the broker runs queries. This is due to how they are transformed on the way down, whereQueryScheduler.prioritizeAndLaneQuery
happens on in the deepest part of the cluster and local walkers runners right before they callQueryScheduler.run
because that is when the segments participating in the query are available. I'm not quite sure what this will look like yet.This PR has:
Key changed/added classes in this PR
LocalQuerySegmentWalker