Support multi-coordinator while serving task info #18146
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
asjadsyed
force-pushed
the
asjadsyed_disagg_pti
branch
2 times, most recently
from
a9616e7
to
ff9118a
Compare
asjadsyed
commented
Aug 5, 2022
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
asjadsyed
commented
Aug 5, 2022
presto-main/src/main/java/com/facebook/presto/resourcemanager/DistributedQueryInfoResource.java
Outdated
Show resolved
Hide resolved
asjadsyed
commented
Aug 5, 2022
presto-main/src/main/java/com/facebook/presto/server/QueryStateInfoResource.java
Outdated
Show resolved
Hide resolved
asjadsyed
force-pushed
the
asjadsyed_disagg_pti
branch
from
ff9118a
to
186ed47
Compare
asjadsyed
commented
Aug 5, 2022
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
swapsmagic
requested changes
Aug 18, 2022
presto-main/src/main/java/com/facebook/presto/resourcemanager/DistributedQueryInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/resourcemanager/DistributedQueryResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/resourcemanager/DistributedTaskInfoResource.java
Outdated
Show resolved
Hide resolved
|
LGTM % @swapsmagic feedback |
ajaygeorge
reviewed
Aug 19, 2022
presto-main/src/main/java/com/facebook/presto/resourcemanager/DistributedQueryInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/resourcemanager/DistributedTaskInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/QueryStateInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
...tests/src/test/java/com/facebook/presto/resourcemanager/TestDistributedTaskInfoResource.java
Outdated
Show resolved
Hide resolved
...tests/src/test/java/com/facebook/presto/resourcemanager/TestDistributedTaskInfoResource.java
Show resolved
Hide resolved
swapsmagic
reviewed
Oct 20, 2022
...tests/src/test/java/com/facebook/presto/resourcemanager/TestDistributedTaskInfoResource.java
Show resolved
Hide resolved
...tests/src/test/java/com/facebook/presto/resourcemanager/TestDistributedTaskInfoResource.java
Outdated
Show resolved
Hide resolved
asjadsyed
requested review from
a team,
vinothchandar and
7c00
as code owners
|
|
asjadsyed
force-pushed
the
asjadsyed_disagg_pti
branch
7 times, most recently
from
72c7b3c
to
d43d33c
Compare
ajaygeorge
reviewed
Oct 26, 2022
presto-main/src/main/java/com/facebook/presto/resourcemanager/DistributedTaskInfoResource.java
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Outdated
Show resolved
Hide resolved
presto-tests/src/test/java/com/facebook/presto/utils/QueryExecutionClientUtil.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/resourcemanager/DistributedTaskInfoResource.java
Outdated
Show resolved
Hide resolved
asjadsyed
force-pushed
the
asjadsyed_disagg_pti
branch
5 times, most recently
from
7eababb
to
07fe3e1
Compare
ajaygeorge
reviewed
Oct 31, 2022
presto-main/src/main/java/com/facebook/presto/server/TaskInfoResource.java
Show resolved
Hide resolved
ajaygeorge
approved these changes
Oct 31, 2022
asjadsyed
force-pushed
the
asjadsyed_disagg_pti
branch
8 times, most recently
from
835a949
to
857b8ea
Compare
tdcmeehan
approved these changes
Nov 2, 2022
asjadsyed
force-pushed
the
asjadsyed_disagg_pti
branch
from
857b8ea
to
63fb402
Compare
asjadsyed
force-pushed
the
asjadsyed_disagg_pti
branch
from
63fb402
to
d2d848b
Compare
swapsmagic
approved these changes
Nov 2, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
The
/v1/taskInfo/{taskId}endpoint serves task info available within a particular coordinator.With multi-coordinator, if a different coordinator is running the query, the endpoint currently returns 404. This leads to a broken UI under multi-coordinator, so fixing this issue will help with debugging. This PR fixing this by enabling the endpoint to retrieve cluster-wide task info.
We do this by implementing a proxying endpoint
/v1/taskInfo/{taskId}in the resource manager: if the query isn't available on the queried coordinator, the coordinator will proxy the request to the RM, which will proxy it to the right coordinator.We also prevent an infinite-loop situation where the RM and coordinator could end up handing the request back and forth in case the coordinator no longer has info about the query, which might happen if the query was recently finished. The RM avoids the situation by adding a query parameter to let the coordinator know not to proxy it back again.
This PR is very similar to #16163, which fixed this issue for
/v1/queryState/*. DistributedTaskInfoResource is to TaskInfoResource as DistributedQueryInfoResource is to QueryStateInfoResource.Test plan:
I've added a unit test (based off of TestDistributedQueryInfoResource) which starts a query on one coordinator and verifies we can retrieve the taskInfo on the second coordinator.
Release Notes: