Skip to content

[SPARK-28183][CORE][UI] Add a task status filter for taskList in REST API#24984

Closed
LantaoJin wants to merge 3 commits intoapache:masterfrom
LantaoJin:SPARK-28183
Closed

[SPARK-28183][CORE][UI] Add a task status filter for taskList in REST API#24984
LantaoJin wants to merge 3 commits intoapache:masterfrom
LantaoJin:SPARK-28183

Conversation

@LantaoJin
Copy link
Contributor

@LantaoJin LantaoJin commented Jun 27, 2019

What changes were proposed in this pull request?

We have a scenario that our application needs to query failed tasks by REST API /applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskList when Spark job is running. In a large Stage, it may filter out dozens of failed tasks from hundred thousands total tasks. It consumes much unnecessary memory and time both in Spark and App side.

This work splits to two PRs, the previous is #24982 (even though I commit them together here) since filter should be handled before pagination. (Assume 100 items per page, filtering after pagination may get four pages but each of page only contains several items. Actually one page could contains all the filtered item).

How was this patch tested?

Add UT.
And manually test:
(In local mode, there is only one running task by default.)
Before:

http://localhost:4040/api/v1/applications/local-1562040123322/stages/3/0/taskList?offset=0&length=10

Return 1 running task and 9 success tasks or 10 success tasks.
After:

http://localhost:4040/api/v1/applications/local-1562040123322/stages/3/0/taskList?offset=0&length=10&sortBy=status&status=RUNNING

Only return 1 running task or empty if all tasks completed.

@LantaoJin
Copy link
Contributor Author

@vanzin @srowen I refactor this implementation follow the comments of #24982 (comment) and #24982 (comment)

@LantaoJin
Copy link
Contributor Author

Gently ping @vanzin @srowen

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@github-actions
Copy link

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just
a way of keeping the PR queue manageable.

If you'd like to revive this PR, please reopen it!

@github-actions github-actions bot added the Stale label Dec 28, 2019
@github-actions github-actions bot closed this Dec 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants