Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-26588][CORE] Idle executor should properly be removed when no job is su… #23509

Closed
wants to merge 1 commit into from

Conversation

TopGunViper
Copy link

What changes were proposed in this pull request?

I enable dynamic allocation feature with spark-shell and do not submit any task. After spark.dynamicAllocation.executorIdleTimeout seconds(default 60s), there is still one active executor, which is abnormal. All idle executors are timeout and should be removed.(default spark.dynamicAllocation.minExecutors=0). The spark-shell command show below:

spark-shell --master=yarn --conf spark.ui.port=8040 --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.maxExecutors=8 --conf spark.dynamicAllocation.initialExecutors=4 --conf spark.shuffle.service.enabled=true

How was this patch tested?

unit tests

@srowen
Copy link
Member

srowen commented Jan 10, 2019

CC @jerryshao

@SparkQA
Copy link

SparkQA commented Jan 10, 2019

Test build #4502 has finished for PR 23509 at commit bd300d6.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jerryshao
Copy link
Contributor

Can you please describe how to reproduce your problem?

@TopGunViper
Copy link
Author

Can you please describe how to reproduce your problem?

Execute the following command and waiting for 60 seconds. The problem can be observed from Spark Web UI under the Executors Tab.

spark-shell --master=yarn --conf spark.ui.port=8040 --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.maxExecutors=8 --conf spark.dynamicAllocation.initialExecutors=4 --conf spark.shuffle.service.enabled=true

@jerryshao
Copy link
Contributor

  1. please fix the PR title to follow Spark's way. 2) Can you please figure out which patch introduce this regression? I did remember that this should be OK in my previous test.

@TopGunViper TopGunViper changed the title SPARK-26588:Idle executor should properly be killed when no job is su… [SPARK-26588][CORE] Idle executor should properly be killed when no job is su… Jan 12, 2019
@TopGunViper
Copy link
Author

TopGunViper commented Jan 12, 2019

  1. please fix the PR title to follow Spark's way. 2) Can you please figure out which patch introduce this regression? I did remember that this should be OK in my previous test.

Hi @jerryshao I guess this problem was introduced by the patch SPARK-21656.

@TopGunViper
Copy link
Author

CC @tgravescs

@TopGunViper TopGunViper changed the title [SPARK-26588][CORE] Idle executor should properly be killed when no job is su… [SPARK-26588][CORE] Idle executor should properly be removed when no job is su… Jan 12, 2019
@jerryshao
Copy link
Contributor

Jenkins, retest this please.

@jerryshao
Copy link
Contributor

I can reproduce this issue, will spend time on it. Thanks.

@SparkQA
Copy link

SparkQA commented Jan 15, 2019

Test build #101223 has finished for PR 23509 at commit bd300d6.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jerryshao
Copy link
Contributor

Jenkins, retest this please.

@jerryshao
Copy link
Contributor

The change itself looks fine to me. @tgravescs @vanzin would you please also take a look.

@SparkQA
Copy link

SparkQA commented Jan 16, 2019

Test build #101284 has finished for PR 23509 at commit bd300d6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@vanzin
Copy link
Contributor

vanzin commented Feb 12, 2019

Already fixed by SPARK-26758.

@vanzin vanzin closed this Feb 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants