Skip to content

[SPARK-22467]Added a config to support whether stdout_stream and stderr_stream output to disk#19690

Closed
10110346 wants to merge 1 commit intoapache:masterfrom
10110346:stdout_err
Closed

[SPARK-22467]Added a config to support whether stdout_stream and stderr_stream output to disk#19690
10110346 wants to merge 1 commit intoapache:masterfrom
10110346:stdout_err

Conversation

@10110346
Copy link
Contributor

@10110346 10110346 commented Nov 8, 2017

What changes were proposed in this pull request?

We should add a switch to control the stdout_stream and stdout_stream output to disk.
In my environment,due to disk I/O blocking, the stdout_stream output is very slow, so it can not be timely cleaning,and this leads the executor process to be blocked.

How was this patch tested?

Added a unit test

@SparkQA
Copy link

SparkQA commented Nov 8, 2017

Test build #83588 has finished for PR 19690 at commit 7b67148.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 8, 2017

Test build #83590 has finished for PR 19690 at commit 7b67148.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@10110346
Copy link
Contributor Author

10110346 commented Nov 9, 2017

retest this please

@SparkQA
Copy link

SparkQA commented Nov 9, 2017

Test build #83621 has finished for PR 19690 at commit 7b67148.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 11, 2017

Test build #83716 has finished for PR 19690 at commit 37fba60.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@10110346
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Nov 11, 2017

Test build #83720 has finished for PR 19690 at commit 37fba60.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987
Copy link
Contributor

Under most conditions the users shouldn't ignore the printed error/warning messages, have you observed much redundant buzz texts?

@10110346
Copy link
Contributor Author

10110346 commented Feb 6, 2018

yea, so we make the default value for this configuration is true.
I set spark.executor.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintHeapAtGC, and a lot of GC prints are produced

@10110346 10110346 changed the title [SPARK-22467]Added a switch to support whether stdout_stream and stderr_stream output to disk [SPARK-22467]Added a config to support whether stdout_stream and stderr_stream output to disk Feb 6, 2018
@github-actions
Copy link

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

@github-actions github-actions bot added the Stale label Jan 14, 2020
@github-actions github-actions bot closed this Jan 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants