Default log4j.properties incorrectly sends all output to stderr and none to stdout

[ash211]

https://issues.apache.org/jira/browse/SPARK-1899

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[rxin]

I actually thought this was intentional and it's been the case since Spark's inception. @mateiz can you comment on this?

[mridulm]

Even I assumed it was intentional since output from user code goes
typically to stdout.

Regards
Mridul
On 22-May-2014 11:20 am, "Reynold Xin" notifications@github.com wrote:

I actually thought this was intentional and it's been the case since
Spark's inception. @mateiz https://github.com/mateiz can you comment on
this?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/852#issuecomment-43850841
.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15137/

[mateiz]

This behavior has been in there for a while, so I'm curious, is there a strong reason to change it? It would be a change in behavior that some users might not expect. Users can always configure their own log4j.properties if they don't want this one.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[ash211]

The only reason to change it is that it doesn't follow people's
expectations, or at least my expectations. I always look first into stdout
for progress/logging of the job, and then remember that everything is
written into stderr regardless of its log level. It's pretty unusual to
see DEBUG and INFO level logs going into a file named stderr.

If you think the continuity break of changing the logging will jar existing
Spark users then we can keep it as is, but I think it's a worthwhile change
and more closely matches my expectations of how log4j log levels should map
to stdout and stderr.

On Fri, May 23, 2014 at 5:08 PM, Matei Zaharia notifications@github.comwrote:

This behavior has been in there for a while, so I'm curious, is there a
strong reason to change it? It would be a change in behavior that some
users might not expect. Users can always configure their own
log4j.properties if they don't want this one.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/852#issuecomment-44060793
.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15175/

[zsxwing]

The only reason to change it is that it doesn't follow people's
expectations, or at least my expectations.

But it follows expectations of people who come from Hadoop. Hadoop has already used such log strategy for a long time: https://github.com/apache/hadoop-common/blob/d92a8a29978e35ed36c4d4721a21c356c1ff1d4d/hadoop-common-project/hadoop-minikdc/src/main/resources/log4j.properties

[pwendell]

It's likely we already have many users scripting around this default behavior of sending spark logs to stderr. If this was something where we clearly got it wrong, we could consider changing it, but here it seems like more of a preference than a bug.

So I think we probably shouldn't change this at this point, even though I see the reason why sending all logs to stdout could be preferable.

[ash211]

Sounds good, I'll close. Would changing the 2 digit years to 4 digit cause heartache?