-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-13071] [hadoop-2.7] Coalescing HadoopRDD overwrites existing input metrics #10973
Conversation
Test build #50335 has finished for PR 10973 at commit
|
test this please |
maybe also put hadoop2.7 in the title, so jenkins can test it? |
Test build #50348 has finished for PR 10973 at commit
|
like this? I didn't know Jenkins can do that |
retest this please |
let's try again. retest this please |
LGTM |
Test build #50394 has finished for PR 10973 at commit
|
Test build #50395 has finished for PR 10973 at commit
|
retest this please |
Test build #50408 has finished for PR 10973 at commit
|
Test build #2478 has finished for PR 10973 at commit
|
Merging into master. |
This issue is causing tests to fail consistently in master with Hadoop 2.6 / 2.7. This is because for Hadoop 2.5+ we overwrite existing values of
InputMetrics#bytesRead
in each call toHadoopRDD#compute
. In the case of coalesce, e.g.we will call
compute
multiple times in the same task, overwritingbytesRead
values from previous calls tocompute
.For a regression test, see
InputOutputMetricsSuite.input metrics for old hadoop with coalesce
. I did not add a new regression test because it's impossible without significant refactoring; there's a lot of existing duplicate code in this corner of Spark.This was caused by #10835.