-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-4048] Enhance and extend hadoop-provided profile. #2982
Changes from 24 commits
284dda6
1adf91c
2f95f0d
417d90e
4d67469
d928d62
9e4e001
f7b3bbe
1fc4d0b
5c54a25
82a54b9
d1399ed
1be73d4
7820d58
e3ab2da
115fde5
9640503
8b00b6a
f24e9e7
322f882
7377e7b
83099fc
52f366d
371ebee
9ef79a3
4e38f4e
eb228c0
82eb688
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -990,11 +990,19 @@ private[spark] object Utils extends Logging { | |
for ((key, value) <- extraEnvironment) { | ||
environment.put(key, value) | ||
} | ||
|
||
// When testing, expose the parent class path to the child. This is processed by | ||
// compute-classpath.{cmd,sh} and makes all needed jars available to child processes | ||
// when the assembly is built with the "*-provided" profiles enabled. | ||
if (sys.props.contains("spark.testing")) { | ||
environment.put("SPARK_DIST_CLASSPATH", sys.props("java.class.path")) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this needed in all cases or only when you are actually running tests with a "hadoop provided" build and you've supplied you own Then you would just check if There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I used to run into problems with tests spawning child processes that referenced hadoop classes, and those not being available when the "hadoop-provided" profile was enabled. Let me try without this and see how it goes, since I noticed some changes in the configuration of the test plugins in the root pom. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Even if it is still a problem, my question was, can we just set There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. But |
||
} | ||
|
||
val process = builder.start() | ||
new Thread("read stderr for " + command(0)) { | ||
override def run() { | ||
for (line <- Source.fromInputStream(process.getErrorStream).getLines()) { | ||
System.err.println(line) | ||
logInfo(line) | ||
} | ||
} | ||
}.start() | ||
|
@@ -1089,7 +1097,7 @@ private[spark] object Utils extends Logging { | |
var firstUserLine = 0 | ||
var insideSpark = true | ||
var callStack = new ArrayBuffer[String]() :+ "<unknown>" | ||
|
||
Thread.currentThread.getStackTrace().foreach { ste: StackTraceElement => | ||
// When running under some profilers, the current stack trace might contain some bogus | ||
// frames. This is intended to ensure that we don't crash in these situations by | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can distributions instead just set
spark.executor.extraClassPath
andspark.driver.extraClassPath
? We've intentionally removed/deprecated this type of brute force env variable, and also, it would be good to have only one way of doing this.