Skip to content

[SPARK-20337][CORE] Support upgrade a jar dependency and don't restart SparkContext#17637

Closed
wangyum wants to merge 1 commit intoapache:masterfrom
wangyum:SPARK-20337
Closed

[SPARK-20337][CORE] Support upgrade a jar dependency and don't restart SparkContext#17637
wangyum wants to merge 1 commit intoapache:masterfrom
wangyum:SPARK-20337

Conversation

@wangyum
Copy link
Copy Markdown
Member

@wangyum wangyum commented Apr 14, 2017

What changes were proposed in this pull request?

Support upgrade a jar dependency and don't restart SparkContext, Something like this:

sc.addJar("breeze-natives_2.11-0.12.jar")
// do something
sc.removeJar("spark://192.168.26.200:23420/jar/breeze-natives_2.11-0.12.jar")
sc.addJar("breeze-natives_2.11-0.13.jar")
// do something

How was this patch tested?

unit tests and manual tests

@SparkQA
Copy link
Copy Markdown

SparkQA commented Apr 14, 2017

Test build #75806 has finished for PR 17637 at commit eb4cb86.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@vanzin
Copy link
Copy Markdown
Contributor

vanzin commented Apr 14, 2017

This does not work. Any classes that have already been loaded from the old jar will not be unloaded. So you're going to end up with really odd issues when two classes from different jars don't agree with each other.

@wangyum wangyum closed this Apr 15, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants