[SPARK-1199][REPL] Remove VALId and use the original import style for defined classes. #1179
Conversation
|
Merged build triggered. |
|
Merged build started. |
ScrapCodes
changed the title
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16022/ |
|
Merged build triggered. |
|
Merged build started. |
ScrapCodes
changed the title
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
ScrapCodes
changed the title
| @@ -182,15 +182,26 @@ trait SparkImports { | |||
| // ambiguity errors will not be generated. Also, quote | |||
| // the name of the variable, so that we don't need to | |||
| // handle quoting keywords separately. | |||
| case x: ClassHandler => | |||
| // I am trying to guess if the import is a defined class | |||
| // This is an ugly hack, I am not 100% sure of the consequences. | |||
There was a problem hiding this comment.
What do you think is ugly about this? Is this the original way the REPL used to include classes?
There was a problem hiding this comment.
This portion of code is just about importing, the ugly part is that there should not be special casing for different things ideally. This can lead to situations where we don't know what next needs a special case. Thus it is hard for us to pitch this to change to original scala REPL.
The actual ugly part is next case statement where we put Val id = something and then import it. This was the actual reason for this SPARK-1199 issue. But unfortunately we can not avoid it, because otherwise the remote executors try to pull in classes which they don't need to. But since we are living with that code for a while it should ideally be okay.
Yes importing without the intermediate val creation was the originally followed style, which we changed to solve the problem described above.
Suppose we only used code.append("import " + objName + ".INSTANCE" + req.accessPath + "." + imv + "\n") this to import and no other way, then it is much better than what we are doing and there will be no need for migration of REPL for next upgrade.
…a class or not. and also support " // show"
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
… defined classes. This is an alternate solution to #1176. Author: Prashant Sharma <prashant.s@imaginea.com> Closes #1179 from ScrapCodes/SPARK-1199/repl-fix-second-approach and squashes the following commits: 820b34b [Prashant Sharma] Here we generate two kinds of import wrappers based on whether it is a class or not. (cherry picked from commit d434150) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
… defined classes. This is an alternate solution to apache#1176. Author: Prashant Sharma <prashant.s@imaginea.com> Closes apache#1179 from ScrapCodes/SPARK-1199/repl-fix-second-approach and squashes the following commits: 820b34b [Prashant Sharma] Here we generate two kinds of import wrappers based on whether it is a class or not.
This is an alternate solution to #1176.