Added ability to return pandas dataframes from SparkSQL for Python and Scala #20
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 9 commits
September 29, 2015 19:08
Pandas df is now returned for sql pyspark
It did not display before because magic was not returning it.
1. Create pandas df for scala and pyspark with one call only 2. Create sqlContext from session start 3. If there are no rows, no schema is shown :( We could detect that and default to previous two call method. Need to fix unit tests. Going to sleep row now
Need to add unit tests for new functionality
|
|
||
| .idea/sparkmagic.iml | ||
|
|
||
| .idea/workspace.xml |
There was a problem hiding this comment.
Minor nitpick: these can all be combined to .idea/*
Execution always returns result. Combined .gitignore entries.
|
With this last commit, I think we are good for now, unless you have more comments on the previous points. Please take a look @alope107. Thank you! |
|
I have the mentioned minor nitpicks, but I don't think they need to be included in this PR. Looks good to me. |
aggFTW
added a commit
that referenced
this pull request
Oct 2, 2015
Added ability to return pandas dataframes from SparkSQL for Python and Scala
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With unit tests. Also, made unit tests run faster.