Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add instructions to configure IPython/PySpark for python 3, now supported with Spark 1.4 #2

Closed
donnemartin opened this issue Jun 24, 2015 · 1 comment
Assignees

Comments

@donnemartin
Copy link
Owner

Reported by core_dumpd on Reddit /r/DataScience.

Solution seems to be discussed in Stack Overflow here.

core_dumpd reports the following works, need to confirm and update repo:

I end up running this:
PYSPARK_DRIVER_PYTHON_OPTS="notebook --profile=pyspark" /usr/local/spark/bin/pyspark

With:
PYSPARK_PYTHON=/opt/anaconda/bin/ipython PYSPARK_DRIVER_PYTHON=/opt/anaconda/bin/ipython

I'm running on docker based on sequenceiq/hadoop-docker:latest with Spark/MiniConda added on top. The only real config options in the profile are for the ip = '*' and open_browser = False.

@donnemartin
Copy link
Owner Author

Commit 1405875 adds links to Stack Overflow and Reddit which provides instructions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant