[SPARK-5929] Pyspark: Register a pip requirements file with spark_context #4897

buckhx · 2015-03-05T00:58:51Z

Ships all packages in the requirements file by installing them locally via pip and then ships the packages to the workers via addPyFile

davies · 2015-03-07T00:05:35Z

python/pyspark/context.py

@@ -65,8 +65,9 @@ class SparkContext(object):
    _python_includes = None  # zip and egg files that need to be added to PYTHONPATH

    def __init__(self, master=None, appName=None, sparkHome=None, pyFiles=None,
-                 environment=None, batchSize=0, serializer=PickleSerializer(), conf=None,
-                 gateway=None, jsc=None, profiler_cls=BasicProfiler):
+                 requirementsFile=None, environment=None, batchSize=0,


Insert a parameter in the middle will break compatibility, we should put it in the end.

davies · 2015-03-07T00:12:36Z

Good idea, this is a useful feature to have. Could you also a argument for spark-submit, (for example, --pip or --py-requirements). Also, could you add a test for it?

davies · 2015-03-07T00:12:51Z

Jenkins, Ok to test.

buckhx · 2015-03-10T14:32:08Z

Great, will get to that this week and add tests.

Do you think bundling importlib with pyspark is reasonable? Or is finding another way to track down the local package the way to go?

davies · 2015-03-10T18:44:43Z

I think we can go with __import__, it's similar to importlib

buckhx · 2015-04-03T20:43:49Z

Changed importlib to import and moved most of the requirements up. I left the pip requirement in the method itself as to not require the system has pip unless that method is used. Also added a test that tests the import of a library not found on most systems https://github.com/buckheroux/QuadKey.

buckhx · 2015-04-20T14:53:47Z

Any traction on this?

robcowie · 2015-04-20T17:06:37Z

This could be very useful. One question; Does this behave with namespace packages? At first glance, it seems not. req.name will not necessarily be importable. I can't see a simple way around this but I'll have a think.

Regardless, this is a nice idea and I'd like to see it merged.

buckhx · 2015-05-12T18:01:09Z

@robcowie I wouldn't think so. Do you have a good example of a pypi package that I could try to install that has a namespace? I can try and support it if possible.

andrewor14 · 2015-07-13T22:00:46Z

Jenkins slow test please

SparkQA · 2015-07-13T22:01:49Z

Test build #5 has finished for PR 4897 at commit 76ff637.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

buckhx · 2015-07-13T22:05:31Z

Just saw the test failure, looking into it now

davies · 2015-07-13T22:11:37Z

python/pyspark/context.py

-                 gateway=None, jsc=None, profiler_cls=BasicProfiler):
+                 environment=None, batchSize=0, serializer=PickleSerializer(),
+                 conf=None, gateway=None, jsc=None, profiler_cls=BasicProfiler,
+                 requirementsFile=None):


We already have two much parameters here, Can we just have addRequirementsFile?

I was trying to match the pyFiles interface as closely as possible, although I do see where you are coming from. I'll re-evaluate having the requirementsFile kwarg after add a requirements file arg to the CLI.

andrewor14 · 2015-07-13T22:13:41Z

@buckheroux I believe the test failure is not related. @JoshRosen is looking into a hot fix.

davies · 2015-07-13T22:15:50Z

@buckheroux Do you think that we should also support this in bin/spark-submit? For example, we should accept .txt file (via --py-files) as dependencies.

buckhx · 2015-07-13T22:31:28Z

python/pyspark/context.py

+                if not req.check_if_exists():
+                    pip.main(['install', req.req.__str__()])
+                mod = __import__(req.name)
+                mod_path = mod.__path__[0]


Worried about this working across different types of modules (namespace packages etc)

Indeed mod.path[0] does not work for namespace packages. Working on supporting them by bundling all paths in mod.path into a single tar.

buckhx · 2015-07-13T22:32:45Z

@andrewor14 Sounds good.

@davies It's probably a good idea to support a CLI arg for the requirements file. I'll work on this as well.

JoshRosen · 2015-07-13T22:49:35Z

@andrewor14 the test failure in the dev/run-tests doctest are because we did a shallow clone of the Spark repository into the fresh Jeknins workspace. I think that this is an ephemeral problem that will resolve itself after retesting. We could also disable / remove that test for now or skip it in Jenkins.

JoshRosen · 2015-07-13T22:50:17Z

I've disabled shallow cloning in SlowSparkPullRequestBuilder, so let's try this again...

JoshRosen · 2015-07-13T22:50:21Z

jenkins slow test please

SparkQA · 2015-07-13T22:51:42Z

Test build #6 has finished for PR 4897 at commit 76ff637.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-15T18:52:09Z

Test build #1076 has finished for PR 4897 at commit 76ff637.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

buckhx · 2015-09-25T15:23:48Z

The tests fail because one of them attempts to pip install a package and doesn't have permissions to do so. Is there a way to enable that? Or just leave the pip installing piece untested?

buckhx · 2015-10-02T19:50:42Z

Looking to add support for namespaces, which I have tested locally and confirmed to work. I was also thinking of exposing an add_package function that would add local packages to the spark context.

buckhx · 2015-10-21T20:49:20Z

@davies removed the requirements file from the context constructor. I looked into add --py-requirements to spark-submit, but it looked a bit more in depth than I was thinking it would be. I think it makes sense to add that as a separate commit/PR.

Also, when testing I import a package from pip, but the Spark Jenkins test errors because it doesn't allow for access to the global pypi server. Any thoughts?

davies · 2015-10-21T23:48:54Z

cc @JoshRosen who is more familiar with Jenkins

DoHe · 2015-12-02T15:37:00Z

For the test you could also add a local egg file to the repository and install this instead of relying on the external pypi server.

rxin · 2015-12-31T02:42:56Z

I'm going to close this pull request. If this is still relevant and you are interested in pushing it forward, please open a new pull request. Thanks!

buckhx · 2016-02-19T15:26:59Z

This is a pretty big pain when using pyspark (adding modules to workers) and should definitely be included. I'll open a new pull request and to push this through.

jurriaan · 2016-03-24T09:38:36Z

Yeah, this is quite a useful feature 👍

buck heroux added 6 commits February 18, 2015 12:50

added requirements file to pyspark

0ed060d

tarfile has no contextmanager in python2.

6b8bcde

reqs fix

2773483

temp tar file

0371ad9

bubbled up try finally

f2a46e5

forgot to remove

fca4be6

davies reviewed Mar 7, 2015
View reviewed changes

buck heroux added 2 commits April 3, 2015 16:35

added requirementsFile tests and switch to __import__

d287522

merged tests

76ff637

davies reviewed Jul 13, 2015
View reviewed changes

buckhx reviewed Jul 13, 2015
View reviewed changes

pep8 styling

565bf7f

daradib mentioned this pull request Aug 21, 2015

[SPARK-5929] Pyspark: Register a pip requirements file with spark_context #4743

Closed

buck heroux added 4 commits October 5, 2015 18:07

support namespace packages and extract addModule logic

23771fd

tmp_dir to mod

cd21c5c

remove reqs from context constructor

39f26d9

upstream merge

49a4ed0

asfgit closed this in 7b4452b Dec 31, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-5929] Pyspark: Register a pip requirements file with spark_context #4897

[SPARK-5929] Pyspark: Register a pip requirements file with spark_context #4897

buckhx commented Mar 5, 2015

davies Mar 7, 2015

davies commented Mar 7, 2015

davies commented Mar 7, 2015

buckhx commented Mar 10, 2015

davies commented Mar 10, 2015

buckhx commented Apr 3, 2015

buckhx commented Apr 20, 2015

robcowie commented Apr 20, 2015

buckhx commented May 12, 2015

andrewor14 commented Jul 13, 2015

SparkQA commented Jul 13, 2015

buckhx commented Jul 13, 2015

davies Jul 13, 2015

buckhx Jul 13, 2015

andrewor14 commented Jul 13, 2015

davies commented Jul 13, 2015

buckhx Jul 13, 2015

buckhx Sep 25, 2015

buckhx commented Jul 13, 2015

JoshRosen commented Jul 13, 2015

JoshRosen commented Jul 13, 2015

JoshRosen commented Jul 13, 2015

SparkQA commented Jul 13, 2015

SparkQA commented Jul 15, 2015

buckhx commented Sep 25, 2015

buckhx commented Oct 2, 2015

buckhx commented Oct 21, 2015

davies commented Oct 21, 2015

DoHe commented Dec 2, 2015

rxin commented Dec 31, 2015

buckhx commented Feb 19, 2016

jurriaan commented Mar 24, 2016

[SPARK-5929] Pyspark: Register a pip requirements file with spark_context #4897

[SPARK-5929] Pyspark: Register a pip requirements file with spark_context #4897

Conversation

buckhx commented Mar 5, 2015

davies Mar 7, 2015

Choose a reason for hiding this comment

davies commented Mar 7, 2015

davies commented Mar 7, 2015

buckhx commented Mar 10, 2015

davies commented Mar 10, 2015

buckhx commented Apr 3, 2015

buckhx commented Apr 20, 2015

robcowie commented Apr 20, 2015

buckhx commented May 12, 2015

andrewor14 commented Jul 13, 2015

SparkQA commented Jul 13, 2015

buckhx commented Jul 13, 2015

davies Jul 13, 2015

Choose a reason for hiding this comment

buckhx Jul 13, 2015

Choose a reason for hiding this comment

andrewor14 commented Jul 13, 2015

davies commented Jul 13, 2015

buckhx Jul 13, 2015

Choose a reason for hiding this comment

buckhx Sep 25, 2015

Choose a reason for hiding this comment

buckhx commented Jul 13, 2015

JoshRosen commented Jul 13, 2015

JoshRosen commented Jul 13, 2015

JoshRosen commented Jul 13, 2015

SparkQA commented Jul 13, 2015

SparkQA commented Jul 15, 2015

buckhx commented Sep 25, 2015

buckhx commented Oct 2, 2015

buckhx commented Oct 21, 2015

davies commented Oct 21, 2015

DoHe commented Dec 2, 2015

rxin commented Dec 31, 2015

buckhx commented Feb 19, 2016

jurriaan commented Mar 24, 2016