[CLIPPER-154] Make a pip package #165

nishadsingh1 · 2017-05-19T20:43:22Z

To test out the functionality of this PR:

Create accounts on TestPyPI and PyPI:
https://testpypi.python.org/pypi?%3Aaction=login_form
https://pypi.python.org/pypi?%3Aaction=login_form

Create a file ~/.pypirc with the following contents:

[distutils]
index-servers =
  pypi
  testpypi

[pypi]
repository=https://pypi.python.org/pypi
username=your_username
password=your_password

[pypitest]
repository=https://testpypi.python.org/pypi
username=your_username
password=your_password

Because this file contains your credentials in plaintext, change its read/write permissions:
chmod 600 ~/.pypirc

Upload our pip package to PyPI Testing server:
python setup.py register -r testpypi
python setup.py sdist upload -r testpypi

Install from PyPi Testing server:
pip install -i https://testpypi.python.org/pypi clipper_admin

Give it a go:

import clipper_admin.clipper_manager as cm
clipper = cm.Clipper("localhost", "", "")
clipper.start()

Upload our pip package to real PyPi server:
python setup.py register -r pypi
python setup.py sdist upload -r pypi

Install from real PyPi server:
pip uninstall clipper_admin (to get rid of the one from the testing server)
pip install clipper_admin

https://clipper.atlassian.net/projects/CLIPPER/issues/CLIPPER-154

AmplabJenkins · 2017-05-19T20:45:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/298/
Test FAILed.

AmplabJenkins · 2017-05-19T21:10:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/299/
Test FAILed.

nishadsingh1 · 2017-05-19T21:11:47Z

setup.py

+  url = 'https://github.com/ucbrise/clipper',
+  packages = ['clipper_admin'],
+  keywords = ['prediction', 'model', 'management'],
+  install_requires = ['requests', 'pyparsing', 'appdirs', 'pprint', 'subprocess32', 'sklearn', 'numpy', 'scipy', 'fabric', 'conda', 'pyyaml']


My code formatting script is hanging for some reason -- figured it was best to push this change up early and figure out the formatting issue while it's getting reviewed.

Sounds good. I can format it in a sec.

dcrankshaw · 2017-05-19T21:18:07Z

Okay just formatted the code

AmplabJenkins · 2017-05-19T21:34:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/301/
Test FAILed.

dcrankshaw · 2017-05-19T21:46:37Z

jenkins test this please

dcrankshaw · 2017-05-20T01:05:26Z

jenkins test this please

dcrankshaw

Looks good. I need to do some testing, but here's a first round of comments.

dcrankshaw · 2017-05-20T01:17:01Z

containers/python/PythonContainerDockerfile

@@ -3,7 +3,7 @@ FROM clipper/py-rpc:latest
 MAINTAINER Dan Crankshaw <dscrankshaw@gmail.com>

 COPY python_container.py python_container_entry.sh /container/
-COPY pywrencloudpickle.py python_container_conda_deps.txt /lib/
+COPY ../../clipper_admin/pywrencloudpickle.py python_container_conda_deps.txt /lib/


This won't work. All the paths copied into the container need to be in the directory tree whose root is the directory containing the Dockerfile (https://docs.docker.com/engine/reference/commandline/build/#build-with-path). This is also important to make sure that the automated builds on Docker hub work. Probably the easiest thing to do is to move all the container dockerfiles to the top level of the directory. You'll need to modify the paths in the Dockerfiles, as well as the script that builds all the files.

dcrankshaw · 2017-05-20T01:17:46Z

examples/tutorial/tutorial_part_one.ipynb

@@ -148,7 +148,7 @@
    "# clipper_manager must be on your path:\n",
    "import sys\n",
    "import os\n",
-    "sys.path.append(os.path.abspath('../../management/'))\n",
+    "sys.path.append(os.path.abspath('../../clipper_admin/'))\n",


Do you think we should import the module directly like this or make them install the pip package?

It does seem cleaner to make them install the pip package, especially since the intended tutorial user is a Clipper user, not developer.

dcrankshaw · 2017-05-20T01:18:11Z

examples/tutorial/tutorial_part_two.ipynb

@@ -23,7 +23,7 @@
    "# clipper_manager must be on your path:\n",
    "import sys\n",
    "import os\n",
-    "sys.path.append(os.path.abspath('../../management/'))\n",
+    "sys.path.append(os.path.abspath('../../clipper_admin/'))\n",


Same question about direct import vs make them install the pip package

dcrankshaw · 2017-05-20T01:19:13Z

setup.py

+setup(
+    name='clipper_admin',
+    version='0.1',
+    description='A low latency online prediction serving system',


Make the description "Admin commands for the Clipper prediction-serving system."

dcrankshaw · 2017-05-20T01:19:48Z

setup.py

+    version='0.1',
+    description='A low latency online prediction serving system',
+    author='Dan Crankshaw',
+    author_email='dcrankshaw@berkeley.edu',


Use crankshaw@cs.berkeley.edu for my email

dcrankshaw · 2017-05-20T01:20:29Z

setup.py

+    description='A low latency online prediction serving system',
+    author='Dan Crankshaw',
+    author_email='dcrankshaw@berkeley.edu',
+    url='https://github.com/ucbrise/clipper',


Make the url http://clipper.ai

dcrankshaw · 2017-05-20T01:20:45Z

setup.py

+    author_email='dcrankshaw@berkeley.edu',
+    url='https://github.com/ucbrise/clipper',
+    packages=['clipper_admin'],
+    keywords=['prediction', 'model', 'management'],


add clipper as a keyword

AmplabJenkins · 2017-05-20T03:13:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/310/
Test FAILed.

AmplabJenkins · 2017-05-20T06:10:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/317/
Test FAILed.

AmplabJenkins · 2017-05-20T06:15:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/318/
Test FAILed.

dcrankshaw

Can you update examples/tutorial/requirements.txt to reflect the new set of Python dependencies that include the clipper-admin package?

dcrankshaw · 2017-05-20T16:24:17Z

containers/python/build_container_images.sh

+time docker build -t clipper/noop-container -f ../../NoopDockerfile ../../
+time docker build -t clipper/python-container -f ../../PythonContainerDockerfile ../../
+time docker build -t clipper/sklearn_cifar_container -f ../../SklearnCifarDockerfile ../../
+time docker build -t clipper/tf_cifar_container -f ../../TensorFlowCifarDockerfile ../../


Let's consolidate this script into bin/build_docker_images.sh

AmplabJenkins · 2017-05-20T17:07:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/319/
Test FAILed.

AmplabJenkins · 2017-05-20T18:02:15Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/320/
Test PASSed.

… consolidated container build scripts

dcrankshaw · 2017-05-22T17:06:32Z

Tested with pip install --extra-index-url https://testpypi.python.org/pypi clipper_admin and it works great. LGTM. Will merge once tests pass. This will break our container builds on docker hub so I'll need to update those as well.

AmplabJenkins · 2017-05-22T17:27:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/326/
Test PASSed.

nishadsingh1 added the status: needs review label May 19, 2017

nishadsingh1 self-assigned this May 19, 2017

nishadsingh1 requested a review from dcrankshaw May 19, 2017 20:43

nishadsingh1 force-pushed the pip_package branch 2 times, most recently from 4598512 to 7a6f5e4 Compare May 19, 2017 21:09

nishadsingh1 commented May 19, 2017

View reviewed changes

dcrankshaw requested changes May 20, 2017

View reviewed changes

dcrankshaw removed the status: needs review label May 20, 2017

nishadsingh1 force-pushed the pip_package branch 2 times, most recently from 77ca261 to f55d05c Compare May 20, 2017 06:12

dcrankshaw requested changes May 20, 2017

View reviewed changes

Nishad Singh and others added 7 commits May 22, 2017 10:04

Setup for pip package

e1d90da

format code

8562c28

Made requested changes

938f583

Fixup

0d5eaf7

Made requested changes -- added clipper_admin to requirements.txt and…

0f453a9

… consolidated container build scripts

fixed formatting

7e8d9a2

Fixed import

fd3c6e7

dcrankshaw force-pushed the pip_package branch from 8001719 to fd3c6e7 Compare May 22, 2017 17:05

dcrankshaw approved these changes May 22, 2017

View reviewed changes

dcrankshaw merged commit bb49084 into ucbrise:develop May 22, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CLIPPER-154] Make a pip package #165

[CLIPPER-154] Make a pip package #165

nishadsingh1 commented May 19, 2017 •

edited

Loading

AmplabJenkins commented May 19, 2017

AmplabJenkins commented May 19, 2017

nishadsingh1 May 19, 2017

dcrankshaw May 19, 2017

dcrankshaw commented May 19, 2017

AmplabJenkins commented May 19, 2017

dcrankshaw commented May 19, 2017

dcrankshaw commented May 20, 2017

dcrankshaw left a comment

dcrankshaw May 20, 2017

dcrankshaw May 20, 2017

nishadsingh1 May 20, 2017

dcrankshaw May 20, 2017

dcrankshaw May 20, 2017

dcrankshaw May 20, 2017

dcrankshaw May 20, 2017

dcrankshaw May 20, 2017

AmplabJenkins commented May 20, 2017

AmplabJenkins commented May 20, 2017

AmplabJenkins commented May 20, 2017

dcrankshaw left a comment

dcrankshaw May 20, 2017

AmplabJenkins commented May 20, 2017

AmplabJenkins commented May 20, 2017

dcrankshaw commented May 22, 2017

AmplabJenkins commented May 22, 2017

[CLIPPER-154] Make a pip package #165

[CLIPPER-154] Make a pip package #165

Conversation

nishadsingh1 commented May 19, 2017 • edited Loading

AmplabJenkins commented May 19, 2017

AmplabJenkins commented May 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcrankshaw commented May 19, 2017

AmplabJenkins commented May 19, 2017

dcrankshaw commented May 19, 2017

dcrankshaw commented May 20, 2017

dcrankshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented May 20, 2017

AmplabJenkins commented May 20, 2017

AmplabJenkins commented May 20, 2017

dcrankshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented May 20, 2017

AmplabJenkins commented May 20, 2017

dcrankshaw commented May 22, 2017

AmplabJenkins commented May 22, 2017

nishadsingh1 commented May 19, 2017 •

edited

Loading