Clipper is a prediction serving system that sits between user-facing applications and a wide range of commonly used machine learning models and frameworks. Learn more about Clipper and view documentation at our website http://clipper.ai.
-
Clipper simplifies integration of machine learning techniques into user facing applications by providing a simple standard REST interface for prediction and feedback across a wide range of commonly used machine learning frameworks. Clipper makes product teams happy.
-
Clipper simplifies model deployment and helps reduce common bugs by using the same tools and libraries used in model development to render live predictions. Clipper makes data scientists happy.
-
Clipper improves throughput and ensures reliable millisecond latencies by introducing adaptive batching, caching, and straggler mitigation techniques. Clipper makes the infra-team less unhappy.
-
Clipper improves prediction accuracy by introducing state-of-the-art bandit and ensemble methods to intelligently select and combine predictions and achieve real-time personalization across machine learning frameworks. Clipper makes users happy.
Note: This quickstart works for the latest version of code. For a quickstart that works with the released version of Clipper available on PyPi, go to our website
This quickstart requires Docker and supports Python 2.7, 3.5, 3.6 and 3.7.
-
Basic query: https://github.com/ucbrise/clipper/tree/develop/examples/basic_query
-
Image query: https://github.com/ucbrise/clipper/tree/develop/examples/image_query
-
Examples including metrics: https://github.com/ucbrise/clipper/tree/develop/examples
Install Clipper
You can either install Clipper directly from GitHub:
pip install git+https://github.com/ucbrise/clipper.git@develop#subdirectory=clipper_adminor by cloning Clipper and installing directly from the file system:
pip install -e </path/to/clipper_repo>/clipper_adminStart a local Clipper cluster
First start a Python interpreter session.
$ python
# Or start one with iPython
$ conda install ipython
$ ipythonCreate a ClipperConnection object and start Clipper. Running this command for the first time will
download several Docker containers, so it may take some time.
from clipper_admin import ClipperConnection, DockerContainerManager
clipper_conn = ClipperConnection(DockerContainerManager())
clipper_conn.start_clipper()17-08-30:15:48:41 INFO [docker_container_manager.py:95] Starting managed Redis instance in Docker
17-08-30:15:48:43 INFO [clipper_admin.py:105] Clipper still initializing.
17-08-30:15:48:44 INFO [clipper_admin.py:107] Clipper is runningRegister an application called "hello-world". This will create a prediction REST endpoint at http://localhost:1337/hello-world/predict
clipper_conn.register_application(name="hello-world", input_type="doubles", default_output="-1.0", slo_micros=100000)17-08-30:15:51:42 INFO [clipper_admin.py:182] Application hello-world was successfully registeredInspect Clipper to see the registered apps
clipper_conn.get_all_apps()[u'hello-world']Define a simple model that just returns the sum of each feature vector. Note that the prediction function takes a list of feature vectors as input and returns a list of strings.
def feature_sum(xs):
return [str(sum(x)) for x in xs]Import the python deployer package
from clipper_admin.deployers import python as python_deployerDeploy the "feature_sum" function as a model. Notice that the application and model
must have the same input type.
python_deployer.deploy_python_closure(clipper_conn, name="sum-model", version=1, input_type="doubles", func=feature_sum)17-08-30:15:59:56 INFO [deployer_utils.py:50] Anaconda environment found. Verifying packages.
17-08-30:16:00:04 INFO [deployer_utils.py:150] Fetching package metadata .........
Solving package specifications: .
17-08-30:16:00:04 INFO [deployer_utils.py:151]
17-08-30:16:00:04 INFO [deployer_utils.py:59] Supplied environment details
17-08-30:16:00:04 INFO [deployer_utils.py:71] Supplied local modules
17-08-30:16:00:04 INFO [deployer_utils.py:77] Serialized and supplied predict function
17-08-30:16:00:04 INFO [python.py:127] Python closure saved
17-08-30:16:00:04 INFO [clipper_admin.py:375] Building model Docker image with model data from /tmp/python_func_serializations/sum-model
17-08-30:16:00:05 INFO [clipper_admin.py:378] Pushing model Docker image to sum-model:1
17-08-30:16:00:07 INFO [docker_container_manager.py:204] Found 0 replicas for sum-model:1. Adding 1
17-08-30:16:00:07 INFO [clipper_admin.py:519] Successfully registered model sum-model:1
17-08-30:16:00:07 INFO [clipper_admin.py:447] Done deploying model sum-model:1.Possible Error
If start_clipper() is stuck at this logs, try pip install -U cloudpickle==0.5.3
18-05-21:12:19:59 INFO [deployer_utils.py:44] Saving function to /tmp/clipper/tmpx6d_zqeq
18-05-21:12:19:59 INFO [deployer_utils.py:54] Serialized and supplied predict function
18-05-21:12:19:59 INFO [python.py:192] Python closure saved
18-05-21:12:19:59 INFO [python.py:206] Using Python 3.6 base image
18-05-21:12:19:59 INFO [clipper_admin.py:451] Building model Docker image with model data from /tmp/clipper/tmpx6d_zqeq
18-05-21:12:20:00 INFO [clipper_admin.py:455] {'stream': 'Step 1/2 : FROM clipper/python36-closure-container:develop'}
18-05-21:12:20:00 INFO [clipper_admin.py:455] {'stream': '\n'}
18-05-21:12:20:00 INFO [clipper_admin.py:455] {'stream': ' ---> 1aaddfa3945e\n'}
18-05-21:12:20:00 INFO [clipper_admin.py:455] {'stream': 'Step 2/2 : COPY /tmp/clipper/tmpx6d_zqeq /model/'}
18-05-21:12:20:00 INFO [clipper_admin.py:455] {'stream': '\n'}
18-05-21:12:20:00 INFO [clipper_admin.py:455] {'stream': ' ---> b7c29f531d2e\n'}
18-05-21:12:20:00 INFO [clipper_admin.py:455] {'aux': {'ID': 'sha256:b7c29f531d2eaf59dd39579dbe512538be398dcb5fdd182db14e4d58770d2055'}}
18-05-21:12:20:00 INFO [clipper_admin.py:455] {'stream': 'Successfully built b7c29f531d2e\n'}
18-05-21:12:20:00 INFO [clipper_admin.py:455] {'stream': 'Successfully tagged sum-model:1\n'}
18-05-21:12:20:00 INFO [clipper_admin.py:457] Pushing model Docker image to sum-model:1
18-05-21:12:20:02 INFO [docker_container_manager.py:247] Found 0 replicas for sum-model:1. Adding 1
It is because of cloudpickle dependency version issue. You may see this error logs from model container docker log.
$ docker logs 439ba722d79a # model container logs. For this example, it will be simple-example model container
Starting Python Closure container
Connecting to Clipper with default port: 7000
Traceback (most recent call last):
File "/container/python_closure_container.py", line 56, in <module>
rpc_service.get_input_type())
File "/container/python_closure_container.py", line 28, in __init__
self.predict_func = load_predict_func(predict_path)
File "/container/python_closure_container.py", line 17, in load_predict_func
return cloudpickle.load(serialized_func_file)
File "/usr/local/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 1060, in _make_skel_func
base_globals['__builtins__'] = __builtins__
TypeError: 'str' object does not support item assignment
Tell Clipper to route requests for the "hello-world" application to the "sum-model"
clipper_conn.link_model_to_app(app_name="hello-world", model_name="sum-model")17-08-30:16:08:50 INFO [clipper_admin.py:224] Model sum-model is now linked to application hello-worldYour application is now ready to serve predictions
Now that you've deployed your first model, you can start requesting predictions at the REST endpoint that clipper created for your application: http://localhost:1337/hello-world/predict
With cURL:
$ curl -X POST --header "Content-Type:application/json" -d '{"input": [1.1, 2.2, 3.3]}' 127.0.0.1:1337/hello-world/predictWith Python:
import requests, json, numpy as np
headers = {"Content-type": "application/json"}
requests.post("http://localhost:1337/hello-world/predict", headers=headers, data=json.dumps({"input": list(np.random.random(10))})).json()If you closed the Python REPL you were using to start Clipper, you will need to start a new Python REPL and create another connection to the Clipper cluster. If you still have the Python REPL session active from earlier, you can re-use your existing ClipperConnection object.
Create a new connection. If you have still have the Python REPL from earlier, you can skip this step.
from clipper_admin import ClipperConnection, DockerContainerManager
clipper_conn = ClipperConnection(DockerContainerManager())
clipper_conn.connect()Stop all Clipper docker containers
clipper_conn.stop_all()17-08-30:16:15:38 INFO [clipper_admin.py:1141] Stopped all Clipper cluster and all model containersTo file a bug or request a feature, please file a GitHub issue. Pull requests are welcome. Additional help and instructions for contributors can be found on our website at http://clipper.ai/contributing.
You can contact us at clipper-dev@googlegroups.com
This research is supported in part by DHS Award HSHQDC-16-3-00083, DOE Award SN10040 DE-SC0012463, NSF CISE Expeditions Award CCF-1139158, and gifts from Ant Financial, Amazon Web Services, CapitalOne, Ericsson, GE, Google, Huawei, Intel, IBM, Microsoft and VMware.