ray-serve/e2e/tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "warming-operation",
   "metadata": {},
   "source": [
    "\n",
    "<a id='end-to-end-tutorial'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "domestic-washington",
   "metadata": {},
   "source": [
    "[Open in colab](https://colab.research.google.com/github/anyscale/academy/blob/main/ray-serve/e2e/tutorial.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "international-jefferson",
   "metadata": {},
   "source": [
    "# Model Deployment with Ray Serve: Introduction\n",
    "\n",
    "This tutorial has a few goals in mind. \n",
    "\n",
    "1. Introduce you to the landscape of ML serving tools, and where Ray Serve fits in.\n",
    "2. Teach you how to deploy any Python based model with Ray Serve, and compose them for production ready pipelines.\n",
    "3. Show you the concrete steps required to deploy models for interactive REST endpoint.\n",
    "\n",
    "If you have any questions about this tutorial, or any follow-up questions, please feel free to ask them in the [Ray discussion forum](https://discuss.ray.io/c/ray-serve/6) or [Ray Slack](https://forms.gle/9TSdDYUgxYs8SA9e8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "structural-garlic",
   "metadata": {},
   "source": [
    "# 1 Landscape of ML Deployment Tools\n",
    "\n",
    "Where does Ray Serve fit in the landscape of machine learning deployment tools? \n",
    "\n",
    "Commonly there's a spectrum of tools:\n",
    "- People typically starts with either framework specific servers (TFServing, TorchServer) or web servers (Flask, FastAPI) as an easy start to deploy a single model. \n",
    "- For more \"production-readiness\", various custom toolings are added (Docker, K8s, Golang based microservices). \n",
    "- But you can't just maintain a glued-together system. Folks starting looking for special purpose deployment tools (KubeFlow, KServe, Triton, etc) to manage and deploy many models in production. \n",
    "\n",
    "Over the spectrum, our team observe that you have to trade-off ease of development with production scalability. Ray Serve lets you easily develop locally and then transparently scale to production.\n",
    "\n",
    "![Serve aims at both ease of development and ready for production.](serve-position.svg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "liquid-suggestion",
   "metadata": {},
   "source": [
    "# 2 Model Serving with Ray Serve\n",
    "\n",
    "Adapted from our [documentation](https://docs.ray.io/en/master/serve/index.html#rayserve)\n",
    "\n",
    "By the end of this tutorial you will have learned how to deploy a machine\n",
    "learning model locally via Ray Serve.\n",
    "\n",
    "First, install Ray Serve and all of its dependencies by running the following\n",
    "command in your terminal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wanted-omega",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "%pip install -qq \"ray[serve]\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hourly-return",
   "metadata": {},
   "source": [
    "For this tutorial, we’ll use [HuggingFace’s SummarizationPipeline](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.SummarizationPipeline)\n",
    "to access a model that summarizes text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "worthy-solution",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qq transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hollywood-music",
   "metadata": {},
   "source": [
    "## Example Model\n",
    "\n",
    "Let’s first take a look at how the model works, without using Ray Serve.\n",
    "This is the code for the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "exposed-chess",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "\n",
    "def summarize(text):\n",
    "    # Load model\n",
    "    summarizer = pipeline(\"summarization\", model=\"t5-small\")\n",
    "\n",
    "    # Run inference\n",
    "    summary_list = summarizer(text)\n",
    "\n",
    "    # Post-process output to return only the summary text\n",
    "    summary = summary_list[0][\"summary_text\"]\n",
    "\n",
    "    return summary\n",
    "\n",
    "\n",
    "article_text = (\n",
    "    \"HOUSTON -- Men have landed and walked on the moon. \"\n",
    "    \"Two Americans, astronauts of Apollo 11, steered their fragile \"\n",
    "    \"four-legged lunar module safely and smoothly to the historic landing \"\n",
    "    \"yesterday at 4:17:40 P.M., Eastern daylight time. Neil A. Armstrong, the \"\n",
    "    \"38-year-old commander, radioed to earth and the mission control room \"\n",
    "    \"here: \\\"Houston, Tranquility Base here. The Eagle has landed.\\\" The \"\n",
    "    \"first men to reach the moon -- Armstrong and his co-pilot, Col. Edwin E. \"\n",
    "    \"Aldrin Jr. of the Air Force -- brought their ship to rest on a level, \"\n",
    "    \"rock-strewn plain near the southwestern shore of the arid Sea of \"\n",
    "    \"Tranquility. About six and a half hours later, Armstrong opened the \"\n",
    "    \"landing craft\\'s hatch, stepped slowly down the ladder and declared as \"\n",
    "    \"he planted the first human footprint on the lunar crust: \\\"That\\'s one \"\n",
    "    \"small step for man, one giant leap for mankind.\\\" His first step on the \"\n",
    "    \"moon came at 10:56:20 P.M., as a television camera outside the craft \"\n",
    "    \"transmitted his every move to an awed and excited audience of hundreds \"\n",
    "    \"of millions of people on earth.\")\n",
    "\n",
    "summary = summarize(article_text)\n",
    "print(summary)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "frank-large",
   "metadata": {},
   "source": [
    "The Python file, called `local_model.py`, uses the `summarize` function to\n",
    "generate summaries of text.\n",
    "\n",
    "- The `summarizer` variable on line 7 inside `summarize` points to a\n",
    "  function that uses the [t5-small](https://huggingface.co/t5-small)\n",
    "  model to summarize text.  \n",
    "- When `summarizer` is called on a Python String, it returns summarized text\n",
    "  inside a dictionary formatted as `[{\"summary_text\": \"...\", ...}, ...]`.  \n",
    "- `summarize` then extracts the summarized text on line 13 by indexing into\n",
    "  the dictionary.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "equipped-drunk",
   "metadata": {},
   "source": [
    "Keep in mind that the `SummarizationPipeline` is an example machine learning\n",
    "model for this tutorial. You can follow along using arbitrary models in any\n",
    "framework that has a Python API. Check out our tutorials on sckit-learn,\n",
    "PyTorch, and Tensorflow for more info and examples:\n",
    "\n",
    "- [Keras and Tensorflow Tutorial](https://docs.ray.io/en/latest/serve/tutorials/tensorflow.html)\n",
    "- [PyTorch Tutorial](https://docs.ray.io/en/latest/serve/tutorials/pytorch.html)\n",
    "- [Scikit-Learn Tutorial](https://docs.ray.io/en/latest/serve/tutorials/sklearn.html)\n",
    "- [Batching Tutorial](https://docs.ray.io/en/latest/serve/tutorials/batch.html)\n",
    "- [RLlib Tutorial](https://docs.ray.io/en/latest/serve/tutorials/rllib.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efficient-colon",
   "metadata": {},
   "source": [
    "## Converting to Ray Serve Deployment\n",
    "\n",
    "This tutorial’s goal is to deploy this model using Ray Serve, so it can be\n",
    "scaled up and queried over HTTP. We’ll start by converting the above Python\n",
    "function into a Ray Serve deployment that can be launched locally on a laptop.\n",
    "\n",
    "First, we need to import `ray` and `ray serve`, to use features in Ray Serve such as `deployments`, which provide HTTP access to our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "living-excellence",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import ray\n",
    "from ray import serve"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accessory-scene",
   "metadata": {},
   "source": [
    "After these imports, we can include our model code from above.\n",
    "We won’t call our `summarize` function just yet though!\n",
    "We will soon add logic to handle HTTP requests, so the `summarize` function\n",
    "can operate on article text sent via HTTP request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "identical-lightweight",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "def summarize(text):\n",
    "    summarizer = pipeline(\"summarization\", model=\"t5-small\")\n",
    "    summary_list = summarizer(text)\n",
    "    summary = summary_list[0][\"summary_text\"]\n",
    "    return summary\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "quantitative-scotland",
   "metadata": {},
   "source": [
    "Ray Serve needs to run on top of a Ray cluster, so we create a local one.\n",
    "See [Deploying Ray Serve](https://docs.ray.io/en/latest/serve/deployment.html) to learn more about starting a Ray Serve\n",
    "instance and deploying to a Ray cluster.\n",
    "\n",
    ">**Note**\n",
    ">\n",
    ">You can use Ray to perform data processing, hyperparameter-tuning, and distributed model training as well! Learn more at http://ray.io/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "still-serve",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "ray.init()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "marked-enclosure",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "serve.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "anticipated-malta",
   "metadata": {},
   "source": [
    "Now that we have defined our `summarize` function, connected to a Ray\n",
    "Cluster, and started the Ray Serve runtime, we can define a function that\n",
    "accepts HTTP requests and routes them to the `summarize` function. We\n",
    "define a function called `router` that takes in a [Starlette `request`\n",
    "object](https://www.starlette.io/requests/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "brilliant-minneapolis",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "from starlette.requests import Request\n",
    "\n",
    "@serve.deployment\n",
    "def router(request: Request):\n",
    "    txt = request.query_params[\"txt\"]\n",
    "    return summarize(txt)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wooden-parameter",
   "metadata": {},
   "source": [
    "- In line 1, we add the decorator `@serve.deployment`\n",
    "  to the `router` function to turn the function into a Serve `Deployment`\n",
    "  object.  \n",
    "- In line 3, `router` uses the `\"txt\"` query parameter in the `request`\n",
    "  to get the article text to summarize.  \n",
    "- In line 4, it then passes this article text into the `summarize` function\n",
    "  and returns the value.  \n",
    "\n",
    "\n",
    ">**Note**\n",
    ">\n",
    ">Lines 3 and 4 define our HTTP request schema. The HTTP requests sent to this\n",
    "endpoint must have a `\"txt\"` query parameter that contains a string.\n",
    "In general, you can accept HTTP data using query parameters or the\n",
    "request body. Additionally, you can add other Serve deployments with\n",
    "different names to create more endpoints that can accept different schemas.\n",
    "For more complex validation, you can also use FastAPI (see\n",
    "[FastAPI HTTP Deployments](https://docs.ray.io/en/latest/serve/http-servehandle.html#fastapi-http-deployments) for more info).\n",
    "\n",
    "This routing function’s name doesn’t have to be `router`. Serve uses your function name to namespace the route. For example the router will be accessible via `http://localhost:8000/router?txt=your-data`. You can change the function name or explicitly provider a `route_prefix` to Ray Serve via `@serve.deployment(route_prefix=\"/...\")`. \n",
    "\n",
    "Since `@serve.deployment` makes `router` a `Deployment` object, it can be\n",
    "deployed using `router.deploy()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "starting-venice",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "router.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "front-start",
   "metadata": {},
   "source": [
    "Once we deploy `router`, we can query the model over HTTP. Currently our router has 1 replica. This means that there is one Python process serving the model. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "warming-wright",
   "metadata": {},
   "source": [
    "## Testing the Ray Serve Deployment\n",
    "\n",
    "We can now test our model over HTTP. The structure of our HTTP query is:\n",
    "\n",
    "`http://127.0.0.1:8000/[Deployment Name]?[Parameter Name-1]=[Parameter Value-1]&[Parameter Name-2]=[Parameter Value-2]&...&[Parameter Name-n]=[Parameter Value-n]`\n",
    "\n",
    "Since the cluster is deployed locally in this tutorial, the `127.0.0.1:8000`\n",
    "refers to a localhost with port 8000. The `[Deployment Name]` refers to\n",
    "either the name of the function that we called `.deploy()` on (in our case,\n",
    "this is `router`), or the `name` keyword parameter’s value in\n",
    "`@serve.deployment` (see the Tip under the `router` function definition\n",
    "above for more info).\n",
    "\n",
    "Each `[Parameter Name]` refers to a field’s name in the\n",
    "request’s `query_params` dictionary for our deployed function. In our\n",
    "example, the only parameter we need to pass in is `txt`. This parameter is\n",
    "referenced in the `txt = request.query_params[\"txt\"]` line in the `router`\n",
    "function. Each [Parameter Name] object has a corresponding [Parameter Value]\n",
    "object. The `txt`’s [Parameter Value] is a string containing the article\n",
    "text to summarize. We can chain together any number of the name-value pairs\n",
    "using the `&` symbol in the request URL.\n",
    "\n",
    "Now that the `summarize` function is deployed on Ray Serve, we can make HTTP\n",
    "requests to it. Here’s a client script that requests a summary from the same\n",
    "article as the original Python script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "stable-peter",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "\n",
    "import requests\n",
    "\n",
    "article_text = (\n",
    "    \"HOUSTON -- Men have landed and walked on the moon. \"\n",
    "    \"Two Americans, astronauts of Apollo 11, steered their fragile \"\n",
    "    \"four-legged lunar module safely and smoothly to the historic landing \"\n",
    "    \"yesterday at 4:17:40 P.M., Eastern daylight time. Neil A. Armstrong, the \"\n",
    "    \"38-year-old commander, radioed to earth and the mission control room \"\n",
    "    \"here: \\\"Houston, Tranquility Base here. The Eagle has landed.\\\" The \"\n",
    "    \"first men to reach the moon -- Armstrong and his co-pilot, Col. Edwin E. \"\n",
    "    \"Aldrin Jr. of the Air Force -- brought their ship to rest on a level, \"\n",
    "    \"rock-strewn plain near the southwestern shore of the arid Sea of \"\n",
    "    \"Tranquility. About six and a half hours later, Armstrong opened the \"\n",
    "    \"landing craft\\'s hatch, stepped slowly down the ladder and declared as \"\n",
    "    \"he planted the first human footprint on the lunar crust: \\\"That\\'s one \"\n",
    "    \"small step for man, one giant leap for mankind.\\\" His first step on the \"\n",
    "    \"moon came at 10:56:20 P.M., as a television camera outside the craft \"\n",
    "    \"transmitted his every move to an awed and excited audience of hundreds \"\n",
    "    \"of millions of people on earth.\")\n",
    "\n",
    "response = requests.get(\"http://127.0.0.1:8000/router?txt=\" +\n",
    "                        article_text).text\n",
    "\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "above-austria",
   "metadata": {},
   "source": [
    "## Using Classes in the Ray Serve Deployment\n",
    "\n",
    "Our application is still a bit inefficient though. In particular, the\n",
    "`summarize` function loads the model on each call when it sets the\n",
    "`summarizer` variable. However, the model never changes, so it would be more\n",
    "efficient to define `summarizer` only once and keep its value in memory\n",
    "instead of reloading it for each HTTP query.\n",
    "\n",
    "We can achieve this by converting our `summarize` function into a class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "reduced-candy",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "@serve.deployment\n",
    "class Summarizer:\n",
    "    def __init__(self):\n",
    "        self.summarize = pipeline(\"summarization\", model=\"t5-small\")\n",
    "\n",
    "    def __call__(self, request):\n",
    "        txt = request.query_params[\"txt\"]\n",
    "        summary_list = self.summarize(txt)\n",
    "        summary = summary_list[0][\"summary_text\"]\n",
    "        return summary\n",
    "\n",
    "\n",
    "Summarizer.deploy()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "academic-camera",
   "metadata": {},
   "source": [
    "In this configuration, we can query the `Summarizer` class directly.\n",
    "The `Summarizer` is initialized once (after calling `Summarizer.deploy()`).\n",
    "In line 13, its `__init__` function loads and stores the model in\n",
    "`self.summarize`. HTTP queries for the `Summarizer` class are routed to its\n",
    "`__call__` method by default, which takes in the Starlette `request`\n",
    "object. The `Summarizer` class can then take the request’s `txt` data and\n",
    "call the `self.summarize` function on it without loading the model on each\n",
    "query.\n",
    "\n",
    ">**Note**\n",
    ">\n",
    ">Instance variables can also store state. For example, to\n",
    ">count the number of requests served, a `@serve.deployment` class can define\n",
    ">a `self.counter` instance variable in its `__init__` function and set it\n",
    ">to 0. When the class is queried, it can increment the `self.counter`\n",
    ">variable inside of the function responding to the query. The `self.counter`\n",
    ">will keep track of the number of requests served across requests.\n",
    "\n",
    "HTTP queries for the Ray Serve class deployments follow a similar format to Ray\n",
    "Serve function deployments. Here’s an example client script for the\n",
    "`Summarizer` class. Notice that the only difference from the `router`’s\n",
    "client script is that the URL uses the `Summarizer` path instead of\n",
    "`router`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "judicial-federation",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "response = requests.get(\"http://127.0.0.1:8000/Summarizer?txt=\" +\n",
    "                        article_text).text\n",
    "\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alternate-agriculture",
   "metadata": {},
   "source": [
    "## Adding Functionality with FastAPI\n",
    "\n",
    "Now suppose we want to expose additional functionality in our model. In\n",
    "particular, the `summarize` function also has `min_length` and\n",
    "`max_length` parameters. Although we could expose these options as additional\n",
    "parameters in URL, Ray Serve also allows us to add more route options to the\n",
    "URL itself and handle each route separately.\n",
    "\n",
    "Because this logic can get complex, Serve integrates with\n",
    "[FastAPI](https://fastapi.tiangolo.com/). This allows us to define a Serve\n",
    "deployment by adding the `@serve.ingress` decorator to a FastAPI app. For\n",
    "more info about FastAPI with Serve, please see [FastAPI HTTP Deployments](https://docs.ray.io/en/latest/serve/http-servehandle.html#fastapi-http-deployments).\n",
    "\n",
    "FastAPI uses [Pydantic](https://pydantic-docs.helpmanual.io/) for schema validation. It allows you to annotate your input value with validators and handle all the request parsing for you.\n",
    "\n",
    "As an example of FastAPI, here’s a modified version of our `Summarizer` class\n",
    "with route options to request a minimum or maximum length in the\n",
    "summaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "indian-waterproof",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import ray\n",
    "from ray import serve\n",
    "from fastapi import FastAPI\n",
    "from transformers import pipeline\n",
    "from pydantic import PositiveInt, constr\n",
    "\n",
    "app = FastAPI()\n",
    "\n",
    "\n",
    "@serve.deployment\n",
    "@serve.ingress(app)\n",
    "class Summarizer:\n",
    "    def __init__(self):\n",
    "        self.summarize = pipeline(\"summarization\", model=\"t5-small\")\n",
    "\n",
    "    @app.get(\"/\")\n",
    "    def get_summary(\n",
    "        self,\n",
    "        text: constr(min_length=1, strip_whitespace=True),\n",
    "        min_length: PositiveInt = 1,\n",
    "        max_length: PositiveInt = 256,\n",
    "    ):\n",
    "        summary_list = self.summarize(\n",
    "            text,\n",
    "            min_length=min_length or 0,\n",
    "            max_length=max_length or 256,\n",
    "        )\n",
    "        summary = summary_list[0][\"summary_text\"]\n",
    "        return summary\n",
    "\n",
    "\n",
    "Summarizer.deploy()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wooden-vancouver",
   "metadata": {},
   "source": [
    "The class now exposes the same route but with the following feature:\n",
    "\n",
    "- Takes a JSON formatted POST method. \n",
    "- Validate the input text is not empty.\n",
    "- Accept optional min_length and max_length values and validate them.\n",
    "\n",
    "Notice that `Summarizer`’s methods no longer take in a Starlette `request`\n",
    "object. Instead, they take in the URL’s txt parameter directly with FastAPI’s\n",
    "[body parameter](https://fastapi.tiangolo.com/tutorial/body/)\n",
    "feature.\n",
    "\n",
    "Since we still deploy our model locally, the full URL still uses the\n",
    "localhost IP. This means each of our three routes comes after the\n",
    "`http://127.0.0.1:8000` IP and port address. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "signal-nigeria",
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "response = requests.post(\n",
    "    \"http://127.0.0.1:8000/Summarizer\", json={\"text\": article_text, \"max_length\": 10}\n",
    ").text\n",
    "\n",
    "response\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "viral-lancaster",
   "metadata": {},
   "source": [
    "Congratulations! You just built and deployed a machine learning model on Ray\n",
    "Serve! You should now have enough context to dive into the [Core API: Deployments](https://docs.ray.io/en/latest/serve/core-apis.html) to\n",
    "get a deeper understanding of Ray Serve."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "atomic-electronics",
   "metadata": {},
   "source": [
    "# 3 Deploying to Cloud Endpoint\n",
    "\n",
    "Now the Ray Serve endpoint is ready for production! It has request validation, it efficiently use memory, and you can transparently scale it out. But you can't just keep this running as a notebook for live serving instances. \n",
    "\n",
    "Here are some practical recommednations to deployment:\n",
    "- For quick demo, use [ngrok](https://ngrok.com/) to expose your local port to the internet. \n",
    "- For running on GCP for quick proof of concept, use Google Cloud Run to run a containerized application. \n",
    "- For production deployment, use [Kubernetes with Ray operator](https://docs.ray.io/en/latest/serve/deployment.html#deploying-on-kubernetes) or hosted provider like [Anyscale](anysclae.com).\n",
    "\n",
    "### Cloud Run Example\n",
    "You can take a look at the [example directory](https://github.com/anyscale/academy/tree/main/ray-serve/e2e/deploy-cloud-run) for a complete example containerizing the application and deploy to Google Coud Run. It contains\n",
    "- `deploy.py` for the application.\n",
    "- `test-query.py` for calling the application as a client.\n",
    "- `requirements.txt` for Python dependencies.\n",
    "- `Dockerfile` for describing how to containerize the application and make it runnable as docker containers.\n",
    "\n",
    "### Anyscale Example\n",
    "For deploying to Anyscale hosted service, all you need is to change set the environment variable `RAY_ADDRESS=\"anyscale://your-cluster-name` or use [production services construct](https://docs.anyscale.com/user-guide/run-and-monitor/production-services). You do need an invite at the moment though. \n",
    "\n",
    "### ngrok Example\n",
    "Let's dive deeper into what ngrok is doing here. When you expose an http endpoint to the internet, you need a stable and public accessible IP address. The IP address on the laptop  is typically ephemeral. The IP address of Google Colab notebook is typically private. This means we need some way to allow anyone to access this through their browser.\n",
    "\n",
    "ngrok is a service that offer free tunneling for developers. It exposes your http endpoint to a ngrok domain name and ip address."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44fdac50-2df9-4e9a-97d9-c69bde3962f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q pyngrok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3a726f9-310e-4a42-8e8c-7c0713bc5655",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyngrok import ngrok\n",
    "ngrok.connect(8000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "gross-broadcasting",
   "metadata": {},
   "source": [
    "# 4 Bonus: Other Features of Ray Serve\n",
    "In the tutorial, we demostrated the ability to serve a single model. Ray Serve is much more than that! In order to scale ML model deployments, Serve enables you:\n",
    "- Scale out to muliple replicas by setting `num_replicas=10` or autoscale. ML models need this because they are typically low throughput.\n",
    "- Use GPU resource, and multiple them with `ray_actor_options={\"num_gpus\": 0.5}`. You use this to improve utilization of expensive GPUs.\n",
    "- Rolling upgrade your deployments and rollback in case it fails. You need to for production stability.\n",
    "\n",
    "![serve features list](serve-features.svg)\n",
    "\n",
    "You can learn more about the features of Ray Serve at [rayserve.org](http://rayserve.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "piano-quality",
   "metadata": {},
   "source": [
    "# 5 Bonus: Real World Model Composition\n",
    "\n",
    "As you start to build end-to-end machine learning applications, you will quickly face the fact that models doesn't exist in isolation. To perform an end-to-end task, you have to glue together multiple ML models and steps. Many existing systems doesn't support model composition natively. Ray Serve supports that natively and it is easy to expressive your logic. \n",
    "\n",
    "![model-composition](model-composition.svg)\n",
    "\n",
    "You can learn more from this [Ray Summit talk](https://www.youtube.com/watch?v=mM4hJLelzSw)."
   ]
  }
 ],
 "metadata": {
  "date": 1643246257.62908,
  "filename": "end_to_end_tutorial.rst",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "title": "End-to-End Tutorial"
 },
 "nbformat": 4,
 "nbformat_minor": 5
}