advanced_ml_and_api_approaches/model_migration_across_dr_instances/Model_Migration_Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Migrate a model to a new cluster\n",
    "\n",
    "This notebook demonstrates how to migrate a model from one DataRobot cluster to another of the same version.\n",
    "\n",
    "## Setup\n",
    "\n",
    "Note that the model you choose to migrate must by using Python 3 or a newer version.\n",
    "\n",
    "Additionally, `This notebook will not work using https://app.datarobot.com`.\n",
    "\n",
    "### Supported paths\n",
    "\n",
    "The following migration paths are currently known to work by default:\n",
    "\n",
    "- DataRobot v8.x -> 8.x\n",
    "- v8.x -> v9.alpha\n",
    "\n",
    "Note that there will be an extra required process when migrating from v7.3.2+ to v8.0.10+ (shown later in this notebook).\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "- This notebook must be able to write to the `model` directory, located in the same directory where this notebook is run from. For the best results, run this notebook from the local file system.\n",
    "- Ensure that the model you choose to migrate must be a deployed model.\n",
    "- Provide API keys for both the source and destination clusters.\n",
    "- The **Source** and **Destination** users must have the \"**Enable Experimental API access**\" feature flag enabled to follow this workflow.\n",
    "- The notebook must have connectivity to the Source and Destination clusters\n",
    "- DataRobot versions on the clusters must be consistent with the **Supported Paths** above.\n",
    "- For models on clusters of DataRobot v7.x, you must have SSH access to the App Node of the cluster.\n",
    "- The **Source** and **Destination** DataRobot clusters must have the following in the config.yaml:\n",
    "\n",
    "```yaml\n",
    "app_configuration:\n",
    "  drenv_override:\n",
    "    WHITELIST_EXPERIMENTAL_API: true\n",
    "    EXPERIMENTAL_API_ACCESS: true\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install libraries\n",
    "\n",
    "If you are using VS Code, you can install the required DataRobot packages with the cell below. Otherwise, use the cell that follows it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!{sys.executable} -m pip install --upgrade pip\n",
    "# !{sys.executable} -m pip install \"datarobot>=2.28,<2.29\"\n",
    "!{sys.executable} -m pip install datarobot\n",
    "!{sys.executable} -m pip install datetime requests --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install --upgrade pip\n",
    "# !pip3 install \"datarobot>=2.28,<2.29\"\n",
    "!pip3 install datarobot\n",
    "!pip3 install datetime requests --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "from datetime import date\n",
    "import json\n",
    "import os\n",
    "import sys\n",
    "import time\n",
    "from timeit import default_timer\n",
    "import urllib.parse\n",
    "\n",
    "from IPython.display import display, HTML\n",
    "import datarobot as dr\n",
    "import requests\n",
    "\n",
    "print(\"Started: %s\" % (str(datetime.datetime.now())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure source settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Provide the URL protocol, address (IP or FQDN), and path\n",
    "# Example: source_host = \"http://1.2.3.4\"\n",
    "source_host = \"https://source.datarobot.example.com\"\n",
    "# Do not use https://app.datarobot.com, because users do not have access to the \"Enable Experimental API access\" permission\n",
    "\n",
    "# Provide an API key from a user with permission from this cluster\n",
    "source_apikey = \"\"\n",
    "\n",
    "# Provide the source project ID\n",
    "deployment_id = \"\"\n",
    "\n",
    "#### Local save file path ####\n",
    "# Saves to the model directory by default, using the deployment_id as the file name\n",
    "model_path = \"model/%s.mlpkg\" % deployment_id\n",
    "\n",
    "#### Destination Settings ####\n",
    "# Example: destination_host = \"http://4.3.2.1\"\n",
    "destination_host = \"https://destination.datarobot.example.com\"\n",
    "\n",
    "# Provide an API key from the nodes referenced above from a user with the permissions referenced above\n",
    "destination_apikey = \"\"\n",
    "\n",
    "print(\"DataRobot client version: %s\" % dr.__version__)\n",
    "print(\"Source url: %s | deployment_id: %s\" % (source_host, deployment_id))\n",
    "print(\"Output path: %s\" % (model_path))\n",
    "print(\"Destinastion url: %s\" % (destination_host))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code block to ensure that the model directory exists\n",
    "os.makedirs(os.path.dirname(model_path), exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the deployed model package\n",
    "\n",
    "The following cell downloads the generated data that represents the deployed model given by the source URL and the deployment_id. It is then saved to `models/{deployment_id}.mlpkg`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the headers and provide the token\n",
    "headers = {}\n",
    "headers[\"Authorization\"] = \"Bearer {}\".format(source_apikey)\n",
    "# Optional - helps DataRobot track usage of this sample\n",
    "headers[\"User-Agent\"] = \"AIA-E2E-MIGRATION-19\"\n",
    "\n",
    "# Create a new session\n",
    "session = requests.Session()\n",
    "\n",
    "session.headers.update(headers)\n",
    "\n",
    "print(\"Downloading the mlpkg file from: %s\" % source_host)\n",
    "\n",
    "# Download Code\n",
    "# Makes request to generate an .mlpkg for download on the target server\n",
    "# Returns a URL in the location attribute in response header or None\n",
    "\n",
    "\n",
    "def _request_model_package_download(session, host, deployment_id):\n",
    "    apiEndpoint = urllib.parse.urljoin(\n",
    "        host, \"/api/v2/deployments/%s/modelPackageFileBuilds/\" % deployment_id\n",
    "    )\n",
    "    print(\"using download apiEndpoint: %s\" % apiEndpoint)\n",
    "\n",
    "    ssl_verify = True if (urllib.parse.urlparse(host)).scheme == \"https\" else False\n",
    "\n",
    "    try:\n",
    "        r = session.post(apiEndpoint, verify=ssl_verify)\n",
    "        r.raise_for_status()\n",
    "        return r.headers.get(\"Location\")\n",
    "    except requests.exceptions.HTTPError as err:\n",
    "        print(\"Error: %s\" % err)\n",
    "        return None\n",
    "\n",
    "\n",
    "# Downloads an .mlpkg file to the local system from the target server\n",
    "# Returns the binary data to be downloaded or None\n",
    "def get_model_package(session, host, deployment_id):\n",
    "    location = _request_model_package_download(session, host, deployment_id)\n",
    "    print(\"using location: %s\" % location)\n",
    "    ssl_verify = True if (urllib.parse.urlparse(host)).scheme == \"https\" else False\n",
    "    attempts = 0\n",
    "    wait_length = 30\n",
    "    r = None\n",
    "    while attempts <= 10:\n",
    "        try:\n",
    "            r = session.get(location, verify=ssl_verify)\n",
    "            r.raise_for_status()\n",
    "            print(r.json())\n",
    "            print(\"sleeping %s seconds\" % wait_length)\n",
    "            time.sleep(wait_length)\n",
    "            attempts += 1\n",
    "        except ValueError:\n",
    "            print(\"looks like no json, time to download\")\n",
    "            return r\n",
    "        except:\n",
    "            attempts += 1\n",
    "            print(\"exception, sleeping for 60 seconds\")\n",
    "            time.sleep(60)\n",
    "    print(\n",
    "        \"Number of check attempts exceeded. please check the target instance to see if the package is still being assembled or not\"\n",
    "    )\n",
    "    return None\n",
    "\n",
    "\n",
    "start = default_timer()\n",
    "\n",
    "output = get_model_package(session, source_host, deployment_id)\n",
    "\n",
    "# if output is None:\n",
    "#     print(\"download failed\")\n",
    "\n",
    "print(\"Saving data to: %s\" % model_path)\n",
    "\n",
    "with open(model_path, \"wb\") as f:\n",
    "    f.write(output.content)\n",
    "\n",
    "print(\n",
    "    \"%s took %s seconds to download %s megs\"\n",
    "    % (\n",
    "        model_path,\n",
    "        default_timer() - start,\n",
    "        str(round(os.path.getsize(model_path) / (1024 * 1024), 2)),\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload the model to the Model Registry\n",
    "\n",
    "The following cell uploads the .mlpkg file produced earlier to the `destination_host` provided above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {}\n",
    "headers[\"Authorization\"] = \"Bearer {}\".format(destination_apikey)\n",
    "# Optional - helps DataRobot track usage of this sample\n",
    "headers[\"User-Agent\"] = \"AIA-E2E-MIGRATION-19\"\n",
    "\n",
    "session = requests.Session()\n",
    "session.headers.update(headers)\n",
    "\n",
    "model_name = \"\"\n",
    "\n",
    "# Upload code\n",
    "# Makes a request to upload the .mlpkg file to the target server\n",
    "# Returns a URL in the location attribute of the response header or None\n",
    "\n",
    "\n",
    "def _request_package_upload(session, host, fileLocation):\n",
    "    apiEndpoint = urllib.parse.urljoin(host, \"/api/v2/modelPackages/fromFile/\")\n",
    "    print(\"using upload apiEndpoint: %s\" % apiEndpoint)\n",
    "\n",
    "    ssl_verify = True if (urllib.parse.urlparse(host)).scheme == \"https\" else False\n",
    "\n",
    "    f = {\"file\": open(fileLocation, \"rb\")}\n",
    "\n",
    "    try:\n",
    "        r = session.post(apiEndpoint, files=f, verify=ssl_verify)\n",
    "        r.raise_for_status()\n",
    "        return r.headers.get(\"Location\")\n",
    "    except requests.exceptions.HTTPError as err:\n",
    "        print(\"ERROR: %s\" % err)\n",
    "        return None\n",
    "\n",
    "\n",
    "# Uploads the .mlpkg file to the target server\n",
    "# Returns the ID of the new model package or None\n",
    "\n",
    "\n",
    "def upload_model_package(session, host, fileLocation):\n",
    "    location = _request_package_upload(session, host, fileLocation)\n",
    "    print(\"Location: %s\" % location)\n",
    "    ssl_verify = True if (urllib.parse.urlparse(host)).scheme == \"https\" else False\n",
    "\n",
    "    attempts = 0\n",
    "    wait_length = 25\n",
    "\n",
    "    while attempts < 10:\n",
    "        try:\n",
    "            r = session.get(location, verify=ssl_verify)\n",
    "            r.raise_for_status()\n",
    "            data = r.json()\n",
    "            # Check if you get a status or if it's redirected to the package object\n",
    "            if data.get(\"status\") is not None:\n",
    "                print(data)\n",
    "            else:\n",
    "                print(\"Model Package Uploaded\")\n",
    "                return data.get(\"id\"), data.get(\"importance\"), data.get(\"name\")\n",
    "            attempts += 1\n",
    "            print(\"sleeping %s seconds\" % wait_length)\n",
    "            time.sleep(wait_length)\n",
    "        except:\n",
    "            attempts += 1\n",
    "            print(\"exception, sleeping 60\")\n",
    "            time.sleep(60)\n",
    "\n",
    "    print(\n",
    "        \"ERROR: Number of check attempts exceeded. please check the target instance to see if there are errors\"\n",
    "    )\n",
    "    return None\n",
    "\n",
    "\n",
    "# Upload the .mlpkg\n",
    "start = default_timer()\n",
    "print(\"Uploading file: %s to: %s\" % (model_path, destination_host))\n",
    "\n",
    "(\n",
    "    destination_model_id,\n",
    "    destination_model_importance,\n",
    "    destination_model_name,\n",
    ") = upload_model_package(session, destination_host, model_path)\n",
    "\n",
    "if destination_model_id is None:\n",
    "    print(\"upload failed\")\n",
    "else:\n",
    "    link = urllib.parse.urljoin(\n",
    "        destination_host, \"/model-registry/model-packages/%s\" % destination_model_id\n",
    "    )\n",
    "    print(\"#----------------------------------------------------------------------\")\n",
    "    print(\"Click the link below to see your model in the model registry\")\n",
    "    display(HTML(\"\"\"<a href=\"{link}\">{link}</a>\"\"\".format(link=link)))\n",
    "    print(\"#----------------------------------------------------------------------\")\n",
    "    print(\"Upload took %s seconds\" % (default_timer() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find the dedicated prediction engine ID\n",
    "\n",
    "The next step is to find the prediction server used in the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpeEndpoint = \"%s/api/v2/predictionServers/\" % (destination_host)\n",
    "prediction_environment_id = None\n",
    "prediction_environment_url = None\n",
    "\n",
    "ssl_verify = (\n",
    "    True if (urllib.parse.urlparse(destination_host)).scheme == \"https\" else False\n",
    ")\n",
    "\n",
    "print(\"finding dpe with: %s\" % dpeEndpoint)\n",
    "try:\n",
    "    r = session.get(dpeEndpoint, verify=ssl_verify)\n",
    "    r.raise_for_status()\n",
    "\n",
    "    data = json.loads(r.text)\n",
    "\n",
    "    prediction_environment_id = data[\"data\"][data[\"count\"] - 1][\"id\"]\n",
    "    prediction_environment_url = data[\"data\"][data[\"count\"] - 1][\"url\"]\n",
    "\n",
    "except requests.exceptions.HTTPError as err:\n",
    "    print(\"Error: %s\" % err)\n",
    "    raise Exception(\"Error: %s\" % err)\n",
    "\n",
    "## Debug\n",
    "# print(\"data: %s\" % data )\n",
    "\n",
    "print(\n",
    "    \"Found DPE id: %s | url: %s\"\n",
    "    % (prediction_environment_id, prediction_environment_url)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a new deployment from the target model package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Returns Deployment ID or None\n",
    "\n",
    "\n",
    "def deploy_model(session, pid, mid, imp):\n",
    "    apiEndpoint = \"%s/api/v2/deployments/fromModelPackage/\" % destination_host\n",
    "    print(\"deploy from: %s\" % apiEndpoint)\n",
    "    ssl_verify = (\n",
    "        True if (urllib.parse.urlparse(destination_host)).scheme == \"https\" else False\n",
    "    )\n",
    "\n",
    "    body_payload = {\n",
    "        \"label\": \"%s\" % (destination_model_name),\n",
    "        \"description\": \"Cloned from: %s\" % (urllib.parse.urlparse(source_host).netloc),\n",
    "        \"modelPackageId\": mid,\n",
    "        \"importance\": imp,\n",
    "    }\n",
    "    print(\"deployment settings: %s\" % body_payload)\n",
    "\n",
    "    try:\n",
    "        r = session.post(\n",
    "            apiEndpoint,\n",
    "            data=json.dumps(body_payload),\n",
    "            headers={\"Content-Type\": \"application/json\", \"Accept\": \"application/json\"},\n",
    "            verify=ssl_verify,\n",
    "        )\n",
    "        r.raise_for_status()\n",
    "        return r.text\n",
    "    except requests.exceptions.HTTPError as err:\n",
    "        print(\"ERROR: %s\" % err)\n",
    "        print(r.text)\n",
    "        print(r.headers)\n",
    "        return None\n",
    "\n",
    "\n",
    "start = default_timer()\n",
    "\n",
    "if destination_model_importance is None:\n",
    "    destination_model_importance = \"LOW\"\n",
    "\n",
    "output = deploy_model(\n",
    "    session,\n",
    "    prediction_environment_id,\n",
    "    destination_model_id,\n",
    "    destination_model_importance,\n",
    ")\n",
    "\n",
    "\n",
    "print(\"#----------------------------------------------------------------------\")\n",
    "print(\"# Click the link below in the browser to see your newly deployed model.\")\n",
    "link = urllib.parse.urljoin(\n",
    "    destination_host, \"/deployments/%s/overview\" % ((json.loads(output))[\"id\"])\n",
    ")\n",
    "display(HTML(\"\"\"<a href=\"{link}\">{link}</a>\"\"\".format(link=link)))\n",
    "print(\"#----------------------------------------------------------------------\")\n",
    "print(\"Deplyment of: %s took: %s seconds\" % (output, default_timer() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataRobot v7.x extra steps\n",
    "\n",
    "As mentioned in the prerequisite section, there is an additional required process to finalize the migration. Upon executing the block below, you will have the commands required after SSHing in to the `destination_host`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Debug command\n",
    "# destination_model_id = \"foo\"\n",
    "app_node = (urllib.parse.urlparse(destination_host)).netloc\n",
    "\n",
    "print(\"# Copy the commands below and paste them to the\")\n",
    "print(\"# ssh command prompt on: %s\" % app_node)\n",
    "print(\"\")\n",
    "print(\"sudo su - datarobot\")\n",
    "print(\n",
    "    'docker exec -it app /entrypoint /bin/bash -c \"python3 support/upgrade_model_packages.py --save %s\"'\n",
    "    % destination_model_id\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upon successfu completion, you should see output like this:\n",
    "```bash\n",
    "Total seconds: 1.097603 | Avg 1.097603 seconds to process a package\n",
    "Successfully updated 1 packages\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Copyright 2023 DataRobot Inc. All Rights Reserved.\n",
    "\n",
    "**This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES**\n",
    "\n",
    "**OR CONDITIONS OF ANY KIND, express or implied**\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "7e1998ff7f8aa20ada591c520b972326324e5ea05489af9e422744c7c09f6dad"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}