examples/cv_mars_rover/Quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "270f6225",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-05T19:40:27.805732Z",
     "start_time": "2021-11-05T19:40:26.564946Z"
    }
   },
   "outputs": [],
   "source": [
    "# imports\n",
    "\n",
    "import os\n",
    "from datetime import date, datetime, timedelta\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "from util.constants import CLASS_LABELS, COLORS\n",
    "from util.data import load_inference_data, output_to_arthur_format, download_model, \\\n",
    "    download_inference_dataset, download_reference_data, plot_color_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f7894b2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-05T19:40:27.869519Z",
     "start_time": "2021-11-05T19:40:27.807911Z"
    }
   },
   "outputs": [],
   "source": [
    "# download ultralytics/yolov5 library\n",
    "\n",
    "from util.yololib import download_yolo_library\n",
    "\n",
    "download_yolo_library()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "255532b6",
   "metadata": {},
   "source": [
    "## The Data\n",
    "\n",
    "We'll download some image to run the model against. You can use a sample of images provided by Arthur, or download recent images from NASA."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc322df6",
   "metadata": {},
   "source": [
    "### Load Sample Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cc6eadb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-05T19:40:30.257356Z",
     "start_time": "2021-11-05T19:40:30.253407Z"
    }
   },
   "outputs": [],
   "source": [
    "# this will populate the 'api-data' folder with some sample images if none are present\n",
    "\n",
    "download_inference_dataset()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7513f7da",
   "metadata": {},
   "source": [
    "### Viewing the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "396f1b9a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-05T19:40:31.338400Z",
     "start_time": "2021-11-05T19:40:31.048084Z"
    }
   },
   "outputs": [],
   "source": [
    "SAMPLE_IMAGE_ID = \"877390\"\n",
    "\n",
    "sample_image_path = f\"./api-data/{SAMPLE_IMAGE_ID}/image.jpg\"\n",
    "sample_image = cv2.imread(sample_image_path)\n",
    "plt.imshow(sample_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7457869c",
   "metadata": {},
   "source": [
    "### Fetch New Mars Rover Images (Optional)\n",
    "\n",
    "Next we'll fetch recent images from the [NASA Mars Rover Photos API](https://api.nasa.gov/#mars-rover-photos). You will need a NASA API Key, which you can obtain through the free signup at the top of the page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "953fe94b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-05T19:40:34.050890Z",
     "start_time": "2021-11-05T19:40:34.049143Z"
    }
   },
   "outputs": [],
   "source": [
    "# Fill in your NASA API Key here to fetch fresh images\n",
    "\n",
    "# os.environ['NASA_API_KEY'] = \"<YOUR_API_KEY>\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7323a4fb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-05T19:40:34.591901Z",
     "start_time": "2021-11-05T19:40:34.589657Z"
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# this cell is tagged 'parameters' which allows us to parameterize it through Papermill\n",
    "\n",
    "lookback_days = 15\n",
    "mars_model_id = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bacd3e6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-04T15:00:02.457068Z",
     "start_time": "2021-11-04T14:59:52.313232Z"
    }
   },
   "outputs": [],
   "source": [
    "new_image_ids = None\n",
    "\n",
    "if 'NASA_API_KEY' in os.environ:\n",
    "    from util import api\n",
    "\n",
    "    # compute the specific days to query\n",
    "    today = date.today()\n",
    "    start_date = today - timedelta(days=lookback_days)\n",
    "\n",
    "    # download the images\n",
    "    new_image_ids = api.download_photos_in_day_range(start_date, today, camera=\"NAVCAM\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a1683e2",
   "metadata": {},
   "source": [
    "## The Model\n",
    "\n",
    "Next we'll load the YOLO Model and our predict function. YOLO is a popular model that performs well at object detection tasks: drawing bounding boxes around different types of objects. We'll be applying this to the Mars images by detecting different types of terrain in the Martian landscape."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cb46c94",
   "metadata": {},
   "source": [
    "### Load Model\n",
    "\n",
    "We'll download and load the PyTorch model from Arthur, pre-trained for the Martian Terrain object detection task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c526a6cb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-05T19:40:41.798871Z",
     "start_time": "2021-11-05T19:40:40.046545Z"
    }
   },
   "outputs": [],
   "source": [
    "# load the model\n",
    "\n",
    "from predict import MarsPredictor\n",
    "\n",
    "download_model()\n",
    "model = MarsPredictor(\"./model/model_weights.pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "676bde08",
   "metadata": {},
   "source": [
    "### Prediction Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4da431d3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-05T19:40:45.313377Z",
     "start_time": "2021-11-05T19:40:41.801360Z"
    }
   },
   "outputs": [],
   "source": [
    "# fetch prediction from model\n",
    "prediction = model.predict(sample_image_path, conf_thres=0.1)[0]\n",
    "\n",
    "# plot bounding boxes on image\n",
    "for bbox in prediction:\n",
    "    x_start, y_start, x_end, y_end, confidence, class_idx = bbox\n",
    "    class_label = CLASS_LABELS[int(class_idx)]\n",
    "    color = COLORS[class_label]\n",
    "    cv2.rectangle(sample_image, (int(x_start), int(y_start)), (int(x_end), int(y_end)),\n",
    "                  color, 10)\n",
    "\n",
    "plt.imshow(sample_image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "100e7dea",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-05T19:40:45.616740Z",
     "start_time": "2021-11-05T19:40:45.315584Z"
    }
   },
   "outputs": [],
   "source": [
    "# plot color key for box labels\n",
    "\n",
    "plot_color_key()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "088d5ba4",
   "metadata": {},
   "source": [
    "## Arthur Integration\n",
    "\n",
    "Next we'll onboard our model to the Arthur platform!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a1146a9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T17:59:06.913189Z",
     "start_time": "2021-10-27T17:59:06.909936Z"
    }
   },
   "outputs": [],
   "source": [
    "# Arthur API Tokens are too long for some AWS environment variable specs, so load one from a file if present\n",
    "#  (this section only applies to SageMaker deployments)\n",
    "if \"arthur-api-key.txt\" in os.listdir():\n",
    "    print(\"loading api key from file\")\n",
    "    with open(\"arthur-api-key.txt\", 'r') as f:\n",
    "        os.environ['ARTHUR_API_KEY'] = f.read().strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d480ab5c",
   "metadata": {},
   "source": [
    "### Model Registration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1bad3465",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-04T15:00:19.052555Z",
     "start_time": "2021-11-04T15:00:18.149909Z"
    }
   },
   "outputs": [],
   "source": [
    "# arthur imports\n",
    "\n",
    "from arthurai import ArthurAI\n",
    "from arthurai.common.constants import InputType, OutputType, Stage, ValueType, Enrichment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "db2aeaae",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-04T15:00:19.394190Z",
     "start_time": "2021-11-04T15:00:19.204632Z"
    }
   },
   "outputs": [],
   "source": [
    "# create a connection to the Arthur API\n",
    "# credentials are being passed to the client via environment variables\n",
    "\n",
    "arthur = ArthurAI()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00195ebe",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T20:30:04.997543Z",
     "start_time": "2021-11-01T20:30:04.952556Z"
    }
   },
   "outputs": [],
   "source": [
    "# define an Arthur Model object\n",
    "\n",
    "# define a unique model ID based on the current timestamp\n",
    "partner_model_id = f\"MarsRover-{datetime.now().strftime('%Y%m%d%H%M%S')}\"\n",
    "\n",
    "# plug in some basic metadata about our model: the input and output types as well as image dimensions\n",
    "arthur_model = arthur.model(partner_model_id = partner_model_id,\n",
    "                           display_name = \"Martian Terrain\",\n",
    "                           input_type = InputType.Image,\n",
    "                           output_type = OutputType.ObjectDetection,\n",
    "                           pixel_width = 1024,\n",
    "                           pixel_height = 1024)\n",
    "\n",
    "# add our input image attribute\n",
    "arthur_model.add_image_attribute(\"image\")\n",
    "\n",
    "# create prediction and ground truth columns for the detected objects\n",
    "# each row of these attributes will contain a list of bounding boxes along with their associted class\n",
    "predicted_attribute_name = \"objects_detected\"\n",
    "ground_truth_attribute_name = \"label\"\n",
    "arthur_model.add_object_detection_output_attributes(\n",
    "    predicted_attribute_name, \n",
    "    ground_truth_attribute_name, \n",
    "    CLASS_LABELS)\n",
    "\n",
    "# add additional metadata attributes that we'll supply with our inferences\n",
    "arthur_model.add_attribute(\"martian_sol\", stage=Stage.NonInputData,\n",
    "                           value_type=ValueType.Integer)\n",
    "arthur_model.add_attribute(\"image_id\", stage=Stage.NonInputData,\n",
    "                           value_type=ValueType.Integer)\n",
    "\n",
    "arthur_model.review()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6172cb9d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T20:30:29.940614Z",
     "start_time": "2021-11-01T20:30:20.411507Z"
    }
   },
   "outputs": [],
   "source": [
    "# if we specified a model ID through papermill above, we'll only be sending inferences to it and can \n",
    "#  fetch it to overwrite the unsaved model we just created\n",
    "\n",
    "if mars_model_id is None:\n",
    "    arthur_model.save()\n",
    "else:\n",
    "    # (this section only applies to SageMaker deployments)\n",
    "    arthur_model = arthur.get_model(mars_model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "764edb80",
   "metadata": {},
   "source": [
    "### Add Reference Dataset\n",
    "\n",
    "Next we'll download a reference dataset from Arthur. This includes a set of images and predictions that are used as a baseline for anomaly detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7370d9a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T20:31:54.142875Z",
     "start_time": "2021-11-01T20:30:38.058391Z"
    }
   },
   "outputs": [],
   "source": [
    "# if we just created the model (no previously-created model was passed in), also set the reference data\n",
    "\n",
    "if mars_model_id is None:\n",
    "    download_reference_data()\n",
    "    arthur_model.set_reference_data(directory_path=\"./reference-data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bc0aab5",
   "metadata": {},
   "source": [
    "### Making Predictions\n",
    "\n",
    "Finally we can use our loaded model and downloaded images to make predictions.\n",
    "\n",
    "We'll then send these inferences to the Arthur platform to register them with the model we created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfc3a9d1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-04T16:29:21.800758Z",
     "start_time": "2021-11-04T16:29:21.778461Z"
    }
   },
   "outputs": [],
   "source": [
    "inference_df = load_inference_data(new_image_ids)\n",
    "inference_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98b4f0de",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-04T16:29:40.982815Z",
     "start_time": "2021-11-04T16:29:22.514384Z"
    }
   },
   "outputs": [],
   "source": [
    "predictions = model.predict(list(inference_df['image']), conf_thres=0.1)\n",
    "inference_df['objects_detected'] = [output_to_arthur_format(x) for x in predictions]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38fd6e80",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-04T16:29:41.985002Z",
     "start_time": "2021-11-04T16:29:41.001384Z"
    }
   },
   "outputs": [],
   "source": [
    "inference_df.rename(columns={'date': 'inference_timestamp'}, inplace=True)\n",
    "inference_df['image_id'] = inference_df['image_id'].astype(int)\n",
    "\n",
    "arthur_model.send_inferences(inference_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "503e4b26",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f8d44f6b",
   "metadata": {},
   "source": [
    "## Explore Your Data in the Arthur UI\n",
    "\n",
    "Now that you've onboarded the model with Arthur, you can view the performance metrics and registered inferences in the platform. You can view the model's Mean Average Precision for the bounding boxes, detect drift through each image's Anomaly Score.\n",
    "\n",
    "Use your rich visual interface to easily explore the images your model evaluated with the predicted bounding boxes and ground truth bounding boxes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "26f65475",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### Visit [your Arthur Dashboard](https://dev-v3.arthur.ai)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from urllib.parse import urlparse\n",
    "from IPython.display import Markdown\n",
    "\n",
    "api_url = urlparse(arthur.client.api_base_url)\n",
    "dashboard_url = f\"{api_url.scheme}://{api_url.netloc}\"\n",
    "\n",
    "Markdown(f\"### Visit [your Arthur Dashboard]({dashboard_url})\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b18d8d3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}