 archived/3D-point-cloud-input-data-processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a 3D Point Cloud Labeling Job for Object Tracking with Amazon SageMaker Ground Truth\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "This notebook will demonstrate how you can pre-process your 3D point cloud input data to create an [object tracking labeling job](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-object-tracking.html) and include sensor and camera data for sensor fusion. \n",
    "\n",
    "In object tracking, you are tracking the movement of an object (e.g., a pedestrian on the side walk) while your point of reference (e.g., the autonomous vehicle) is moving. When performing object tracking, your data must be in a global reference coordinate system such as [world coordinate system](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-sensor-fusion-details.html#sms-point-cloud-world-coordinate-system) because the ego vehicle itself is moving in the world. You can transform point cloud data in local coordinates to the world coordinate system by multiplying each of the points in a 3D frame with the extrinsic matrix for the LiDAR sensor. \n",
    "\n",
    "In this notebook, you will transform 3D frames from a local coordinate system to a world coordinate system using extrinsic matrices. You will use the KITTI dataset<sup>[1](#The-Dataset-and-Input-Manifest-Files)</sup>, an open source autonomous driving dataset. The KITTI dataset provides an extrinsic matrix for each 3D point cloud frame. You will use [pykitti](https://github.com/utiasSTARS/pykitti) and the [numpy matrix multiplication function](https://numpy.org/doc/1.18/reference/generated/numpy.matmul.html) to multiple this matrix with each point in the frame to translate that point to the world coordinate system used by the KITTI dataset. \n",
    "\n",
    "You include camera image data and provide workers with more visual information about the scene they are labeling. Through sensor fusion, workers will be able to adjust labels in the 3D scene and in 2D images, and label adjustments will be mirrored in the other view. \n",
    "\n",
    "Ground Truth computes your sensor and camera extrinsic matrices for sensor fusion using sensor and camera **pose data** - position and heading. The KITTI raw dataset includes rotation matrix and translations vectors for extrinsic transformations for each frame. This notebook will demonstrate how you can extract **position** and **heading** from KITTI rotation matrices and translations vectors using [pykitti](https://github.com/utiasSTARS/pykitti).\n",
    "\n",
    "In summary, you will:\n",
    "* Convert a dataset to a world coordinate system.\n",
    "* Learn how you can extract pose data from your LiDAR and camera extrinsict matrices for sensor fusion.\n",
    "* Create a sequence input manifest file for an object tracking labeling job. \n",
    "* Create an object tracking labeling job.\n",
    "* Preview the worker UI and tools provided by Ground Truth.\n",
    "\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "To run this notebook, you can simply execute each cell in order. To understand what's happening, you'll need:\n",
    "* An S3 bucket you can write to -- please provide its name in `BUCKET`. The bucket must be in the same region as this SageMaker Notebook instance. You can also change the `EXP_NAME` to any valid S3 prefix. All the files related to this experiment will be stored in that prefix of your bucket. **Important: you must attach the CORS policy to this bucket. See the next section for more information**.\n",
    "* Familiarity with the [Ground Truth 3D Point Cloud Labeling Job](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud.html).\n",
    "* Familiarity with Python and [numpy](http://www.numpy.org/).\n",
    "* Basic familiarity with [AWS S3](https://docs.aws.amazon.com/s3/index.html).\n",
    "* Basic understanding of [AWS Sagemaker](https://aws.amazon.com/sagemaker/).\n",
    "* Basic familiarity with [AWS Command Line Interface (CLI)](https://aws.amazon.com/cli/) -- ideally, you should have it set up with credentials to access the AWS account you're running this notebook from.\n",
    "\n",
    "This notebook has only been tested on a SageMaker notebook instance. The runtimes given are approximate. We used an `ml.t2.medium` instance in our tests. However, you can likely run it on a local instance by first executing the cell below on SageMaker and then copying the `role` string to your local copy of the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMPORTANT: Attach CORS policy to your bucket\n",
    "\n",
    "You must attach the following CORS policy to your S3 bucket for the labeling task to render. To learn how to add a CORS policy to your S3 bucket, follow the instructions in [How do I add cross-domain resource sharing with CORS?](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/add-cors-configuration.html). Paste the following policy in the CORS configuration editor:\n",
    "\n",
    "```\n",
    "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
    "<CORSConfiguration xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\">\n",
    "<CORSRule>\n",
    "    <AllowedOrigin>*</AllowedOrigin>\n",
    "    <AllowedMethod>GET</AllowedMethod>\n",
    "    <AllowedMethod>HEAD</AllowedMethod>\n",
    "    <AllowedMethod>PUT</AllowedMethod>\n",
    "    <MaxAgeSeconds>3000</MaxAgeSeconds>\n",
    "    <ExposeHeader>Access-Control-Allow-Origin</ExposeHeader>\n",
    "    <AllowedHeader>*</AllowedHeader>\n",
    "</CORSRule>\n",
    "<CORSRule>\n",
    "    <AllowedOrigin>*</AllowedOrigin>\n",
    "    <AllowedMethod>GET</AllowedMethod>\n",
    "</CORSRule>\n",
    "</CORSConfiguration>\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install boto3==1.14.8\n",
    "!pip install -U botocore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import time\n",
    "import pprint\n",
    "import json\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from datetime import datetime, timezone\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "\n",
    "sagemaker_client = boto3.client(\"sagemaker\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET = \"\"\n",
    "EXP_NAME = \"\"  # Any valid S3 prefix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure the bucket is in the same region as this notebook.\n",
    "sess = sagemaker.session.Session()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.session.Session().region_name\n",
    "s3 = boto3.client(\"s3\")\n",
    "bucket_region = s3.head_bucket(Bucket=BUCKET)[\"ResponseMetadata\"][\"HTTPHeaders\"][\n",
    "    \"x-amz-bucket-region\"\n",
    "]\n",
    "assert (\n",
    "    bucket_region == region\n",
    "), \"Your S3 bucket {} and this notebook need to be in the same region.\".format(BUCKET)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Dataset and Input Manifest Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset and resources used in this notebook are located in the following Amazon S3 bucket: https://aws-ml-blog.s3.amazonaws.com/artifacts/gt-point-cloud-demos/.\n",
    "\n",
    "This bucket contains a single scene from the [KITTI datasets](http://www.cvlibs.net/datasets/kitti/raw_data.php). KITTI created datasets for computer vision and machine learning research, including for 2D and 3D object detection and object tracking. The datasets are captured by driving around the mid-size city of Karlsruhe, in rural areas and on highways.\n",
    "\n",
    "\\[1\\] The KITTI dataset is subject to its own license.  Please make sure that any use of the dataset conforms to the license terms and conditions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and unzip data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rm -rf sample_data*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://aws-ml-blog.s3.amazonaws.com/artifacts/gt-point-cloud-demos/sample_data.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!unzip -o sample_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the sample_data folder. You'll see that we have images which can be used for sensor fusion, and point cloud data in ASCII format (.txt files). We will use a script to convert this point cloud data from the LiDAR sensor's local coordinates to a world coordinate system. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls sample_data/2011_09_26/2011_09_26_drive_0005_sync/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls sample_data/2011_09_26/2011_09_26_drive_0005_sync/oxts/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the Kitti2GT script to convert the raw data to Ground Truth format\n",
    "\n",
    "You can use this script to do the following:\n",
    "* Transform the KITTI dataset with respect to the LIDAR sensor's orgin in the first frame as the world cooridinate system ( global frame of reference ), so that it can be consumed by SageMaker Ground Truth. \n",
    "* Extract pose data in world coordinate system using the camera and LiDAR extrinsic matrices. You will supply this pose data in your sequence file to enable sensor fusion.\n",
    "\n",
    "First, the script uses [pykitti](https://github.com/utiasSTARS/pykitti) python module to load the KITTI raw data and calibrations. Let's look at the two main data-transformation functions of the script:\n",
    "\n",
    "### Data Transformation to a World Coordinate System\n",
    "\n",
    "In general, multiplying a point in a LIDAR frame with a LIDAR extrinsic matrix transforms it into world coordinates.\n",
    " \n",
    "Using pykitti `dataset.oxts[i].T_w_imu` gives the lidar extrinsic transform for the `i`th frame. This matrix can be multiplied with the points of the frame to convert it to a world frame using the numpy matrix multiplication, function, [matmul](https://numpy.org/doc/1.18/reference/generated/numpy.matmul.html): `matmul(lidar_transform_matrix, points)`. Let's look at the function that performs this transformation:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transform points from lidar to global frame using lidar_extrinsic_matrix\n",
    "def generate_transformed_pcd_from_point_cloud(points, lidar_extrinsic_matrix):\n",
    "    tps = []\n",
    "    for point in points:\n",
    "        transformed_points = np.matmul(\n",
    "            lidar_extrinsic_matrix,\n",
    "            np.array([point[0], point[1], point[2], 1], dtype=np.float32).reshape(4, 1),\n",
    "        ).tolist()\n",
    "        if len(point) > 3 and point[3] is not None:\n",
    "            tps.append(\n",
    "                [\n",
    "                    transformed_points[0][0],\n",
    "                    transformed_points[1][0],\n",
    "                    transformed_points[2][0],\n",
    "                    point[3],\n",
    "                ]\n",
    "            )\n",
    "\n",
    "    return tps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your point cloud data includes more than four elements for each point, for example, (x,y,z) and r,g,b, modify the `if` statement in the function above to ensure your r, g, b values are copied. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting Pose Data from LiDAR and Camera Extrinsic for Sensor Fusion\n",
    "\n",
    "\n",
    "For sensor fusion, you provide your extrinsic matrix in the form of sensor-pose in terms of origin position (for translation) and heading in quaternion (for rotation of the 3 axis). The following is an example of the pose JSON you use in the sequence file. \n",
    "\n",
    "```\n",
    "\n",
    "{\n",
    "    \"position\": {\n",
    "        \"y\": -152.77584902657554,\n",
    "        \"x\": 311.21505956090624,\n",
    "        \"z\": -10.854137529636024\n",
    "      },\n",
    "      \"heading\": {\n",
    "        \"qy\": -0.7046155108831117,\n",
    "        \"qx\": 0.034278837280808494,\n",
    "        \"qz\": 0.7070617895701465,\n",
    "        \"qw\": -0.04904659893885366\n",
    "      }\n",
    "}\n",
    "```\n",
    "\n",
    "All of the positional coordinates (x, y, z) are in meters. All the pose headings (qx, qy, qz, qw) are measured in Spatial Orientation in Quaternion. Separately for each camera, you provide pose data extracted from the extrinsic of that camera.\n",
    "\n",
    "Both LIDAR sensors and and cameras have their own extrinsic matrices, and they are used by SageMaker Ground Truth to enable the sensor fusion feature. In order to project a label from 3D point cloud to camera image plane Ground Truth needs to transform 3D points from LIDAR\u2019s own coordinate system to the camera\u2019s coordinate system. This is typically done by first transforming 3D points from LIDAR\u2019s own coordinate to a world coordinate system using the LIDAR extrinsic matrix. Then we use the camera inverse extrinsic (world to camera) to transform the 3D points from the world coordinate system we obtained in previous step into camera image plane. If your 3D data is already transformed into world coordinate system then the first transformation doesn\u2019t have any impact and label translation depends only on the camera extrinsic.\n",
    "\n",
    "If you have a rotation matrix (made up of the axis rotations) and translation vector (or origin) in world coordinate system instead of a single 4x4 rigid transformation matrix, then you can directly use rotation and translation to compute pose. For example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m pip install --user numpy scipy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "rotation = [\n",
    "    [9.96714314e-01, -8.09890350e-02, 1.16333982e-03],\n",
    "    [8.09967396e-02, 9.96661051e-01, -1.03090934e-02],\n",
    "    [-3.24531964e-04, 1.03694477e-02, 9.99946183e-01],\n",
    "]\n",
    "\n",
    "origin = [1.71104606e00, 5.80000039e-01, 9.43144935e-01]\n",
    "\n",
    "\n",
    "from scipy.spatial.transform import Rotation as R\n",
    "\n",
    "# position is the origin\n",
    "position = origin\n",
    "r = R.from_matrix(np.asarray(rotation))\n",
    "# heading in WCS using scipy\n",
    "heading = r.as_quat()\n",
    "print(f\"position:{position}\\nheading: {heading}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you indeed have a 4x4 extrinsic transformation matrix then the transformation matrix is just in the form of ```[R T; 0 0 0 1]``` where R is the rotation matrix and T is the origin translation vector. That means you can extract rotation matrix and translation vector from the transformation matrix as follows "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "transformation = [\n",
    "    [9.96714314e-01, -8.09890350e-02, 1.16333982e-03, 1.71104606e00],\n",
    "    [8.09967396e-02, 9.96661051e-01, -1.03090934e-02, 5.80000039e-01],\n",
    "    [-3.24531964e-04, 1.03694477e-02, 9.99946183e-01, 9.43144935e-01],\n",
    "    [0, 0, 0, 1],\n",
    "]\n",
    "\n",
    "transformation = np.array(transformation)\n",
    "rotation = transformation[0:3, 0:3]\n",
    "origin = transformation[0:3, 3]\n",
    "\n",
    "from scipy.spatial.transform import Rotation as R\n",
    "\n",
    "# position is the origin\n",
    "position = origin\n",
    "r = R.from_matrix(np.asarray(rotation))\n",
    "# heading in WCS using scipy\n",
    "heading = r.as_quat()\n",
    "print(f\"position:{position}\\nheading: {heading}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For convenience, in this blog you will use [pykitti](https://github.com/utiasSTARS/pykitti) development kit to load the raw data and calibrations. With pykitti you will extract sensor pose in the world coordinate system from KITTI extrinsic which is provided as a rotation matrix and translation vector in the raw calibrations data. We will then format this pose data using the JSON format required for the 3D point cloud sequence input manifest.  \n",
    "\n",
    "With pykitti the ```dataset.oxts[i].T_w_imu``` gives the LiDAR extrinsic matrix ( lidar_extrinsic_transform ) for the i'th frame. Similarly, with pykitti the camera extrinsic matrix ( camera_extrinsic_transform  ) for cam0 in i'th frame can be calculated by ```inv(matmul(dataset.calib.T_cam0_velo, inv(dataset.oxts[i].T_w_imu)))``` and this can be converted into heading and position for cam0.\n",
    "\n",
    "In the script, the following functions are used to extract this pose data from the LiDAR extrinsict and camera inverse extrinsic matrices. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# utility to convert extrinsic matrix to pose heading quaternion and position\n",
    "def convert_extrinsic_matrix_to_trans_quaternion_mat(lidar_extrinsic_transform):\n",
    "    position = lidar_extrinsic_transform[0:3, 3]\n",
    "    rot = np.linalg.inv(lidar_extrinsic_transform[0:3, 0:3])\n",
    "    quaternion = R.from_matrix(np.asarray(rot)).as_quat()\n",
    "    trans_quaternions = {\n",
    "        \"translation\": {\"x\": position[0], \"y\": position[1], \"z\": position[2]},\n",
    "        \"rotation\": {\n",
    "            \"qx\": quaternion[0],\n",
    "            \"qy\": quaternion[1],\n",
    "            \"qz\": quaternion[2],\n",
    "            \"qw\": quaternion[3],\n",
    "        },\n",
    "    }\n",
    "    return trans_quaternions\n",
    "\n",
    "\n",
    "def convert_camera_inv_extrinsic_matrix_to_trans_quaternion_mat(camera_extrinsic_transform):\n",
    "    position = camera_extrinsic_transform[0:3, 3]\n",
    "    rot = np.linalg.inv(camera_extrinsic_transform[0:3, 0:3])\n",
    "    quaternion = R.from_matrix(np.asarray(rot)).as_quat()\n",
    "    trans_quaternions = {\n",
    "        \"translation\": {\"x\": position[0], \"y\": position[1], \"z\": position[2]},\n",
    "        \"rotation\": {\n",
    "            \"qx\": quaternion[0],\n",
    "            \"qy\": quaternion[1],\n",
    "            \"qz\": quaternion[2],\n",
    "            \"qw\": -quaternion[3],\n",
    "        },\n",
    "    }\n",
    "    return trans_quaternions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a Sequence File\n",
    "\n",
    "\n",
    "After you've converted your data to a world coordinate system and extracted sensor and camera pose data for sensor fusion, you can create a sequence file. This is accomplished with the function `convert_to_gt` in the python script.\n",
    "\n",
    "A **sequence** specifies a temporal series of point cloud frames. When a task is created using a sequence file, all point cloud frames in the sequence are sent to a worker to label. Your input manifest file will contain a single sequence per line. To learn more about the sequence input manifest format, see [Create a Point Cloud Frame Sequence Input Manifest](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-multi-frame-input-data.html).\n",
    "\n",
    "\n",
    "If you want to use this script to create a frame input manifest file, which is required for 3D point cloud object tracking and semantic segmentation labeling jobs, you can modify the for-loop in the function `convert_to_gt`\n",
    "to produce the required content for `source-ref-metadata`. To learn more about the frame input manifest format, see [Create a Point Cloud Frame Input Manifest File](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-single-frame-input-data.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's download the script and run it on the KITTI dataset to process the data you'll use for your labeling job. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!wget https://aws-ml-blog.s3.amazonaws.com/artifacts/gt-point-cloud-demos/kitti2gt.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize kitti2gt.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install pykitti"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pykitti"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from kitti2gt import *\n",
    "\n",
    "if EXP_NAME == \"\":\n",
    "    s3loc = f\"s3://{BUCKET}/frames/\"\n",
    "else:\n",
    "    s3loc = f\"s3://{BUCKET}/{EXP_NAME}/frames/\"\n",
    "\n",
    "convert_to_gt(\n",
    "    basedir=\"sample_data\",\n",
    "    date=\"2011_09_26\",\n",
    "    drive=\"0005\",\n",
    "    output_base=\"sample_data_out\",\n",
    "    s3prefix=s3loc,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following folders that will contain the data you'll use for the labeling job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!ls sample_data_out/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls sample_data_out/frames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you'll upload the data to your bucket in S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "if(EXP_NAME == ''):\n",
    "    !aws s3 cp sample_data_out/kitti-gt-seq.json s3://{BUCKET}/\n",
    "else:\n",
    "    !aws s3 cp sample_data_out/kitti-gt-seq.json s3://{BUCKET}/{EXP_NAME}/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if(EXP_NAME == ''):\n",
    "    !aws s3 sync sample_data_out/frames/ s3://{BUCKET}/frames/\n",
    "else:\n",
    "    !aws s3 sync sample_data_out/frames s3://{BUCKET}/{EXP_NAME}/frames/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if(EXP_NAME == ''):\n",
    "    !aws s3 sync sample_data_out/images/ s3://{BUCKET}/frames/images/\n",
    "else:\n",
    "    !aws s3 sync sample_data_out/images s3://{BUCKET}/{EXP_NAME}/frames/images/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write and Upload Multi-Frame Input Manifest File\n",
    "\n",
    "Now, let's create a **sequence input manifest file**. Each line in the input manifest (in this demo, there is only one) will point to a sequence file in your S3 bucket, `BUCKET/EXP_NAME`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"manifest.json\", \"w\") as f:\n",
    "    if EXP_NAME == \"\":\n",
    "        json.dump({\"source-ref\": \"s3://{}/kitti-gt-seq.json\".format(BUCKET)}, f)\n",
    "    else:\n",
    "        json.dump({\"source-ref\": \"s3://{}/{}/kitti-gt-seq.json\".format(BUCKET, EXP_NAME)}, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our manifest file is one line long, and identifies a single sequence file in your S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat manifest.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "if(EXP_NAME == ''):\n",
    "    !aws s3 cp manifest.json s3://{BUCKET}/\n",
    "    input_manifest_s3uri = f's3://{BUCKET}/manifest.json'\n",
    "else:\n",
    "    !aws s3 cp manifest.json s3://{BUCKET}/{EXP_NAME}/\n",
    "    input_manifest_s3uri = f's3://{BUCKET}/{EXP_NAME}/manifest.json'\n",
    "input_manifest_s3uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Labeling Job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following cell, we specify object tracking as our [3D Point Cloud Task Type](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-task-types.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "task_type = \"3DPointCloudObjectTracking\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify Resources for Labeling Job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specify Human Task UI ARN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following will be used to identify the HumanTaskUiArn. When you create a 3D point cloud labeling job, Ground Truth provides a worker UI that is specific to your task type. You can learn more about this UI and the assistive labeling tools that Ground Truth provides for Object Tracking on the [Object Tracking task type page](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-object-tracking.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Set up human_task_ui_arn map, to be used in case you chose UI_CONFIG_USE_TASK_UI_ARN\n",
    "## Supported for GA\n",
    "## Set up human_task_ui_arn map, to be used in case you chose UI_CONFIG_USE_TASK_UI_ARN\n",
    "human_task_ui_arn = (\n",
    "    f\"arn:aws:sagemaker:{region}:394669845002:human-task-ui/PointCloudObjectTracking\"\n",
    ")\n",
    "human_task_ui_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label Category Configuration File\n",
    "\n",
    "Your label category configuration file is used to specify labels, or classes, for your labeling job.\n",
    "\n",
    "When you use the object detection or object tracking task types, you can also include **label attributes** in your [label category configuration file](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-label-category-config.html). Workers can assign one or more attributes you provide to annotations to give more information about that object. For example, you may want to use the attribute *occluded* to have workers identify when an object is partially obstructed. \n",
    "\n",
    "Let's look at an example of the label category configuration file for an object detection or object tracking labeling job. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://aws-ml-blog.s3.amazonaws.com/artifacts/gt-point-cloud-demos/label-category-config/label-category.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"label-category.json\", \"r\") as j:\n",
    "    json_data = json.load(j)\n",
    "    print(\n",
    "        \"\\nA label category configuration file: \\n\\n\",\n",
    "        json.dumps(json_data, indent=4, sort_keys=True),\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if(EXP_NAME == ''):\n",
    "    !aws s3 cp label-category.json s3://{BUCKET}/label-category.json\n",
    "    label_category_config_s3uri = f's3://{BUCKET}/label-category.json'\n",
    "else:\n",
    "    !aws s3 cp label-category.json s3://{BUCKET}/{EXP_NAME}/label-category.json\n",
    "    label_category_config_s3uri = f's3://{BUCKET}/{EXP_NAME}/label-category.json'\n",
    "label_category_config_s3uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To learn more about the label category configuration file, see [Create a Label Category Configuration File](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-label-category-config.html)\n",
    "\n",
    "Run the following cell to identify the labeling category configuration file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up a private work team\n",
    "\n",
    "If you want to preview the worker task UI, create a private work team and add yourself as a worker. \n",
    "\n",
    "If you have already created a private workforce, follow the instructions in [Add or Remove Workers](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management-private-console.html#add-remove-workers-sm) to add yourself to the work team you use to create a lableing job. \n",
    "\n",
    "#### Create a private workforce and add yourself as a worker\n",
    "\n",
    "To create and manage your private workforce, you can use the **Labeling workforces** page in the Amazon SageMaker console. When following the instructions below, you will have the option to create a private workforce by entering worker emails or importing a pre-existing workforce from an Amazon Cognito user pool. To import a workforce, see [Create a Private Workforce (Amazon Cognito Console)](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-create-private-cognito.html).\n",
    "\n",
    "To create a private workforce using worker emails:\n",
    "\n",
    "* Open the Amazon SageMaker console at https://console.aws.amazon.com/sagemaker/.\n",
    "\n",
    "* In the navigation pane, choose **Labeling workforces**.\n",
    "\n",
    "* Choose Private, then choose **Create private team**.\n",
    "\n",
    "* Choose **Invite new workers by email**.\n",
    "\n",
    "* Paste or type a list of up to 50 email addresses, separated by commas, into the email addresses box.\n",
    "\n",
    "* Enter an organization name and contact email.\n",
    "\n",
    "* Optionally choose an SNS topic to subscribe the team to so workers are notified by email when new Ground Truth labeling jobs become available. \n",
    "\n",
    "* Click the **Create private team** button.\n",
    "\n",
    "After you import your private workforce, refresh the page. On the Private workforce summary page, you'll see your work team ARN. Enter this ARN in the following cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##Use Beta Private Team till GA\n",
    "workteam_arn = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task Time Limits\n",
    "\n",
    "3D point cloud annotation jobs can take workers hours. Workers will be able to save their work as they go, and complete the task in multiple sittings. Ground Truth will also automatically save workers' annotations periodically as they work. \n",
    "\n",
    "When you configure your task, you can set the total amount of time that workers can work on each task when you create a labeling job using `TaskTimeLimitInSeconds`. The maximum time you can set for workers to work on tasks is 7 days. The default value is 3 days. It is recommended that you create labeling tasks that can be completed within 12 hours. \n",
    "\n",
    "If you set `TaskTimeLimitInSeconds` to be greater than 8 hours, you must set `MaxSessionDuration` for your IAM execution to at least 8 hours. To update your execution role's `MaxSessionDuration`, use [UpdateRole](https://docs.aws.amazon.com/IAM/latest/APIReference/API_UpdateRole.html) or use the [IAM console](https://docs.aws.amazon.com/IAM/latest/UserGuide/roles-managingrole-editing-console.html#roles-modify_max-session-duration). You an identify the name of your role at the end of your role ARN. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# See your execution role ARN. The role name is located at the end of the ARN.\n",
    "role"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ac_arn_map = {\n",
    "    \"us-west-2\": \"081040173940\",\n",
    "    \"us-east-1\": \"432418664414\",\n",
    "    \"us-east-2\": \"266458841044\",\n",
    "    \"eu-west-1\": \"568282634449\",\n",
    "    \"ap-northeast-1\": \"477331159723\",\n",
    "}\n",
    "\n",
    "prehuman_arn = \"arn:aws:lambda:{}:{}:function:PRE-{}\".format(region, ac_arn_map[region], task_type)\n",
    "acs_arn = \"arn:aws:lambda:{}:{}:function:ACS-{}\".format(region, ac_arn_map[region], task_type)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set Up HumanTaskConfig\n",
    "\n",
    "`HumanTaskConfig` is used to specify your work team, and configure your labeling job tasks. Modify the following cell to identify a `task_description`, `task_keywords`, `task_title`, and `job_name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "## Set up Human Task Config\n",
    "\n",
    "## Modify the following\n",
    "task_description = \"add a task description here\"\n",
    "# example keywords\n",
    "task_keywords = [\"lidar\", \"pointcloud\"]\n",
    "# add a task title\n",
    "task_title = \"Add a Task Title Here - This is Displayed to Workers\"\n",
    "# add a job name to identify your labeling job\n",
    "job_name = \"add-job-name\"\n",
    "\n",
    "human_task_config = {\n",
    "    \"AnnotationConsolidationConfig\": {\n",
    "        \"AnnotationConsolidationLambdaArn\": acs_arn,\n",
    "    },\n",
    "    \"UiConfig\": {\n",
    "        \"HumanTaskUiArn\": human_task_ui_arn,\n",
    "    },\n",
    "    \"WorkteamArn\": workteam_arn,\n",
    "    \"PreHumanTaskLambdaArn\": prehuman_arn,\n",
    "    \"MaxConcurrentTaskCount\": 200,  # 200 images will be sent at a time to the workteam.\n",
    "    \"NumberOfHumanWorkersPerDataObject\": 1,  # One worker will work on each task\n",
    "    \"TaskAvailabilityLifetimeInSeconds\": 18000,  # Your workteam has 5 hours to complete all pending tasks.\n",
    "    \"TaskDescription\": task_description,\n",
    "    \"TaskKeywords\": task_keywords,\n",
    "    \"TaskTimeLimitInSeconds\": 3600,  # Each seq must be labeled within 1 hour.\n",
    "    \"TaskTitle\": task_title,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json.dumps(human_task_config, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up Create Labeling Request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following formats your labeling job request. For Object Tracking task types, the `LabelAttributeName` must end in `-ref`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if EXP_NAME == \"\":\n",
    "    s3_output_path = f\"s3://{BUCKET}\"\n",
    "else:\n",
    "    s3_output_path = f\"s3://{BUCKET}/{EXP_NAME}\"\n",
    "s3_output_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Set up Create Labeling Request\n",
    "\n",
    "labelAttributeName = job_name + \"-ref\"\n",
    "\n",
    "if (\n",
    "    task_type == \"3DPointCloudObjectDetection\"\n",
    "    or task_type == \"Adjustment3DPointCloudObjectDetection\"\n",
    "):\n",
    "    labelAttributeName = job_name\n",
    "\n",
    "\n",
    "ground_truth_request = {\n",
    "    \"InputConfig\": {\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"ManifestS3Uri\": input_manifest_s3uri,\n",
    "            }\n",
    "        },\n",
    "        \"DataAttributes\": {\n",
    "            \"ContentClassifiers\": [\"FreeOfPersonallyIdentifiableInformation\", \"FreeOfAdultContent\"]\n",
    "        },\n",
    "    },\n",
    "    \"OutputConfig\": {\n",
    "        \"S3OutputPath\": s3_output_path,\n",
    "    },\n",
    "    \"HumanTaskConfig\": human_task_config,\n",
    "    \"LabelingJobName\": job_name,\n",
    "    \"RoleArn\": role,\n",
    "    \"LabelAttributeName\": labelAttributeName,\n",
    "    \"LabelCategoryConfigS3Uri\": label_category_config_s3uri,\n",
    "}\n",
    "\n",
    "print(json.dumps(ground_truth_request, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Call CreateLabelingJob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_client.create_labeling_job(**ground_truth_request)\n",
    "print(f\"Labeling Job Name: {job_name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check Status of Labeling Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## call describeLabelingJob\n",
    "describeLabelingJob = sagemaker_client.describe_labeling_job(LabelingJobName=job_name)\n",
    "print(describeLabelingJob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start Working on tasks\n",
    "\n",
    "When you add yourself to a private work team, you recieve an email invitation to access the worker portal that looks similar to this [image](https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2020/04/16/a2i-critical-documents-26.gif). Use this invitation to sign in to the protal and view your 3D point cloud annotation tasks. Tasks may take up to 10 minutes to show up the worker portal. \n",
    "\n",
    "Once you are done working on the tasks, click **Submit**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Output Data\n",
    "\n",
    "Once you have completed all of the tasks, you can view your output data in the S3 location you specified in `OutputConfig`. \n",
    "\n",
    "To read more about Ground Truth output data format for your task type, see [Output Data](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-data-output.html#sms-output-point-cloud-object-tracking). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acknowledgments\n",
    "\n",
    "We would like to thank the KITTI team for letting us use this dataset to demonstrate how to prepare your 3D point cloud data for use in SageMaker Ground Truth."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/ground_truth_labeling_jobs|3d_point_cloud_input_data_processing|3D-point-cloud-input-data-processing.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}