 archived/ground_truth_annotation_dense_point_cloud_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "motivated-cattle",
   "metadata": {},
   "source": [
    "## Annotation of Dense Point Clouds Using Amazon SageMaker Ground Truth\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "motivated-cattle",
   "metadata": {},
   "source": [
    "\n",
    "This notebook walks through how to annotate dense point clouds using SageMaker Ground Truth. We define dense point clouds as point clouds with a high number of points per scene (> 500k) and have enough points relative to the feature size of a scene that labelers can still make out objects in the scene (cars, pedestrians, etc) while only looking at a subset of the scene's total points. When these conditions are met, you can often improve the labeling experience and throughput by reducing your scene's point cloud size through downsampling techniques. After labeling, you can either directly use labels on the full-size scene for some modalities (like object detection bounding boxes), or you can \"upsample\" your labels by doing some in-sample prediction of the unlabeled points.\n",
    "\n",
    "Recommended Kernel: `conda_python3`\n",
    "\n",
    "Let's start by importing required libraries and initializing session and other variables used in this notebook. By default, the notebook uses the default Amazon S3 bucket in the same AWS Region you use to run this notebook. If you want to use a different S3 bucket, make sure it is in the same AWS Region you use to complete this tutorial, and specify the bucket name for `bucket`. It is also recommended to use an instance size with at least 16 GB of RAM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "purple-community",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pylab inline\n",
    "import boto3\n",
    "import json\n",
    "import uuid\n",
    "import zlib\n",
    "from array import array\n",
    "\n",
    "import sagemaker as sm\n",
    "import scipy.stats\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from mpl_toolkits.mplot3d import Axes3D"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "serious-tumor",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "You will create some of the resources you need to launch a Ground Truth audit labeling job in this notebook. You must create the following resources before executing this notebook:\n",
    "\n",
    "* A work team. A work team is a group of workers that complete labeling tasks. If you want to preview the worker UI and execute the labeling task you will need to create a private work team, add yourself as a worker to this team, and provide the work team ARN below. This [GIF](images/create-workteam-loop.gif) demonstrates how to quickly create a private work team on the Amazon SageMaker console. To learn more about private, vendor, and Amazon Mechanical Turk workforces, see [Create and Manage Workforces\n",
    "](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "stable-prague",
   "metadata": {},
   "outputs": [],
   "source": [
    "WORKTEAM_ARN = \"<<ADD WORK TEAM ARN HERE>>\"\n",
    "\n",
    "# Make sure workteam arn is populated if private work team is chosen\n",
    "assert WORKTEAM_ARN != \"<<ADD WORK TEAM ARN HERE>>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stretch-principle",
   "metadata": {},
   "source": [
    "* The IAM execution role you used to create this notebook instance must have the following permissions:\n",
    "    * If you do not require granular permissions for your use case, you can attach [AmazonSageMakerFullAccess](https://console.aws.amazon.com/iam/home?#/policies/arn:aws:iam::aws:policy/AmazonSageMakerFullAccess) to your IAM user or role. If you are running this example in a SageMaker notebook instance, this is the IAM execution role used to create your notebook instance.If you need granular permissions, please see [Assign IAM Permissions to Use Ground Truth](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-security-permission.html#sms-security-permissions-get-started) for granular policy to use Ground Truth.\n",
    "    * AWS managed policy [AmazonSageMakerGroundTruthExecution](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AmazonSageMakerGroundTruthExecution). Run the following code-block to see your IAM execution role name. This [GIF](images/add-policy-loop.gif) demonstrates how to attach this policy to an IAM role in the IAM console. You can also find instructions in the IAM User Guide: [Adding and removing IAM identity permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html#add-policies-console).\n",
    "    * When you create your role, you specify Amazon S3 permissions. Make sure that your IAM role has access to the S3 bucket that you plan to use in this example. If you do not specify an S3 bucket in this notebook, the default bucket in the AWS region you are running this notebook instance will be used. If you do not require granular permissions, you can attach [AmazonS3FullAccess](https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AmazonS3FullAccess) to your role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "continuing-bahamas",
   "metadata": {},
   "outputs": [],
   "source": [
    "role = sm.get_execution_role()\n",
    "role_name = role.split(\"/\")[-1]\n",
    "print(\n",
    "    \"IMPORTANT: Make sure this IAM role has one or more IAM policies with the permissions described above attached.\"\n",
    ")\n",
    "print(\"********************************************************************************\")\n",
    "print(\"The IAM execution role name:\", role_name)\n",
    "print(\"The IAM execution role ARN:\", role)\n",
    "print(\"********************************************************************************\")\n",
    "\n",
    "sagemaker_cl = boto3.client(\"sagemaker\")\n",
    "# Make sure the bucket is in the same region as this notebook.\n",
    "bucket = \"<< YOUR S3 BUCKET NAME >>\"\n",
    "\n",
    "sm_session = sm.Session()\n",
    "s3 = boto3.client(\"s3\")\n",
    "\n",
    "if bucket == \"<< YOUR S3 BUCKET NAME >>\":\n",
    "    bucket = sm_session.default_bucket()\n",
    "region = boto3.session.Session().region_name\n",
    "bucket_region = s3.head_bucket(Bucket=bucket)[\"ResponseMetadata\"][\"HTTPHeaders\"][\n",
    "    \"x-amz-bucket-region\"\n",
    "]\n",
    "assert (\n",
    "    bucket_region == region\n",
    "), f\"Your S3 bucket {bucket} and this notebook need to be in the same region.\"\n",
    "print(f\"IMPORTANT: make sure the role {role_name} has the access to read and write to this bucket.\")\n",
    "print(\n",
    "    \"********************************************************************************************************\"\n",
    ")\n",
    "print(f\"This notebook will use the following S3 bucket: {bucket}\")\n",
    "print(\n",
    "    \"********************************************************************************************************\"\n",
    ")\n",
    "\n",
    "print(\n",
    "    \"If you don't have a CORS policy attached, you will be unable to label data in the task, use the following policy.\"\n",
    ")\n",
    "print(\n",
    "    \"\"\"\n",
    "[\n",
    "    {\n",
    "        \"AllowedHeaders\": [\n",
    "            \"*\"\n",
    "        ],\n",
    "        \"AllowedMethods\": [\n",
    "            \"GET\",\n",
    "            \"HEAD\",\n",
    "            \"PUT\"\n",
    "        ],\n",
    "        \"AllowedOrigins\": [\n",
    "            \"*\"\n",
    "        ],\n",
    "        \"ExposeHeaders\": [\n",
    "            \"Access-Control-Allow-Origin\"\n",
    "        ],\n",
    "        \"MaxAgeSeconds\": 3000\n",
    "    }\n",
    "]\n",
    "\"\"\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "considerable-holocaust",
   "metadata": {},
   "source": [
    "## Section 1: Download Data\n",
    "\n",
    "You can download from the following location. There is an object called `rooftop_12_49_41.xyz` that you will visualize, downsample, and then label. This data is a scan of an apartment building rooftop generated using the 3d Scanner App on an iPhone12 Pro. The app allows users to scan a given area and then export a pointcloud file. In this case the pointcloud data is in xyzrgb format, an accepted format for SageMaker Ground Truth point cloud, for more information on the data types allowed in SageMaker Ground Truth point cloud refer to the following documentation:\n",
    "\n",
    "https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-raw-data-types.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "orange-willow",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://aws-ml-blog.s3.amazonaws.com/artifacts/annotating-dense-point-cloud-data/rooftop_12_49_41.xyz -O pointcloud.xyz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caring-association",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's read our dataset into a numpy file, we'll call our pointclouds \"pc\" for short.\n",
    "# Our point cloud consists of 3D euclidean coordinates of each point and RGB color values for each point.\n",
    "# For example:\n",
    "# p1_x, p1_y, p1_z, p1_r, p1_g, p1_b\n",
    "# p2_x, p2_y, p2_z, p2_r, p2_g, p2_b\n",
    "# p3_x, p3_y, p3_z, p3_r, p3_g, p3_b\n",
    "# ...\n",
    "pc = np.loadtxt(\"pointcloud.xyz\", delimiter=\",\")\n",
    "\n",
    "print(f\"Loaded points of shape {pc.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "clinical-disney",
   "metadata": {},
   "source": [
    "### Visualize our pointcloud\n",
    "\n",
    "You can visualize the point cloud using [matplotlib's scatter3d](https://matplotlib.org/stable/gallery/mplot3d/scatter3d.html) function. The pointcloud file contains all of the correct points but in this case is not rotated correctly. You can rotate the object around it's axis by multiplying the pointcloud by a rotation matrix. You can obtain a rotation matrix using [scipy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.transform.Rotation.html) and specifying the degree changes you would like to make to each axis using the from_euler method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vertical-lighting",
   "metadata": {},
   "outputs": [],
   "source": [
    "# playing with view of 3D scene\n",
    "\n",
    "from scipy.spatial.transform import Rotation\n",
    "\n",
    "\n",
    "def plot_pointcloud(\n",
    "    pc, rot=[[0, 0, 0]], color=True, title=\"Simple Downsampling 1\", figsize=(50, 25), verbose=False\n",
    "):\n",
    "    rot1 = Rotation.from_euler(\"zyx\", rot, degrees=True)\n",
    "    R1 = rot1.as_matrix()\n",
    "    if verbose:\n",
    "        print(\"Rotation matrix:\", \"\\n\", R1)\n",
    "\n",
    "    # matrix multiplication between our rotation matrix and pointcloud\n",
    "    pc_show = np.matmul(R1, pc.copy()[:, :3].transpose()).transpose()\n",
    "    if color:\n",
    "        rot_color = pc.copy()[:, 3:]\n",
    "\n",
    "    fig = plt.figure(figsize=figsize)\n",
    "    ax = fig.add_subplot(111, projection=\"3d\")\n",
    "    ax.set_title(title, fontdict={\"fontsize\": 20})\n",
    "    if color:\n",
    "        # need to create list of tuples for matplotlib to plot RGB color values\n",
    "        color_tuple_list = []\n",
    "        for i in range(pc_show.shape[0]):\n",
    "            color_tuple = (\n",
    "                int(np.abs(rot_color[i, 0])) / 255,\n",
    "                int(np.abs(rot_color[i, 1])) / 255,\n",
    "                int(np.abs(rot_color[i, 2])) / 255,\n",
    "            )\n",
    "            color_tuple_list.append(color_tuple)\n",
    "        ax.scatter(pc_show[:, 0], pc_show[:, 1], pc_show[:, 2], c=color_tuple_list, s=0.05)\n",
    "    else:\n",
    "        ax.scatter(pc_show[:, 0], pc_show[:, 1], pc_show[:, 2], c=\"blue\", s=0.05)\n",
    "\n",
    "\n",
    "# rotate in z direction 30 degrees, y direction 90 degrees, and x direction 60 degrees\n",
    "# only run this once!\n",
    "rot = Rotation.from_euler(\"zyx\", [[30, 90, 60]], degrees=True)\n",
    "R = rot.as_matrix()\n",
    "print(\"Rotation matrix:\", \"\\n\", R)\n",
    "pc_rot = np.matmul(R, pc[:, :3].transpose()).transpose().squeeze()\n",
    "pc_color = pc.copy()[:, 3:]\n",
    "pc = np.zeros_like(pc)\n",
    "pc[:, :3] = pc_rot\n",
    "pc[:, 3:] = pc_color"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caroline-immunology",
   "metadata": {},
   "outputs": [],
   "source": [
    "# view full pointcloud, will take ~10 seconds to render depending on the instance type you are using\n",
    "\n",
    "plot_pointcloud(pc, rot=[[0, 0, 0]], color=True, title=\"Full Pointcloud\", figsize=(50, 30))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "natural-charm",
   "metadata": {},
   "source": [
    "## Section 2: Downsample Approaches\n",
    "\n",
    "Just like in image processing, there are many different approaches to performing downsampling in a point cloud.\n",
    "Certain approaches may preserve more information relevant for labelers, or may have different noise characteristics. You'll walk through two approaches, a very basic linear sampling and a more advanced interpolation method.\n",
    "\n",
    "Other alternatives include 3d box filtering, median filtering, and even random sampling.\n",
    "\n",
    "\n",
    "### I: Basic Approach\n",
    "The simplest form of downsampling is to choose values using a fixed step size based on how large we want our resulting point cloud to be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "gentle-producer",
   "metadata": {},
   "outputs": [],
   "source": [
    "target_num_pts = 500_000\n",
    "subsample = int(np.ceil(len(pc) / target_num_pts))\n",
    "pc_downsample_simple = pc[::subsample]\n",
    "print(f\"We've subsampled to {len(pc_downsample_simple)} points\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "selective-graduate",
   "metadata": {},
   "source": [
    "### II: Advanced Approach\n",
    "\n",
    "A more advanced approach is to break the input space into cubes, otherwise known as voxels, and choose a single point per box using an averaging function.\n",
    "\n",
    "In this example you will use cube sizes of 1.3 cm, which will partition the entire pointcloud scene into 1.3 cm by 1.3 cm cubes. Then for each cube, you will get a list of the points contained within the cube. Given the coordinates of this list of points, you take the centroid, or mean across each coordinate dimension to get a mean point that is representative of the overall set of points within the voxel. You run the operation twice, once over the xyz values then again over the color channel RGB values since you want to use the mins and maxes of the xyz values, but also preserve the color.\n",
    "\n",
    "You will use scipy's `binned_statistic_dd` to efficiently compute these representative points per voxel, then construct a new point cloud using these representative points and use it for downstream labeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "executed-agreement",
   "metadata": {},
   "outputs": [],
   "source": [
    "boxsize = 0.013  # 1.3 cm box size.\n",
    "mins = pc[:, :3].min(axis=0)\n",
    "maxes = pc[:, :3].max(axis=0)\n",
    "volume = maxes - mins\n",
    "num_boxes_per_axis = np.ceil(volume / boxsize).astype(\"int32\").tolist()\n",
    "\n",
    "print(\"Number of boxes per axis\", num_boxes_per_axis)\n",
    "\n",
    "# For each voxel or \"box\", use the mean of the box to chose which points are in the box.\n",
    "means, _, _ = scipy.stats.binned_statistic_dd(\n",
    "    pc[:, :3],\n",
    "    [pc[:, 0], pc[:, 1], pc[:, 2]],\n",
    "    statistic=\"mean\",\n",
    "    bins=num_boxes_per_axis,\n",
    ")\n",
    "\n",
    "# run a second operation over the color channels\n",
    "means2, _, _ = scipy.stats.binned_statistic_dd(\n",
    "    pc[:, :3],\n",
    "    [pc[:, 3], pc[:, 4], pc[:, 5]],\n",
    "    statistic=\"mean\",\n",
    "    bins=num_boxes_per_axis,\n",
    ")\n",
    "\n",
    "x_means = means[0, ~np.isnan(means[0])].flatten()\n",
    "y_means = means[1, ~np.isnan(means[1])].flatten()\n",
    "z_means = means[2, ~np.isnan(means[2])].flatten()\n",
    "r_means = means2[0, ~np.isnan(means2[0])].flatten()\n",
    "b_means = means2[1, ~np.isnan(means2[1])].flatten()\n",
    "g_means = means2[2, ~np.isnan(means2[2])].flatten()\n",
    "\n",
    "pc_downsample_adv = np.column_stack([x_means, y_means, z_means, r_means, b_means, g_means])\n",
    "print(\"downsampled pointcloud shape:\", pc_downsample_adv.shape)\n",
    "\n",
    "del x_means, y_means, z_means, r_means, b_means, g_means, means, num_boxes_per_axis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "working-piece",
   "metadata": {},
   "source": [
    "## Section 3: Visualize 3D Rendering\n",
    "\n",
    "You can visualize pointclouds using a 3D scatter plot of the points. You will see that the advanced voxel mean method creates a \"smoother\" point cloud as averaging has a noise reduction effect. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "armed-british",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(50, 30))\n",
    "ax = fig.add_subplot(121, projection=\"3d\")\n",
    "ax.set_title(\"Simple Downsampling 1\", fontdict={\"fontsize\": 20})\n",
    "color_tuple_list = []\n",
    "for i in range(pc_downsample_simple.shape[0]):\n",
    "    color_tuple = (\n",
    "        int(pc_downsample_simple[i, 3]) / 255,\n",
    "        int(pc_downsample_simple[i, 4]) / 255,\n",
    "        int(pc_downsample_simple[i, 5]) / 255,\n",
    "    )\n",
    "    color_tuple_list.append(color_tuple)\n",
    "ax.scatter(\n",
    "    pc_downsample_simple[:, 0],\n",
    "    pc_downsample_simple[:, 1],\n",
    "    pc_downsample_simple[:, 2],\n",
    "    c=color_tuple_list,\n",
    "    s=0.05,\n",
    ")\n",
    "\n",
    "ax = fig.add_subplot(122, projection=\"3d\")\n",
    "ax.set_title(\"Voxel Mean Downsampling 1\", fontdict={\"fontsize\": 20})\n",
    "color_tuple_list = []\n",
    "for i in range(pc_downsample_adv.shape[0]):\n",
    "    color_tuple = (\n",
    "        int(pc_downsample_adv[i, 3]) / 255,\n",
    "        int(pc_downsample_adv[i, 4]) / 255,\n",
    "        int(pc_downsample_adv[i, 5]) / 255,\n",
    "    )\n",
    "    color_tuple_list.append(color_tuple)\n",
    "ax.scatter(\n",
    "    pc_downsample_adv[:, 0],\n",
    "    pc_downsample_adv[:, 1],\n",
    "    pc_downsample_adv[:, 2],\n",
    "    c=color_tuple_list,\n",
    "    s=0.05,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hundred-machine",
   "metadata": {},
   "source": [
    "Below you can look at the point clouds from different perspectives by running the following function, which will multiply the pointclouds by the specified  rotation matrices. The current setting will rotate the pointcloud 90 degrees to the right.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ranging-armenia",
   "metadata": {},
   "outputs": [],
   "source": [
    "# try some different angles\n",
    "plot_pointcloud(\n",
    "    pc_downsample_adv, rot=[[90, 0, 0]], color=True, title=\"Advanced Downsampling\", figsize=(50, 30)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "understanding-grant",
   "metadata": {},
   "source": [
    "## Section 4: Amazon SageMaker Ground Truth (SMGT) Labeling Job\n",
    "\n",
    "You can submit this downsampled point cloud as a SageMaker Ground Truth job by converting to SMGT labeling format, then creating a labeling job using this point cloud as input. You only save the points themselves since you are going to launch a semantic segmentation labeling task where you will label the points using distinct colors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "metropolitan-technology",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save in Ground Truth format\n",
    "# See: https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-raw-data-types.html\n",
    "\n",
    "np.savetxt(\"pointcloud.txt\", pc_downsample_adv[:, :3], delimiter=\" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "looking-opposition",
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp pointcloud.txt s3://{bucket}/input/pointcloud.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "composed-helena",
   "metadata": {},
   "source": [
    "In the following cell, you generate an input manifest file. This file tells Ground Truth where 3D point cloud input data is located and contains metadata about your input data. To learn more, see [Create an Input Manifest File for a 3D Point Cloud Labeling Job](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-input-manifest.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adjustable-bench",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate an input manifest.\n",
    "input_manifest = json.dumps(\n",
    "    {\n",
    "        \"source-ref\": f\"s3://{bucket}/input/pointcloud.txt\",\n",
    "        \"source-ref-metadata\": {\n",
    "            \"format\": \"text/xyz\",\n",
    "            \"unix-timestamp\": 1566861644.759115,\n",
    "        },\n",
    "    }\n",
    ")\n",
    "\n",
    "input_manifest_name = \"input.manifest\"\n",
    "with open(input_manifest_name, \"w\") as f:\n",
    "    f.write(input_manifest)\n",
    "\n",
    "input_manifest_s3_uri = f\"s3://{bucket}/input/input.manifest\"\n",
    "!aws s3 cp {input_manifest_name} {input_manifest_s3_uri}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hybrid-evening",
   "metadata": {},
   "source": [
    "In the following cell, you generate a label category configuration file. This file contains the label categories workers use to label the 3D point cloud, worker instructions, and other information about your task. To learn more about all of the options you have when you create a label category configuration file, see [Create a Labeling Category Configuration File with Label Category and Frame Attributes](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-label-cat-config-attributes.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "athletic-carolina",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate label category configuration.\n",
    "label_category_config = json.dumps(\n",
    "    {\n",
    "        \"labels\": [\n",
    "            {\"label\": \"Floor\", \"attributes\": []},\n",
    "            {\"label\": \"Wall\", \"attributes\": []},\n",
    "            {\"label\": \"Furniture\", \"attributes\": []},\n",
    "            {\"label\": \"Plant\", \"attributes\": []},\n",
    "        ],\n",
    "        \"frameAttributes\": [],\n",
    "        \"categoryGlobalAttributes\": [],\n",
    "        \"instructions\": {\n",
    "            \"shortInstruction\": \"Label\",\n",
    "            \"fullInstruction\": '<ul><li>After you\u2019ve painted an object, use the <strong>Unpaint</strong> category to remove paint.&nbsp;</li><li>If you need to paint an object that is partially blocked by another object that you have painted, select the Paint cloud as background tool before you start painting.</li><li>Use the <strong>Shortcuts</strong> menu to see keyboard shortcuts that you can use to label objects faster.</li><li>Use the <strong>View</strong> menu to modify your view of the 3D point cloud and the worker portal.</li><li>Use the <strong>3D Point Cloud&nbsp;</strong>menu to modify the perspective of and pixel-attributes you see in the 3D point cloud.</li><li>Use this <a href=\"https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-worker-instructions-semantic-segmentation.html\" rel=\"noopener noreferrer\" target=\"_blank\">resource</a> to learn about worker portal navigation, tools available to complete your task, icons, and view options.</li></ul>',\n",
    "        },\n",
    "        \"document-version\": \"2020-03-01\",\n",
    "    }\n",
    ")\n",
    "\n",
    "label_category_config_filename = \"labels.json\"\n",
    "with open(label_category_config_filename, \"w\") as f:\n",
    "    f.write(label_category_config)\n",
    "\n",
    "label_category_config_s3_uri = f\"s3://{bucket}/input/labels.json\"\n",
    "!aws s3 cp {label_category_config_filename} {label_category_config_s3_uri}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "yellow-scene",
   "metadata": {},
   "source": [
    "In the following cell you configure the labeling job request. To learn more about the parameters used to configure a labeling job, see [Create a Labeling Job (API)](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-create-labeling-job-api.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "enclosed-discrimination",
   "metadata": {},
   "outputs": [],
   "source": [
    "hash_str = str(uuid.uuid4())[:4]\n",
    "job_name = f\"downsample-demo-{hash_str}\"\n",
    "\n",
    "lambda_account_id = {\n",
    "    \"us-east-1\": \"432418664414\",\n",
    "    \"us-east-2\": \"266458841044\",\n",
    "    \"us-west-2\": \"081040173940\",\n",
    "    \"ca-central-1\": \"918755190332\",\n",
    "    \"eu-west-1\": \"568282634449\",\n",
    "    \"eu-west-2\": \"487402164563\",\n",
    "    \"eu-central-1\": \"203001061592\",\n",
    "    \"ap-northeast-1\": \"477331159723\",\n",
    "    \"ap-northeast-2\": \"845288260483\",\n",
    "    \"ap-south-1\": \"565803892007\",\n",
    "    \"ap-southeast-1\": \"377565633583\",\n",
    "    \"ap-southeast-2\": \"454466003867\",\n",
    "}[region]\n",
    "\n",
    "job_creation_request = {\n",
    "    \"LabelingJobName\": job_name,\n",
    "    \"LabelAttributeName\": f\"{job_name}-ref\",\n",
    "    \"InputConfig\": {\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"ManifestS3Uri\": input_manifest_s3_uri,\n",
    "            }\n",
    "        },\n",
    "        \"DataAttributes\": {\"ContentClassifiers\": []},\n",
    "    },\n",
    "    \"OutputConfig\": {\n",
    "        \"S3OutputPath\": f\"s3://{bucket}/output\",\n",
    "        \"KmsKeyId\": \"\",\n",
    "    },\n",
    "    \"RoleArn\": role,\n",
    "    \"LabelCategoryConfigS3Uri\": label_category_config_s3_uri,\n",
    "    \"StoppingConditions\": {\"MaxPercentageOfInputDatasetLabeled\": 100},\n",
    "    \"HumanTaskConfig\": {\n",
    "        \"WorkteamArn\": WORKTEAM_ARN,\n",
    "        \"UiConfig\": {\n",
    "            \"HumanTaskUiArn\": f\"arn:aws:sagemaker:{region}:394669845002:human-task-ui/PointCloudSemanticSegmentation\"\n",
    "        },\n",
    "        \"PreHumanTaskLambdaArn\": f\"arn:aws:lambda:{region}:{lambda_account_id}:function:PRE-3DPointCloudSemanticSegmentation\",\n",
    "        \"TaskKeywords\": [\"Point cloud\", \"segmentation\"],\n",
    "        \"TaskTitle\": job_name,\n",
    "        \"TaskDescription\": \"Create a semantic segmentation mask by painting objects in a 3D point cloud\",\n",
    "        \"NumberOfHumanWorkersPerDataObject\": 1,\n",
    "        \"TaskTimeLimitInSeconds\": 604800,\n",
    "        \"TaskAvailabilityLifetimeInSeconds\": 864000,\n",
    "        \"MaxConcurrentTaskCount\": 1000,\n",
    "        \"AnnotationConsolidationConfig\": {\n",
    "            \"AnnotationConsolidationLambdaArn\": f\"arn:aws:lambda:{region}:{lambda_account_id}:function:ACS-3DPointCloudSemanticSegmentation\"\n",
    "        },\n",
    "    },\n",
    "}\n",
    "\n",
    "# Review the request for any errors before submitting.\n",
    "print(json.dumps(job_creation_request, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "undefined-harmony",
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_cl.create_labeling_job(**job_creation_request)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "criminal-creation",
   "metadata": {},
   "source": [
    "### Label Data in the Worker Portal"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "assigned-nutrition",
   "metadata": {},
   "source": [
    "Run the following cell to see the worker portal URL of your private workforce. Use this URL to login to the worker portal to see your labeling tasks. See the [3D point cloud worker instructions](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-worker-instructions.html) to learn more about the feature you'll see in the worker UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "virgin-shock",
   "metadata": {},
   "outputs": [],
   "source": [
    "workforce = sagemaker_cl.describe_workforce(WorkforceName=\"default\")\n",
    "worker_portal_url = workforce[\"Workforce\"][\"SubDomain\"]\n",
    "print(f\"Sign-in by going here: {worker_portal_url}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prerequisite-submission",
   "metadata": {},
   "source": [
    "### Label Data in the Worker Portal\n",
    "After submitting the job in the worker portal, wait until the labeling job has status \"complete\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "solved-civilian",
   "metadata": {},
   "outputs": [],
   "source": [
    "job = sagemaker_cl.describe_labeling_job(LabelingJobName=job_name)\n",
    "status = job[\"LabelingJobStatus\"]\n",
    "print(f\"Status: '{status}'\")\n",
    "assert status == \"Completed\"\n",
    "\n",
    "label_attribute_name = job[\"LabelAttributeName\"]\n",
    "\n",
    "output_manifest_s3_uri = job[\"LabelingJobOutput\"][\"OutputDatasetS3Uri\"]\n",
    "!aws s3 cp {output_manifest_s3_uri} output.manifest\n",
    "\n",
    "with open(\"output.manifest\") as f:\n",
    "    manifest_line = f.read().splitlines()[0]\n",
    "\n",
    "data_object = json.loads(manifest_line)\n",
    "labels_s3_uri = data_object[label_attribute_name]\n",
    "\n",
    "!aws s3 cp {labels_s3_uri} pointcloud.zlib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "white-relay",
   "metadata": {},
   "source": [
    "### Load Annotations From S3\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "changing-bearing",
   "metadata": {},
   "outputs": [],
   "source": [
    "def semantic_segmentation_to_classes(filename):\n",
    "    \"\"\"Convert from class map labels to numpy array.\"\"\"\n",
    "    with open(filename, \"rb\") as f:\n",
    "        binary_content = zlib.decompress(f.read())\n",
    "    my_classes = array(\"B\", binary_content)\n",
    "    return np.array(my_classes)\n",
    "\n",
    "\n",
    "annotations = semantic_segmentation_to_classes(\"pointcloud.zlib\")\n",
    "\n",
    "# Verify we have a label for each point in the downsampled point cloud.\n",
    "assert len(annotations) == len(pc_downsample_adv)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "norman-shock",
   "metadata": {},
   "source": [
    "## Section 5: KNN Upsample Approach"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fatal-spoke",
   "metadata": {},
   "source": [
    "Now that we have the downsampled labels, we will train a K-NN classifier from SKLearn to predict the full dataset labels by treating our annotated points as training data and performing inference on the remainder of the unlabeled points in our full size point cloud. Note that you can tune the number of points used as well as the distance metric and weighting scheme to influence how label inference is performed. If you label a few tiles in the full-size dataset, you can use those labeled tiles as ground truth to evaluate the accuracy of the K-NN predictions. This accuracy metric can then be used for hyperparameter tuning of K-NN or to try different inference algorithms to reduce your number of misclassified points between object boundaries, resulting in the lowest possible in-sample error rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "suburban-scotland",
   "metadata": {},
   "outputs": [],
   "source": [
    "# There's a lot of possibility to tune KNN further\n",
    "# 1) Prevent classification of points far away from all other points (random unfiltered ground point)\n",
    "# 2) Perform a non-uniform weighted vote\n",
    "# 3) Tweak number of neighbors\n",
    "knn = KNeighborsClassifier(n_neighbors=3)\n",
    "print(f\"Training on {len(pc_downsample_adv)} labeled points\")\n",
    "knn.fit(pc_downsample_adv[:, :3], annotations)\n",
    "\n",
    "print(f\"Upsampled to {len(pc)} labeled points\")\n",
    "annotations_full = knn.predict(pc[:, :3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "traditional-ordinary",
   "metadata": {},
   "source": [
    "## Section 6: Visualize 3D Rendering\n",
    "\n",
    "Now that you've performed upsampling of the labeled data, you can visualize a tile of the original full size point cloud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "welsh-technician",
   "metadata": {},
   "outputs": [],
   "source": [
    "pc_downsample_annotated = np.column_stack((pc_downsample_adv[:, :3], annotations))\n",
    "pc_annotated = np.column_stack((pc[:, :3], annotations_full))\n",
    "\n",
    "labeled_area = pc_downsample_annotated[pc_downsample_annotated[:, 3] != 255]\n",
    "min_bounds = np.min(labeled_area, axis=0)\n",
    "max_bounds = np.max(labeled_area, axis=0)\n",
    "\n",
    "min_bounds = [-2, -2, -5.5, -1]\n",
    "max_bounds = [2, 2, 0, 256]\n",
    "\n",
    "\n",
    "def extract_tile(point_cloud, min_bounds, max_bounds):\n",
    "    return point_cloud[\n",
    "        (point_cloud[:, 0] > min_bounds[0])\n",
    "        & (point_cloud[:, 1] > min_bounds[1])\n",
    "        & (point_cloud[:, 2] > min_bounds[2])\n",
    "        & (point_cloud[:, 0] < max_bounds[0])\n",
    "        & (point_cloud[:, 1] < max_bounds[1])\n",
    "        & (point_cloud[:, 2] < max_bounds[2])\n",
    "    ]\n",
    "\n",
    "\n",
    "tile_downsample_annotated = extract_tile(pc_downsample_annotated, min_bounds, max_bounds)\n",
    "tile_annotated = extract_tile(pc_annotated, min_bounds, max_bounds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bizarre-tomato",
   "metadata": {},
   "outputs": [],
   "source": [
    "down_rot = tile_downsample_annotated.copy()\n",
    "down_rot_color = tile_downsample_annotated.copy()[:, 3:].astype(\"int32\")\n",
    "# change the color scheme to clearly display classes\n",
    "down_rot_color[down_rot_color == 0] = 50\n",
    "down_rot_color[down_rot_color == 1] = 100\n",
    "down_rot_color[down_rot_color == 2] = 150\n",
    "down_rot_color[down_rot_color == 3] = 200\n",
    "\n",
    "full_rot = tile_annotated.copy()\n",
    "full_rot_color = tile_annotated.copy()[:, 3:].astype(\"int32\")\n",
    "full_rot_color[full_rot_color == 0] = 50\n",
    "full_rot_color[full_rot_color == 1] = 100\n",
    "full_rot_color[full_rot_color == 2] = 150\n",
    "full_rot_color[full_rot_color == 3] = 200\n",
    "\n",
    "fig = plt.figure(figsize=(50, 20))\n",
    "ax = fig.add_subplot(121, projection=\"3d\")\n",
    "ax.set_title(\"Downsampled Annotations\", fontdict={\"fontsize\": 25})\n",
    "ax.scatter(down_rot[:, 0], down_rot[:, 1], down_rot[:, 2], c=down_rot_color[:, 0], s=0.05)\n",
    "ax = fig.add_subplot(122, projection=\"3d\")\n",
    "ax.set_title(\"Upsampled Annotations\", fontdict={\"fontsize\": 25})\n",
    "ax.scatter(full_rot[:, 0], full_rot[:, 1], full_rot[:, 2], c=full_rot_color[:, 0], s=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "surgical-touch",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This notebook demonstrated a couple approaches to performing downsampling on pointclouds for data labeling use cases. These approaches can also be applied as preprocessing steps before training and inference. The notebook demonstrated a simple step based approach as well as a a more complex approach where you broke the pointcloud bounds into voxels and replaced each voxel with a summary point created by taking the mean of all points in the voxel.\n",
    "\n",
    "You then learned how to convert these pointclouds into a SageMaker Ground Truth Semantic Segmentation job. Finally you learned how to take the resulting labels and apply them to the full-size point cloud using some in-sample prediction based on the K Nearest Neighbors algorithm.\n",
    "\n",
    "Using these approaches, you can save time and effort labeling high-density pointclouds by avoiding tiling and instead labeling downsampled pointclouds.\n",
    "\n",
    "## Cleanup\n",
    "\n",
    "If for some reason you didn't complete the labeling job, you can run the following command to stop it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "impressive-representation",
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_cl.stop_labeling_job(job_name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/ground_truth_labeling_jobs|3d_dense_point_cloud_downsampling_tutorial|ground_truth_annotation_dense_point_cloud_tutorial.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}