Python/client_examples/lidar_API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Aircontrol Lidar API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Aircontrol Lidar API facilitates communication with simulated LIDAR in Unity.\n",
    "The lidar script uses the Raycast feature of Unity Game Engine. The distance array, of float type, consists of 360 members, the ith member storing the distance at which the ray hit an object at the ith degree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "import PIL.Image as Image\n",
    "import base64\n",
    "import numpy as np\n",
    "from io import BytesIO\n",
    "from matplotlib.pyplot import  imshow\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from airctrl import environment \n",
    "from airctrl import sample_generator\n",
    "from airctrl.utils import unity\n",
    "from airctrl import sample_generator\n",
    "port=8999\n",
    "sample = sample_generator.samples()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mNow call method `.get_connected(port=<Default 8053>)` to get connected\n",
      "\u001b[0m\n",
      "\u001b[32mLoading environment from /home/supatel/Games/AirControl_2021/Build/1.3.0/Linux/v1.3.0-AirControl.x86_64 at port 8999 client ip 127.0.1.1 client port 8999\n",
      "\u001b[0m\n",
      "['/home/supatel/Games/AirControl_2021/Build/1.3.0/Linux/v1.3.0-AirControl.x86_64', '--serverPort', '8999', '--clientIP', '127.0.1.1', '--clientPort', '8999']\n",
      "\u001b[32mSleeping for 5 seconds to allow environment load\n",
      "\u001b[0m\n",
      "[UnityMemory] Configuration Parameters - Can be set up in boot.config\n",
      "    \"memorysetup-bucket-allocator-granularity=16\"\n",
      "    \"memorysetup-bucket-allocator-bucket-count=8\"\n",
      "    \"memorysetup-bucket-allocator-block-size=4194304\"\n",
      "    \"memorysetup-bucket-allocator-block-count=1\"\n",
      "    \"memorysetup-main-allocator-block-size=16777216\"\n",
      "    \"memorysetup-thread-allocator-block-size=16777216\"\n",
      "    \"memorysetup-gfx-main-allocator-block-size=16777216\"\n",
      "    \"memorysetup-gfx-thread-allocator-block-size=16777216\"\n",
      "    \"memorysetup-cache-allocator-block-size=4194304\"\n",
      "    \"memorysetup-typetree-allocator-block-size=2097152\"\n",
      "    \"memorysetup-profiler-bucket-allocator-granularity=16\"\n",
      "    \"memorysetup-profiler-bucket-allocator-bucket-count=8\"\n",
      "    \"memorysetup-profiler-bucket-allocator-block-size=4194304\"\n",
      "    \"memorysetup-profiler-bucket-allocator-block-count=1\"\n",
      "    \"memorysetup-profiler-allocator-block-size=16777216\"\n",
      "    \"memorysetup-profiler-editor-allocator-block-size=1048576\"\n",
      "    \"memorysetup-temp-allocator-size-main=4194304\"\n",
      "    \"memorysetup-job-temp-allocator-block-size=2097152\"\n",
      "    \"memorysetup-job-temp-allocator-block-size-background=1048576\"\n",
      "    \"memorysetup-job-temp-allocator-reduction-small-platforms=262144\"\n",
      "    \"memorysetup-temp-allocator-size-background-worker=32768\"\n",
      "    \"memorysetup-temp-allocator-size-job-worker=262144\"\n",
      "    \"memorysetup-temp-allocator-size-preload-manager=262144\"\n",
      "    \"memorysetup-temp-allocator-size-nav-mesh-worker=65536\"\n",
      "    \"memorysetup-temp-allocator-size-audio-worker=65536\"\n",
      "    \"memorysetup-temp-allocator-size-cloud-worker=32768\"\n",
      "    \"memorysetup-temp-allocator-size-gfx=262144\"\n",
      "\u001b[32mConnecting with port 8999\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "env =  environment.Trigger()\n",
    "L = unity.Launch()\n",
    "process = L.launch_executable(\"/home/supatel/Games/AirControl_2021/Build/1.3.0/Linux/v1.3.0-AirControl.x86_64\", server_port=port)\n",
    "env.get_connected(port=port)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lidar Placement\n",
    "Lidar is placed in the cockpit, Attached to the cockpit camera. This ensures that all the distance are measured with respect to the cockpit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python API\n",
    "- InputControlType (str, optional): It can be either `Code` or `Other`. This is to control the internal mechanism and prevent repeated calling in already set variables. \\\n",
    "        If `InputControlType` is set to 'Code', the camera cannot be controlled from Keyboard or Joystick. If `InputControlType` is set to 'Other', the camera can be only controlled from Keyboard or Joystick.  Defaults to \"Code\". \n",
    "- Range (float, optional): Range of the Lidar. Defaults to 100000.0.\n",
    "- Density (int, optional): Number of Raycast spread across 360 degrees. Defaults to 360.\n",
    "- IsActive (bool, optional): If lidar is set to active or not. Defaults to False.\n",
    "- IsOutput (bool, optional): By default `step` function only sets the internal state. `step` only provides log output and not the actual captured image. `set_control` when called it returns the actual output. IF you want to force `step` to return the image, set `IsOutput` to True. Defaults to False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 296.13327,\n",
      " 185.8067,\n",
      " 135.403366,\n",
      " 106.535721,\n",
      " 87.83601,\n",
      " 74.7402,\n",
      " 65.05968,\n",
      " 57.61505,\n",
      " 51.7131348,\n",
      " 46.92129,\n",
      " 42.95404,\n",
      " 39.6163559,\n",
      " 36.77036,\n",
      " 34.31585,\n",
      " 32.177578,\n",
      " 30.2988453,\n",
      " 28.6354046,\n",
      " 27.153286,\n",
      " 25.8244038,\n",
      " 24.6267738,\n",
      " 23.5418587,\n",
      " 22.5552273,\n",
      " 21.6543789,\n",
      " 20.828886,\n",
      " 20.0699081,\n",
      " 19.3697357,\n",
      " 18.7224026,\n",
      " 17.6607,\n",
      " 17.11607,\n",
      " 16.6089325,\n",
      " 16.1357517,\n",
      " 15.6934414,\n",
      " 15.2792549,\n",
      " 14.8907967,\n",
      " 14.5259142,\n",
      " 14.1826944,\n",
      " 13.8594522,\n",
      " 13.5546484,\n",
      " 13.2669239,\n",
      " 12.9950275,\n",
      " 12.7378559,\n",
      " 12.4944048,\n",
      " 12.2637453,\n",
      " 12.0450516,\n",
      " 11.8375616,\n",
      " 11.6405878,\n",
      " 11.4534912,\n",
      " 11.2756939,\n",
      " 11.1066666,\n",
      " 10.9459133,\n",
      " 10.7929907,\n",
      " 10.6474819,\n",
      " 10.5089979,\n",
      " 10.3771935,\n",
      " 10.2517376,\n",
      " 10.132329,\n",
      " 10.0186853,\n",
      " 9.91055,\n",
      " 9.807681,\n",
      " 9.709848,\n",
      " 9.616855,\n",
      " 9.528494,\n",
      " 9.444596,\n",
      " 9.364991,\n",
      " 9.28952,\n",
      " 9.218044,\n",
      " 9.150424,\n",
      " 9.086538,\n",
      " 9.02627,\n",
      " 8.969505,\n",
      " 8.916151,\n",
      " 8.866115,\n",
      " 8.819306,\n",
      " 8.775649,\n",
      " 8.735069,\n",
      " 8.697502,\n",
      " 8.662881,\n",
      " 8.631155,\n",
      " 8.602271,\n",
      " 8.576183,\n",
      " 8.552852,\n",
      " 8.532236,\n",
      " 8.514309,\n",
      " 8.499041,\n",
      " 8.486404,\n",
      " 8.476389,\n",
      " 8.468969,\n",
      " 8.464142,\n",
      " 8.461896,\n",
      " 8.462227,\n",
      " 8.465137,\n",
      " 8.470627,\n",
      " 8.47871,\n",
      " 8.489402,\n",
      " 8.50270748,\n",
      " 8.518659,\n",
      " 8.537271,\n",
      " 8.55858,\n",
      " 8.582617,\n",
      " 8.609418,\n",
      " 8.6390295,\n",
      " 8.671493,\n",
      " 8.706865,\n",
      " 8.7452,\n",
      " 8.786564,\n",
      " 8.831021,\n",
      " 8.878651,\n",
      " 8.929534,\n",
      " 8.983754,\n",
      " 9.041408,\n",
      " 9.102594,\n",
      " 9.167428,\n",
      " 9.236028,\n",
      " 9.308514,\n",
      " 9.385033,\n",
      " 9.465726,\n",
      " 9.550752,\n",
      " 9.640284,\n",
      " 9.734503,\n",
      " 9.833607,\n",
      " 9.937808,\n",
      " 10.0473318,\n",
      " 10.1624308,\n",
      " 10.2833633,\n",
      " 10.41042,\n",
      " 10.5439062,\n",
      " 10.6841583,\n",
      " 10.8315344,\n",
      " 10.9864254,\n",
      " 11.1492586,\n",
      " 11.320488,\n",
      " 11.5006227,\n",
      " 11.690197,\n",
      " 11.8898077,\n",
      " 12.1001081,\n",
      " 12.3217974,\n",
      " 12.5556593,\n",
      " 12.8025465,\n",
      " 13.0633936,\n",
      " 13.3392382,\n",
      " 13.631237,\n",
      " 13.9406357,\n",
      " 14.2688532,\n",
      " 14.61746,\n",
      " 14.988204,\n",
      " 15.38307,\n",
      " 15.8042259,\n",
      " 16.2541885,\n",
      " 16.7357769,\n",
      " 17.2521935,\n",
      " 17.80708,\n",
      " 18.404644,\n",
      " 19.0497055,\n",
      " 19.7478676,\n",
      " 20.5056438,\n",
      " 21.330637,\n",
      " 22.2318573,\n",
      " 23.2199669,\n",
      " 24.3077545,\n",
      " 25.5106468,\n",
      " 26.8473663,\n",
      " 28.3227062,\n",
      " 30.00095,\n",
      " 31.9009266,\n",
      " 34.06898,\n",
      " 36.5887,\n",
      " 39.4943428,\n",
      " 42.9154778,\n",
      " 47.0011749,\n",
      " 51.96433,\n",
      " 58.1190376,\n",
      " 65.95037,\n",
      " 76.24769,\n",
      " 90.38823,\n",
      " 115.557961,\n",
      " 150.338654,\n",
      " 179.335876,\n",
      " 182.953522,\n",
      " 185.972717,\n",
      " 189.151932,\n",
      " 201.652527,\n",
      " 204.916885,\n",
      " 205.803421,\n",
      " 207.084885,\n",
      " 208.630981,\n",
      " 210.2648,\n",
      " 212.386749,\n",
      " 214.70665,\n",
      " 216.986,\n",
      " 219.343109,\n",
      " 222.895447,\n",
      " 230.7293,\n",
      " 230.256439,\n",
      " 234.7577,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0,\n",
      " 500.0]\n"
     ]
    }
   ],
   "source": [
    "output = env.set_lidar(InputControlType=\"Code\", Range=100000.0, Density=360, IsActive=False,IsOutput=True)\n",
    "lidar_output = output['LidarPointCloud'] # get lidar output\n",
    "pprint(lidar_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively one can specify `IsOutput`  to `False` and get output using reply to `schemaDef.set_control` function call. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Just log output {'Log': 'Active scene camera - 1 Capture camera - 0 Width - 256  Height - 256: '}\n"
     ]
    }
   ],
   "source": [
    "output = env.set_lidar(InputControlType=\"Code\", Range=10000.0, Density=360, IsActive=False,IsOutput=False)\n",
    "# just log output as `IsOutput` is set to false\n",
    "print(\"Just log output\", output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use set_control function\n",
    "output = A.step(IsOutput=True)\n",
    "lidar_output = output['LidarPointCloud'] # get lidar output\n",
    "pprint(lidar_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the Airplane is a tail dagger, the backside rays of lidar interact with a ground plane. Due to this hit distance for some of the rays in the above-shown lidar output is very near to airplane."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](../../docs/images/lidar/5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, lidar interacts with buildings and provides a sense of nearness."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](../../docs/images/lidar/9.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "1. [Physics.Raycast](https://docs.unity3d.com/ScriptReference/Physics.Raycast.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "e25000c05b9059318314f17b52328a5a24133f627d403c4e70496d99daf098c4"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}