notebooks/01 - Pre-processing - Mapping columns and data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This notebook shows you how to how to scan over your raw data in a pre-processing step to understand what's inside.\n",
    "\n",
    "## This is useful to avoid crashing with multiple column names for the same item (e.g. lon and LON), and to map raw data values to standard values (e.g. string descriptions to integers).\n",
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import trackio as tio\n",
    "import glob\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## First, define the raw data files. We'll start with some free AIS data from Marine Cadastre:\n",
    "\n",
    "https://marinecadastre.gov/ais/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./files\\\\AIS_2021_01_01.csv']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#define raw data files\n",
    "data_path = './files'\n",
    "files = glob.glob(f'{data_path}/AIS_*.csv')\n",
    "\n",
    "files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Use the below function to making a dictionary mapper for all of the column names encountered in the raw data files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92mMaking column mapper\u001b[0m: 100%|\u001b[32m██████████\u001b[0m| 1/1 [00:00<00:00, 667.03it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'BaseDateTime': 'BaseDateTime',\n",
       " 'COG': 'COG',\n",
       " 'CallSign': 'CallSign',\n",
       " 'Cargo': 'Cargo',\n",
       " 'Draft': 'Draft',\n",
       " 'Heading': 'Heading',\n",
       " 'IMO': 'IMO',\n",
       " 'LAT': 'LAT',\n",
       " 'LON': 'LON',\n",
       " 'Length': 'Length',\n",
       " 'MMSI': 'MMSI',\n",
       " 'SOG': 'SOG',\n",
       " 'Status': 'Status',\n",
       " 'TranscieverClass': 'TranscieverClass',\n",
       " 'VesselName': 'VesselName',\n",
       " 'VesselType': 'VesselType',\n",
       " 'Width': 'Width'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#make a column mapper for raw data\n",
    "col_mapper = tio.make_col_mapper(files, \n",
    "                                 ncores=1)\n",
    "                                 \n",
    "col_mapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "## Every raw data file must have X,Y,Time columns, or have column names mapped to these 3 columns. In this case, you would simply edit the column mapper as below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'BaseDateTime': 'Time',\n",
       " 'COG': 'COG',\n",
       " 'CallSign': 'CallSign',\n",
       " 'Cargo': 'Cargo',\n",
       " 'Draft': 'Draft',\n",
       " 'Heading': 'Heading',\n",
       " 'IMO': 'IMO',\n",
       " 'LAT': 'Y',\n",
       " 'LON': 'X',\n",
       " 'Length': 'Length',\n",
       " 'MMSI': 'MMSI',\n",
       " 'SOG': 'SOG',\n",
       " 'Status': 'Status',\n",
       " 'TranscieverClass': 'TranscieverClass',\n",
       " 'VesselName': 'VesselName',\n",
       " 'VesselType': 'VesselType',\n",
       " 'Width': 'Width'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_mapper['LON'] = 'X'\n",
    "col_mapper['LAT'] = 'Y'\n",
    "col_mapper['BaseDateTime'] = 'Time'\n",
    "\n",
    "col_mapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Alternatively, you can use the built-in column mapper. This has been built over time by encountering new column names and data fields in raw data files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92mMaking column mapper\u001b[0m: 100%|\u001b[32m██████████\u001b[0m| 1/1 [00:00<00:00, 333.81it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'BaseDateTime': 'Time',\n",
       " 'COG': 'Coursing',\n",
       " 'CallSign': 'CallSign',\n",
       " 'Cargo': 'Cargo',\n",
       " 'Draft': 'Draft',\n",
       " 'Heading': 'Heading',\n",
       " 'IMO': 'IMO',\n",
       " 'LAT': 'Y',\n",
       " 'LON': 'X',\n",
       " 'Length': 'Length',\n",
       " 'MMSI': 'MMSI',\n",
       " 'SOG': 'Speed',\n",
       " 'Status': 'Status',\n",
       " 'TranscieverClass': 'TranscieverClass',\n",
       " 'VesselName': 'Name',\n",
       " 'VesselType': 'AISCode',\n",
       " 'Width': 'Width'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#make a column mapper for raw data, use the built in mapper\n",
    "col_mapper = tio.make_col_mapper(files, \n",
    "                                 ncores=4,\n",
    "                                 fill_mapper=tio.mappers.columns)\n",
    "\n",
    "col_mapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Notice that X,Y,Time fields have been automtically detected, as well as some other fields."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You can also update the built in column mapper so you can save newly encountered column names and data fields, and be able to automatically detect them next time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'BaseDateTime': 'Time',\n",
       " 'basedatetime': 'Time',\n",
       " 'BASEDATETIME': 'Time',\n",
       " 'Basedatetime': 'Time',\n",
       " 'dt_pos_utc': 'Time',\n",
       " 'DT_POS_UTC': 'Time',\n",
       " 'Dt_Pos_Utc': 'Time',\n",
       " 'dt pos utc': 'Time',\n",
       " 'dt-pos-utc': 'Time',\n",
       " 'DATE TIME (UTC)': 'Time',\n",
       " 'date time (utc)': 'Time',\n",
       " 'Date Time (Utc)': 'Time',\n",
       " 'Date time stamp': 'Time',\n",
       " 'date time stamp': 'Time',\n",
       " 'DATE TIME STAMP': 'Time',\n",
       " 'Date Time Stamp': 'Time',\n",
       " 'POSITION_UTC_DATE': 'Time',\n",
       " 'position_utc_date': 'Time',\n",
       " 'Position_Utc_Date': 'Time',\n",
       " 'POSITION UTC DATE': 'Time',\n",
       " 'POSITION-UTC-DATE': 'Time',\n",
       " 'Time': 'Time',\n",
       " 'time': 'Time',\n",
       " 'TIME': 'Time',\n",
       " 'MovementDateTime': 'Time',\n",
       " 'movementdatetime': 'Time',\n",
       " 'MOVEMENTDATETIME': 'Time',\n",
       " 'Movementdatetime': 'Time',\n",
       " 'X': 'X',\n",
       " 'x': 'X',\n",
       " 'Longitude': 'X',\n",
       " 'longitude': 'X',\n",
       " 'LONGITUDE': 'X',\n",
       " 'Lon': 'X',\n",
       " 'lon': 'X',\n",
       " 'LON': 'X',\n",
       " 'Longitude (DDD.ddd)': 'X',\n",
       " 'longitude (ddd.ddd)': 'X',\n",
       " 'LONGITUDE (DDD.DDD)': 'X',\n",
       " 'Longitude (Ddd.Ddd)': 'X',\n",
       " 'Y': 'Y',\n",
       " 'y': 'Y',\n",
       " 'Latitude': 'Y',\n",
       " 'latitude': 'Y',\n",
       " 'LATITUDE': 'Y',\n",
       " 'Lat': 'Y',\n",
       " 'lat': 'Y',\n",
       " 'LAT': 'Y',\n",
       " 'Latitude (DD.ddd)': 'Y',\n",
       " 'latitude (dd.ddd)': 'Y',\n",
       " 'LATITUDE (DD.DDD)': 'Y',\n",
       " 'Latitude (Dd.Ddd)': 'Y',\n",
       " 'Draught': 'Draft',\n",
       " 'draught': 'Draft',\n",
       " 'DRAUGHT': 'Draft',\n",
       " 'Draft': 'Draft',\n",
       " 'draft': 'Draft',\n",
       " 'DRAFT': 'Draft',\n",
       " 'Max draught': 'Draft',\n",
       " 'max draught': 'Draft',\n",
       " 'MAX DRAUGHT': 'Draft',\n",
       " 'Max Draught': 'Draft',\n",
       " 'Name': 'Name',\n",
       " 'name': 'Name',\n",
       " 'NAME': 'Name',\n",
       " 'VesselName': 'Name',\n",
       " 'vesselname': 'Name',\n",
       " 'VESSELNAME': 'Name',\n",
       " 'Vesselname': 'Name',\n",
       " 'Vessel_Name': 'Name',\n",
       " 'vessel_name': 'Name',\n",
       " 'VESSEL_NAME': 'Name',\n",
       " 'Vessel Name': 'Name',\n",
       " 'Vessel-Name': 'Name',\n",
       " 'vessel name': 'Name',\n",
       " 'vessel-name': 'Name',\n",
       " 'MMSI': 'MMSI',\n",
       " 'mmsi': 'MMSI',\n",
       " 'Mmsi': 'MMSI',\n",
       " 'VESSEL_ID': 'MMSI',\n",
       " 'vessel_id': 'MMSI',\n",
       " 'Vessel_Id': 'MMSI',\n",
       " 'VESSEL ID': 'MMSI',\n",
       " 'VESSEL-ID': 'MMSI',\n",
       " 'IMO': 'IMO',\n",
       " 'imo': 'IMO',\n",
       " 'Imo': 'IMO',\n",
       " 'IMO number': 'IMO',\n",
       " 'imo number': 'IMO',\n",
       " 'IMO NUMBER': 'IMO',\n",
       " 'Imo Number': 'IMO',\n",
       " 'LRIMOShipNo': 'IMO',\n",
       " 'lrimoshipno': 'IMO',\n",
       " 'LRIMOSHIPNO': 'IMO',\n",
       " 'Lrimoshipno': 'IMO',\n",
       " 'AISTYPE': 'AISCode',\n",
       " 'aistype': 'AISCode',\n",
       " 'Aistype': 'AISCode',\n",
       " 'VesselType': 'AISCode',\n",
       " 'vesseltype': 'AISCode',\n",
       " 'VESSELTYPE': 'AISCode',\n",
       " 'Vesseltype': 'AISCode',\n",
       " 'vessel_type_code': 'AISCode',\n",
       " 'VESSEL_TYPE_CODE': 'AISCode',\n",
       " 'Vessel_Type_Code': 'AISCode',\n",
       " 'vessel type code': 'AISCode',\n",
       " 'vessel-type-code': 'AISCode',\n",
       " 'Type of ship': 'AISCode',\n",
       " 'type of ship': 'AISCode',\n",
       " 'TYPE OF SHIP': 'AISCode',\n",
       " 'Type Of Ship': 'AISCode',\n",
       " 'AISCode': 'AISCode',\n",
       " 'aiscode': 'AISCode',\n",
       " 'AISCODE': 'AISCode',\n",
       " 'Aiscode': 'AISCode',\n",
       " 'ShipType': 'AISCode',\n",
       " 'shiptype': 'AISCode',\n",
       " 'SHIPTYPE': 'AISCode',\n",
       " 'Shiptype': 'AISCode',\n",
       " 'COG': 'Coursing',\n",
       " 'cog': 'Coursing',\n",
       " 'Cog': 'Coursing',\n",
       " 'Course': 'Coursing',\n",
       " 'course': 'Coursing',\n",
       " 'COURSE': 'Coursing',\n",
       " 'Coursing': 'Coursing',\n",
       " 'coursing': 'Coursing',\n",
       " 'COURSING': 'Coursing',\n",
       " 'COG (deg)': 'Coursing',\n",
       " 'cog (deg)': 'Coursing',\n",
       " 'COG (DEG)': 'Coursing',\n",
       " 'Cog (Deg)': 'Coursing',\n",
       " 'Heading': 'Heading',\n",
       " 'heading': 'Heading',\n",
       " 'HEADING': 'Heading',\n",
       " 'Hdng': 'Heading',\n",
       " 'hdng': 'Heading',\n",
       " 'HDNG': 'Heading',\n",
       " 'True heading': 'Heading',\n",
       " 'true heading': 'Heading',\n",
       " 'TRUE HEADING': 'Heading',\n",
       " 'True Heading': 'Heading',\n",
       " 'Speed': 'Speed',\n",
       " 'speed': 'Speed',\n",
       " 'SPEED': 'Speed',\n",
       " 'SOG': 'Speed',\n",
       " 'sog': 'Speed',\n",
       " 'Sog': 'Speed',\n",
       " 'SOG (knts)': 'Speed',\n",
       " 'sog (knts)': 'Speed',\n",
       " 'SOG (KNTS)': 'Speed',\n",
       " 'Sog (Knts)': 'Speed',\n",
       " 'SPEED_KNOTS': 'Speed',\n",
       " 'speed_knots': 'Speed',\n",
       " 'Speed_Knots': 'Speed',\n",
       " 'SPEED KNOTS': 'Speed',\n",
       " 'SPEED-KNOTS': 'Speed',\n",
       " 'ROT (deg/min)': 'Turning Rate',\n",
       " 'rot (deg/min)': 'Turning Rate',\n",
       " 'ROT (DEG/MIN)': 'Turning Rate',\n",
       " 'Rot (Deg/Min)': 'Turning Rate',\n",
       " 'Turning Rate': 'Turning Rate',\n",
       " 'LOA': 'Length',\n",
       " 'loa': 'Length',\n",
       " 'Loa': 'Length',\n",
       " 'Length': 'Length',\n",
       " 'length': 'Length',\n",
       " 'LENGTH': 'Length',\n",
       " 'Beam': 'Width',\n",
       " 'beam': 'Width',\n",
       " 'BEAM': 'Width',\n",
       " 'Width': 'Width',\n",
       " 'width': 'Width',\n",
       " 'WIDTH': 'Width',\n",
       " 'A': 'A',\n",
       " 'a': 'A',\n",
       " 'DimensionA': 'A',\n",
       " 'dimensiona': 'A',\n",
       " 'DIMENSIONA': 'A',\n",
       " 'Dimensiona': 'A',\n",
       " 'B': 'B',\n",
       " 'b': 'B',\n",
       " 'DimensionB': 'B',\n",
       " 'dimensionb': 'B',\n",
       " 'DIMENSIONB': 'B',\n",
       " 'Dimensionb': 'B',\n",
       " 'C': 'C',\n",
       " 'c': 'C',\n",
       " 'DimensionC': 'C',\n",
       " 'dimensionc': 'C',\n",
       " 'DIMENSIONC': 'C',\n",
       " 'Dimensionc': 'C',\n",
       " 'D': 'D',\n",
       " 'd': 'D',\n",
       " 'DimensionD': 'D',\n",
       " 'dimensiond': 'D',\n",
       " 'DIMENSIOND': 'D',\n",
       " 'Dimensiond': 'D',\n",
       " 'NavStat': 'Status',\n",
       " 'navstat': 'Status',\n",
       " 'NAVSTAT': 'Status',\n",
       " 'Navstat': 'Status',\n",
       " 'Status': 'Status',\n",
       " 'status': 'Status',\n",
       " 'STATUS': 'Status',\n",
       " 'nav_status_code': 'Status',\n",
       " 'NAV_STATUS_CODE': 'Status',\n",
       " 'Nav_Status_Code': 'Status',\n",
       " 'nav status code': 'Status',\n",
       " 'nav-status-code': 'Status',\n",
       " 'Navigational status': 'Status',\n",
       " 'navigational status': 'Status',\n",
       " 'NAVIGATIONAL STATUS': 'Status',\n",
       " 'Navigational Status': 'Status',\n",
       " 'Msg Type': 'MessageType',\n",
       " 'msg type': 'MessageType',\n",
       " 'MSG TYPE': 'MessageType',\n",
       " 'Message Type': 'MessageType',\n",
       " 'message type': 'MessageType',\n",
       " 'MESSAGE TYPE': 'MessageType',\n",
       " 'Msg type': 'MessageType',\n",
       " 'MessageType': 'MessageType',\n",
       " '__file__': 'c:\\\\Users\\\\dere\\\\Miniconda3\\\\envs\\\\trackio\\\\lib\\\\site-packages\\\\trackio/supporting/column_mapper.csv'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "builtin_col_mapper = tio.mappers.columns\n",
    "\n",
    "builtin_col_mapper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "False\n",
      "Updated mapper in c:\\Users\\dere\\Miniconda3\\envs\\trackio\\lib\\site-packages\\trackio/supporting/column_mapper.csv\n"
     ]
    }
   ],
   "source": [
    "#check if they are already in the built-in\n",
    "print('longitude [deg]' in builtin_col_mapper.keys())\n",
    "print('latitude [deg]' in builtin_col_mapper.keys())\n",
    "\n",
    "#make new mappings\n",
    "add = {'longitude [deg]': 'X',\n",
    "       'latitude [deg]': 'Y'}\n",
    "\n",
    "#append to built in\n",
    "tio.mappers.update(tio.mappers.columns, add)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Now you can auto-detect these fields next time by passing `fill_mapper=tio.mappers.columns` to `tio.make_col_mapper`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next, you can perform the same operation on any of the raw data fields. This allows you to do a cursory scan over data fields you know might be problematic (e.g. mix of integers, floats, strings, nans, etc.) and map them to consistent values when processing the raw data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92mQCing data columns\u001b[0m: 100%|\u001b[32m██████████\u001b[0m| 1/1 [00:00<00:00, 395.47it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{0.0: None,\n",
       " 1.0: None,\n",
       " 3.0: None,\n",
       " 5.0: None,\n",
       " 8.0: None,\n",
       " 9.0: None,\n",
       " 11.0: None,\n",
       " 12.0: None,\n",
       " 15.0: None,\n",
       " nan: None}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#make a data mapper for raw data\n",
    "data_mapper = tio.make_raw_data_mapper(files,\n",
    "                                       col_mapper=col_mapper,\n",
    "                                       data_cols=['Status'],\n",
    "                                       fill_mapper={},\n",
    "                                       ncores=4)\n",
    "\n",
    "data_mapper\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## You can do the same thing for multiple data fields at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92mQCing data columns\u001b[0m: 100%|\u001b[32m██████████\u001b[0m| 1/1 [00:00<00:00, 666.40it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'Status': {0.0: None,\n",
       "  1.0: None,\n",
       "  3.0: None,\n",
       "  5.0: None,\n",
       "  8.0: None,\n",
       "  9.0: None,\n",
       "  11.0: None,\n",
       "  12.0: None,\n",
       "  15.0: None,\n",
       "  nan: None},\n",
       " 'AISCode': {31.0: None,\n",
       "  36.0: None,\n",
       "  37.0: None,\n",
       "  59.0: None,\n",
       "  60.0: None,\n",
       "  70.0: None,\n",
       "  80.0: None,\n",
       "  90.0: None,\n",
       "  nan: None}}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#make a data mapper for raw data\n",
    "data_mapper = tio.make_raw_data_mapper(files,\n",
    "                                       col_mapper=col_mapper,\n",
    "                                       data_cols=['Status', 'AISCode'],\n",
    "                                       fill_mapper={},\n",
    "                                       ncores=4)\n",
    "\n",
    "data_mapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## You can now edit these dictionaries manually and use them when processing the data. \n",
    "\n",
    "## There are also two built in mappers specifically meant for AIS data as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92mQCing data columns\u001b[0m: 100%|\u001b[32m██████████\u001b[0m| 1/1 [00:00<00:00, 666.61it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'Status': {0.0: 0,\n",
       "  1.0: 1,\n",
       "  3.0: 3,\n",
       "  5.0: 5,\n",
       "  8.0: 8,\n",
       "  9.0: 9,\n",
       "  11.0: 11,\n",
       "  12.0: 12,\n",
       "  15.0: 15,\n",
       "  nan: None},\n",
       " 'AISCode': {31.0: 31,\n",
       "  36.0: 36,\n",
       "  37.0: 37,\n",
       "  59.0: 59,\n",
       "  60.0: 60,\n",
       "  70.0: 70,\n",
       "  80.0: 80,\n",
       "  90.0: 90,\n",
       "  nan: None},\n",
       " 'Draft': {2.3: 'small',\n",
       "  2.5: 'small',\n",
       "  2.7: 'small',\n",
       "  2.9: 'small',\n",
       "  3.0: 'small',\n",
       "  3.3: 'large',\n",
       "  3.4: 'large',\n",
       "  3.5: 'large',\n",
       "  3.6: 'large',\n",
       "  3.7: 'large',\n",
       "  3.8: 'large',\n",
       "  3.9: 'large',\n",
       "  4.0: 'large',\n",
       "  4.1: 'large',\n",
       "  4.2: 'large',\n",
       "  4.3: 'large',\n",
       "  4.4: 'large',\n",
       "  4.5: 'large',\n",
       "  4.6: 'large',\n",
       "  4.9: 'large',\n",
       "  5.0: 'large',\n",
       "  5.2: 'large',\n",
       "  5.5: 'large',\n",
       "  5.8: 'large',\n",
       "  6.1: 'large',\n",
       "  9.4: 'large',\n",
       "  9.8: 'large',\n",
       "  9.9: 'large',\n",
       "  10.1: 'large',\n",
       "  12.0: 'large',\n",
       "  14.0: 'large',\n",
       "  14.5: 'large',\n",
       "  14.9: 'large',\n",
       "  nan: 'unknown'}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#make a data mapper for raw data\n",
    "data_mappers = tio.make_raw_data_mapper(files,\n",
    "                                        col_mapper=col_mapper,\n",
    "                                        data_cols=['Status','AISCode','Draft'],\n",
    "                                        fill_mapper={'Status': tio.mappers.ais['Status'],\n",
    "                                                     'AISCode': tio.mappers.ais['AISCode']},\n",
    "                                        ncores=4)\n",
    "\n",
    "#make a descriptor for draft\n",
    "for key in data_mappers['Draft'].keys():\n",
    "    if key <= 3:\n",
    "        data_mappers['Draft'][key] = 'small'\n",
    "    elif np.isnan(key):\n",
    "        data_mappers['Draft'][key] = 'unknown'\n",
    "    else:\n",
    "        data_mappers['Draft'][key] = 'large'\n",
    "        \n",
    "data_mappers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Please see notebook [`04 - Grouping Points and Splitting Tracks.ipynb`](https://github.com/derekeden/trackio/blob/main/notebooks/04%20-%20Grouping%20Points%20and%20Splitting%20Tracks.ipynb) to see how these mappers get used for the processing of raw data.\n",
    "___"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "trackio",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}