SARIG Geochemistry Data Entry Errors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SARIG Geochemistry Data Entry Errors Analysis\n",
    " \n",
    "## Author: Michael Rodda - Caldera Analytics\n",
    "\n",
    "\n",
    "<i> So a notebook that has the Data Entry Errors in its title - possibly the most boring title for a notebook on the topic that is mind numbingly dull at the best of times. But data quality is critical for data science, so why should you care? </i>\n",
    "\n",
    "The geochemistry datasets through the states of Australia are massive datasets that have accumulated assay results over the past 40+ years. Although the majority of the data is correct, there are numerous data entry errors that are scattered throughout these datasets. In the past, it is my assumption that obvious data entry errors have been cleaned up on the local scale by Exploration companies utilising historical assay data in their desktop studies. \n",
    "\n",
    "However with the rise of data science and the need to use millions of rows of data at once for the machine learning algorithms, data entry errors have the potential to cause big headaches at the state level, destabilise model training and <b>conceal valuable anomalous levels of desired metals</b>. \n",
    "\n",
    "Typically, the approach to deal with data entry errors would be to drop the most obvious outliers, but this is not a good approach, as the outliers in these geochemistry datasets are a mixture of data entry errors and <i> assay results belonging to deposits/mines</i>, which are outliers by definition.\n",
    "\n",
    "The primary purpose of this notebook is to explore the data entry errors common in state geochemistry datasets, come up with a strategy to deal with these errors and attempt to hunt down the errors smarter and more efficiently. This notebook is based on the SARIG geochemistry dataset, utilising the extract done for the ExplorerSA competition.\n",
    "\n",
    "\n",
    "This notebook has four sections:\n",
    "\n",
    "1 - Wrangling the databases into a format better for data science/analysis\n",
    "\n",
    "2 - Looking for and documenting errors that were found\n",
    "\n",
    "3 - Coming up with better ways to hunt outliers\n",
    "\n",
    "4 - Summarising the type of errors, root causes and strategies to improve the data quality\n",
    "\n",
    "Because this notebook morphed into something massive, you can probably skip to the second section if you're only interested in data entry errors.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Section One -  Format Conversion\n",
    "\n",
    "In this section, we convert the geochemistry data into three seperate tables\n",
    "\n",
    "#### Collars (locs)\n",
    "\n",
    "Table that contains information about the collar such as location, depth, name, azimuth and depth\n",
    "\n",
    "\n",
    "#### Samples (samples)\n",
    "\n",
    "Table that contains information about the samples (sample id, downhole depth start of sample), downhole depth end of sample)\n",
    "\n",
    "#### Assays (assays)\n",
    "\n",
    "Table that contains information about the assay, such as what element, what was the assay result, what was the original unit and what was the lab method used for the assay.\n",
    "\n",
    "\n",
    "\n",
    "Our focus is on the downhole geochemistry, rather than the surface chemistry.\n",
    "\n",
    "\n",
    "The first step is to load the csv file that contains information about the collars such as location, azimuth etc.\n",
    "\n",
    "<b> Note: From past experience, loading geo data with default settings in pandas can cause some rounding errors with latitude and longitudes - round_trip for float_precision seems to fix that </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DRILLHOLE_NO</th>\n",
       "      <th>DH_NAME</th>\n",
       "      <th>DH_OTHER_NAME</th>\n",
       "      <th>PACE_DH</th>\n",
       "      <th>PACE_ROUND_NO</th>\n",
       "      <th>REPRESENTATIVE_DH</th>\n",
       "      <th>REPRESENTATIVE_DH_COMMENTS</th>\n",
       "      <th>DH_UNIT_NO</th>\n",
       "      <th>MAX_DRILLED_DEPTH</th>\n",
       "      <th>MAX_DRILLED_DEPTH_DATE</th>\n",
       "      <th>...</th>\n",
       "      <th>LONGITUDE_GDA2020</th>\n",
       "      <th>LATITUDE_GDA2020</th>\n",
       "      <th>LONGITUDE_GDA94</th>\n",
       "      <th>LATITUDE_GDA94</th>\n",
       "      <th>HORIZ_ACCRCY_M</th>\n",
       "      <th>ELEVATION_M</th>\n",
       "      <th>INCLINATION</th>\n",
       "      <th>AZIMUTH</th>\n",
       "      <th>SURVEY_METHOD_CODE</th>\n",
       "      <th>SURVEY_METHOD</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>GINGERAH HILL 1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3359 1</td>\n",
       "      <td>1473.50</td>\n",
       "      <td>14/09/1986</td>\n",
       "      <td>...</td>\n",
       "      <td>122.366742</td>\n",
       "      <td>-19.469838</td>\n",
       "      <td>122.366734</td>\n",
       "      <td>-19.469824</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>BROOKE 1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3458 1</td>\n",
       "      <td>2035.10</td>\n",
       "      <td>21/07/1988</td>\n",
       "      <td>...</td>\n",
       "      <td>122.916370</td>\n",
       "      <td>-19.749266</td>\n",
       "      <td>122.916362</td>\n",
       "      <td>-19.749252</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>SAHARA 1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3555 1</td>\n",
       "      <td>2120.19</td>\n",
       "      <td>26/02/1965</td>\n",
       "      <td>...</td>\n",
       "      <td>123.392991</td>\n",
       "      <td>-21.026384</td>\n",
       "      <td>123.392983</td>\n",
       "      <td>-21.026370</td>\n",
       "      <td>200.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MAP</td>\n",
       "      <td>Map Plot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>NYALAYI 1/90</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3743 1</td>\n",
       "      <td>96.00</td>\n",
       "      <td>06/12/1990</td>\n",
       "      <td>...</td>\n",
       "      <td>124.485432</td>\n",
       "      <td>-27.189890</td>\n",
       "      <td>124.485424</td>\n",
       "      <td>-27.189876</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>GAMBANGA 1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3833 1</td>\n",
       "      <td>391.06</td>\n",
       "      <td>05/03/1960</td>\n",
       "      <td>...</td>\n",
       "      <td>124.743123</td>\n",
       "      <td>-32.290334</td>\n",
       "      <td>124.743116</td>\n",
       "      <td>-32.290320</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 51 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   DRILLHOLE_NO          DH_NAME DH_OTHER_NAME PACE_DH  PACE_ROUND_NO  \\\n",
       "0             1  GINGERAH HILL 1           NaN       N            NaN   \n",
       "1             2         BROOKE 1           NaN       N            NaN   \n",
       "2             3         SAHARA 1           NaN       N            NaN   \n",
       "3             4     NYALAYI 1/90           NaN       N            NaN   \n",
       "4             5       GAMBANGA 1           NaN       N            NaN   \n",
       "\n",
       "  REPRESENTATIVE_DH REPRESENTATIVE_DH_COMMENTS DH_UNIT_NO  MAX_DRILLED_DEPTH  \\\n",
       "0                 N                        NaN     3359 1            1473.50   \n",
       "1                 N                        NaN     3458 1            2035.10   \n",
       "2                 N                        NaN     3555 1            2120.19   \n",
       "3                 N                        NaN     3743 1              96.00   \n",
       "4                 N                        NaN     3833 1             391.06   \n",
       "\n",
       "  MAX_DRILLED_DEPTH_DATE  ...  LONGITUDE_GDA2020 LATITUDE_GDA2020  \\\n",
       "0             14/09/1986  ...         122.366742       -19.469838   \n",
       "1             21/07/1988  ...         122.916370       -19.749266   \n",
       "2             26/02/1965  ...         123.392991       -21.026384   \n",
       "3             06/12/1990  ...         124.485432       -27.189890   \n",
       "4             05/03/1960  ...         124.743123       -32.290334   \n",
       "\n",
       "  LONGITUDE_GDA94 LATITUDE_GDA94 HORIZ_ACCRCY_M ELEVATION_M INCLINATION  \\\n",
       "0      122.366734     -19.469824            NaN         NaN         NaN   \n",
       "1      122.916362     -19.749252            NaN         NaN         NaN   \n",
       "2      123.392983     -21.026370          200.0         NaN         NaN   \n",
       "3      124.485424     -27.189876            NaN         NaN         NaN   \n",
       "4      124.743116     -32.290320            NaN         NaN         NaN   \n",
       "\n",
       "  AZIMUTH SURVEY_METHOD_CODE SURVEY_METHOD  \n",
       "0     NaN                NaN           NaN  \n",
       "1     NaN                NaN           NaN  \n",
       "2     NaN                MAP      Map Plot  \n",
       "3     NaN                NaN           NaN  \n",
       "4     NaN                NaN           NaN  \n",
       "\n",
       "[5 rows x 51 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "locs = pd.read_csv('sarig_dh_details_exp.csv',\n",
    "                 encoding='cp1252',\n",
    "                 low_memory=False,\n",
    "                 float_precision='round_trip') \n",
    "\n",
    "locs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have two choices of location data - GDA94 and GDA2020. GDA2020 is chosen as the more recent holes are null for GDA94. The primary key for this table is DRILLHOLE_NO - this is appended with SA in case other states data is used at a later stage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep only relevant data\n",
    "locs = locs[['DRILLHOLE_NO', 'DH_NAME',\n",
    "            'MAX_DRILLED_DEPTH', 'MAX_DRILLED_DEPTH_DATE',\n",
    "            'LONGITUDE_GDA2020', 'LATITUDE_GDA2020',\n",
    "            'INCLINATION', 'AZIMUTH']]\n",
    "\n",
    "locs.columns = ['collar_id', 'collar_name', 'max_depth', 'date_drilled',\n",
    "               'lon', 'lat', 'dip', 'azi']\n",
    "\n",
    "locs['collar_id'] = locs['collar_id'].astype(str) + '_SA'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reference csv is loaded and joined to the collar dataframe as this contains the ID (Usually ENV xxxxx) to the source documents. This will be needed later when potential data entry errors are cross referenced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "loc_refs = pd.read_csv('sarig_dh_reference_exp.csv',\n",
    "                 encoding='cp1252',\n",
    "                 low_memory=False,\n",
    "                 float_precision='round_trip')\n",
    "\n",
    "loc_refs = loc_refs[['DRILLHOLE_NO', 'REFERENCE']].drop_duplicates()\n",
    "loc_refs.columns = ['collar_id', 'reference']\n",
    "loc_refs['collar_id'] = loc_refs['collar_id'].astype(str) + '_SA'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reference data is not in a 1:1 relationship with the drillhole location. A hacky way is to take the last row for each collar_id, with the assumption that this will be the reference id to the most recent documentation, and therefore the most recent and up to date data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "loc_refs = loc_refs.groupby('collar_id').tail(1)\n",
    "locs = locs.merge(loc_refs, \n",
    "                  how = 'left', \n",
    "                  validate = '1:1',\n",
    "                  on = 'collar_id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the geochemistry data is loaded in. Unfortunately the SARIG data has all the downhole and surface geochem merged into one file, so these needed to be pulled apart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SAMPLE_NO</th>\n",
       "      <th>SAMPLE_SOURCE_CODE</th>\n",
       "      <th>SAMPLE_SOURCE</th>\n",
       "      <th>ROCK_GROUP_CODE</th>\n",
       "      <th>ROCK_GROUP</th>\n",
       "      <th>LITHO_CODE</th>\n",
       "      <th>LITHO_CONF</th>\n",
       "      <th>LITHOLOGY_NAME</th>\n",
       "      <th>LITHO_MODIFIER</th>\n",
       "      <th>MAP_SYMBOL</th>\n",
       "      <th>...</th>\n",
       "      <th>LATITUDE_GDA94</th>\n",
       "      <th>SAMPLE_ANALYSIS_NO</th>\n",
       "      <th>OTHER_ANALYSIS_ID</th>\n",
       "      <th>ANALYSIS_TYPE_DESC</th>\n",
       "      <th>LABORATORY</th>\n",
       "      <th>CHEM_CODE</th>\n",
       "      <th>VALUE</th>\n",
       "      <th>UNIT</th>\n",
       "      <th>CHEM_METHOD_CODE</th>\n",
       "      <th>CHEM_METHOD_DESC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>589</td>\n",
       "      <td>RO</td>\n",
       "      <td>Rock outcrop / float</td>\n",
       "      <td>SSA</td>\n",
       "      <td>Sediment Siliciclastic Arenite</td>\n",
       "      <td>SDST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Sandstone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-27.069759</td>\n",
       "      <td>158</td>\n",
       "      <td>NaN</td>\n",
       "      <td>GEOCHEMISTRY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ti</td>\n",
       "      <td>0.15</td>\n",
       "      <td>%</td>\n",
       "      <td>AES</td>\n",
       "      <td>AES</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>589</td>\n",
       "      <td>RO</td>\n",
       "      <td>Rock outcrop / float</td>\n",
       "      <td>SSA</td>\n",
       "      <td>Sediment Siliciclastic Arenite</td>\n",
       "      <td>SDST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Sandstone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-27.069759</td>\n",
       "      <td>158</td>\n",
       "      <td>NaN</td>\n",
       "      <td>GEOCHEMISTRY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ag</td>\n",
       "      <td>500</td>\n",
       "      <td>ppb</td>\n",
       "      <td>AES</td>\n",
       "      <td>AES</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>589</td>\n",
       "      <td>RO</td>\n",
       "      <td>Rock outcrop / float</td>\n",
       "      <td>SSA</td>\n",
       "      <td>Sediment Siliciclastic Arenite</td>\n",
       "      <td>SDST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Sandstone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-27.069759</td>\n",
       "      <td>158</td>\n",
       "      <td>NaN</td>\n",
       "      <td>GEOCHEMISTRY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ba</td>\n",
       "      <td>1200</td>\n",
       "      <td>ppm</td>\n",
       "      <td>AES</td>\n",
       "      <td>AES</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>589</td>\n",
       "      <td>RO</td>\n",
       "      <td>Rock outcrop / float</td>\n",
       "      <td>SSA</td>\n",
       "      <td>Sediment Siliciclastic Arenite</td>\n",
       "      <td>SDST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Sandstone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-27.069759</td>\n",
       "      <td>158</td>\n",
       "      <td>NaN</td>\n",
       "      <td>GEOCHEMISTRY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Co</td>\n",
       "      <td>2</td>\n",
       "      <td>ppm</td>\n",
       "      <td>AES</td>\n",
       "      <td>AES</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>589</td>\n",
       "      <td>RO</td>\n",
       "      <td>Rock outcrop / float</td>\n",
       "      <td>SSA</td>\n",
       "      <td>Sediment Siliciclastic Arenite</td>\n",
       "      <td>SDST</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Sandstone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-27.069759</td>\n",
       "      <td>158</td>\n",
       "      <td>NaN</td>\n",
       "      <td>GEOCHEMISTRY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cr</td>\n",
       "      <td>1000</td>\n",
       "      <td>ppm</td>\n",
       "      <td>AES</td>\n",
       "      <td>AES</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   SAMPLE_NO SAMPLE_SOURCE_CODE         SAMPLE_SOURCE ROCK_GROUP_CODE  \\\n",
       "0        589                 RO  Rock outcrop / float             SSA   \n",
       "1        589                 RO  Rock outcrop / float             SSA   \n",
       "2        589                 RO  Rock outcrop / float             SSA   \n",
       "3        589                 RO  Rock outcrop / float             SSA   \n",
       "4        589                 RO  Rock outcrop / float             SSA   \n",
       "\n",
       "                       ROCK_GROUP LITHO_CODE LITHO_CONF LITHOLOGY_NAME  \\\n",
       "0  Sediment Siliciclastic Arenite       SDST        NaN      Sandstone   \n",
       "1  Sediment Siliciclastic Arenite       SDST        NaN      Sandstone   \n",
       "2  Sediment Siliciclastic Arenite       SDST        NaN      Sandstone   \n",
       "3  Sediment Siliciclastic Arenite       SDST        NaN      Sandstone   \n",
       "4  Sediment Siliciclastic Arenite       SDST        NaN      Sandstone   \n",
       "\n",
       "  LITHO_MODIFIER MAP_SYMBOL  ... LATITUDE_GDA94 SAMPLE_ANALYSIS_NO  \\\n",
       "0            NaN        NaN  ...     -27.069759                158   \n",
       "1            NaN        NaN  ...     -27.069759                158   \n",
       "2            NaN        NaN  ...     -27.069759                158   \n",
       "3            NaN        NaN  ...     -27.069759                158   \n",
       "4            NaN        NaN  ...     -27.069759                158   \n",
       "\n",
       "  OTHER_ANALYSIS_ID ANALYSIS_TYPE_DESC LABORATORY  CHEM_CODE VALUE  UNIT  \\\n",
       "0               NaN       GEOCHEMISTRY        NaN         Ti  0.15     %   \n",
       "1               NaN       GEOCHEMISTRY        NaN         Ag   500   ppb   \n",
       "2               NaN       GEOCHEMISTRY        NaN         Ba  1200   ppm   \n",
       "3               NaN       GEOCHEMISTRY        NaN         Co     2   ppm   \n",
       "4               NaN       GEOCHEMISTRY        NaN         Cr  1000   ppm   \n",
       "\n",
       "   CHEM_METHOD_CODE  CHEM_METHOD_DESC  \n",
       "0               AES               AES  \n",
       "1               AES               AES  \n",
       "2               AES               AES  \n",
       "3               AES               AES  \n",
       "4               AES               AES  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rs = pd.read_csv('sarig_rs_chem_exp.csv',\n",
    "           low_memory=False)\n",
    "rs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With sample source we can roughly see the type of geochemistry assay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Drill cuttings                                                  10956887\n",
       "Sawn half drill core                                             7256976\n",
       "Drill core                                                       6533477\n",
       "Calcrete                                                         2154614\n",
       "Soil                                                              989296\n",
       "Drilled interval rock sample, type unspecified                    970508\n",
       "Rock outcrop / float                                              537537\n",
       "Sawn quarter drill core                                           237103\n",
       "Drillhole log data - used in calculating values                   212746\n",
       "Stream sediment                                                   164251\n",
       "Auger sample from near surface                                    105514\n",
       "Vegetation                                                         71018\n",
       "Core sludge                                                        56217\n",
       "A full face slice of core                                          52531\n",
       "Gravel                                                             37740\n",
       "Pulp / powdered rock of known origin, typically a lab return       16428\n",
       "Magnetic lag fraction                                              10135\n",
       "Duplicate: a split from an existing sample.                         8646\n",
       "Sample specifically for bulk leach extractable gold analysis        7826\n",
       "Aircore: consolidated sample from aircore drilling method           7488\n",
       "Analytical standard check sample                                    6358\n",
       "Mine rock sample                                                    6203\n",
       "Channel (linear rock chip/outcrop sampling)                         4453\n",
       "Rock outcrop sample                                                 4086\n",
       "Lag sample, surface sample consisting of gravelly material.         3918\n",
       "Costean (trench)                                                    3746\n",
       "Filleted, shaved or ground core sample                              3478\n",
       "Mine mullock / well spoil                                           3086\n",
       "Loam                                                                2679\n",
       "Mine stockpile                                                      2248\n",
       "Lake floor sediment                                                 1799\n",
       "Excrement of animals                                                 772\n",
       "Mine tailings                                                        347\n",
       "Rock float sample (not in situ)                                      335\n",
       "Drillhole                                                            332\n",
       "Smelter slag                                                         293\n",
       "Rock subcrop sample (in situ, not attached to bedrock)                67\n",
       "Single mineral                                                        40\n",
       "Bulk (high volume) sample, diamond exploration                        19\n",
       "Core Library archival rock, original source unknown                   12\n",
       "Name: SAMPLE_SOURCE, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rs['SAMPLE_SOURCE'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The quick and effective way to seperate is to look at if there is a corresponding collar id in the geochemistry data. Keep only the ones that do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "rs = rs[~(rs['DRILLHOLE_NUMBER'].isnull()) &\n",
    "   ~(rs['DH_NAME'].isnull())]\n",
    "\n",
    "# Do some minor wrangling\n",
    "# Keep the relevant columns\n",
    "rs = rs[['SAMPLE_NO', 'SAMPLE_SOURCE', 'DRILLHOLE_NUMBER',\n",
    "    'DH_DEPTH_FROM', 'DH_DEPTH_TO', 'CHEM_CODE', 'VALUE', 'UNIT',\n",
    "    'CHEM_METHOD_CODE']]\n",
    "\n",
    "# Change the sample and collar ids to our format that requires the state to be appended to the end\n",
    "rs['SAMPLE_NO'] = rs['SAMPLE_NO'].astype('int').astype('str') + '_SA'\n",
    "rs['DRILLHOLE_NUMBER'] = rs['DRILLHOLE_NUMBER'].astype('int').astype('str') + '_SA'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As per the target format of having three seperate tables of collars, samples and assays, the rs dataframe is split into samples and assays.\n",
    "\n",
    "Samples contains:\n",
    "- Sample ID\n",
    "- Sample Name\n",
    "- Drillhole Name\n",
    "- Drillhole Number\n",
    "- Start of sample (Downhole depth)\n",
    "- End of sample (Downhole depth)\n",
    "\n",
    "Assays contains:\n",
    "- Sample ID\n",
    "- Element\n",
    "- Unit (ppm/ppb)\n",
    "- Lab method\n",
    "- Result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = rs[['SAMPLE_NO', 'SAMPLE_SOURCE', 'DRILLHOLE_NUMBER',\n",
    "              'DH_DEPTH_FROM', 'DH_DEPTH_TO']].drop_duplicates()\n",
    "\n",
    "assays = rs[['SAMPLE_NO', 'CHEM_CODE', 'VALUE', 'UNIT', 'CHEM_METHOD_CODE']].drop_duplicates()\n",
    "\n",
    "del rs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 - Assay Cleaning\n",
    "\n",
    "The assay results need some standardisation:\n",
    "\n",
    "- Some elements are listed as oxides. As the majority of assay results are for the base element, any oxide assay results are converted to their base element.\n",
    "\n",
    "- Assay results are in different units. These needs to be standardised to ppm to be in an appropriate format for data analysis.\n",
    "\n",
    "The unit standardisation is dealt with first. What units are present?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ppm    19923316\n",
       "%       6089681\n",
       "ppb      292301\n",
       "g/T       41321\n",
       "cps        3632\n",
       "Name: UNIT, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assays['UNIT'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before any conversion happens, the result column has to be converted into a numeric format, but cant be done yet because there are numerous assay results with > or < in them, denoting a result that is above  or below the detection limit of the lab method.\n",
    "\n",
    "For the following cases:\n",
    "- String starts with > (Above Detection limit): Drop the >\n",
    "- String starts with < (Less than detection limit): Drop the <, set to zero.\n",
    "\n",
    "The method for dealing with the below detection limit is not ideal. Theoretically the assay result is going to sit somewhere between 0 and the lower detection limit of the assay, so this should be kept in mind as our method will alter the means and medians. Possible workarounds would be to get the lower detection limit of each assay and take a random number between 0 and the lower limit, but this would be a lot of manual work and is outside of the scope of the notebook.\n",
    "\n",
    "Once the data is converted into number format, any assay value in the negatives is also converted to 0, as this usually means a below detection limit / trace value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "assays['isUDL'] = np.where(assays['VALUE'].str.contains('>'), 1, 0)\n",
    "\n",
    "assays['VALUE'] = assays['VALUE'].str.replace('>', '')\n",
    "\n",
    "assays.loc[assays['VALUE'].str.contains('<'), 'VALUE'] = 0\n",
    "assays['VALUE'] = pd.to_numeric(assays['VALUE'])\n",
    "assays.loc[assays['VALUE'] < 0, 'VALUE'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the assay data can be standardised to ppm:\n",
    "\n",
    "- ppb is divided by 1000\n",
    "- percents are multiplied by 10000\n",
    "- g/t is left the same\n",
    "- cps is associated with Uranium assays and are considered outside the scope of the notebook.\n",
    "\n",
    "The original unit is kept for error checking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "assays.loc[assays['UNIT'] == 'ppb', 'VALUE'] = assays.loc[assays['UNIT'] == 'ppb', 'VALUE'] / 1000\n",
    "assays.loc[assays['UNIT'] == '%', 'VALUE'] = assays.loc[assays['UNIT'] == '%', 'VALUE'] * 10000\n",
    "assays = assays[assays['UNIT'] != 'cps']  #Uranium\n",
    "assays = assays.rename({'UNIT': 'old_unit'}, axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SAMPLE_NO</th>\n",
       "      <th>CHEM_CODE</th>\n",
       "      <th>VALUE</th>\n",
       "      <th>old_unit</th>\n",
       "      <th>CHEM_METHOD_CODE</th>\n",
       "      <th>isUDL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2078806_SA</td>\n",
       "      <td>U3O8</td>\n",
       "      <td>5.0</td>\n",
       "      <td>ppm</td>\n",
       "      <td>XRF1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>2078802_SA</td>\n",
       "      <td>U3O8</td>\n",
       "      <td>5.0</td>\n",
       "      <td>ppm</td>\n",
       "      <td>XRF1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>2078803_SA</td>\n",
       "      <td>U3O8</td>\n",
       "      <td>5.0</td>\n",
       "      <td>ppm</td>\n",
       "      <td>XRF1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>2078804_SA</td>\n",
       "      <td>U3O8</td>\n",
       "      <td>5.0</td>\n",
       "      <td>ppm</td>\n",
       "      <td>XRF1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>2078805_SA</td>\n",
       "      <td>U3O8</td>\n",
       "      <td>5.0</td>\n",
       "      <td>ppm</td>\n",
       "      <td>XRF1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30445423</th>\n",
       "      <td>2979044_SA</td>\n",
       "      <td>Te</td>\n",
       "      <td>0.1</td>\n",
       "      <td>ppm</td>\n",
       "      <td>IC1M</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30445424</th>\n",
       "      <td>2979044_SA</td>\n",
       "      <td>Tl</td>\n",
       "      <td>0.5</td>\n",
       "      <td>ppm</td>\n",
       "      <td>IC1M</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30445425</th>\n",
       "      <td>2979044_SA</td>\n",
       "      <td>Tm</td>\n",
       "      <td>0.5</td>\n",
       "      <td>ppm</td>\n",
       "      <td>IC4R</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30445426</th>\n",
       "      <td>2979044_SA</td>\n",
       "      <td>Yb</td>\n",
       "      <td>5.0</td>\n",
       "      <td>ppm</td>\n",
       "      <td>IC4R</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30445427</th>\n",
       "      <td>2979044_SA</td>\n",
       "      <td>Zn</td>\n",
       "      <td>67.0</td>\n",
       "      <td>ppm</td>\n",
       "      <td>IC1M</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>26346619 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           SAMPLE_NO CHEM_CODE  VALUE old_unit CHEM_METHOD_CODE  isUDL\n",
       "14        2078806_SA      U3O8    5.0      ppm             XRF1      0\n",
       "37        2078802_SA      U3O8    5.0      ppm             XRF1      0\n",
       "38        2078803_SA      U3O8    5.0      ppm             XRF1      0\n",
       "39        2078804_SA      U3O8    5.0      ppm             XRF1      0\n",
       "40        2078805_SA      U3O8    5.0      ppm             XRF1      0\n",
       "...              ...       ...    ...      ...              ...    ...\n",
       "30445423  2979044_SA        Te    0.1      ppm             IC1M      0\n",
       "30445424  2979044_SA        Tl    0.5      ppm             IC1M      0\n",
       "30445425  2979044_SA        Tm    0.5      ppm             IC4R      0\n",
       "30445426  2979044_SA        Yb    5.0      ppm             IC4R      0\n",
       "30445427  2979044_SA        Zn   67.0      ppm             IC1M      0\n",
       "\n",
       "[26346619 rows x 6 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for element conversion. The following needs to be done:\n",
    "\n",
    "- Drop weird elements\n",
    "- Convert Oxides to their Base Element\n",
    "\n",
    "What needs to be dropped? The 50 least common values in the element column are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gd           93845\n",
      "Er           93638\n",
      "Tm           93600\n",
      "Ho           93558\n",
      "Li           65813\n",
      "MnO          64472\n",
      "Cl           60273\n",
      "Re           55325\n",
      "Fe2O3        55298\n",
      "Ge           44285\n",
      "Hg           34945\n",
      "Cr2O3        32438\n",
      "CO2          31789\n",
      "B            31251\n",
      "V2O5         20903\n",
      "F            15161\n",
      "SO3          11794\n",
      "BaO           7307\n",
      "Total         6618\n",
      "SrO           2986\n",
      "FeO           2964\n",
      "NiO           2593\n",
      "ZrO2          2593\n",
      "ZnO           2593\n",
      "CoO           2593\n",
      "C             1521\n",
      "ThO2           549\n",
      "Ru             439\n",
      "WO3            394\n",
      "Nb2O5          362\n",
      "Ta2O5          358\n",
      "TOT/C          357\n",
      "TOT/S          351\n",
      "CaCO3          251\n",
      "Ir             182\n",
      "Br             181\n",
      "GPSM           153\n",
      "H2O_minus      144\n",
      "MgCO3          138\n",
      "H2O_plus       132\n",
      "Insol           90\n",
      "GoI             65\n",
      "Rh              63\n",
      "Os              60\n",
      "CO3             57\n",
      "NaCl            56\n",
      "H2O             54\n",
      "SO4              4\n",
      "Sr87_86          3\n",
      "CaSO4            3\n",
      "Name: CHEM_CODE, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(assays['CHEM_CODE'].value_counts()[-50:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Old Length: 26346619\n",
      "New Length after dropping non-compliant elements: 25874780\n"
     ]
    }
   ],
   "source": [
    "elements_to_keep = ['B', 'S', 'U', 'P', 'Si',  'Au', 'Cu', 'Zn', 'Pb', 'Ag', 'As', 'Fe', 'Ni',\n",
    "                    'Mn', 'Co', 'Mo', 'Bi', 'Cr', 'La', 'Ce', 'Mg',\n",
    "                   'Ba', 'K', 'Ca', 'V', 'Sb', 'Na', 'Sr', 'Ti', 'Th',\n",
    "                   'Al', 'Cd', 'Y', 'W', 'Sc', 'Ga', 'Zr', 'Tl', 'Be',\n",
    "                   'Sn', 'Se', 'Te', 'Nb', 'Rb', 'Cs', 'Ta', 'Pd', 'In',\n",
    "                   'Pt', 'Hf', 'Nd', 'Yb', 'Sm', 'Pr', 'Lu', 'Tb', 'Dy',\n",
    "                   'Eu', 'Gd', 'Er', 'Tm', 'Ho', 'Li', 'Re', 'Ge', 'Hg',\n",
    "                   'Ru', 'Ir', 'Rh', 'Os', 'F']\n",
    "oxides_to_convert = ['MgO', 'CaO', 'Al2O3', 'TiO2', 'K2O', 'Na2O', 'MnO',\n",
    "                     'Fe2O3', 'V2O5', 'Cr2O3', 'Nb2O5', 'ZnO', 'NiO', 'ThO2',\n",
    "                    'WO3', 'BaO', 'Ta2O5', 'U3O8', 'P2O5', 'SiO2', 'FeO', 'SrO']\n",
    "\n",
    "print(\"Old Length: {}\".format(len(assays)))\n",
    "assays = assays[assays['CHEM_CODE'].isin(elements_to_keep + oxides_to_convert)]\n",
    "print(\"New Length after dropping non-compliant elements: {}\".format(len(assays)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the oxides are converted to their base elements and multiplied by the required molar mass difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from geochem_utils import *\n",
    "\n",
    "#Convert our oxides\n",
    "assays = clean_oxides(assays, 'CHEM_CODE', 'VALUE')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final step is to rename the column names to the same format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples.columns = ['sample_id', 'sample_source', 'collar_id', 'from', 'to']\n",
    "assays.columns = ['sample_id', 'ele', 'result', 'old_unit', 'method', 'isUDL']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section Two -  Data Entry Errors\n",
    "\n",
    "So now for the real stuff. Data entry errors. We know that errors will be in the database - after Section One there are ~26 million assay results for South Australia, so it is just not possible that this will be error free. But how deep does this rabbit hole go?\n",
    "\n",
    "From my past experience dealing with the QLD and NSW geochem databases, my understanding was that the South Australia data was relevatively error free. But after hunting for errors for this notebook, this is not the case.\n",
    "\n",
    "The strategy to start off with is to simply look at the very highest values and the very lowest values that are above zero. We create a new column called fixed_result that is initially null. When an exploration report is looked at and the data is legitimate, the original result is carried over to the fixed_result column. But if the data is wrong -\n",
    "\n",
    "- If it can be corrected, the correct value is set in the fixed_result column\n",
    "- If it cant be corrected (real value is unknown), it is set to be -777\n",
    "\n",
    "\n",
    "It generally helps to understand the typical distribution of elements - if there is a gold assay with 600,000 ppm, you've either found the best gold mine in the world or found a data entry error. Likewise if you find a iron assay of 0.001ppm, chances are its a data entry error. \n",
    "\n",
    "\n",
    "Create a temp dataframe to collate all our data for error checking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp = locs.merge(samples,\n",
    "                 on = 'collar_id',\n",
    "                 validate = '1:m').merge(assays,\n",
    "                                         on = 'sample_id',\n",
    "                                         validate = '1:m').drop(['lon', 'lat', 'dip', 'azi'], axis = 1)\n",
    "\n",
    "tmp['fixed_result'] = np.nan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 - Error Checking & Evidence\n",
    "\n",
    "### Collar Name - RC03SF07, Tasman Resources - Data Release: ENV 09786\n",
    "\n",
    "\n",
    "<b>What I noticed:</b> Really high values of La and Sb\n",
    "\n",
    "![title](img/rc03tasman.png)\n",
    "\n",
    "<b>What was the root cause:</b> Column headers (element) has shifted causing misalignment.\n",
    "\n",
    "![title](img/rc03actual.png)\n",
    "\n",
    "It may be hard to see, but the assays are correct until the blank column, where everything is misaligned after that. So the high values for La are actually Ca assays and the high values for Sb is actually Fe.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "messed_up_columns = ['Ag', 'S', 'Mn', 'Co', 'Ti', 'Sb', 'Fe', 'Cu', 'Ce', 'Ca', 'La', 'U', 'Y',\n",
    "                     'Ni', 'Mo', 'Sr', 'Au']\n",
    "\n",
    "for ele in messed_up_columns:\n",
    "    tmp.loc[(tmp['collar_name'] == 'RC03SF007') & \n",
    "        (tmp['ele'] == ele), 'fixed_result'] = -777"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Collar Name 15_SHR_006 - Data Release: ENV 11877\n",
    "\n",
    "<b>What I noticed:</b> Really high values of Te\n",
    "\n",
    "![title](img/ex2SARIG.png)\n",
    "\n",
    "<b>What was the root cause:</b> Column misalignment resulting from lack of S repeat assays\n",
    "\n",
    "\n",
    "![title](img/ex2Actual.png)\n",
    "\n",
    "When S repeat assays are not present, the columns to the right of S become misaligned so Te on SARIG is actual Ti assays\n",
    "\n",
    "\n",
    "<b>Whats really bad about this error?:</b> Zinc assays have been cut off and are not present - and one of the non-present Zn assays has a ppm value of >8000, which could be a potential indicator.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mark Errors\n",
    "tmp.loc[(tmp['collar_id'] == '295497_SA') & \n",
    "    (tmp['ele'].isin(['Sb', 'Sc', 'Sn', 'Sr', 'Te', 'Ti', 'Tl', 'V', 'W', 'Zn'])) & \n",
    "     ~(tmp['sample_id'].isin(['2369457_SA', '2369463_SA', '2369459_SA', '2369458_SA'])), 'fixed_result'] = -777\n",
    "\n",
    "# Carry over the correct assays\n",
    "tmp.loc[(tmp['collar_id'] == '295497_SA') & \n",
    "        (tmp['fixed_result'].isnull()), 'fixed_result'] = tmp.loc[(tmp['collar_id'] == '295497_SA') & \n",
    "                                                        (tmp['fixed_result'].isnull()), 'result']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Release ENV02073 - Sediment Hosted Copper Drilling\n",
    "\n",
    "<b>What I noticed:</b> Very low Cu values mixed in with high Cu values in the same drill hole\n",
    "\n",
    "\n",
    "\n",
    "<b>What was the root cause:</b> When the base metal values (Cu, Zn, Pb) start to increase, the drill logs switch over to % units rather than the initial ppm units. SARIG has all of the assays as ppm.\n",
    "\n",
    "![title](img/ex3actual.png)\n",
    "\n",
    "<b>Whats really bad about this error?:</b> The highish base metal assays are smaller by a factor of 10000, masking anomalous assay levels which could be potential deposit indicators\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fix the copper values\n",
    "tmp.loc[(tmp['reference'] == 'ENV 02703') &\n",
    "    (tmp['result'] > 0) &\n",
    "    (tmp['result'] < 3), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 02703') &\n",
    "                                            (tmp['result'] > 0) &\n",
    "                                            (tmp['result'] < 3), 'result'] * 10000\n",
    "\n",
    "# Carry over the correct values\n",
    "tmp.loc[(tmp['reference'] == 'ENV 02703') &\n",
    "    (tmp['result'] >= 3), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 02703') &\n",
    "                                            (tmp['result'] >= 3), 'result'] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Investigator Resources Silver Assays - Data Release ENV 12043\n",
    "\n",
    "<b>What I noticed:</b> All Silver assays are at extreme levels (including a 256000ppm value)\n",
    "\n",
    "![ex4sarig](img/ex4sarig.png)\n",
    "\n",
    "<b>What was the root cause:</b> Silver assays are listed in SARIG as being %, so they thave been multiplied by 10000 when we have standardised assays to ppm. The source is from the assay data, where Al is listed as ppm and Ag is listed as %, when it should be the other way around.\n",
    "\n",
    "![ex4actual](img/ex4actual.png)\n",
    "\n",
    "<b>Whats really bad about this error?:</b> The huge amount of assays at extreme levels will cause issues for machine learning if untreated Silver assays are used as an input\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp.loc[(tmp['reference'] == 'ENV 12043') & \n",
    "        (tmp['ele'] == 'Ag'), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 12043') & \n",
    "                                            (tmp['ele'] == 'Ag'), 'result'] / 10000\n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 12043') & \n",
    "        (tmp['ele'] == 'Al'), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 12043') & \n",
    "                                            (tmp['ele'] == 'Al'), 'result'] * 10000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ENV 11042 Data Release\n",
    "\n",
    "\n",
    "<b>What I noticed:</b> High Gold Assays\n",
    "\n",
    "\n",
    "![sarig](img/11042sarig.png)\n",
    "\n",
    "\n",
    "<b>What was the root cause:</b> All the Au assays that are listed as B/ETA should be ppb. Other Au assays in the data release with the lab method as B/ETA have their unit as ppb. Checking other ppm B/ETA assays supports this. Unforunately no reference data is there to support this but the Annual Report highlights grades of 1.4ppm, not 80ppm.\n",
    "\n",
    "\n",
    "<b>Whats really bad about this error?:</b> Gold assays are really high, but it is associated with a report for a historical prospect, so someone might think this high assays are normal\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If the method is B/ETA and units are ppm, divide by 1000\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['reference'] == 'ENV 11042') & \n",
    "    (tmp['method'] == 'B/ETA') &\n",
    "    (tmp['old_unit'] == 'ppm'), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                                (tmp['reference'] == 'ENV 11042') & \n",
    "                                                (tmp['method'] == 'B/ETA') &\n",
    "                                                (tmp['old_unit'] == 'ppm'), 'result'] / 1000\n",
    "\n",
    "# Carry over the rest\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['reference'] == 'ENV 11042') & \n",
    "    (tmp['fixed_result'].isnull()), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                                        (tmp['reference'] == 'ENV 11042') & \n",
    "                                                        (tmp['fixed_result'].isnull()), 'result']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PDH 2 Drill Hole\n",
    "\n",
    "![title](img/pdh2.png)\n",
    "\n",
    "\n",
    "<b>What I noticed:</b> High gold assays\n",
    "\n",
    "<b>What was the root cause:</b> Not sure exactly but the switch over to ppb in the same hole seems to indicate that the ppm assays are off by 1000. Unforunately 0 reference documents, but the lack of drilling in the area for insane (but plausible) gold assays indicates that the original units are probably ppb rather than ppm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['collar_name'] == 'PDH 2') &\n",
    "    (tmp['old_unit'] == 'ppm'), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                                (tmp['collar_name'] == 'PDH 2') &\n",
    "                                                (tmp['old_unit'] == 'ppm'), 'result'] / 1000\n",
    "\n",
    "\n",
    "\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['collar_name'] == 'PDH 2') &\n",
    "    (tmp['old_unit'] != 'ppm'), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                                (tmp['collar_name'] == 'PDH 2') &\n",
    "                                                (tmp['old_unit'] != 'ppm'), 'result']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ENV 06960 Prom Hill Data Release\n",
    "\n",
    "### DD09/DD10/DD11 Au/Pt/Pd Assays\n",
    "\n",
    "\n",
    "<b>What I noticed:</b> Very high gold assays\n",
    "\n",
    "![title](img/ptpdau.png)\n",
    "\n",
    "<b>What was the root cause:</b> Assay that produced results for Au/Pt/Pd is in ppm when it should be ppb.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['collar_name'].str.startswith('DD')) &\n",
    "    (tmp['reference'] == 'ENV 06960') &\n",
    "    (tmp['method'] == 'FA2') &\n",
    "    (tmp['result'] > 0) &\n",
    "    (tmp['result'] >= 1), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                        (tmp['collar_name'].str.startswith('DD')) &\n",
    "                                        (tmp['reference'] == 'ENV 06960') &\n",
    "                                        (tmp['method'] == 'FA2') &\n",
    "                                        (tmp['result'] > 0) &\n",
    "                                        (tmp['result'] >= 1), 'result'] / 1000\n",
    "\n",
    "\n",
    "#Anything below one is correct\n",
    "# These are the assays from 2012 onwards\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['collar_name'].str.startswith('DD')) &\n",
    "    (tmp['reference'] == 'ENV 06960') &\n",
    "    (tmp['method'] == 'FA2') &\n",
    "    (tmp['result'] > 0) &\n",
    "    (tmp['result'] < 1), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                        (tmp['collar_name'].str.startswith('DD')) &\n",
    "                                        (tmp['reference'] == 'ENV 06960') &\n",
    "                                        (tmp['method'] == 'FA2') &\n",
    "                                        (tmp['result'] > 0) &\n",
    "                                        (tmp['result'] < 1), 'result'] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ENV  09020 Data Release\n",
    "\n",
    "<b>What I noticed:</b> Exceptionally low Au assays\n",
    "\n",
    "<b>What was the root cause:</b>: The NHRC holes have the units listed as ppb when its actually ppm (verified in the reference data), causing the Au assays to be low by a factor of 1000\n",
    "\n",
    "<b>Whats really bad about this error?:</b> Some of these 'low' assays turn out to be Au 1-4ppm, which are anomalous results and could be potential indicators for a Gold deposit\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['result'] > 0) &\n",
    "    (tmp['collar_name'].str.startswith('NHRC')) &\n",
    "    (tmp['reference'] == 'ENV 09020') &\n",
    "    (tmp['fixed_result'].isnull()), 'fixed_result'] =  tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                            (tmp['result'] > 0) &\n",
    "                                            (tmp['collar_name'].str.startswith('NHRC')) &\n",
    "                                            (tmp['reference'] == 'ENV 09020') &\n",
    "                                            (tmp['fixed_result'].isnull()), 'result'] * 1000 \n",
    "\n",
    "\n",
    "# Rest are correct\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['result'] > 0) &\n",
    "    ~(tmp['collar_name'].str.startswith('NHRC')) &\n",
    "    (tmp['reference'] == 'ENV 09020') &\n",
    "    (tmp['fixed_result'].isnull()), 'fixed_result'] =  tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                            (tmp['result'] > 0) &\n",
    "                                            ~(tmp['collar_name'].str.startswith('NHRC')) &\n",
    "                                            (tmp['reference'] == 'ENV 09020') &\n",
    "                                            (tmp['fixed_result'].isnull()), 'result']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GFDH Drill Holes - Lincoln Minerals/ Gum Flat Project\n",
    "\n",
    "- What I noticed: Highish Gold assays from a XRF source\n",
    "\n",
    "![lincoln](img/lincoln.png)\n",
    "\n",
    "\n",
    "- Root Cause: Unable to verify due to no reference documents available, but it is clear that the assays for holes with collar names starting with GFDH that have their original unit as % are wrong.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['result'] > 0) &\n",
    "    (tmp['result'] >= 100) &\n",
    "    (tmp['collar_name'].str.startswith('GFDH')) &\n",
    "    (tmp['fixed_result'].isnull()), 'fixed_result'] = -777\n",
    "\n",
    "#Rest are correct\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['result'] > 0) &\n",
    "    (tmp['result'] < 100) &\n",
    "    (tmp['collar_name'].str.startswith('GFDH')) &\n",
    "    (tmp['fixed_result'].isnull()), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                            (tmp['result'] > 0) &\n",
    "                                            (tmp['result'] < 100) &\n",
    "                                            (tmp['collar_name'].str.startswith('GFDH')) &\n",
    "                                            (tmp['fixed_result'].isnull()), 'result']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Errors\n",
    "\n",
    "The other errors and non errors are changed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Changing:  Na  to  Ag\n",
      "Changing:  Ag  to  As\n",
      "Changing:  U  to  Ba\n",
      "Changing:  Co  to  Ce\n",
      "Changing:  As  to  Co\n",
      "Changing:  Ba  to  Cr\n",
      "Changing:  K  to  Fe\n",
      "Changing:  La  to  K\n",
      "Changing:  Mg  to  La\n",
      "Changing:  Ni  to  Mg\n",
      "Changing:  Co  to  Na\n",
      "Changing:  Fe  to  Ni\n",
      "Changing:  As  to  Pb\n",
      "Changing:  Ce  to  U\n"
     ]
    }
   ],
   "source": [
    "### Data Release  RB 93/00004\n",
    "###What I noticed: Very low gold assays (0.000001 ppm)\n",
    "###Root Cause: Section of the data release has assays listed as ppb when they are actually already in ppm, causing them to be lower by a factor of 1000\n",
    "\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['result'] > 0) &\n",
    "    (tmp['result'] < 0.001) &\n",
    "    (tmp['reference'] == 'RB 93/00004') &\n",
    "    (tmp['fixed_result'].isnull()), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                            (tmp['result'] > 0) &\n",
    "                                            (tmp['result'] < 0.001) &\n",
    "                                            (tmp['reference'] == 'RB 93/00004') &\n",
    "                                            (tmp['fixed_result'].isnull()), 'result'] * 1000\n",
    "\n",
    "# Rest are correct\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['result'] >= 0.001) &\n",
    "    (tmp['reference'] == 'RB 93/00004') &\n",
    "    (tmp['fixed_result'].isnull()), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                            (tmp['result'] >= 0.001) &\n",
    "                                            (tmp['reference'] == 'RB 93/00004') &\n",
    "                                            (tmp['fixed_result'].isnull()), 'result'] \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "#12023\n",
    "# Two holes with Zinc Values at 99/100%\n",
    "# The rest of the zinc values appear to be normal\n",
    "# Set these to -777, our unknown but wrong value\n",
    "# We have not checked the tail end, but it appears to be correct\n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 12023') & \n",
    "    (tmp['ele'] == 'Zn') &\n",
    "    (tmp['result'] > 50000), 'fixed_result'] = -777\n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 12023') & \n",
    "    (tmp['ele'] == 'Zn') &\n",
    "    (tmp['result'] < 50000) &\n",
    "    (tmp['result'] > 500), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 12023') & \n",
    "                                                (tmp['ele'] == 'Zn') &\n",
    "                                                (tmp['result'] < 50000) &\n",
    "                                                (tmp['result'] > 500), 'result']\n",
    "\n",
    "\n",
    "# Collar ID -312946\n",
    "# Collar Name - RD2785\n",
    "# Olympic Dam\n",
    "# Lead Values Appear to be ppm but are listed as %?\n",
    "tmp.loc[(tmp['collar_id'] == '312946_SA') & \n",
    "    (tmp['ele'] == 'Pb'), 'fixed_result'] = tmp.loc[(tmp['collar_id'] == '312946_SA') & \n",
    "                                            (tmp['ele'] == 'Pb'), 'result'] / 10000\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# ENV09793\n",
    "# HNWPR95A Has Zinc Values at 80-220% \n",
    "# Most likely a percentage - ppm error?\n",
    "# We would divide by 10000, but it looks like the 220% values have been capped to 100%\n",
    "tmp.loc[(tmp['reference'] == 'ENV 09793') & \n",
    "    (tmp['ele'] == 'Zn') &\n",
    "    (tmp['result'] > 500000), 'fixed_result'] = -777\n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 09793') & \n",
    "    (tmp['ele'] == 'Zn') &\n",
    "    (tmp['result'] < 500000), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 09793') & \n",
    "                                            (tmp['ele'] == 'Zn') &\n",
    "                                            (tmp['result'] < 500000), 'result']\n",
    "\n",
    "\n",
    "\n",
    "# 11TDRC001 Collar Name\n",
    "# 301204 Collar ID\n",
    "# 100% Lead Values when stuff like Fe is 4%\n",
    "# Confirmed to be correct in source documents and isnt XRF- probably just variance at high lead values?\n",
    "tmp.loc[(tmp['collar_name'] == '11TDRC001'), 'fixed_result']   = tmp.loc[(tmp['collar_name'] == '11TDRC001'), 'result'] \n",
    "#Also The Same for 11TDRC007\n",
    "tmp.loc[(tmp['collar_name'] == '11TDRC007'), 'fixed_result']   = tmp.loc[(tmp['collar_name'] == '11TDRC007') , 'result'] \n",
    "#Also The Same for 11TDRC009\n",
    "tmp.loc[(tmp['collar_name'] == '11TDRC009'), 'fixed_result']   = tmp.loc[(tmp['collar_name'] == '11TDRC009') , 'result'] \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# RC03SF07\n",
    "# Real K is Sarig Ag\n",
    "# Ag is S \n",
    "# S is Mn\n",
    "# Mn is Co\n",
    "# Co is Co\n",
    "# Co is Ti\n",
    "# Ti is Sb\n",
    "# Sb is Fe\n",
    "# Fe is Cu\n",
    "# Cu is Ce\n",
    "# Ce is Ca\n",
    "# Ca is La\n",
    "# La is U\n",
    "# U is Ag\n",
    "# Ag is is Cu\n",
    "# Cu is Y\n",
    "# Y is Ni\n",
    "# Ni is Mo\n",
    "# Mo is Sr\n",
    "# Sr is La\n",
    "# La is Ce\n",
    "# Ce is Au\n",
    "messed_up_columns = ['Ag', 'S', 'Mn', 'Co', 'Ti', 'Sb', 'Fe', 'Cu', 'Ce', 'Ca', 'La', 'U', 'Y',\n",
    "                     'Ni', 'Mo', 'Sr', 'Au']\n",
    "\n",
    "for ele in messed_up_columns:\n",
    "    tmp.loc[(tmp['collar_name'] == 'RC03SF007') & \n",
    "        (tmp['ele'] == ele), 'fixed_result'] = -777\n",
    "\n",
    "    \n",
    "# ENV 11877\n",
    "# CollarName 15_SHR_006\n",
    "# The samples where zinc is present - \n",
    "# Are Good\n",
    "# The Samples where zinc is not present - \n",
    "# Sb Sc Sn Sr Te Ti Tl V W Zn are wrong\n",
    "# The cause of this is a suplhur repeat assay\n",
    "tmp.loc[(tmp['collar_id'] == '295497_SA') & \n",
    "    (tmp['ele'].isin(['Sb', 'Sc', 'Sn', 'Sr', 'Te', 'Ti', 'Tl', 'V', 'W', 'Zn'])) & \n",
    "     ~(tmp['sample_id'].isin(['2369457_SA', '2369463_SA', '2369459_SA', '2369458_SA'])), 'fixed_result'] = -777\n",
    "\n",
    "tmp.loc[(tmp['collar_id'] == '295497_SA') & \n",
    "        (tmp['fixed_result'].isnull()), 'fixed_result'] = tmp.loc[(tmp['collar_id'] == '295497_SA') & \n",
    "                                                        (tmp['fixed_result'].isnull()), 'result']\n",
    "# Can potentially add in the other drillholes from the reference as they appear to be okay\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# BHT Min PACE Drilling Program\n",
    "# Sc appears to be out by 10000 (ppm not %)\n",
    "# Evidence: one drillhole where Sc is 40% but the sum of percentages is at least 120%\n",
    "# Lab Max for Sc is 10000ppm\n",
    "tmp.loc[(tmp['collar_name'].str.startswith('WD16')) &\n",
    "        (tmp['ele'] == 'Sc'), 'fixed_result'] = tmp.loc[(tmp['collar_name'].str.startswith('WD16')) &\n",
    "                                                       (tmp['ele'] == 'Sc'), 'result'] / 10000\n",
    "\n",
    "#Oxiano Prom Hill Regional Drilling\n",
    "# Moly >= 15000 is actually ppm\n",
    "# All occur in 2002 drill holes?\n",
    "tmp.loc[(tmp['reference'] == 'ENV 06960') &\n",
    "        (tmp['ele'] == 'Mo') & \n",
    "        (tmp['result'] >= 15000) , 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 06960') &\n",
    "                                                (tmp['ele'] == 'Mo') & \n",
    "                                                (tmp['result'] >= 15000) , 'result'] / 10000\n",
    "#Set everything below 15000 and above 100 ppm to be correct\n",
    "#This is because we haven't checked for  the low ppm values that are actually %\n",
    "tmp.loc[(tmp['reference'] == 'ENV 06960') &\n",
    "        (tmp['ele'] == 'Mo') & \n",
    "        (tmp['result'] < 15000) &\n",
    "        (tmp['result'] > 100), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 06960') &\n",
    "                                                (tmp['ele'] == 'Mo') & \n",
    "                                                (tmp['result'] < 15000) &\n",
    "                                                (tmp['result'] > 100), 'result'] \n",
    "\n",
    "#ENV11449\n",
    "#KI001,KI002,KI003,KI004\n",
    "#Al and As are good\n",
    "#Everything else is shifted by 2-3 columns \n",
    "# The big values for Chromium are actually Fe\n",
    "# The big values for Lead are actually Si\n",
    "tmp.loc[(tmp['collar_name'].isin(['KI001', 'KI002', 'KI003', 'KI004'])) & \n",
    "    ~(tmp['ele'].isin(['Al', 'As'])), 'fixed_result'] = -777\n",
    "tmp.loc[(tmp['collar_name'].isin(['KI001', 'KI002', 'KI003', 'KI004'])) & \n",
    "    (tmp['ele'].isin(['Al', 'As'])), 'fixed_result'] = tmp.loc[(tmp['collar_name'].isin(['KI001', 'KI002', 'KI003', 'KI004'])) & \n",
    "                                                               (tmp['ele'].isin(['Al', 'As'])), 'result']\n",
    "\n",
    "\n",
    "# ENV 11450\n",
    "# Nickel values of 20-50%\n",
    "# Aircore data suffering from same mistakes\n",
    "# As other Centrex data\n",
    "# Starts with MHAC\n",
    "# XRF but not As/Al are the errors \n",
    "tmp.loc[(tmp['collar_name'].str.startswith('MHAC')) &\n",
    "        (tmp['reference'] == 'ENV 11450') & \n",
    "        (tmp['method'].str.contains('XRF')) & \n",
    "    ~(tmp['ele'].isin(['Al', 'As'])), 'fixed_result'] =  -777\n",
    "\n",
    "\n",
    "tmp.loc[(tmp['collar_name'].str.startswith('MHAC')) &\n",
    "        (tmp['reference'] == 'ENV 11450') & \n",
    "        (tmp['fixed_result'].isnull()), 'fixed_result'] =  tmp.loc[(tmp['collar_name'].str.startswith('MHAC')) &\n",
    "                                                    (tmp['reference'] == 'ENV 11450') & \n",
    "                                                    (tmp['fixed_result'].isnull()), 'result'] \n",
    "\n",
    "#ENV 11936\n",
    "# No actual assay files to verify with for Hole number one\n",
    "#But Cd, Sb and Te look to be clear outliers\n",
    "tmp.loc[(tmp['reference'] == 'ENV 11936') & \n",
    "    (tmp['ele'].isin(['Cd', 'Sb', 'Te'])) &\n",
    "    (tmp['collar_name'] == 'EWHDDH01'), 'fixed_result'] = -777\n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 11936') & \n",
    "    (tmp['ele'].isin(['Cd', 'Sb', 'Te'])) &\n",
    "    (tmp['collar_name'] == 'EWHDDH02'), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 11936') & \n",
    "                                            (tmp['ele'].isin(['Cd', 'Sb', 'Te'])) &\n",
    "                                            (tmp['collar_name'] == 'EWHDDH02'), 'result']\n",
    "\n",
    "\n",
    "\n",
    "# ENV 06960 Mount Woods\n",
    "# Two Sets of Reports have Pd as ppm when it should be ppb\n",
    "# Everything >= 1 should be divided by 1000\n",
    "tmp.loc[(tmp['ele'] == 'Pd') & \n",
    "        (tmp['reference'] == 'ENV 06960') &\n",
    "        (tmp['result'] >= 1), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Pd') & \n",
    "                                                        (tmp['reference'] == 'ENV 06960') &\n",
    "                                                        (tmp['result'] >= 1), 'result'] / 1000\n",
    "\n",
    "\n",
    "tmp.loc[(tmp['ele'] == 'Pd') & \n",
    "        (tmp['reference'] == 'ENV 06960') &\n",
    "        (tmp['result'] < 1), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Pd') & \n",
    "                                                        (tmp['reference'] == 'ENV 06960') &\n",
    "                                                        (tmp['result'] < 1), 'result'] \n",
    "\n",
    "\n",
    "# Exact same for Platinum\n",
    "tmp.loc[(tmp['ele'] == 'Pt') & \n",
    "        (tmp['reference'] == 'ENV 06960') &\n",
    "        (tmp['result'] >= 1), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Pt') & \n",
    "                                                        (tmp['reference'] == 'ENV 06960') &\n",
    "                                                        (tmp['result'] >= 1), 'result'] / 1000\n",
    "\n",
    "\n",
    "tmp.loc[(tmp['ele'] == 'Pt') & \n",
    "        (tmp['reference'] == 'ENV 06960') &\n",
    "        (tmp['result'] < 1), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Pt') & \n",
    "                                                        (tmp['reference'] == 'ENV 06960') &\n",
    "                                                        (tmp['result'] < 1), 'result'] \n",
    "\n",
    "#Tantalum\n",
    "#High Spike in Joe Dams East\n",
    "#No reference to anom value in report, but lithology supports the correct value\n",
    "tmp.loc[(tmp['ele'] == 'Ta') & \n",
    "        (tmp['reference'] == 'ENV 06960'), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Ta') & \n",
    "                                                    (tmp['reference'] == 'ENV 06960'), 'result']\n",
    "\n",
    "# Ultima Dam Drilling\n",
    "# Date listed as 2002 but actually 1982\n",
    "# Two sets of W values, 1.43 and 4.5% for the same assay\n",
    "# Both are in the source documents, so maybe its badly calibrated XRF results\n",
    "# But still correct\n",
    "tmp.loc[(tmp['ele'] == 'W') & \n",
    "    (tmp['collar_id'].isin(['218198_SA', \n",
    "                            '218197_SA', \n",
    "                            '218196_SA'])), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'W') & \n",
    "                                                                        (tmp['collar_id'].isin(['218198_SA',\n",
    "                                                                                                '218197_SA',\n",
    "                                                                                                '218196_SA'])), 'result']\n",
    "\n",
    "\n",
    "# 90 Neptune RC1\n",
    "# Ag has ppm values of 3400\n",
    "# 2 samples have AG and P incorrect\n",
    "# Rest are all good\n",
    "tmp.loc[(tmp['ele'].isin(['P', 'Ag'])) &\n",
    "    (tmp['reference'] == 'ENV 06960') &\n",
    "    (tmp['collar_id'] == '189400_SA') &\n",
    "    (tmp['sample_id'].isin(['2061727_SA', '2061728_SA'])), 'fixed_result'] = -777\n",
    "\n",
    "tmp.loc[ (tmp['reference'] == 'ENV 06960') &\n",
    "        (tmp['collar_id'] == '189400_SA') &\n",
    "        (tmp['fixed_result'].isnull()), 'fixed_result'] = tmp.loc[ (tmp['reference'] == 'ENV 06960') &\n",
    "                                                    (tmp['collar_id'] == '189400_SA') &\n",
    "                                                    (tmp['fixed_result'].isnull()), 'result']\n",
    "\n",
    "#D01PKN Hole\n",
    "# Column misalignment for the below elements\n",
    "D01PKN_fix = {\n",
    "    'Ag' : 'Na',\n",
    "    'As' : 'Ag',\n",
    "    'Ba' : 'U',\n",
    "    'Ce' : 'Co',\n",
    "    'Co' : 'As',\n",
    "    'Cr' : 'Ba',\n",
    "    'Fe' : 'K',\n",
    "    'K' : 'La',\n",
    "    'La' : 'Mg',\n",
    "    'Mg' : 'Ni',\n",
    "    'Na' : 'Co',\n",
    "    'Ni' : 'Fe',\n",
    "    'Pb' : 'As',\n",
    "    'U' : 'Ce'\n",
    "}\n",
    "\n",
    "for sarig_ele, actual_ele in D01PKN_fix.items():\n",
    "    print('Changing: ', actual_ele, ' to ', sarig_ele)\n",
    "    tmp.loc[(tmp['ele'] == actual_ele) &\n",
    "        (tmp['reference'] == 'ENV 06960') &\n",
    "        (tmp['collar_id'].isin(['269229_SA', '269228_SA'])), 'fixed_result'] =  tmp.loc[(tmp['ele'] == sarig_ele) &\n",
    "                                                                                (tmp['reference'] == 'ENV 06960') &\n",
    "                                                                                (tmp['collar_id'].isin(['269229_SA', '269228_SA'])), 'result'].values \n",
    "\n",
    "# Rest of elements are correct\n",
    "tmp.loc[(tmp['ele'].isin(['Au', 'Cu', 'P', 'Ti', 'Zn'])) &\n",
    "        (tmp['reference'] == 'ENV 06960') &\n",
    "        (tmp['collar_id'].isin(['269229_SA', '269228_SA'])), 'fixed_result'] = tmp.loc[(tmp['ele'].isin(['Au', 'Cu', 'P', 'Ti', 'Zn'])) &\n",
    "                                            (tmp['reference'] == 'ENV 06960') &\n",
    "                                            (tmp['collar_id'].isin(['269229_SA', '269228_SA'])), 'result']\n",
    "\n",
    "\n",
    "# ENV9008\n",
    "# What appears to be wrong? \n",
    "# Within same sample id, Au values range from 1ppm to 130ppm\n",
    "# What went wrong?\n",
    "# The high Au ppm values are actually Pb values?\n",
    "# Why is this bad?\n",
    "# Well its actually a gold project with legit values 60ppm +\n",
    "# The Pb values labelled as Au are intermixed within the good Au values, \n",
    "# making it hard to tell whats legit and whats false\n",
    "# How?\n",
    "# Verified by look at previous years format\n",
    "# How to fix?\n",
    "# Any Au assays with method 'B/AAS' are the lead assays\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['reference'] == 'ENV 09008') &\n",
    "    (tmp['method'].isin(['B/AAS'])), 'fixed_result'] = -777\n",
    "\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['reference'] == 'ENV 09008') &\n",
    "     (tmp['fixed_result'].isnull()), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                                    (tmp['reference'] == 'ENV 09008') &\n",
    "                                                    (tmp['fixed_result'].isnull()), 'result']\n",
    "\n",
    "\n",
    "# Drill holes HDD-001 and HDD002\n",
    "# Whats the problem?\n",
    "# P assays with high ppm values and really low ppm values?\n",
    "# What caused it?\n",
    "# The low P assays are actually Pt assays\n",
    "# Verified?\n",
    "# From source documents and no Pt assays for these holes\n",
    "tmp.loc[(tmp['ele'] == 'P') & \n",
    "    (tmp['collar_id'].isin(['292938_SA', '292939_SA'])) &\n",
    "    (tmp['result'] < 1), 'fixed_result'] = -777\n",
    "        \n",
    "tmp.loc[(tmp['ele'] == 'P') & \n",
    "    (tmp['collar_id'].isin(['292938_SA', '292939_SA'])) &\n",
    "    (tmp['result'] > 1), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'P') & \n",
    "                                        (tmp['collar_id'].isin(['292938_SA', '292939_SA'])) &\n",
    "                                        (tmp['result'] > 1), 'result']\n",
    "\n",
    "\n",
    "# Drillhole BTC183\n",
    "# Zinc Deposit\n",
    "# High Variance between Zinc assays for a single assay\n",
    "# Look to be just a follow up repeat assay, cant find much difference\n",
    "# All Good\n",
    "tmp.loc[(tmp['ele'] == 'Zn') & \n",
    "    (tmp['collar_id'].isin(['295718_SA'])), 'fixed_result'] =  tmp.loc[(tmp['ele'] == 'Zn') & \n",
    "                            (tmp['collar_id'].isin(['295718_SA'])), 'result']\n",
    "\n",
    "\n",
    "\n",
    "# ENV02073\n",
    "# Whats the problem?\n",
    "# Really low Cu values mixed in with high\n",
    "# What happened?\n",
    "# Assay logs went to percentage when ppm values started to go higher\n",
    "# Also occurs for Zn and Pb\n",
    "# How to fix?\n",
    "# Anything less than 3 ppm and greater than 0ppm is actually % - times by 10000\n",
    "\n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 02703') &\n",
    "    (tmp['result'] > 0) &\n",
    "    (tmp['result'] < 3), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 02703') &\n",
    "                                            (tmp['result'] > 0) &\n",
    "                                            (tmp['result'] < 3), 'result'] * 10000\n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 02703') &\n",
    "    (tmp['result'] >= 3), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 02703') &\n",
    "                                            (tmp['result'] >= 3), 'result'] \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Mount Gunson PY 9\n",
    "# Whats wrong?\n",
    "# Really low values for lots of elements\n",
    "# What went wrong?\n",
    "# These Assays are actually in percentage not ppm, so should be times by 10000\n",
    "# How to fix?\n",
    "# Any element with their old_unit as ppm is x10000\n",
    "tmp.loc[(tmp['collar_id'] == '193679_SA') & \n",
    "        (tmp['old_unit'] == 'ppm'), 'fixed_result'] = tmp.loc[(tmp['collar_id'] == '193679_SA') & \n",
    "                                                (tmp['old_unit'] == 'ppm'), 'result'] * 10000\n",
    "\n",
    "tmp.loc[(tmp['collar_id'] == '193679_SA') & \n",
    "        (tmp['old_unit'] != 'ppm'), 'fixed_result'] = tmp.loc[(tmp['collar_id'] == '193679_SA') & \n",
    "                                                (tmp['old_unit'] != 'ppm'), 'result'] \n",
    "\n",
    "#GUNSON PY10 probably looks good?\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "#ENV 11574\n",
    "# What is wrong? Lots of really low Copper Values\n",
    "#Fe is ppb when it should be % ( * 10000 * 1000)\n",
    "# Si is ppm wihen it should be %\n",
    "# Al is ppm when it should be %\n",
    "# Ca is ppm when it should be %\n",
    "# Mg is ppm when it should be %\n",
    "# Mn is ppm when it should be %\n",
    "# K is correct\n",
    "# Na is ppm when it should be %\n",
    "# Same for P S Ti \n",
    "# Te U and Z just repeating????\n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 11574') &\n",
    "        ~(tmp['collar_name']).str.startswith('ACEDD')&\n",
    "         (tmp['ele'].isin(['Te', 'U', 'Zn'])), 'fixed_result'] = -777\n",
    "\n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 11574') &\n",
    "        ~(tmp['collar_name']).str.startswith('ACEDD')&\n",
    "         (tmp['ele'].isin(['Fe'])), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 11574') &\n",
    "                                        ~(tmp['collar_name']).str.startswith('ACEDD')&\n",
    "                                         (tmp['ele'].isin(['Fe'])), 'result'] * 1000 * 10000 \n",
    "\n",
    "ppm_to_pct = ['Si', 'Al', 'Ca', 'Mg', 'Mn', 'Na', 'P', 'S', 'Ti', 'Cu']\n",
    "\n",
    "for ele in ppm_to_pct:\n",
    "    tmp.loc[(tmp['reference'] == 'ENV 11574') &\n",
    "            ~(tmp['collar_name']).str.startswith('ACEDD')&\n",
    "             (tmp['ele'].isin([ele])), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 11574') &\n",
    "                                            ~(tmp['collar_name']).str.startswith('ACEDD') &\n",
    "                                             (tmp['ele'].isin([ele])), 'result'] * 10000 \n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 11574') &\n",
    "        ~(tmp['collar_name']).str.startswith('ACEDD')&\n",
    "         (tmp['ele'].isin(['K'])), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 11574') &\n",
    "                                        ~(tmp['collar_name']).str.startswith('ACEDD')&\n",
    "                                         (tmp['ele'].isin(['K'])), 'result']\n",
    "\n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 11574') &\n",
    "        (tmp['collar_name']).str.startswith('ACEDD'), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 11574') &\n",
    "                                        (tmp['collar_name']).str.startswith('ACEDD'), 'result']\n",
    "\n",
    "\n",
    "\n",
    "# Burra Copper Mine Aircore Drilling\n",
    "# High copper values but legit\n",
    "tmp.loc[(tmp['reference'] == 'ENV 12045') & \n",
    "    (tmp['collar_name'].str.startswith('AT')), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 12045') & \n",
    "    (tmp['collar_name'].str.startswith('AT')), 'result']\n",
    "\n",
    "# Moonta mining\n",
    "# High Copper values but all good\n",
    "tmp.loc[(tmp['reference'] == 'ENV 08203'), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 08203'), 'result']\n",
    "\n",
    "\n",
    "# BHP Eagle/Copper Pedy Drilling\n",
    "# All Good\n",
    "tmp.loc[(tmp['reference'] == 'ENV 08434') & \n",
    "    (tmp['collar_name'].str.startswith('CD93')), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 08434') & \n",
    "                                                                    (tmp['collar_name'].str.startswith('CD93')), 'result']\n",
    "\n",
    "tmp.loc[(tmp['reference'] == 'ENV 08434') & \n",
    "    (tmp['collar_name'].str.startswith('CR93')), 'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 08434') & \n",
    "                                                                    (tmp['collar_name'].str.startswith('CR93')), 'result']\n",
    "\n",
    "\n",
    "# One sample from these holes is wrong\n",
    "# original unit is actually ppb so needs to be divided by 1000\n",
    "tmp.loc[(tmp['fixed_result'].isnull()) & \n",
    "    (tmp['sample_id'] == '928214_SA') & \n",
    "    (tmp['old_unit'] == 'ppm'), 'fixed_result'] = tmp.loc[(tmp['fixed_result'].isnull()) & \n",
    "                                        (tmp['sample_id'] == '928214_SA') & \n",
    "                                        (tmp['old_unit'] == 'ppm'), 'result'] / 1000\n",
    "\n",
    "\n",
    "\n",
    "# Gunsight Drilling, North Flinders, BHP - Checks out despite very high LREE values\n",
    "tmp.loc[(tmp['collar_name'].str.startswith('GUNSIGHT')), 'fixed_result'] = tmp.loc[ (tmp['collar_name'].str.startswith('GUNSIGHT')), 'result']\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# More Moonta mining\n",
    "# High Copper values but all good\n",
    "tmp.loc[(tmp['reference'] == 'ENV 07001'),\n",
    "        'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 07001'), 'result']\n",
    "\n",
    "#Alpha Dale Drilling\n",
    "tmp.loc[(tmp['reference'] == 'ENV 11215'),\n",
    "        'fixed_result'] = tmp.loc[(tmp['reference'] == 'ENV 07001'), 'result']\n",
    "\n",
    "\n",
    "# Challenger Gold Mine - High Gold Assays\n",
    "# All Good\n",
    "tmp.loc[(tmp['ele'] == 'Au') &\n",
    "        (tmp['reference'] == 'ENV 11460'), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') &\n",
    "                                                    (tmp['reference'] == 'ENV 11460'), 'result']\n",
    "\n",
    "tmp.loc[(tmp['ele'] == 'Au') &\n",
    "        (tmp['reference'] == 'ENV 09006'), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') &\n",
    "                                                    (tmp['reference'] == 'ENV 09006'), 'result']\n",
    "tmp.loc[(tmp['ele'] == 'Au') &\n",
    "        (tmp['reference'] == 'ENV 09774'), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') &\n",
    "                                                    (tmp['reference'] == 'ENV 09774'), 'result']\n",
    "\n",
    "\n",
    "\n",
    "# Tarcoola Gold Project - All Good\n",
    "tmp.loc[(tmp['ele'] == 'Au') &\n",
    "        (tmp['collar_name'].str.startswith('TAD00')), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') &\n",
    "                                                    (tmp['collar_name'].str.startswith('TAD00')), 'result']\n",
    "\n",
    "\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['collar_name'].str.startswith('QR')) &\n",
    "    (tmp['method'] == 'FA'), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                        (tmp['collar_name'].str.startswith('QR')) &\n",
    "                                        (tmp['method'] == 'FA'), 'result']\n",
    "\n",
    "tmp.loc[(tmp['ele'] == 'Au') &\n",
    "        (tmp['collar_name'].str.startswith('TCD0')), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') &\n",
    "                                                    (tmp['collar_name'].str.startswith('TCD0')), 'result']\n",
    "\n",
    "\n",
    "\n",
    "tmp.loc[(tmp['ele'] == 'Au') & \n",
    "    (tmp['collar_name'].str.startswith('GPMET')) &\n",
    "    (tmp['method'] == 'FA'), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') & \n",
    "                                        (tmp['collar_name'].str.startswith('GPMET')) &\n",
    "                                        (tmp['method'] == 'FA'), 'result']\n",
    "\n",
    "\n",
    "# Portia Gold Project\n",
    "tmp.loc[(tmp['ele'] == 'Au') &\n",
    "        (tmp['reference'] == 'ENV 11046'), 'fixed_result'] = tmp.loc[(tmp['ele'] == 'Au') &\n",
    "                                                    (tmp['reference'] == 'ENV 11046'), 'result']\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section Three - Better Ways to Detect Outliers\n",
    "\n",
    "So far the errors have been checked by simply filtering to a specific element, sorting from high to low and going from there. However eventually this method will hit a zone where you start running into assays from deposits. For example with Gold, most of the high legitimate results are from the Challenger gold mine.\n",
    "\n",
    "How can we do this in a smarter way?\n",
    "\n",
    "\n",
    "## 3.1 Distance To Deposit/Occurrence\n",
    "\n",
    "For each element, we can take the distance to the nearest deposit/occurrence associated with that element. To demonstrate this, for all the Gold asssays checked so far, we plot them against the distance to a gold deposit/occurrence\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "au_df = tmp[(tmp['ele'] == 'Au') & ~(tmp['fixed_result'].isnull())]\n",
    "au_df.loc[:, 'is_error'] = np.where(au_df['result'] != au_df['fixed_result'], 1, 0).copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the Occurrence and Gold Resource database from SARIG. Join them together and only keep location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import geopandas as gpd\n",
    "occ = gpd.read_file('occurence/mindep_activity_all_occur.shp')\n",
    "#Filter to only gold occurrences\n",
    "occ = occ[occ['COMM_CODE'].str.contains('Au')]\n",
    "dep = gpd.read_file('occurence/Gold.shp')['geometry']\n",
    "\n",
    "\n",
    "gold = occ['geometry'].append(dep).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the au_df, convert into geopandas format, downsample the dataframe and get the distance to nearest gold occurrence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from shapely.geometry import Point\n",
    "local_crs = {'init' : 'epsg:28353'}\n",
    "latlon_crs = {'init': 'epsg:4326'}\n",
    "\n",
    "\n",
    "# Get back the lat/lons\n",
    "au_df = au_df[['collar_id', 'collar_name',\n",
    "               'result', 'fixed_result',\n",
    "               'is_error']].merge(locs[['collar_id', 'lat', 'lon']],\n",
    "                         validate = 'm:1')\n",
    "\n",
    "# Convert into geopandas format\n",
    "au_df = gpd.GeoDataFrame(au_df,\n",
    "                      crs=latlon_crs,\n",
    "                      geometry=[Point(xy) for xy in zip(au_df.lon, au_df.lat)])\n",
    "\n",
    "# Downsample the dataframe otherwise it will take forever without multiprocessing\n",
    "au_df = au_df.sample(frac = 0.05)\n",
    "\n",
    "# Get the minimum distance for each sample to each gold location\n",
    "# Taking the min over axis 0 reduces it down to min dist for each sample\n",
    "au_df['dist_to_occurrence'] = gold.apply(lambda x: au_df.distance(x)).min(axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmwAAAFeCAYAAAA4z0MeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzde5yUdd3/8deH5QwqBxENRDzgERWVWylREUrN7tAo19N66kCSZSb3L+vuZHV3Zxma1p1m5alVdC0yKjuYhofSChVRtEQBBeQkCAgsx/38/vhew1w7zM7O7F6zszPzfj4e85id6zSfufaamc98j+buiIiIiEjn1aXUAYiIiIhIbkrYRERERDo5JWwiIiIinZwSNhEREZFOTgmbiIiISCenhE1ERESkk1PCJiJSYmZ2mpk9YmZrzazJzNzMLi11XJ2FmY2LzsmiNu6/KNp/XLKRiXQcJWySlZndGX3A5XO7qtTxVgMzG55x3ifm2Pag2HbDE4xhnJlda2Znt/MY+V5bc5KKvbMys5OA3wPjgb7AKmAF0FjKuLIxsxozO9/M7jGzV81snZltNbOVZvZXM5tmZu8udZzFYmZnx67Nh0sdj1SXrqUOQDq9bcCaVrbZ2BGByC6+YWa/8Y4d/Xoc8DXgLuDBBI73FrCjlfWV7krCj+cG4FJ373SJGoCZjQF+DhwUW7wVeAcYALwnul1tZv8ELnD3Vzs80OK6JPb3eDMb4u5LSxaNVBUlbNKav7n7uFIHIVkdDdQC95c6kHb4D3dfVOogSuyI6P7nnThZO4OQoPcAlgLfAX6T+t+ZmQGHAmcAU4D/AEYCFZOwmdmewAcIP1B/DVwAXARcV8q4pHqoSlSkPP0+uv+6mdWUNBJpr17R/YaSRtECM9sXuJeQrD0NHOXuP4gn2h687O43EhK3z1J5Je/nA92AmcCPo2WXtLy5SLKUsEnizGxWqtG0mfUzs++Y2b/MbJOZrY1tt7MhsJkNMbMfmdkCM9uS2XbJzLqY2cfM7DEzW2Nmm81soZndZmYH7RrFrg2Vzez9Zvb7qL1NU2tt78ysp5mtj47xn61s+69ou89mLD/FzH5hZkuitj7rzGy+mT1oZp80s7a+B79N+II/BLi4LQcws8FRm6PU/2admf3DzKaaWY+MbYebmROqQwEuydLebHgbX0shMV8aPdes6PGF0TWxOlp+drT82ujxndG18+nota2Nlo/KOO4xZlZvZouj6+8tM/ujmX04Ryx5X78t7O/ROR0eLfpL7FzOyti2w6//mC8C/YH1wEfcPWcTCXdvcveb3T1rGy8z62FmV5vZ36NrrtHM/m1mN5jZ3nnGlO24F5rZ02a2ITpHj5rZB9p6vCxSydk9wBPAG8ChZnZ8K3EV9BlgZruZ2VfM7Bkzeyfa500zm21m15vZyIzta6L/7Y+jfVbE9vmVmY3PElOffD7bLFgYbTe5Pa9LEuDuuum2yw24E3BgVhv2nRXt+/+A16K/NxM+8NfGtlsUrZtMaGjthF/lG4A5se16A3+M1juh3cza2ONG4KwscYyL1i8CpkZ/NwFvA9uBq/J4LXdF+92bY5tjo222A3vHlk+OxRh/bfFlPQs4r8Nj+x0KfCv6eyHQLWPbg2LbDs9yrOOB1bFt1kfnMfV4DrBXbPt9geWx+Bujx/Hbvnm+jnG5Ymtl30tT1yVwc/T3DkI7yx3A2dF210brUm3tUv+ft6O/R2X8n3bEYkpdH6nHPwdqssSyiDyu3xyvJXXeUs+9JrZsRie5/rtHr8mBmxL4XBkEPBuLPfW5kHq8BhiT67W0cNwfxo6xI3qNTdHjK2P/q3FtjPuIaP+3iN5rhKpQB/4vx34FfQYAewDzMl5L6tpOLbsu4zlGZhxvXZbn+GKW2G6L1v0yR/wTYnHv3tbXpVsyt5IHoFvnvJFMwvYO4VfoGUCXaN1Bse0WxbabC7wnti6+3a2xD/dPAj2i5QcDf4l9YBycEUfqQ76R8AX1f8DgaF1PYGger+X06BgbgN4tbHN9tM3DsWW9o9flwM+IJTOEBtpnEKqZuhdwXofHPgwPBfqRTkA+lbFtiwkbobTkzWjdXEI7MoAa4CPRF0Sz1xPb99po3Z3tuLbGtRRbHvteGrtmmoCvAv2idbsTJZmxON+Jrpspqf8fsBfRlw+hkXzqy/CB1DVB6K35JdJf+l/OEkte128eryl1nHEtrC/l9T829r86ra3/89jxfk86MTuHKBEGRkfn0AkJ654tvJZFWY55YSzG62PXw2BCwr6VdNKZ9RznEfd3o/1/FFt2ZLRsNVnex7ThMyC6nh1YSWgv1zVa3g0YAVwDfCLjeQ6Ojn8azZOqvYAvR//7JuCEjP2OJ/0DYM8WXnd9tM3d7XlduiVzK3kAunXOG+mEbSu7lqRk3nbP2HdWbN+ROZ5jEekSjcEtbDOc9BfqJ7Os701o2NzsQyVal/qQd3KUkLVyHmoIQyw4cH6W9UZISh24LLY89WG4gSylM22MZXjs9RwaLfty9PhNoFds21wJ21di533vLM9zWmzf8RnrriXZhG1VK9dW5pfTpbF9/zfHc1wb225yju0eibZ5Mtv/Cfhf0klZ5nXe6vWb5/lIHWdcJ7z+PxE7xi7XSoHHOil2rNOzrB9M+sfCN1p4LYsyllvs9e9yTUbrH4497y7nOI+4a0j/wBmbsS6VZH44y34FfwYAD0X7XNOec51xzNT7/Y4s656P1n02y7o9gE2Z560tr0u3ZG6qY5bWdCN8kOa6tXQd/d7dX8zjOe529xUtrPtQdPzlwE8zV7r7JsKvX4BJ1nID/OvziGMX7r6DMNwChEbHmcYSqgs3AzNiy9dH992AgW157jx9n5D07ANckec+H4nuf+ruyzNXuvufgKeih7XtjjC3Pcl9bfVpYb8dwA15HH81cHu2FWY2ADg1evjt6H+d6TuE/21f4MwWniPX9dteJb3+CSUmKW9n28DMLjaz5VluMzI2TV13s939j5nHic7hrdHDfK+7UcCB0d/fznJMJyTd7fE+wvvrdeCvGevuie4vybJfWz4DUvvsU0iArfhNdH9ilnWpa+qyLOvOI3SIeQ14LLa8oz7bJIMSNmnNY+5urdzWtrDvUy0sL2S7Y6P7J1r4QgV4NLrvQ2iEn6mR8Euyre6N7k+PvuTjLojuH3L3dbHl86Nbd+ApM/ucmR1qZtaOOHbh7hsISQXANWa2W67tzaw7oc0LhOq0lqTO6bE5tknC/q1cW99vYb9X3T2fMdpmu/v2FtYdQyiBcZp/Ie0U/U+fiR62dC7yvc7bojNc/63pTfZkO/O9knot+Vx3B5tZS8l6tmOucPd/t7DN3wjVgm11aXQ/PUoA46YTrp/3m9mgjHVt+Qx4KLq/0sx+HnUmyPmeBjCzXtHxZ0WdSrbFOrU8F232riy71hN+kBxtZpnX90ej+zsyXneHfLbJrpSwSTGtSmC71IdgrsEpl2TZPm61uzflGcsu3P0pQsP+7sDOXoNm1pV0qcG9GfvsICRzS4EDCKVBLwNvmdkDZjYxwQ+4HxGqbPYEWuv5N4D0+z6fc5rtfHYGSV5b66LEtyWtnYt8Y2mLUl//8R6h/bNt4O63xpNsQmeYbAp5LUa4nluTOuabLW3g7lto4wDMZrYHcFb08N7M9e7+BqHHaFfSP95S6wr+DHD3uwmdAQyoIyRwa83sOTP7hpntUvIWLZsTHf8UwjnZQnrGjNRr3yUBdve3SdcM7CxlM7MjCFWfTYR2gO16XZIMJWxSTLlGsC90u54dEEcu06P7+Ify+whfKuuB32Xu4O6zCQ2F64C7gQWEhOkjhIE3f5ejCitvHgZbTX1JTjWzrF+sWbTnnJZaktdWj9Y3afdztFeprv+XY38f1Y7jxJXTdXcu6XjnWpbp04CTo/W7VIu25TPA3T9JKAX/BqE98BZC1e9XgPlm9r6Mp/k+oePBAsIPygHu3tfd93L3vYExrbzGn0T3F1h6OJ9U8vYnd1+SuUNHfbZJc0rYpLNLlV4My7HN0CzbJy316/pkM0tVLaTatM1w983ZdnL3Rne/x90vcfcDCb9Iv01UjQJcnlB8PyU0Xt8D+HyO7dYQfjVDfue0mKVHpZZ6bb2yVGfFlfJclPr6/weh4TmEXovtUchrcfIrFUsdM1t1H7CzGUA+pXXZZGub1pJjzOzIzIVt+Qxw93nu/jV3P5XQG/yDwAuEUrK7zKwb7HxtqRLAC919RlRqFjc4V9DuPovQcWMAMDGqOaiLVmdt/9nW1yXto4RNOrtno/sTzKx3C9ukBobcCLTUjqVd3H0eoUdYF+A8M+sJpCZA36WqJMdxFrr7f5OeTuqUhOLbSvhFDvAZWviQjrZLdQQ5Nds2kdQ5fTZjeSrZq4Qqj+cIXy7QwrmIqsSOix5mnouOUNLrP7peUlVil5rZkHYcLvVaTslRZZZ6La+4ez4zJaSOOdjMDm5hm/fQhmkYzWxEtC+EEq7+OW6phv2tJniFfga4+1Z3/y1hGBQIHRJGRH/vSbqE+LnMfSPvbS0mwvAcENqtfYDw+bGaUFqWl2J9tkmaEjbp7GYQkoSBhMEam4m+xP5fatscDbOTkErMzif84t2N0Hvv0cwNo1++uaTmjGxvdVzc3YQv7D6E0elb8ovo/tIW2sScBrw7etiQsTrVQ6xfO+LsFDyM2J9qAH9NCyOzX0OoEttAukF4R+oM1/+3CT1Edwd+kaXjTb5S190RpEuFdjKzwaRLZTKvu5bMIT1f6TVZjmnAFwoLc6fUDCLPu/vz7r62pRthDD+AC1NVgW35DGhln/g8s6l9UuOhQRgXrpno/f2ZVuKAMIzTdsKQPqnzVR8l7JnHLMVnm6CETTo5d3+d0AgX4Dozm5xqZxH9ov4dYcyxTcD/FDmcVI+w0aQTooYWviTPNLOnzOwTZrZfaqGZ9TazTxAG+4Qwgn0iojiujR7mqr76IbCM0GX/D2Y2OoqtxsJUTPdF2/3Z3TOT0XnR/dioBKLcfYWQEB0L3GdmQwHMrK+Z/TfpL6/r3H19C8coms5w/bv7YkLbzS2E9lBzzexKy5iKzMz2NbPLCROiZzvOE8Afooe3m9lHYsnNccCfCKVVK4Cb8ozNSV/zH7UwDV6/6JiDCVV640lX6+YlSvRSryNzeJJsfgNsA/YmDLYNbfsM+LOZ3WxmJ5tZr9g+RxCSKgjv3RcA3P0dwvyuEM7pqGj7LmY2gdD7udXS8Gh4n98ScoJUm7eWqkM7/LNNIt4JBoPTrfPdKGzg3Jsy9p0V7XtpK8+xiDwGsyQMG/An0oNfbiU9wr8TuqXnnJonwfPyROx5nYzRw2PbnZ2x3SbS7cdSy35HNJJ5ns89PLbvoS1sY6QHw8w6cG603fGkByl1dp2a6nliU1PF9utGeqDSJsKI7IuiW6sj52f8X/IZOHd5xr6XRvvNauU5riXPAX4JswekBqdtis5LfGqqenJPTZXz+s3j+XMep7Nc/4Qv8vkZ11aqN+KmjOVPAcdnOcYg0lXRHl1zmVNTvbvQ10Lzqam20/y9VvDUVIQq8tTxjshznz9E29/f1s8AQolhal1qWqr4+3IjMCHjeU/IOP8bYo9XE0oznSi/zRH/B2LHmJ1ju8Q/23TL76YSNmlNPgPn7lHMADwMDvp+4OOEhGkT4UvsdUJj+yPdPe+2Fu0Ub6/2mrv/vYXtHiX8Qr+L8Gt4E6EKdTVh5PWLgQ96y2OEtYmHT9Sv5LHdP4DDgRuBVwj/5+3AbEIV2wnuvjLLftsI8wv+nNCtvz+wX3QruJ0QrQ+cm7PBdBLc/cfAfxD+t8sIg+SuI/yfznH3Oi9uVXtr8XWK69/dnwYOI5S2TScMqLqF8P7fQCjpuZEwF+i7o2ss8xirCNXt/0W41rYRhsuZT+jteISHYXQKje3ThIbyf49iMkLp0n+6+82FHo90W7RXPLRfzccvo/uJUSlfWz4DPg58jVBV/wahFBzgX4SkdKS7PxJ/0ugz6N2EOXPfJryXVwI/JrS9y3cMvj+QLolssbNBG1+XJMCijFlERESqlJmdSJiibTOwj7c8ILqUiErYREREJNXh4wEla51TyRK2qIHqX8zsJTObZ2afjZYPMLOHzWx+dN8/Wm5RY8xXzWyu7TqNhoiIiBTIzE4nPa5kXh0+pOOVsoRtOzDV3Q8nNGa9wswOJ/TKesTdRwCPkO6l9X7C2DMjCN3bb+n4kEVERCqDmS0ysxWE9ms1wM/d/ZlWdpMSKVnC5u7L3P3Z6O93CFOgDCH0aEkN1HgX6cFJzwLu9uBpoF+2MaREREQkL/sReu4uAaYRekxLJ9WWXl2Ji8bzOYbQw2ewuy+LVi0n3UtsCLA4ttuSaNkyREREpCDuXgkzllSNkidsZtaX0B36KndfH5+xxN1Tk+sWcrzJRCOC9+nT57hDDz00yXBFREREiuKZZ555y92zzm1c0oQtmsD2l8A97p4aTXqFme3j7suiKs/UWFBLgX1juw+NljXj7rcRjQw+evRonz17dtHiFxEREUmKmb3e0rpS9hI1woSzL7v7DbFVM0kPWngJ6clnZwIXR71FxwDrYlWnIiIiIhWrlL1ETySMljzezOZEtzOB64D3mdl84L3RYwgTLy8gTIvzE+BTJYi5ferrYeLEcC8iIiKSp4qe6aDTVYlOnAi9ekFjI8ycWepoREREpBMxs2fcfXS2dSXvdFBVamuhoSHci4iIVIht27axZMkSNm/eXOpQykLPnj0ZOnQo3bp1y3sflbCJiIhIuyxcuJDddtuNgQMHEh/tQXbl7qxevZp33nmH/fffv9m6XCVsmktURERE2mXz5s1K1vJkZgwcOLDg0kglbCIiItJuStby15ZzpTZsIiIiUvZqamo48sgjdz4+77zz+MIXvpBjj/KihE1ERETKXq9evZgzZ07ObXbs2EFNTc3Ox9u3b6dr19ZToXy3KyZViYqIiEjFGj58ONdccw3HHnssDzzwAOPGjeOqq65i9OjR3HTTTSxatIjx48dz1FFHMWHCBN544w0ALr30Ui6//HJOOOEEPv/5z5f4VaiETURERCpAY2Mjo0aN2vn4i1/8Iueeey4AAwcO5NlnnwXg1ltvZevWraRGkfjgBz/IJZdcwiWXXMLtt9/OlVdeyYMPPgjAkiVL+Nvf/tasVK5UlLCJiIhIx6uvT49NWlfX7sPlqhJNJW7ZHj/11FPMmBGmM7/oooualaadc845nSJZA1WJioiISCk0NITZfxoaiv5Uffr0yfk43/1KSQmbiIiIdLza2jBVY4ln/3nPe97DfffdB8A999zDSSedVNJ4WqIqUREREel4dXWJVIWmZLZhO+OMM7juuuta3e8HP/gBl112Gddffz2DBg3ijjvuSCymJClhExERkbK3Y8eOrMsXLVrU7PGsWbOaPd5vv/149NFHd9nvzjvvTCiyZKhKVERERKSTU8KWoPp6mDgx3IuIiIgkRQlbgjqww4uIiIhUESVsCeokHV5ERESkwqjTQYIS7vAiIiIiAqiETURERKTTU8ImIiIiZc/MmDp16s7H3/ve97j22mvz3v/OO+9k0KBBjBo1auftpZdeKkKkbaOETURERMpejx49mDFjBm+99Vabj3HuuecyZ86cnbfDDz+82frt27c3e+zuNDU15XXslsaJy5cSNhERESl7Xbt2ZfLkydx44427rFu0aBHjx4/nqKOOYsKECbzxxht5H3fWrFmcdNJJTJw4kcMPP5xFixZxyCGHcPHFFzNy5EgWL17M9OnTOfLIIxk5ciTXXHPNzn379u3L1KlTOfroo3nqqafa9fqUsImIiEhFuOKKK7jnnntYt25ds+Wf+cxnuOSSS5g7dy4XXnghV155Zdb977///mZVoo2NjQA8++yz3HTTTbzyyisAzJ8/n0996lPMmzePbt26cc011/Doo48yZ84c/vnPf/Lggw8CsHHjRk444QSef/55xo4d267XpoRNREREOlwxBpvffffdufjii7n55pubLX/qqae44IILALjooot48skns+6fWSXaq1cvAI4//nj233//ndvtt99+jBkzBoB//vOfjBs3jkGDBtG1a1cuvPBCHn/8cQBqamr48Ic/nMhrU8ImIiIiHa5Yg81fddVV/OxnP2Pjxo2JHbNPnz45H7ekZ8+e1NTUJBJDyRI2M7vdzFaa2YuxZfeb2ZzotsjM5kTLh5tZY2zdraWKW0RERNqvWIPNDxgwgNraWn72s5/tXPae97yH++67D4B77rmHk046KbHnO/7443nsscd466232LFjB9OnT+eUU05J7PgppRw4907gh8DdqQXufm7qbzObBsQroV9z91EdFp2IiIgUTTEHm586dSo//OEPdz7+wQ9+wGWXXcb111/PoEGDuOOOO7Lud//99zerLv3Rj37U6nPts88+XHfddZx66qm4Ox/4wAc466yz2v8iMpi7J37QvJ/cbDjwW3cfmbHcgDeA8e4+v6XtWjN69GifPXt2MsHmo74+lO3W1mrKAxERqRovv/wyhx12WKnDKCvZzpmZPePuo7Nt31nbsJ0ErHD3+bFl+5vZc2b2mJklV5aZJM3+LiIiIkXQWRO284HpscfLgGHufgxwNXCvme2ebUczm2xms81s9qpVqzog1BjN/i4iIiJF0OkSNjPrCkwC7k8tc/ct7r46+vsZ4DXg4Gz7u/tt7j7a3UcPGjSoI0JOq6uDmTOppy7xrsoiIiJSvQpK2Myst5kdY2anm9lp0d+9E47pvcC/3H1J7HkHmVlN9PcBwAhgQcLPmxjVjIqISLUpZZv4ctOWc9VqwmZm/c3sc2b2N+BtYDbwEPD76O+3zexvZnaVmfXP94nNbDrwFHCImS0xs49Fq86jeXUowMnA3GiYj18Al7v7mnyfq6OpZlRERKpJz549Wb16tZK2PLg7q1evpmfPngXt12IvUTPbA/gK8CmgJ/BvQoL1GrAaMGAAcBAwBjgE2Az8H/A/7r4uy2E7VIf3EhUREalC27ZtY8mSJWzevLnUoZSFnj17MnToULp169Zsea5eornGYXsN2AJ8G6h394W5njyqqrwImAxcBuxZQOwiIiJSprp169Zs6iZJXq6E7RvAj919Sz4HcvcFwNfN7Drg8iSCExEREZEcCZu739zSulyiBO+mNkckIiIiIs10umE9RERERKS5gucSNbMRhGE1BhI6HjTj7nfvspOIiIiItFneCZuZ7QPcBUxILcqymRObzF1ERERE2q+QErbbgFOB7wNPEMZkExEREZEiKyRhGw/c5O7/VaxgRERERGRXhXQ62AC8WqxARERERCS7QhK23xLm+RQRERGRDlRIwjYV2N/MbjSzA8wsW6cDEREREUlY3gmbu68l9BK9EpgPbDezHRm37cUKVERERKRaFTKsx+cJ84quAP6BeomKiIiIdIhCeol+BpgFnOHu24oTjoiIiIhkKqQN2wCgQcmaiIiISMcqJGF7HhhWrEBEREREJLtCErYvAZPNbHSxghERERGRXRXShu0iYCnwtJk9BSwAdmRs4+7+saSCExEREZHCErZLY3+fGN0yOaCETURERCRBeSds7l5I9amIiIiIJERJmIiIiEgn16aEzcwOMbP3R7dDkg6qHNXXw8SJ4V5EREQkSQUlbGY23szmAS8RJoP/LfCSmc0zswnFCLBcNDRAr17hXkRERCRJeSdsZjYe+ANhLLafAJ+Lbj8F9gV+H21TlWprYf58WLw4VsqmYjcRERFJgLl7fhuaPQ0MAca4+9KMdUOBp4HF7v7uxKNso9GjR/vs2bM77PkmTgylbI2NMHNmtgUiIiIi2ZnZM+6edbzbQqpEjwJ+nJmsAbj7EuDHwNEFBHW7ma00sxdjy641s6VmNie6nRlb90Uze9XM/m1mpxcQd4eprQ25WW1tSwtEREREClfIOGzrgHdyrF8PrC3geHcCPwTuzlh+o7t/L77AzA4HzgOOAN4F/NnMDnb3zIF7S6quLtxaXiAiIiJSuEJK2B4AzjezXZI8M+sGnB9tkxd3fxxYk+fmZwH3ufsWd18IvAocn+9ziYiIiJSzQhK2Wwklco+b2TlmdmR0qwUeB2qAW81sWPzWhpg+bWZzoyrT/tGyIcDi2DZLomXlQx0QREREpI0KqRJ9kTD1lAH3Zayz2DaZagp4jluAb0bP801gGvDRAvbHzCYDkwGGDWtLvlgk8XE/VE0qIiIiBSgkYfsGIZEqGndfkfrbzH5CGOcNwqTz+8Y2HRoty3aM24DbIPQSLU6kbVBbG5I1dUAQERGRAhUyl+i1RYwDADPbx92XRQ8/RLrEbiZwr5ndQOh0MAL4R7HjSZQ6IIiIiEgbFVLCligzmw6MA/Y0syXA14BxZjaKUJK3CPgkgLvPM7MGwgwL24ErOlsP0Wbq65uXpqX+VsImIiIibZD3wLk7dwidDD4EHBAtWgD8yt073aRMHT1w7k7xAXNBg+eKiIhIqxIZONfM+pjZw8B04FxCteSI6O/pZvaImfVJIuCyFx8wV4PnioiISDsVMjXV94ErgZuB69x9ebR8b+ALqXXuflWRYi1YyUrYRERERAqU1NRU5wIPuPtVqWQNwN2XR0naL6NtRERERCRBhSRsuwN/ybH+0WgbEREREUlQIQnbXEKbtZaMAF5oXzgiIiIikqmQhO3LwCfM7IOZK8zsLODjwH8nFZiIiIiIBIWMw3YhsBB40Mz+DbwcLT8MOIRQulZnZvHBxtzdP5ZIpCIiIiJVqpCE7dLY34dGt7ijolucA0rYRERERNqhkKmpCqk+FREREZGEKAkTERER6eSUsImIiIh0ci0mbGb2czM7oKX1OfYbYWb17QtLRERERFJylbAdBLxsZr8ws4lm1qulDc2sr5l9xMx+DcwD9k86UBEREZFq1WKnA3d/t5ldAHwFeBDYbmbzgNeANYABAwgD5h4G1AAvARe5+/3FDrys1ddDQ0OYEL6urvXtRUREpKrlNfm7mU0AzgFOBg4mXTLXRBiPbRZhntHHixNm23Tayd8nToRevaCxEWbOLHU0IiIi0gnkmvw9r2E93P0R4JHoYF2AgYQx1lZ7PhmfNFdbmy5hExEREWlFIR/sis8AACAASURBVAPnAuDuTcCqIsRSPerqVBUqIiIiedOwHqVQXx+qRevVmbbddC5FRKQKKGErhYaG0IatoaHUkZQ/nUsREakCSthKobY2dDhQG7b207kUEZEqkFcv0XLVaXuJioiIiGTI1UtUJWwiIiIinVzeCZuZnWxmg3Ks39PMTk4mLBERERFJKaSE7S/A+3KsnxBtIyIiIiIJKiRhs1bW1xBmPhAR6Tw09IuIVIBC27Dl6qHwHuCtfA9kZreb2UozezG27Hoz+5eZzTWzX5lZv2j5cDNrNLM50e3WAuMuPn0piHROGvpFRCpAzoTNzD5rZgvMbEG06Pupxxm3t4EpwG8LeO47gTMylj0MjHT3o4BXgC/G1r3m7qOi2+UFPE/H0JeCSOekoV9EpAK0NjXVWuD16O/hwGpgRcY2DrwIPA3cmO8Tu/vjZjY8Y9mfYg+fBj6S7/FKTvODinROmgpORCpAzoTN3e8C7gIws4XAF9x9ZkcEBnwUuD/2eH8zew5YD3zZ3Z/ooDjyoy8FERERKZK8J3939/2LGUicmX0J2A7cEy1aBgxz99VmdhzwoJkd4e7rs+w7GZgMMGzYsI4KWURERKRoOt3AuWZ2KfCfwIUeTcPg7lvcfXX09zPAa8DB2fZ399vcfbS7jx40qMVh40RERETKRoslbLGOBoVwdz+wrcGY2RnA54FT3H1TbPkgYI277zCzA4ARQFviExERESk7uapE3yD3MB7tYmbTgXHAnma2BPgaoVdoD+BhMwN4OuoRejLwDTPbRhjr7XJ3X1Os2EREREQ6E03+LiIiItIJaPJ3ERERkTKmhK1YNPOBiIiIJCTvYT3y7ITQrk4HFSU+84HGZxMREZF2KKSE7Q3CrAfx21KgO2EWhO3RNgKaDkdEREQSk0inAzM7H5gGnOru/273AROiTgciIiJSLore6cDdpwMPEpI2EREREUlQkp0O5hDGS6teUUeD+ilPqr+BiIiIJCbJhG0UYVDb6hV1NGiY0W1nfwMRERGR9iqkl2hLpWcDgPcCnwBmJBFU2aqtpX7aCpb0Ogh/BaZOja2rrw8ZXG2teo2KiIhIQfLudGBmTWSfqsqi+z8Dde6+MqHY2q0UnQ4mTgyjeTQ2wsyZ+awQERERyd3pIO8SNuCyLMscWAO84u6vtCW4SlNbmy5Iy2+FiIiISG6aS7RIVAMqIiIihSjqsB5mtqeZjWjvccpd5kxU8YkORERERNoj74TNzC42s9syln0bWAH8y8z+ama7JR1guchM0DTRgYiIiCSlkBK2TxJr82Zmo4FrgCeAnwDHA1cnGl0ZyUzQ6upC34Kd1aGaDF5ERETaqJCE7SBgbuzxOYQOB6e5++XAT4GqLU/aJUHLpDpSERERaaNCErY9gHWxxxOAP7v71ujxbGBYUoFVHNWRioiISBsVMqzHcmAEgJkNIsxscEdsfV9gR3KhVZi6OnUXFRERkTYpJGF7FLjCzNYApxLGYPtdbP0hwNIEYxMRERERCkvYvgq8B/hu9Ph/3H0RgJl1BT4M/DLR6EREREQk/4TN3ZeY2RHA4cA6d38jtro3MBl4PuH4RERERKpeISVsuPsO4IUsy9cDv04qKBERERFJK2imAzOriQbQrTezh83smGh5/2j5kOKEWR401JqIiIgUQyEzHfQGHgPuBM4CxgP9o9XrgeuAKQnHV1Y01JqIiIgUQyElbNcCo4EPAQcAlloRVZXOAE5PMrhyo6HWREREpBgKSdjOAW5z918DTVnWvwoML+TJzex2M1tpZi/Glg2IqlvnR/f9o+VmZjeb2atmNtfMji3kuTpC1tkOVE8qIiIi7VRIwvYucvcC3QQUOvn7ncAZGcu+ADzi7iOAR6LHAO8nDNw7gtAj9ZYCn6s0VE8qIiIi7VRIwrYayNWp4AjgzUKe3N0fJ8xHGncWcFf0913A2bHld3vwNNDPzPYp5PlKopj1pCq9ExERqQqFJGyPAJdFnQ+aMbP9gY8Cf0ggpsHuviz6ezkwOPp7CLA4tt0ScieQnUOrs8K3g0rvpK2U7IuIlJVCEravE3qF/pPQG9SBM8zs28CzwBbg20kG5+4ePU/ezGyymc02s9mrVq1KMpz8dOQXoXo5SFsp2RcRKSt5J2zu/iowAdgOfIPQS/S/gGsIJV8T3H1xy0fI24pUVWd0vzJavhTYN7bdULLMXerut7n7aHcfPWjQoATCKVBHfhEWs/ROKpuSfRGRslLQwLnu/oy7Hw0cBZwLnAcc5+5HuXtS01LNBC6J/r6E9AwKM4GLo96iYwjTYy3LdoCS0hehlAMl+yIiZaWgqalS3P1F4MXM5WZmUTVmXsxsOjAO2NPMlgBfIwzA22BmHwNeB1KZz0PAmYThQzYBl7Ul9mKqr4eGhjpqa+v0PSgiIiKJsQLyq5YPYmbAhcBX3P2Qdh8wIaNHj/bZs2d32PNNnBhqQxsbQ+GFiIiISL7M7Bl3H51tXV5VomZ2jJnVmtl7zaxbxrrzgJeAu4G92x1tGVNtqIiIiBRDzoTNzHqZ2R+A2cB04I/AfDM7yMz2NbO/AvcA+wDfAvYvdsCdmWY6ECkxvd9EpEK1VsJ2DXAa8AwwjdABYBhwK/Bn4Gjgm8B+7v4Vd88cBLe6ZPuy0PAJIh1H7zcRqVCtJWwfAZ4Exrj75919EmES+PFAD+Aod7/W3dcVN8wyke3LQvWkIh1H7zcRqVA5Ox2Y2Sbgi+5+U2zZIcDLwKfd/UfFD7HtOrrTQdRNNHxZqJuoiIiIFCBXp4PWhvXoCWROF/BWdD+/vYFVnLo6JWoiIiKSuIIGzs2wI7EoKlF7Gz+r8bSIiIhE8hk49+JoZoGUnoT5PT9tZmdnbOvu/tnEoitn8fZsbSl1a+/+IiIiUjHySdhOi26ZMpM1CImcEjYI7dhS7dlKsb+IiIhUjNY6HexX6AHd/fV2RZSgDu90kC91ThAREZEMbe500JmSr4qi6k4REREpQHs6HUhbaawoERERKUA+bdgkaRr+Q0RERAqgEjYRERGRTk4Jm4hUB41tKCJlTAmbiFQHTQwvImVMCZuIVAd19hGRMpZ3wmZmPzWzE4oZTEUqpBpGVTYixVNXBzNnqsOPiJSlQkrYLgX+ZmYvmtlVZjawSDFVlkKqYVRlIyIiIlkUkrANBb4EdANuAJaY2X1m9r6iRFYpCqmGUZWNiIiIZJFzaqoWdzI7GfgY8GGgF/AGcDtwp7svTjTCdui0U1OJiIiIZMg1NVWbOh24++PufgmwDzAFWAlcCywws4fM7Gwzs7YGXDHUJk1EREQS0N5eoj2B3aObARuBE4BfAs+b2WHtPH7ZyJqbxdukKXkTERGRNio4YbPgA2Y2A1gCfBdYB3wceBeh1O0T0f1PEoy1U8vaXyDeJk0dCkRERKSN8p5L1MwOBD4KpKpC1wO3Abe5+wsZm99uZr2B7yUVaGeXysma9RfInDN0lw0SUl+fPraGLBAREak4eXc6MLOm6M+nCIna/e6+Ocf2/wn8wN33Lyggs0OA+2OLDgC+CvQjlNytipb/t7s/lOtYpeh0UJLcaeLEUHrX2BjGmRIREZGyk1Sng5uBI939RHe/K1eyBuDuvy00WYv2+7e7j3L3UcBxwCbgV9HqG1PrWkvWOlqqidoNN5Sg5lPDgYiIiFS0vBM2d7/K3ecVM5gsJgCvufvrHfy8BUs1UXNP504t9jNIugOCRnAXERGpaG3qJWpmfc1sqJkNy7wlHN95wPTY40+b2Vwzu93M+if8XO2SKuSaOjWdO7XYz0AdEERERKQABSVsZnaemb1I6BX6OrAwyy0RZtYdmAg8EC26BTgQGAUsA6a1sN9kM5ttZrNXrVqVbZOiyFbI1WJNpaowRUREpACFdDo4G5gBvAI8ClwO3EvoaXo2MBf4nbt/PZHAzM4CrnD307KsGw781t1H5jqGZjoQERGRcpFUp4P/Al4mlHB9NVp2u7ufB4wGDgHmtCfQDOcTqw41s31i6z4EvJjgc4mIiIh0WoUkbEcBqd6hqSE+agDc/UXCUB9fTCIoM+sDvI9QopfyXTN7wczmAqcCn0viuUREREQ6u0ISthpgdfR3Y3S/R2z9v4GcVZT5cveN7j7Q3dfFll3k7ke6+1HuPtHdlyXxXMWgWahEREQkSYUkbEuA/QDcvZEw4ftxsfWHEOYSrXrqBCoiIiJJyntqKuBvwHtJt1+bCVxlZo2ExO8K4DfJhldeUrMcDBkCS5eqE6iIiIgko5CE7UfAh8ysV1TC9iXgeODaaP08QseEqpUqWVu6VDNEiYiISHIKmengn+7+31GyhruviqaPGgUcCRzt7ouLFGenV18PS5bAK69klKypQZuIiIi0U5tmOohz97nuPs/dm1rfunI1NMCIEbDvvhkzRKlBm4iIiLRT3gmbmQ00s8Mylu1vZj8ws3vM7PTkwysftbWhdG3JklCYtrNgbcjnNauBiIiItEshMx3UAwe7+/HR477Av4B3RZs0AePd/fFiBNoWHT3TwcSJoTCtMRr0JPX3Lu3ZUr0Tams1YbuIiIgAyc108G7godjjcwnJ2pnR/cvA59saZLmrr4fFi2H+/JCH5ZwuVNWkyVNbQWmNrhERKWOFJGyDgXingvcDs939D+6+HLgTOCbB2MpKQwMcfDAMHRoKzbJNBr+TJn9PnpJgaY2uEREpY4UkbNuAXrHHpwCPxR6vBQYmEVQ5qq0NpWuLF0c/4HP9ms+ZzUmbKAmW1ugaEZEyVkgbtr8Dm4DxwAeBXwEfcPc/ROuvBSa7+7taPEgHK2UbtpmLj4EVK2DwYHjuuQ6LQURERMpTUm3Y/o9QqvY28AtgAfBIbP1JwAttDbISDBkCs2aFe8zCwtS9iIiISBvlPdOBu99tZg6cDawD/tfdt0EY8gPoR5gNoWotXQrjxsHSB/4Ga+dCv35w9dWlDktERETKXCFTU+HuPwd+nmX5appPBF91UjMduMPUtT+G7t3hnXdCO7XMSUbjk42qHZuIiIi0ot0zHZjZcWb2PjPrmURA5arZTAenLIEdO2Ds2PTKXr3g3nth9uxwr95qIiIikqdCZjr4LzP7Tcaye4F/AH8AXjCzwQnHVzaadUB75BHYsiXcx1f27x8e9++v3moiIiKSt0KqRM8D/p56YGbjo2XTCZ0NvkwYOHdqkgGWi1TNZqrQrI6M2QziVaOqChUREZECFFIlOpwwm0HK2cAyoM7drwNuJQz3UbWajcuZbZDO1sZf00jsIsWj95eIlLFCErY+QGPs8Xjgz54eyO0lYEhSgZWjZtWibRmkUyOxixSP3l8iUsYKqRJdChwJYGb7AYcDN8TW9we2JBda+UnVfEaPCq/2rK1NV5mKSLL0/hKRMlbITAc3Ap8CfgKcAIwEhrv7imj9HcDR7n5skWItWEfPdJC1iZrarYmIiEgekprp4BvAk4SkbSRwVSxZ6wV8CPhLO2Mta1lrXFQNU3xqmyQiIhUu74TN3d929wmEGQ12d/cfZ2xyCvA/SQZXbnY2WxvyZDqBqK2FV14Jo+oqodhVEsmWkmIREalwBQ+c6+7rU1NSxZY1AiuBTyYVWFl7+u/pBKKuLoymO2KEEopskki22tLBQ0REpIy0a6YDM6sxsw9FA+q+DnwrmbDK087cw85tnkAooWhZEuemteFSREREylybEjYzO8zMvkfoOfoL4FTg18CFSQVmZovM7AUzm2Nms6NlA8zsYTObH933T+r5krAz9zhhUagCnTYNpkxJzyPa0KBq0UzFSrbUrk0y6ZoQkTJWSC/RvoSZDT4GHA84YMA3ge9E1aLJBWa2CBjt7m/Fln0XWOPu15nZF4D+7n5NS8fo6F6iO+23H/VvnERDl/Op7f4r6vo/BG+/Haak2ntvePbZjo+p2kycGIo7GxtDQiiia0JEOrl29RI1s5OiITuWAbcBPYDPAWMJCdvcpJO1HM4C7or+vosw20Lns3YtDXYevZo20mDnNV+XZ4Is7aRqaMmka0JEyljOgXPN7N/AQcAqQrJ2p7u/EK07sMixOfAnM3Pgx+5+GzDY3ZdF65cDnWuy+WjMtfqh17DklWH4Hnsw9ZzFsHR0qBJdulRfFh2lrg0DF0tl0zUhImWstZkORgCvApPdfVbxw2lmrLsvNbO9gIfN7F/xle7uUTLXjJlNBiYDDBs2rGMiTYl6HUxbUsuKQQcxeDDU3bIfMDYkczfcENq1gb44REREJG+tVYleD/QFHoka+n85mpaq6Nx9aXS/EvgVod3cCjPbByC6X5llv9vcfbS7jx40aFBHhJoWVblY/9AXwiy2rqEBli+HFSs0vIeIiIgUJGfCFjXo3xeYBLwMfBV4zcweJXRAKEqDLDPrY2a7pf4GTgNeBGYCl0SbXULomdppTLnjPxj8x7tYsa4Hq1ftoP/rz6V7pNXWhg4HgwerWlREik+9YkUqSt69RAHMbDBwKXAZcHC0+E/ALcAf3X1zIkGZHUAoVYNQbXuvu3/LzAYCDcAwwrhvte6+pqXjdHQv0cE93qZ7zXaWNg5giL3J1prerBj53jBwbm0t/PWvMGMGjBwJffpoflERKR71ihUpO0nNJYq7r3D377j7oYSpqO4GTgRmAKvMLJG6Pndf4O5HR7cj3P1b0fLV7j7B3Ue4+3tzJWulMGnsSrbu6Mph3RewtaY3k7rODPWiqZH8Z8yA7t3hySc1lZKIFJd6xYpUlIJK2LIeIIzPdj5hfLb/cPeaJAJLQqnGYauf8iQNM7pRO2kbdScuComZSthEREQkh8RK2LJx9w3u/hN3HwMc2d7jVYKGpWPpNe4EGpaObT6S/y23hE4Hl11W6hBFqke1teVKvd4pU6rrdYtUuHYnbHHu/lKSxytXO2sihjyZ/QPzhhtg9uxwLyLFtXOS3yppgpB6vTNmVNfrFqlwiSZs1ay+Ho49Fo45Bvjrk8xkInVPfwZWroQrrggrU794V68OO2nWA5HiGzIEZs0K99Ug9Ytx0iS1YROpIK0NnCt5uuEGmDcPeveGhje7UTeuV+hwsHAhdOkSxmCbMQPGjYOBA2HoUH2QinSEpUvD+27p0lJH0jE0o4NIRVIJW0LcQ7LW1AS1k7aFX7ZXXx1mNhg+PIzBlvrFe8IJpQ5XpHqot6SIVAAlbAmZOjXkZQceCJw4Nt3R4K9/hTffDEnaLbeE5UuXqm2JSEeJd/ypJtXW2UKkwilhS0hdXRgfd8SIjDwsNfbajBnpD9AhQ/SLX6SjVFviknq906bph6FIBck7YTOzYfncihlsZ5e15mXSJNi6Ndynem8tXVqdv/hFSqFae4ma6YehSAUppIRtEbAwj1vVqqsLn40NDWHw3PpjpnHsA1/gmPWzqH9oQLpkbciQ6vrFXwzVVmoibZftl1QlXz+p13v11fphKFJB8p7pwMyuZdfJ3rsCBwJnAS8Av3f3rycZYHuUYqaDndP3zfo7bN3K7LUHhFh4lpmjvgrPPac5/pKgcyjtUQ3Xz5QpoSnGpEmh/ayIdHqJzHTg7te6+9czbl9x9wsIMxzsD3T8PFCdzM4f85O2Uds0nb1tFYNZTm2XB0IVRbONVFXRZjqH0h7VcP3E28+KSNlr91yiOw9k9g3gzJYyw1Lo8BK2+vqd84bWU0fDDUuoXX0LdRtuhe3b4YIL9EtXRDqGSthEyk6uErYkB85dChye4PHKTv20FTSs+Cq1ix+jwZbQa+E8GppOo+7A34fuo9UycKeIlN4ttyhRE6kgSQ7rcTbwdoLHKzsNdi692ESDnUut309jo1G7/R54+eXQTub550MpXCU3eC6EzoOIiEhe8i5hM7OvtrBqADAeGAl8N4mgylXt1UNpaBhKbS3UMZi6G74Ar70GXXrCpk2wbVt6aIHUMAPV3IMrPtxCNZ8HERGRVhRSJXptjnXLgS8D32lXNGUuNYVf/ZQnmThjBLWTbqau/9fgscdCYtKtGyxZAv37w9//HtqWVLPUGCiV3PBbREQkAYUkbPtnWebAGnffkFA85SvV4WDIEBpuP5tefbvQcG936jY9HiZ/79IFjj46JG6zZlXXZNQt0STV0h6xTj45r6N8txMR6cQKGdbj9Sy3N+LJmpn1KE6YZaChgfqV72Pi7WczpMcqGt/eQm23qFv9tm1htoONG0Oytuee4X7IkFJHLVK+8p3BoNpmOkhRG1GRipJIpwMzO87MfgS8mcTxylJtLdNefj+zu57A05uOYubx/0Pdvo/DwQfD7ruH0rUXXwwla2+9pRI2kfbKdyy1ahhzLZtqTVRFKlSbh/UwswFAHfBRwsC5BrySUFxlybZug+1bsH57wF57pb8gpk2DZcvCh+crr8DIkaGErdrbsIm0R75V6tVa9a42oiIVpeCEzcxOJyRpE4HuhCTt68Av3X1esuGVkYYGru7xFA1+FrX7vpCe7uaYY2DFilAdethhsHAhrFkDNTXw9NPNj6G2Nu2ncygSVGuiKlKh8prpwMyGE5K0S4ChwFvAn4ALgHPcvVPOfdKRMx3UT7idG2Ydi3epYczJ3Vna5xCGDIGldz5M7fZ7qOOeMNtBl6gWuqkJ+vaFd95JH6Qa5jcsNp1DEREpU22eS9TMLjSzR4BXgWsIc4V+CBhCGObDkg21fDW8eATLuw9jEcO5/clDWLkyzArTq2cTDTUXhGQNQqLW1BQSt82bmx+kWtvaJEnnUEREKlBrVaI/BxYAVwHT3X11aoWZcrW42knbWHLvGl5jX/bYDZ57ZgcH9VjMrKYxTBr8JGx/F7wZ9cmoqYEdO0JnhDhVYbSfzqFUq9jQQixdqmYBIhWmtV6iW4DhwFnAGWbWq9gBmdm+ZvYXM3vJzOaZ2Wej5dea2VIzmxPdzix2LIWou2UsV49+nAO3/5stazdyTPd5vNXYm3E9nmLp0R+A446DHj2gZ8+QrAGsXVvaoEWkcqR6hc6Yod6hIhWotYRtH0Lp2kBCadtyM/uZmZ1M8apDtwNT3f1wYAxwhZmlJpW/0d1HRbeHivT8bVJfD1Mf/QDdt2/iwB3z2atpBZOYQePmmlA7V1sLffqkq0bzPajGURLJXzW/Z1LNASZNUrMAkQqUs0rU3dcCPwR+aGbHAh8DzgcuBVYRZjrYI8mA3H0ZsCz6+x0ze5nQZq5Ta2iA/bstYeG24Uzr923q9nsCli+nvttlNDS8D2rrqDsx+gX8yCOhs8HYsa0fVHNtiuSvmt8zag4gUtEKmengWXe/glDqdhGQGsLjp1EV5ZfN7Igkg4t6px4D/D1a9Gkzm2tmt5tZ/ySfq71qa2GvkYOZNuoe6mq3gjv13S5j6tovs/Lh52m4aCY88UQYf+3oo+H00+Gyy9IHyFYyUO0N6Ku5tETaZsgQzSIiIhUpr2E9Wty5+XAf+wJN7t7mwXgzjt0XeAz4lrvPMLPBhOFEHPgmsI+7fzTLfpOByQDDhg077vXXX08inNZNmUL9ndto4Fw21vTlxe2H0dM2s8+7ali4YAfTmEod98I++4Rx2QD69YPVUT8ODUexK50TKVQ1XzMag1Ck7LV5WI/WuPsid/8qoWPCmUAi47GZWTfgl8A9qTHe3H2Fu+9w9ybgJ8DxLcR0m7uPdvfRgwYNSiKcvEy583gu2/wjHtk8hsc2jqa7bWWtDWCvDQuYZv8vJGvxMdgA1q9PH6DaS9Oy0TmRQlXzNaOpqEQqWrtK2IrBwnghdwFr3P2q2PJ9ovZtmNnngBPc/bxcx+rIgXP36L6J9dt6AtCFHfTt5VxwSXduObEevvQl6le+j4Y+H6V236eoe+WrsGlTKGF7++0OiU+k6lV6CVSlvz6RKlC0ErYiOZHQRm58xhAe3zWzF8xsLnAq8LmSRpmhW490p9meXbbTc8taTrz3U3DHHbByJQ2bz6LX2jdpeGlkGDC3V6+QtEky1N5NIPt1kFo2bVpll0DV1YVqYCVrIhWp0yVs7v6ku5u7HxUfwsPdL3L3I6PlE1OlbZ1Fn+3r6IIDzqamHmxu6k7DO2fCo4/C5s3Uch+NO7pT2zQ9VIk2NkLv3qUOu3KoOkgg+3WQWmZWvdWlIlL2Ol3CVq4GDu5GV9sBGIaznt15zMcygT8xkV8DMJOzqKu5L+ygErZkVXPbJUnLdh2kll19tUqgRKRsKWFLSP8+W9jmNUDoxgpNdGU7T3ISvbrtoIHzwi/8bdvC1FTZSthUrdd2qg4SyH0d/PWvlf3+0ueHSEVTwpaQF18Ns3Z1wenONg5gAevYne5s4Q/bTmUIb4QNm5rC1FRdusCGDc0Pomq99tEXlmRzww0wezbce29lv78yPz/0fhCpKErYEjLyoEYM6NLFGVvzNEfwL47iBXZjA33YwFL2BfdQygYhcaupaX4QVevlJ/OLqFoalUvbpHrC9+sX3l8bN8LgwTBlSmnjSlrm54d+AIpUFCVsCVnT/V3s3q+G3n27ctknujLzgz/l6r4/YTDL2ZsV1HI/7LEHDBoUJoDv2TNMBh+nar38ZH4RqVG55DJ1KoweDd/6Vnh/vfgidO8eJkmvJJmfH/oBKFJRlLAlZPFiWLvWeWf9DqY9dBjU1lI34h88N/B0nu1yfBg4t6YmlAJdeinsvjtccEGpwy5PmV9EtbUwf34oSdEYVJIpM5EZORJWrgz3lShV4gz6AShSQZSwJWTb+o2A4xj/eqM3Uy5vYr/nZrDH6leZ0nRzaLPWtWv48Hz8cVi1KtynqL1J/rKVRL72GixcGNorieTSpw+cfXa4r0SpNnt6L4hUFCVsCRm6bWH0l7GZ7ty7cSIrGMxWujODD4c2a9u3h4Ts5ZdD9d3LL6cPoPYmbdfQEAYjXrcOOmruWCm9tv7IqfSqwtWrYc2a9DzFIlIRlLAlZAH7A+nZDrbSnb68Q3e2MolfhgStR4+QXBx2WKi+O+yw9AEq/UukmGprQ0LcrVtIiqU6tPVHTqW3FR0wIH0TkYqhNDgT4gAAIABJREFUhC0hO+jW7PHuvENNF+MCm87TjOFY/yf1y8aHHmrz5oUEY9689A65vkSqqbq0La+1rg4+/nHo3z+0C6ym81XNsv3ImTJl1x6g1XY9jBkThg4aM6bUkYhIgjrd5O9J6sjJ3wfaKtawJ+DUsI0murI/C9jAbtSwA3BG2xxmdvsw/OxnLSdmDQ0wZAgsXZpuQD9xYihJaGwMSV0lS+K1VtP5kuYGDw49QLduhRUrwrJqux6q7fWKVJBym/y9LO1GahBcYwfd2Y31LOAAVrIXq+kXhvbw6aHK7ktfCptm/vJPVfHMmNG8qqeaqkuTeK3VdL6kuUmTQrI2aVJ62ZAhMGtWuK8Guv5FKpJK2BKyny3iDYZFj1LnNJUPN+F0DUN5bN0aSgDWrdv1l3BLJWwi0nYqcRKRMqEStg6woesehETNSHc+aAKcPmwInQ42bAi9Gfv2DV8iQ4Y0/yWcasd2yy3N27NVWxucQun8VKds//dsbdiqpcQpdT6mTNH7QaQCqYQtITUW2q2lOQN5ixqamMQvuYVPxzaugb32gi1bwthskyaFJK0lhZYQpErqqqWE7phjQnulwYPhuedKHY10lGzvi2xt2KZMCc0MRo4MY69V6vvi2GNh+fLQsemMM1SiKFKGVMLWATzLqdxIH7bQnafJ6K3Vr1+4X78+vylyCi0hqLYx3VLzs5rl3k4qS7b3RbY2bDNmhPfZk09W9vsic87USi9RFKkyStgS0oeNzR73YDNb6c469mANA6jnAibya+prLoZzzglzG5588q5fLtkUOm5UtVQBpfTvHwYJ7d8/PC73KtJyj7+UbrkllKzFS6xTU1HtuWdldz5IzZl65pmljkREikBVogkx20Hz/LeJVHu2GrYzhCVsozuDbRXPHX0Z7Ltv86qZtlZjlnv1ZyHxx7eF9N9XXJG7M0e5Kff4synGdZrtPKWeZ+PGMMn7pEmhA0+vXiFZGzcOXnll1/dfJUhV/e7YEd4LaiIgUnZyVYkqYUuIWeZ5bCIkcGF5Lxrpx9vszQqe7XNKGN6jTx848cTwxfGlL6XbYX3rW/l/uZX7l3sh8cfbqkH4u1s3eOON8LhLl/BlVU1JbLkoxnWa7TylnufBB0M70a1bYdq05klcKqHZe2949tlkYukM9tgjvN6tW9Nt+Y46qrKuI5EKpzZsHaIp43EX3sViwDGcXmxkNM9yNTeEzgZmoQ1bqk3N2rVh2dq1hbVBK/fqz0Lij7dVS/29dm16fVP0Pyj3qYfKPf5sinGdZjtPqecZOzbd3CC1XZ8+oYRt27awbaX9WO3XL7ymfv1C1Wj//pXdZk+kyihhS0zmqWziTfYFnG5sZT39GMJi6rrcBwcfHJKLgw9Of4ldcEEYp+2CC9JfOkOGlGdbpkLaYGV+6WYbliHl6qvDF9HVV8MJJ4SSkqFD0+t7904m/lLLdQ7KVTGS0GzXWep5Hnkk3ZZtwoQwj+8TT4SSt6FDQ8nsggWVdY7PPDOUso0aFR6PGVPeP+ZEpBlViSZk1yrRuLCuG9u4nY/SsPtkag+bS91eD+/a9iZefTF8OCxbBv+/vTOPs6osH/j3mRlHhl02Q01RwWVcWMsWfoiIqRmIpWQuRWk6ppmJlqX9xPZy6WeBGampQ6RouVu2qBgaaOIGGIKAwbgg68CwDvP8/njewz1z7rn3njvDzL0X3+/ncz7n3ve8532fdz3Ped7l9O0LJ59s81OiW4AU45BoUpni0lxVlZqPtnlz5vsuvtiGQOvrU5Y1gJoaG2Yu5SHFuK0pPOkE21iEhzaDeVzhdlJenqojHTpY3gbn3SmPgykDa9eadc3PYfN4Sg4/JFpwbPHBdir5KlNZWb8nM17s33y12owZtpJt4sTUxpd1dTbXra4Opk6F99+HO+9sHnR0qKkYVhhm+hRQpk9xhYdstm2zh+uWLfYAGjIk5T+8MWh9vQ2HNkWGou+4A665xqwpF19cetZJsNWMdXV2jlIM5VssBC+b4ZfO6dOtbkyfnnIrC3Vz27aZZbaqyvzlWqFdSixfbgrstm22anrx4t3PUpsN3zY8uzleYWtXlEb2YCkHMr7To1BXl+pj9v0WLF0KBx6Y+pZoRYUN3VS4DXnjrKHRoaZi2IOtrs7mCtXVNXePyhY3r6my0uanlZWZteDdd1P+g/s3bkz5j+69tt9+ZmHYvNmuleL8nVWrTNldtSr9WjGUb7EQbGMxcWLKLTyPK+D8820BQs+epsCUl9uQ+rHHZt+wutTYvt2GflUtrZs3J9vncXfBtw3Pbo5X2NoVoZE9WElPvrF+EkOeuoEbL15M1cplzJi23Sxs8+fbW/8bb9jD5fDD4aCD7PaKCpgwoXmQ0bfK8eNh9myYObNwb9aZLGxxCtry5XDTTSn5J0yA3r1N4dt7bxvuCvw3NNgcpPDig27dmsexZAls2mQLOzZtKs09tyoqYMWKlKIepj0XmRS7xSJuXlwwj6t//3TZBw40Be+ss2DRIqt7xZq2ljBsmFmc+/Y1K2KXLs0ttS0pz2KvA2FKfQGWx5MLVS2pAzgJWAgsBq7K5nfo0KHaXkBTXkcZjdqXFTqYuTqmw1+1lrNU7d1YddQo1YoK1fJyO5eVNb8WpmdPVRE7B3Ttqtqhg51byqhRqpWV6fElYcwY1fHj7RxQU2Py7L+/am1tSvYgXUHaOna03x07Ng+zttbyQaT5PbmOTp1MjiDOYqSmRrVPHzurNpc/SmvKJV+i5VhbW1x5GbSTHj1Mptpa1aoqcwvyr3NnawtlZeY+ZkyqLnbvbnUwnPelzAEHWN2oqrJyKyuzo0MHux7XV+Qiri3vaqqrTa7q6raLw+MpEYB/awadpqQsbCJSDkwBTgaqgS+ISHVhpWoZTcAqenEMz/Fwo+1MPpaHmMZZZp1qbLS35MbG5vO0nn66+Rvv6tX2aFq9OmVRq6+3OWD19fkJFX6bnjXLho5mzUp+T0BgCWsIff1h+nTYsAHefjs1ZLF6dfOwZs5k2qZxlg+bxjW/NmOG5UO+i2QaGoprmCQuv+LmXQUEfoOVo089ZXXimWfa3voRLcfWDDkFKzWPP37XyTdrlrWRNWtSe60FcyADNm60ttDUZPlWVWVDhI2Ntsny2rW7z7Dh8uWW/s2b7WhqsmPbNru+fr1ZpdevTx5me1itFiywdr1gQdvF4dn17I6r2YucklLYgI8Ci1V1iapuA+4BTi2wTA4JHUkoYzuV3MrXOKBxETdxBfM5jPO4nYuaJqeGw9x556etOHvnQ3PatJCSB61/6IQfyMOH28Nw+PDk9wTMm2dzhubNS7l17w7l5UzjbMauuCVNx5jGWYwte5QbuZwqNjGDzzf30JoHRjENk8Tl11572YM1+LRWmMBv8D3MwHbUrVvbz9mJlmNrHt5JXwDyYfhwy4tg3uP48TaHsXNnq28iUF3dfNHB5s025aCiwvKwe/dkn4crJcrKbKi4QwfLg8pKcx8xwtI9YkTysNpjT8DOnZufPaVB0CftDi87pUIm01sxHsDpwG2h/+cCkzP5L+YhUXvS5HtPkjCibvmMIKbfFx5dSnpPfm6N7ghfb9R42Rt3Xh/FEwrb3P8dofuC39u1hsk7h1niRmTDxI32BfcEedCxY/poZG2t6pjBy7Vm/0e1Z+fNWl6euh4d6amuVhV26D6VK3XM4OU7R/HGDF6utYNuUK2ttdHRPVZrjdyiWl2ttYNu0E6VW1y6tins0OpOS1Vra7W25p86ps9sra35p6qmRgh79jTZx4xJuYHJEh5dD9yC86BBqftqa1Wruy9XoVH36bgqLW/yHh1tq6HcTIKE44vEHS6zMYOX66hRLR8VjY5mZxOpzYnmcZxwxUixDbV7klEq9WsX0J5JJcuQaEntwyYipwMnqer57v+5wDGqeknIzwXABe7vodh8t3Zg6FB4H+jdPtF94FFNXyIapbERXnnFfg8cCOVlgMDGenhjMdALcEsxD+lvZq6yMnctdI+ETDRNTRZvsPHXIf0709hxE50rmqgQs7Cq2vWhQ1Nyvvhi8//r16ZMP+F4Bw4EbbI4X3nF5OoSWVkBFl5U5iFD7JqIpb1hA3TtnjufdubXdsufhg0WZpdu6fIGeROXX21KqKxyMwSGKKiAzIXI96cC2Tt1SeVR4/ZUnudDtLzC4bdb3hQjeZWXp+D48spKXDtvMw5Q1VhFImYZWlFTB3w49H8/57YTVZ0KTG1PoQJE5N+qb8VueOcpPqy84jco9BQXvqxKC19epYUvr9Kg1OawvQAMEJEDRaQSOBMoku39PR6Px+PxeNqGkrKwqWqjiFwCPAGUA3eo6vwCi+XxeDwej8fTppSUwgagqo8DjxdajgwUZCjW02J8eZUOvqxKC19epYUvrxKgpBYdeDwej8fj8XwQKbU5bB6Px+PxeDwfOLzClicicpKILBSRxSJyVcz1PUXkXnd9joj0a38pPQEJymuEiMwVkUa3bYyngCQor8tFZIGIvCoi/xCRAwohp8dIUF41IvKaiLwsIrNK9cs0uwu5yivk73MioiLiV44WEV5hy4OEn8Y6D1irqv2BXwA/a18pPQEJy+u/wAQg5rtQnvYkYXm9BAxT1aOB+4Gft6+UnoCE5TVdVY9S1UFYWd3UzmJ6HEk/7SgiXYBvAHPaV0JPLrzClh9JPo11KnCX+30/cLxI0o1LPbuYnOWlqstU9VXs866ewpKkvJ5S1U3u72xsL0ZPYUhSXuEPGncC/KTpwpH0044/wAwNW9pTOE9uvMKWH/sCy0P/Vzi3WD+q2gisB3q2i3SeKEnKy1M85Fte5wF/blOJPNlIVF4icrGIvIlZ2C5tJ9k86eQsLxEZAnxYVR9rT8E8yfAKm8fjKTlE5BxgGHB9oWXxZEdVp6jqwcC3gWsKLY8nHhEpw4asJxZaFk88XmHLj5yfxgr7EZEKoBuwul2k80RJUl6e4iFReYnIaOBqYKyqbm0n2Tzp5Nu+7gHGtalEnmzkKq8uwJHA0yKyDPgY8LBfeFA8eIUtP5J8Guth4Evu9+nAk+o3uysU/lNmpUXO8hKRwcBvMGVtZQFk9KRIUl4DQn9PARa1o3ye5mQtL1Vdr6q9VLWfqvbD5oiOVdV/F0ZcTxSvsOWBm5MWfBrrdWCGqs4Xke+LyFjn7Xagp4gsBi4HMi6d9rQtScpLRD4iIiuAM4DfiIj/1FmBSNi+rgc6A/e5rSK8Al4gEpbXJSIyX0RexvrDL2UIztPGJCwvTxHjv3Tg8Xg8Ho/HU+R4C5vH4/F4PB5PkeMVNo/H4/F4PJ4ixytsHo/H4/F4PEWOV9g8Ho/H4/F4ihyvsHk8Ho/H4/EUOV5hK0JEZIKIqIiMLLQsHk+hEJE7RSTRMnYR6efazKRWxqkicmdrwigkpS6/p2WIyMki0igihyX07+sJICLfEJHVIrJXoWVJglfY2hARGekaRnDsEJG1IjJPRO4SkZN29YfhRWSSiOy2u4k7ZfayQsuxOyEig1y96deCez8kIj8SkRdFZJ2IbBeRlSLyDxG5QkT8d3QLTJI+QUSGuT7qxPaSy7NrcF/UuRH4var+p9DylBi/AbYC3yu0IEnwClv78AfgXGAC9kmdfwAjsQ9X/1VEukf81wJVwDMtiOtadu/Pv0wAvMK2axmE1Zt++dwkIicB/wGuxHawnwRcgG1uux74EfDcLpTTk5sq4KsRtyR9wmlYmT3ZFkJ52pQzgMOx74B68kBVtwC3Al8rhZfLikIL8AFhrqpOCzuIyOXAz7Hdv/8AnBxcU9UdwI52ldBT1IhIObCnqm4qtCwAInIE8EfsO7kfV9XXY/zsDVza3rJ9kHEPoJZwGvC4qm7flfIUAhHpoqobMlwToJOqbmxnsdqSrwGvquorhRYkKSJSBWx3X18oNNOA6zBjwI2FFSUHquqPNjowK5oCV2Tx80/nZ3jIbYJzGxly64BZMBYCm4B1wGvA9e56P3dP2hEK4/PYt+P+i5mBVwEPAkfHyLUMeBo4DHgM2IC9gd8PfCjGf1fMovI6sAV7kM8Czoz46wv82smwDXgbmAr0SZCfyzKkMZxPI4C/OVk3A3OB8/IoMwXuBD4OzAQaXFpuAzrH+E+UHmAfrDN4GVjr8mgB8G2gPOI3KP/RmKn+TWA7MCHkZxjwgCvDra5eXA1URMI6ArgP+8jzVuBd4CngFHd9UoY8vTNHPv3R+TuhBe0iURm5ctAY9+HAs+7e94DJ2EerFZiUUIYjgL+48l0D/B7okyntWNuZhbWDTcAc4PQs9Wc09i3GTS7Pb85Qf/phFvX3XPm8CfwY6Bjx1wP4hbsetK8XgSvj4k/aJzh/hzr3MyLu41w+NwAb3e9TM+TnYFfPgnQsx15ED474Ow7rT1a7dCzBPufXy10f6WSZEBNHWn3A+qhlwEFY37Qm8BMOC7gYa29bw3UEGODy/x2s/S7DLMSd4uIGumHtfaWT/1ngmBhZBbN0znF5txHrr78f8bcn8F1gvgtvHfAIMDhhPf6Qk+sn7VzPy7G+6S0n96vu3kku7H4xedcbuMPVkaaIn0TxOr+jgb+6vArironx9wlsFOtd568OeBz4WIzf/wCz8+3L2vvwFrbCczv2ADoFq7CZmAJ8BbgbM31XYJ3NKHf9fWzYtRZTAqfGhHEJ1lFOxSrxwdgQ1rMiMkRVox9m3hfrEB/Ahr0GAhdiytmnAk9uSHcW1jncj3Vo5Vgn/hngHudvf+BfQKVL95tAf+Ai4DgRGaaq67PkwWXAT4BewDdD7q+78Mc4Wd/FlKMN2AeObxORg1T16ixhhxkEPAr8DpiOdfznYZ3MBaF055Oeo4HPOvneBPYATgJ+ij1sLoyR4wbn77dAPaaUISKnAH8CFrt0rsEUzO872c9w/nqSGuK6Fetce2HK3jHYg/NPmNJ5AaYoBJayNzNljoh0wOrrW6r6t0z+MtzbqjISkWOAv7v7foZ12mdi7SKpDAdibWRPTNlbDozBHmxx/n+IKcN/wR5STZhF6j4RuURVp0RuGQKcjpXb3ZiScilwpIicoKpNLtwDgOcxJeAWbFh5JPAd4JMicrymLBD3YYrurdgDqgobBhuJKRhxJOkTcGnZij3cgjR/Detz/oPVKzDF50ERuVBVp4b8fgZT4BuwF5vFmCJxIqZIv+n8XYj1DXXu/BawP5b3+2EvHy2hM/Zy9SxWTn0i1y8DemLl8S5W3ojIUKx9rMPmMtVhfdylWP4fq+kWxyewfP2+C/Ny4DEROVCbW/VqgbMxxeNHLo7DsHrxvy7+PbA69QnnfzJWF76K9ckjNPeH14915+ejF9q4nk8GarCXvxswZewWYGkWWf+G5f8PgE6YEptXvCJyAdYGZmP52gCcAPxaRA5W1Sudv0ND8d2MKYl7Y8/age7+MP8CzhGRzlrM1tdCa4y780EyC9sQ5+ePIbcJpFuO1mBDFrnizGgdIfLW6NwOxzrrWyLuy1xY4yPuU5z7oSG3W5zbBTHhl4V+P4S9me4X8TMMaCSBdQT3Rh3jXo49ANYB+4TcK7GOfAcwIGH+NRF5a8aUm+2ErCT5pAd7wEpMfLVOtr4x5b+QdEtLB6wTeoZ0a9o3w/UGGBtXhjEypNW3HP6Pcv4firnWAVMKw0dFS8qIeIvKc5gl5JDI/c+T0MKGKeEKHBdyE0yRbNZ+SLXPH8eE8yCmSHeJ1B8FxkX83uzczwy5/d65fTri93rnfp773839vyVB2tLaf5xb5Poc4NHQ/72wh+lioGvIvSumfG0Auju3jpgCsxLYNybsMnfeD+tnFgT3ZvA3kvwtbAr8MMZ/ENYaYiz4wCuYQtol4n5aVAZSVqJoP3mGc78w5DbeudUS6v/C6Yy01xMjfrpiFvunE5T3dS6MuFGSNqnn2Iu5YgpWOD1HYW04k4VtWkzY+cTbF7OUTY/xe7OL+yD3/1IX7kdz5aHzf43zPzSJ/0IdftFB4al35645/K0HjhCRI1sakao2gM3jEJGuItIL62wXYhaXKG+r6oyIW2CxGeDCKsMsHK9r6K07FGdgTeiGWdseBraISK/gwJTDxYSsdi1gKPa2foeqvh2Kfxs2V7AMODVhWP9S1TkRtycxq2Y/yD89qrpZgx5TpFJEeji/TzjZhsXI8WtNn7N2Avam+DugeyTex52fIN7AuneyiOSqX/kQhFUfc+18rE6Fj0HuWqvKSET6YJbEh1T1jcj9v0giuKuvY4B/q+pToTDUyRDlbKwjvyuc1y6/Hwa6OJnCLFTVByNuP3Xn00JyjAVeUtXHI35/QsrKADb0uxU4piUrebMhIvsAH8EejgEnYBaQX6rqzjJ2v3+JWbRGO+cTMaX8RlWti4YftH9MsakErlPVdVn8tZQbsly7W1VXhh1E5CjM6j0d2DNSrrMwy01cfxStZ836Q8fZ7nxFNF2R/+dgCuOLkfgrMevQcDfXKxu93XlNJH1tWc8/4843h9Ojqq9h/Vkm4soon3hPx6yFt8f4fQTrP4J6GfR9p7oRgVysdueodbao8EOihSfbwy/MZdgb22sisgQzRT8CPJK0sxORwZg5eiTWIYdZGnPLkhi3oGIHK2p6YW/ksWb2EIdiDeo8d8QRF19SDnTn+THXAreDEoaVJN15pcctvb8K+CI2bBrdziVuH6A3YtwOd+c7MsQJptChqjNF5G7Mgna2iLyADSfeq6oLstyfi2wvGQ9iDyGwtJ4butbaMgquxW1dkDQ9fTCFI2kYh2NllW27hL0j/9MWYKjqOyKyjlQaejs50vJCVdeIyDuBX1XdJraVzc3AUhFZgCkKD6rqP7LIlYRx2APz4ZBbPuUUKCov5Ygnqb+W8H6cEhgiWzu6zh1xRMsVIn2Dqq62dQyEVxgOAN5R1feyyBTIUIW91GSiF24INwPqztH+pC3reVA/Fsb4WUhoAV2ETOWQNN6gzP6ewO89mEL8XeCbIjIbUybvUdW3Yu4L8k9jrhUNXmErPEe7c1zl34mqPuTerj+NzVsYjSkK/xSR0c7KkBE33+oZ7GH7AxdfA1ZB/w9r3FGyrVTNd/+4wP804K4MfjbnGWZbkSTd+abnJuDrwL3Y3IuV2BDrEGwuVpy1O25FaBDvldgChjjC1qsvicj1WCf6P8BE4GoRuUxVJ2e4PxeLMIvPwOgFVV0BrAAQkeEtDL+YEKyNnEzmehGn2OxSVPVWEXkImzt4LGZtuERE7lXVM1sR9GnAc1ELVAHJ9sDM9LzKtXI6Wzu6kcwvm2ujDmor+ONoyX6agi1EuDyLn2zKXPh6D7IrdklkadN6HjNakG+8QR5/EVskEscSF9dW4AQR+ShmBR6BzTucJCJnqeoDkft6uHOu/C4oXmErPIF15rFcHlV1DaYgTHPL038KfAsbRrovx+2nYUrZ2LCJHHZOTt+ap9wBq7COLe3hHWEx1jArVTXbG1IuMnXowZvvETHXqiN+dgX5pudc4Jnow1VE+ucZb7AwpCFpPqrqPGAecL1bIDIH+KmITHFDJHm9VarqFhF5DPism0SfdOFBa8sosALH7eZeHeMWx/vY/KykYSzCFof8V2O2LsnA4VEHEekLdCeVvvexuWBpeSG263pfIgq5qr6DTeq/TWybl1rgCyJyo6q+kFC2aDwjsZXKYcLlFLXgRcspsJoMwlbuZSLsL87SEhAM7fWIuZbUQp6EoB3taGV/FMcb2FDc3jmsbIswS+uTrRgSnufOA7A5eQFtWc+XufOhpLfXQ3Pc25p4gzJblUff9zxuQYaIfBiz8P4Qm8cXpj827zir4aTQ+DlsBUJEykXkBmzVyuOq+mwOv80213UP2mB4Idy5bSS+swveXpq9CYrIV7EVXS3CdTR/AKpFJG1o0CmWqOpqbI7VZ0XkY3H+RKR31D2GjcBeQbgh5mITdb8sIjvT41ZiXYmbJJ8sVblpQXp2kJ73nWi+2jUJT2DWuatEJK2cRaRKRLq43z3cXJaw3OswxacjtkAA3Got4utNJv4Xs1zcLiJpCkogTuR/q8rIPfxmYw/DQ0L3V5IwH52F5FFgmIgcFwpDsJefKLXu/GOnJDVDbK+5KIdK+pcFAqXoQSdHE277BrENiMNchfXND7g4OopIx5h0vOr+5iq3TH3CZ7CX9ujD62+Y9f3rQV1ycnTBrMQbnR8wJW0VMNEppc0ItdP7scUi18bNpwz5W4o9OEdHrn8CSGtnreAlTNmpEZE0RVBEKuLaV0J+784/j7a/SL91N9b3xlrYMtStKDPduVnetHE9f8SdvxFOn9i8wHy/lJFPvDMww8J1cXP7RKSbiOzpfveKiWsFpsjGlevHgBe1mFeI4i1s7cUQETnH/e6CvYWMAw7AOryzctzfBXhHRB7GOpqV2DyCizDr1iMhv7OB0SLybezhqKp6D7ZkfxNQKyKT3X2fxIZY36R1deEabHuR20TkU9ikXcG29aggNY/pInftGTe36iXswXQQZiW8G9vHJxuzsQfNZBF5DlOEnlTVlSJyCfbweUFEpmIWjM9jjfHHmr5tSWvJJz33AxeKyL3YHIy9sW1aVpMHqtogIl/EHvwLReQOzNrXHXub/ixmTX0aGzr4pog84Pxsx4bTTgRmqGowZPsCNsn9amd1aQCWxiy8CMsxX0Q+h80VeUVE/oQtja/HrAYfcXmwHje0pKo7dkEZXe7S9qyITCG1rUc+9fcabAjmURH5FdaRjyE1gTuczhfEvk86CXhZRO7Dhpz7YosoPo1NEg/zGmYF/y1mFTgOG8KciQ2JB3wXm+D/oIjcgpXRCCw/niE11H4IMNOV4zwsPw/H6t9SbOuGbGTqE07DNlxtNn9VVdeJyLewFeFzJPXNyQmYJeJCddvVqOom96J2PzBPRIJtPXpj9ewmbJHICrF5eFOwebh3YyuG98XqyVeAl1V1o4vvfBH5A1bWA4AvYwpqLkt+IlRVReRcbC7gq64dzceanlLNAAADdUlEQVReZPpj7eg72ArHfMO+z7XzLwIDXL+9FivHYKsTsDmJJ2CW71FOlnpsYc7x2IrI48iCqr4vIk9j9fCKyOU2qeeu7U/FtgL6u6uXvbG97l5y/hNZ7fOMd4WIXIRZmV8XkVqsDvXGVqiOw6yHy4Br3LPoUayNiEv7YUQWXYjIwdgzOZp/xUeSpaT+aNlBall5cOzAHmDzsc74pAz3TaD59gyV2Mqx57EH/FasUt5BZKsKrHP7K9bwldAyeOxhEGxOuA4bhj2SmK0yXPhPZ0nThIh7d6whLMbepFdjD5LotiC9sG0L3iC1UeRrWOdVnSBPO2J7nr1Hagn5yND1Y7G3/3oX/ku0YOPcXGWSb3qc3NeT2mhyEWZJOT6an5niisR7JDY8Xufy+z1sy4vvAT2cn0Guni3GlLB6bNhkIvbVhHB4X8ImI2/LlAcZ5OiLzcmbi9Xt7dgLxZNYB9gz5p5EZUTmjXNHuLRucemeQv4b5x6FtZOkG4qeglk315DaGPbPRDbsDO7HLERzSG3u+ysi20c4/wdiVoaVLu+XENk4F5vQ/gtsiHSdC3MxNve0b1z8ufoEbLJ7Q7b8ws1vc/4a3O9xGfx+FHuJCDZy/q/L04Mi/j5FatPkYOPc34brCTZ14zasD9mE9SOfiKsPZNjmJ1tfFfFzALav1zJS/daLWH/74Vx1MUuel2EKzFyXhg2YwnltxF8FtgXFC6F8XuTy7lMJ63KwjUjalhS0XT0vxz55FmzC/qqT4wYXdp+Q34x5l2+8zu8nsZe+oM28jS3Cmwh0CJX9va5cN7tw52Cr2CUS3rWuLqb1VcV2iBPY4/F4PK1ERBS4S1UnFFqWbIjIqZiCNVhVMy1e8ZQAbijxFcxCeU4u/20syyPYaEtXzbxAo2gQ2/JjCbZ6NNvij6LAz2HzeDyeDx6bge96Za30cYrRFdgClEzzSXcpGeaQHY0NwT5ZCsqaowaby/uDQguSBG9h83g8nl1EqVjYPJ7WICI12By9x7CJ/Idhc9rKgE+qalvst/eBxy868Hg8Ho/Hkw9zsTmOl2KrLjdg81av88pa2+EtbB6Px+PxeDxFjp/D5vF4PB6Px1PkeIXN4/F4PB6Pp8jxCpvH4/F4PB5PkeMVNo/H4/F4PJ4ixytsHo/H4/F4PEWOV9g8Ho/H4/F4ipz/B7+//mq6kx92AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['figure.figsize'] = [10,5]\n",
    "plt.scatter(x = au_df[au_df['is_error'] == 1]['dist_to_occurrence'],\n",
    "            y = au_df[au_df['is_error'] == 1]['result'],\n",
    "            color = 'red', alpha = 0.6, s = 4, label = 'Error')\n",
    "\n",
    "\n",
    "plt.scatter(x = au_df[au_df['is_error'] == 0]['dist_to_occurrence'],\n",
    "            y = au_df[au_df['is_error'] == 0]['result'],\n",
    "            color = 'blue', alpha = 0.6, s = 4, label = 'No Error')\n",
    "\n",
    "plt.legend()\n",
    "plt.title('Error vs Not Error for Gold Assays', fontsize = 24)\n",
    "plt.xlabel('Distance to nearest Gold deposit/occurrence (degrees)', fontsize = 18) \n",
    "plt.ylabel('Au Assay Result (ppm)', fontsize = 18)\n",
    "_ = plt.ylim(0,200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly we see that when the gold result is really high but is an error, its far away from a gold occurrence. This means that distance to a deposit is an important statistic to weeding out the Gold assays that may be small (eg 2ppm) but are incorrect.\n",
    "\n",
    "Lets do a quick test on Gold results that are 20ppm "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>collar_id</th>\n",
       "      <th>collar_name</th>\n",
       "      <th>result</th>\n",
       "      <th>fixed_result</th>\n",
       "      <th>is_error</th>\n",
       "      <th>lat</th>\n",
       "      <th>lon</th>\n",
       "      <th>geometry</th>\n",
       "      <th>dists</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>234726_SA</td>\n",
       "      <td>MAC35</td>\n",
       "      <td>20.0</td>\n",
       "      <td>-777.0</td>\n",
       "      <td>1</td>\n",
       "      <td>-32.561317</td>\n",
       "      <td>140.999317</td>\n",
       "      <td>POINT (140.999317 -32.5613166)</td>\n",
       "      <td>0.206771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>272864_SA</td>\n",
       "      <td>DP016</td>\n",
       "      <td>20.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>-29.717121</td>\n",
       "      <td>135.573824</td>\n",
       "      <td>POINT (135.5738239 -29.7171213)</td>\n",
       "      <td>0.004012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>273061_SA</td>\n",
       "      <td>DP006</td>\n",
       "      <td>20.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>-29.717310</td>\n",
       "      <td>135.571758</td>\n",
       "      <td>POINT (135.5717579 -29.7173103)</td>\n",
       "      <td>0.005965</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   collar_id collar_name  result  fixed_result  is_error        lat  \\\n",
       "0  234726_SA       MAC35    20.0        -777.0         1 -32.561317   \n",
       "1  272864_SA       DP016    20.0           NaN         1 -29.717121   \n",
       "2  273061_SA       DP006    20.0           NaN         1 -29.717310   \n",
       "\n",
       "          lon                         geometry     dists  \n",
       "0  140.999317   POINT (140.999317 -32.5613166)  0.206771  \n",
       "1  135.573824  POINT (135.5738239 -29.7171213)  0.004012  \n",
       "2  135.571758  POINT (135.5717579 -29.7173103)  0.005965  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "au_test = tmp[((tmp['fixed_result'].isnull()) | (tmp['collar_name'] == 'MAC35')) &\n",
    "    (tmp['ele'] == 'Au') &\n",
    "    (tmp['result'] == 20)]\n",
    "\n",
    "au_test.loc[:, 'is_error'] = np.where(au_test['result'] != au_test['fixed_result'], 1, 0).copy()\n",
    "\n",
    "# Get back the lat/lons\n",
    "au_test = au_test[['collar_id', 'collar_name',\n",
    "       'result', 'fixed_result',\n",
    "       'is_error']].merge(locs[['collar_id', 'lat', 'lon']],\n",
    "                         validate = 'm:1')\n",
    "\n",
    "# Convert into geopandas format\n",
    "au_test = gpd.GeoDataFrame(au_test,\n",
    "                      crs=latlon_crs,\n",
    "                      geometry=[Point(xy) for xy in zip(au_test.lon, au_test.lat)])\n",
    "\n",
    "\n",
    "# Get the minimum distance for each sample to each gold location\n",
    "# Taking the min over axis 0 reduces it down to min dist for each sample\n",
    "au_test['dists'] = gold.apply(lambda x: au_test.distance(x)).min(axis = 0).copy()\n",
    "\n",
    "au_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have three results - MAC35 which is very far away from a gold occurrence and DP016/DP006 which were drillholes into the Prominent Hill IOCG deposit (and verified to be correct).\n",
    "\n",
    "So is the MAC35 assay correct?\n",
    "\n",
    "Nope!\n",
    "\n",
    "![title](img/mac35.png)\n",
    "\n",
    "It appears that the Zinc assays in the source document are the gold assays in SARIG. Whats interesting is that it is the same company that found Challenger, so this error might have been skipped if the user noticed the company was Dominion and assumed this was related to Challenger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp.loc[(tmp['collar_id'] == '234726_SA') &\n",
    "    (tmp['result'] > 0) &\n",
    "    (tmp['ele'] == 'Au'), 'fixed_result'] = -777"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 - Variance within Samples\n",
    "\n",
    "Another way to check is for the variance within a single sample for an element. If there is the same interval that has a Gold assay of 100ppm and then 0.1ppm, its possible that an error has occurred.\n",
    "\n",
    "To calculate this:\n",
    "- Group by each sample_id, and take the max and min for each element\n",
    "\n",
    "To measure the variance, the difference or the ratio between the max and min can be taken. So far I've found that the ratio works better, because unit conversion errors can be easily found - If there are two Cu assays for a sample, and one is 1000 and one is incorrectly 1ppm, then the sample ratio will be exactly 1000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_minmaxs = tmp.groupby(['sample_id', 'ele']).agg({'result' : [np.nanmax, np.nanmin]}).reset_index()\n",
    "sample_minmaxs.columns = ['sample_id', 'ele', 'max', 'min']\n",
    "sample_minmaxs['maxminratio'] = sample_minmaxs['max'] /sample_minmaxs['min']\n",
    "\n",
    "\n",
    "# Put the maxminratio back into the main dataframe\n",
    "tmp = tmp.merge(sample_minmaxs[['sample_id', 'ele', 'maxminratio']],\n",
    "          how = 'left')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what shows up when we check samples that have a max/min ratio of exactly 1000?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>collar_id</th>\n",
       "      <th>collar_name</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>date_drilled</th>\n",
       "      <th>reference</th>\n",
       "      <th>sample_id</th>\n",
       "      <th>sample_source</th>\n",
       "      <th>from</th>\n",
       "      <th>to</th>\n",
       "      <th>ele</th>\n",
       "      <th>result</th>\n",
       "      <th>old_unit</th>\n",
       "      <th>method</th>\n",
       "      <th>isUDL</th>\n",
       "      <th>fixed_result</th>\n",
       "      <th>maxminratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1334201</th>\n",
       "      <td>202280_SA</td>\n",
       "      <td>RC-DD03SH001</td>\n",
       "      <td>462.5</td>\n",
       "      <td>27/04/2003</td>\n",
       "      <td>NaN</td>\n",
       "      <td>929299_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>104.0</td>\n",
       "      <td>110.0</td>\n",
       "      <td>Pb</td>\n",
       "      <td>4.5</td>\n",
       "      <td>ppb</td>\n",
       "      <td>ICP-OES</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1334218</th>\n",
       "      <td>202280_SA</td>\n",
       "      <td>RC-DD03SH001</td>\n",
       "      <td>462.5</td>\n",
       "      <td>27/04/2003</td>\n",
       "      <td>NaN</td>\n",
       "      <td>929299_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>104.0</td>\n",
       "      <td>110.0</td>\n",
       "      <td>Pb</td>\n",
       "      <td>4500.0</td>\n",
       "      <td>ppm</td>\n",
       "      <td>ICP-MS</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2077804</th>\n",
       "      <td>206266_SA</td>\n",
       "      <td>RC-DD03WP003</td>\n",
       "      <td>148.3</td>\n",
       "      <td>07/04/2003</td>\n",
       "      <td>NaN</td>\n",
       "      <td>928363_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>96.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>Pb</td>\n",
       "      <td>11500.0</td>\n",
       "      <td>ppm</td>\n",
       "      <td>ICP-OES</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2077821</th>\n",
       "      <td>206266_SA</td>\n",
       "      <td>RC-DD03WP003</td>\n",
       "      <td>148.3</td>\n",
       "      <td>07/04/2003</td>\n",
       "      <td>NaN</td>\n",
       "      <td>928363_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>96.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>Pb</td>\n",
       "      <td>11.5</td>\n",
       "      <td>ppb</td>\n",
       "      <td>ICP-MS</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         collar_id   collar_name  max_depth date_drilled reference  sample_id  \\\n",
       "1334201  202280_SA  RC-DD03SH001      462.5   27/04/2003       NaN  929299_SA   \n",
       "1334218  202280_SA  RC-DD03SH001      462.5   27/04/2003       NaN  929299_SA   \n",
       "2077804  206266_SA  RC-DD03WP003      148.3   07/04/2003       NaN  928363_SA   \n",
       "2077821  206266_SA  RC-DD03WP003      148.3   07/04/2003       NaN  928363_SA   \n",
       "\n",
       "          sample_source   from     to ele   result old_unit   method  isUDL  \\\n",
       "1334201  Drill cuttings  104.0  110.0  Pb      4.5      ppb  ICP-OES      0   \n",
       "1334218  Drill cuttings  104.0  110.0  Pb   4500.0      ppm   ICP-MS      0   \n",
       "2077804  Drill cuttings   96.0   99.0  Pb  11500.0      ppm  ICP-OES      0   \n",
       "2077821  Drill cuttings   96.0   99.0  Pb     11.5      ppb   ICP-MS      0   \n",
       "\n",
       "         fixed_result  maxminratio  \n",
       "1334201           NaN       1000.0  \n",
       "1334218           NaN       1000.0  \n",
       "2077804           NaN       1000.0  \n",
       "2077821           NaN       1000.0  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp[(tmp['fixed_result'].isnull())& (tmp['maxminratio'] == 1000)\n",
    "   ].sort_values('maxminratio', ascending=False).iloc[0:500,0:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only two assays show up, and it is apparent that the two samples have the unit conversion error between ppm/ppb.\n",
    "\n",
    "\n",
    "What about a ratio of 100?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>collar_id</th>\n",
       "      <th>collar_name</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>date_drilled</th>\n",
       "      <th>reference</th>\n",
       "      <th>sample_id</th>\n",
       "      <th>sample_source</th>\n",
       "      <th>from</th>\n",
       "      <th>to</th>\n",
       "      <th>ele</th>\n",
       "      <th>result</th>\n",
       "      <th>old_unit</th>\n",
       "      <th>method</th>\n",
       "      <th>isUDL</th>\n",
       "      <th>fixed_result</th>\n",
       "      <th>maxminratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>865343</th>\n",
       "      <td>181510_SA</td>\n",
       "      <td>96ETAR 154</td>\n",
       "      <td>44.0</td>\n",
       "      <td>10/11/1996</td>\n",
       "      <td>ENV 09229</td>\n",
       "      <td>440023_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>12.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>Ag</td>\n",
       "      <td>0.100</td>\n",
       "      <td>ppm</td>\n",
       "      <td>IC3M</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>865393</th>\n",
       "      <td>181510_SA</td>\n",
       "      <td>96ETAR 154</td>\n",
       "      <td>44.0</td>\n",
       "      <td>10/11/1996</td>\n",
       "      <td>ENV 09229</td>\n",
       "      <td>440023_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>12.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>Ag</td>\n",
       "      <td>0.001</td>\n",
       "      <td>ppb</td>\n",
       "      <td>BLEG1C</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>876287</th>\n",
       "      <td>181532_SA</td>\n",
       "      <td>96ETAR 319</td>\n",
       "      <td>29.0</td>\n",
       "      <td>25/11/1996</td>\n",
       "      <td>ENV 09229</td>\n",
       "      <td>439997_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>Ag</td>\n",
       "      <td>0.100</td>\n",
       "      <td>ppm</td>\n",
       "      <td>IC3M</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>876337</th>\n",
       "      <td>181532_SA</td>\n",
       "      <td>96ETAR 319</td>\n",
       "      <td>29.0</td>\n",
       "      <td>25/11/1996</td>\n",
       "      <td>ENV 09229</td>\n",
       "      <td>439997_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>Ag</td>\n",
       "      <td>0.001</td>\n",
       "      <td>ppb</td>\n",
       "      <td>BLEG1C</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2598708</th>\n",
       "      <td>214457_SA</td>\n",
       "      <td>KKRC070</td>\n",
       "      <td>208.0</td>\n",
       "      <td>04/10/2005</td>\n",
       "      <td>ENV 11109</td>\n",
       "      <td>1041266_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>34.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>Mo</td>\n",
       "      <td>0.100</td>\n",
       "      <td>ppm</td>\n",
       "      <td>ARM3</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2598714</th>\n",
       "      <td>214457_SA</td>\n",
       "      <td>KKRC070</td>\n",
       "      <td>208.0</td>\n",
       "      <td>04/10/2005</td>\n",
       "      <td>ENV 11109</td>\n",
       "      <td>1041266_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>34.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>Mo</td>\n",
       "      <td>10.000</td>\n",
       "      <td>ppm</td>\n",
       "      <td>AA1R</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3595430</th>\n",
       "      <td>229024_SA</td>\n",
       "      <td>KKRC101</td>\n",
       "      <td>147.0</td>\n",
       "      <td>14/05/2006</td>\n",
       "      <td>ENV 11335</td>\n",
       "      <td>1485262_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>39.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>Mo</td>\n",
       "      <td>0.100</td>\n",
       "      <td>ppm</td>\n",
       "      <td>ARM3</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3595436</th>\n",
       "      <td>229024_SA</td>\n",
       "      <td>KKRC101</td>\n",
       "      <td>147.0</td>\n",
       "      <td>14/05/2006</td>\n",
       "      <td>ENV 11335</td>\n",
       "      <td>1485262_SA</td>\n",
       "      <td>Drill cuttings</td>\n",
       "      <td>39.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>Mo</td>\n",
       "      <td>10.000</td>\n",
       "      <td>ppm</td>\n",
       "      <td>AA1R</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9606773</th>\n",
       "      <td>284741_SA</td>\n",
       "      <td>WRD53</td>\n",
       "      <td>1009.0</td>\n",
       "      <td>03/04/2012</td>\n",
       "      <td>ENV 09288</td>\n",
       "      <td>2124844_SA</td>\n",
       "      <td>Drill core</td>\n",
       "      <td>685.0</td>\n",
       "      <td>686.0</td>\n",
       "      <td>Cu</td>\n",
       "      <td>200.000</td>\n",
       "      <td>%</td>\n",
       "      <td>IC1R</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9606822</th>\n",
       "      <td>284741_SA</td>\n",
       "      <td>WRD53</td>\n",
       "      <td>1009.0</td>\n",
       "      <td>03/04/2012</td>\n",
       "      <td>ENV 09288</td>\n",
       "      <td>2124844_SA</td>\n",
       "      <td>Drill core</td>\n",
       "      <td>685.0</td>\n",
       "      <td>686.0</td>\n",
       "      <td>Cu</td>\n",
       "      <td>2.000</td>\n",
       "      <td>ppm</td>\n",
       "      <td>ME-4ACD81</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         collar_id collar_name  max_depth date_drilled  reference   sample_id  \\\n",
       "865343   181510_SA  96ETAR 154       44.0   10/11/1996  ENV 09229   440023_SA   \n",
       "865393   181510_SA  96ETAR 154       44.0   10/11/1996  ENV 09229   440023_SA   \n",
       "876287   181532_SA  96ETAR 319       29.0   25/11/1996  ENV 09229   439997_SA   \n",
       "876337   181532_SA  96ETAR 319       29.0   25/11/1996  ENV 09229   439997_SA   \n",
       "2598708  214457_SA     KKRC070      208.0   04/10/2005  ENV 11109  1041266_SA   \n",
       "2598714  214457_SA     KKRC070      208.0   04/10/2005  ENV 11109  1041266_SA   \n",
       "3595430  229024_SA     KKRC101      147.0   14/05/2006  ENV 11335  1485262_SA   \n",
       "3595436  229024_SA     KKRC101      147.0   14/05/2006  ENV 11335  1485262_SA   \n",
       "9606773  284741_SA       WRD53     1009.0   03/04/2012  ENV 09288  2124844_SA   \n",
       "9606822  284741_SA       WRD53     1009.0   03/04/2012  ENV 09288  2124844_SA   \n",
       "\n",
       "          sample_source   from     to ele   result old_unit     method  isUDL  \\\n",
       "865343   Drill cuttings   12.0   13.0  Ag    0.100      ppm       IC3M      0   \n",
       "865393   Drill cuttings   12.0   13.0  Ag    0.001      ppb     BLEG1C      0   \n",
       "876287   Drill cuttings    7.0    8.0  Ag    0.100      ppm       IC3M      0   \n",
       "876337   Drill cuttings    7.0    8.0  Ag    0.001      ppb     BLEG1C      0   \n",
       "2598708  Drill cuttings   34.0   36.0  Mo    0.100      ppm       ARM3      0   \n",
       "2598714  Drill cuttings   34.0   36.0  Mo   10.000      ppm       AA1R      0   \n",
       "3595430  Drill cuttings   39.0   42.0  Mo    0.100      ppm       ARM3      0   \n",
       "3595436  Drill cuttings   39.0   42.0  Mo   10.000      ppm       AA1R      0   \n",
       "9606773      Drill core  685.0  686.0  Cu  200.000        %       IC1R      0   \n",
       "9606822      Drill core  685.0  686.0  Cu    2.000      ppm  ME-4ACD81      0   \n",
       "\n",
       "         fixed_result  maxminratio  \n",
       "865343            NaN        100.0  \n",
       "865393            NaN        100.0  \n",
       "876287            NaN        100.0  \n",
       "876337            NaN        100.0  \n",
       "2598708           NaN        100.0  \n",
       "2598714           NaN        100.0  \n",
       "3595430           NaN        100.0  \n",
       "3595436           NaN        100.0  \n",
       "9606773           NaN        100.0  \n",
       "9606822           NaN        100.0  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp[(tmp['fixed_result'].isnull())& (tmp['maxminratio'] == 100)\n",
    "   ].sort_values('maxminratio', ascending=False).iloc[0:500,0:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample for WRD53 (Belonging to BHPs Wirrda Well deposit) is interesting - no other sample from the drilling program has a lab method of ME-4ACD81 and it cannot be found in the reference documents. This is also a copper assay with completely reasonable assay levels, so without using this 'variance within sample' metric, this discrepancy would probably stay hidden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Mark all the elements associated with that sample as wrong\n",
    "tmp.loc[(tmp['method'] == 'ME-4ACD81') &\n",
    "    (tmp['result'] > 0) &\n",
    "    (tmp['collar_id'] == '284741_SA'), 'fixed_result'] = -777\n",
    "\n",
    "# Rest are all good\n",
    "tmp.loc[(tmp['method'] != 'ME-4ACD81') &\n",
    "    (tmp['result'] > 0) &\n",
    "    (tmp['collar_id'] == '284741_SA'), 'fixed_result'] = tmp.loc[(tmp['method'] != 'ME-4ACD81') &\n",
    "                                            (tmp['result'] > 0) &\n",
    "                                            (tmp['collar_id'] == '284741_SA'), 'result']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One downside of this variance within sample metric is that it detects a lot of samples where a follow up assay using a different assay technique has been used. Usually this different technique is done to measure some aspect of ore recovery so it can be quite different to the first assay - leading to high variance within the sample for an element."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Element Means for Drill Hole\n",
    "\n",
    "\n",
    "We noticed that column swapping and misalignment can be a potential issue - So far I've stumbled upon them because the column swap causes something like Gold or Platinum, elements that tradionally have very low values, to have exceptionally high values.\n",
    "\n",
    "How can we attempt to target these data entry errors? If columns have swapped or become misaligned, it may mean that one drill hole will have higher than normal values for one element, and another one will have lower than normal values.\n",
    "\n",
    "To be able to compare different elements, the following is done.\n",
    "\n",
    "- Take the median value for that element from the entire database\n",
    "- Take the median value for that element for a drillhole\n",
    "- Divide the maximum of those two values by the minimum of those two values\n",
    "    - Why? We dont care if an element is smaller or larger than its typical values, we just want to know that its different\n",
    "    - Downsides to dividing? Elements where the median is 0 (very low trace elements) will divide by zero\n",
    " \n",
    " \n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the median value for each element\n",
    "ele_med = tmp.groupby('ele')['result'].median().reset_index()\n",
    "ele_med.columns = ['ele', 'ele_medians']\n",
    "\n",
    "# Get the median value for each element in each drillhole\n",
    "dh_ele_med = tmp.groupby(['collar_id', 'ele']).agg({'result': [len, np.nanmedian]}).reset_index()\n",
    "dh_ele_med.columns = ['collar_id', 'ele', 'sample_counts','dh_ele_med']\n",
    "\n",
    "# Merge the two tables\n",
    "dh_ele_med = dh_ele_med.merge(ele_med)\n",
    "\n",
    "# Calculate the ratio of the two values\n",
    "dh_ele_med['median_ratio'] = np.where(dh_ele_med['dh_ele_med'] > dh_ele_med['ele_medians'],\n",
    "                                      dh_ele_med['dh_ele_med'] / dh_ele_med['ele_medians'],\n",
    "                                      dh_ele_med['ele_medians'] /dh_ele_med['dh_ele_med'])\n",
    "\n",
    "# Filter the table\n",
    "# Drop the rows where the ratio is 0 of infinite\n",
    "# Drop the rows where the drillhole has less than 20 samples\n",
    "# Drop the rows where we have already detected errors (we want to find new errors!)\n",
    "dh_ele_med = dh_ele_med[(dh_ele_med['sample_counts'] > 20) &\n",
    "                       ~(dh_ele_med['median_ratio'].isnull()) & (dh_ele_med['median_ratio'] > 0) &\n",
    "                       (dh_ele_med['median_ratio'] != np.inf) &\n",
    "                       ~(dh_ele_med['collar_id'].isin(tmp[~tmp['fixed_result'].isnull()]['collar_id'].unique()))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to simply pick a threshold for the ratio, and count how many elements exceed the ratio for each drillhole.\n",
    "\n",
    "A lot of the errors that show up with this method are data releases where the typical % assays (Mg,Fe,Ca, Na, K, Ti) are off by a factor of 10000 - one of these is shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>collar_id</th>\n",
       "      <th>ele</th>\n",
       "      <th>sample_counts</th>\n",
       "      <th>dh_ele_med</th>\n",
       "      <th>ele_medians</th>\n",
       "      <th>median_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>33294</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Ag</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.290</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>1.450000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84537</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Au</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.002</td>\n",
       "      <td>0.005000</td>\n",
       "      <td>2.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137565</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Cu</td>\n",
       "      <td>25.0</td>\n",
       "      <td>783.000</td>\n",
       "      <td>39.000000</td>\n",
       "      <td>20.076923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185413</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Pb</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.500</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>2.857143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>234239</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Zn</td>\n",
       "      <td>25.0</td>\n",
       "      <td>195.000</td>\n",
       "      <td>40.000000</td>\n",
       "      <td>4.875000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>252451</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Ca</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1.790</td>\n",
       "      <td>7457.000000</td>\n",
       "      <td>4165.921788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>268679</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>K</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.250</td>\n",
       "      <td>15440.810227</td>\n",
       "      <td>61763.240910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>285977</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Mg</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.430</td>\n",
       "      <td>9200.000000</td>\n",
       "      <td>1430.793157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300017</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Na</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.650</td>\n",
       "      <td>2522.255193</td>\n",
       "      <td>3880.392604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>325353</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Co</td>\n",
       "      <td>25.0</td>\n",
       "      <td>90.200</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>5.011111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>349070</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Mo</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1.610</td>\n",
       "      <td>1.100000</td>\n",
       "      <td>1.463636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>379341</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Ni</td>\n",
       "      <td>25.0</td>\n",
       "      <td>156.000</td>\n",
       "      <td>18.400000</td>\n",
       "      <td>8.478261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388931</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Sn</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1.100</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.818182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>397904</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>W</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.500</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>437830</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>As</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.300</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>16.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>450190</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Ba</td>\n",
       "      <td>25.0</td>\n",
       "      <td>150.000</td>\n",
       "      <td>390.000000</td>\n",
       "      <td>2.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>461637</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Ce</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.250</td>\n",
       "      <td>86.900000</td>\n",
       "      <td>6.098246</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>476404</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Cr</td>\n",
       "      <td>25.0</td>\n",
       "      <td>93.000</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>2.657143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504051</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Fe</td>\n",
       "      <td>25.0</td>\n",
       "      <td>134500.000</td>\n",
       "      <td>55500.000000</td>\n",
       "      <td>2.423423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>516729</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>La</td>\n",
       "      <td>25.0</td>\n",
       "      <td>5.700</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>7.543860</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>543491</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Mn</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1970.000</td>\n",
       "      <td>500.000000</td>\n",
       "      <td>3.940000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>558290</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Th</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.200</td>\n",
       "      <td>13.900000</td>\n",
       "      <td>69.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>584261</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Nb</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.200</td>\n",
       "      <td>8.500000</td>\n",
       "      <td>2.656250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>588733</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Ta</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.270</td>\n",
       "      <td>0.640000</td>\n",
       "      <td>2.370370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>593473</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Be</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.440</td>\n",
       "      <td>3.200000</td>\n",
       "      <td>7.272727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>626456</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>P</td>\n",
       "      <td>25.0</td>\n",
       "      <td>260.000</td>\n",
       "      <td>589.108047</td>\n",
       "      <td>2.265800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>652693</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>V</td>\n",
       "      <td>25.0</td>\n",
       "      <td>243.000</td>\n",
       "      <td>56.016133</td>\n",
       "      <td>4.338036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>661226</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Y</td>\n",
       "      <td>25.0</td>\n",
       "      <td>20.600</td>\n",
       "      <td>27.141000</td>\n",
       "      <td>1.317524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>676475</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Al</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.650</td>\n",
       "      <td>57422.598571</td>\n",
       "      <td>15732.218787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>693387</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Bi</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.050</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>20.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>706825</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Cd</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.220</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>1.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>718975</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Rb</td>\n",
       "      <td>25.0</td>\n",
       "      <td>2.200</td>\n",
       "      <td>68.100000</td>\n",
       "      <td>30.954545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>748774</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Sr</td>\n",
       "      <td>25.0</td>\n",
       "      <td>91.100</td>\n",
       "      <td>48.000000</td>\n",
       "      <td>1.897917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>762012</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Ti</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.597</td>\n",
       "      <td>2472.000000</td>\n",
       "      <td>4140.703518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>768897</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Zr</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.400</td>\n",
       "      <td>134.000000</td>\n",
       "      <td>9.305556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>774278</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Ga</td>\n",
       "      <td>25.0</td>\n",
       "      <td>15.650</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>1.277955</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>776055</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Ge</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.200</td>\n",
       "      <td>0.220000</td>\n",
       "      <td>1.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>779599</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Hf</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.600</td>\n",
       "      <td>3.400000</td>\n",
       "      <td>5.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>782784</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>In</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.125</td>\n",
       "      <td>0.050000</td>\n",
       "      <td>2.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>785799</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Li</td>\n",
       "      <td>25.0</td>\n",
       "      <td>8.500</td>\n",
       "      <td>15.500000</td>\n",
       "      <td>1.823529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>816039</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>S</td>\n",
       "      <td>25.0</td>\n",
       "      <td>40500.000</td>\n",
       "      <td>500.000000</td>\n",
       "      <td>81.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>822199</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Sc</td>\n",
       "      <td>25.0</td>\n",
       "      <td>39.600</td>\n",
       "      <td>9.600000</td>\n",
       "      <td>4.125000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>843074</th>\n",
       "      <td>325470_SA</td>\n",
       "      <td>Tl</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.020</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>250.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        collar_id ele  sample_counts  dh_ele_med   ele_medians  median_ratio\n",
       "33294   325470_SA  Ag           25.0       0.290      0.200000      1.450000\n",
       "84537   325470_SA  Au           25.0       0.002      0.005000      2.500000\n",
       "137565  325470_SA  Cu           25.0     783.000     39.000000     20.076923\n",
       "185413  325470_SA  Pb           25.0       3.500     10.000000      2.857143\n",
       "234239  325470_SA  Zn           25.0     195.000     40.000000      4.875000\n",
       "252451  325470_SA  Ca           25.0       1.790   7457.000000   4165.921788\n",
       "268679  325470_SA   K           25.0       0.250  15440.810227  61763.240910\n",
       "285977  325470_SA  Mg           25.0       6.430   9200.000000   1430.793157\n",
       "300017  325470_SA  Na           25.0       0.650   2522.255193   3880.392604\n",
       "325353  325470_SA  Co           25.0      90.200     18.000000      5.011111\n",
       "349070  325470_SA  Mo           25.0       1.610      1.100000      1.463636\n",
       "379341  325470_SA  Ni           25.0     156.000     18.400000      8.478261\n",
       "388931  325470_SA  Sn           25.0       1.100      2.000000      1.818182\n",
       "397904  325470_SA   W           25.0       0.500      5.000000     10.000000\n",
       "437830  325470_SA  As           25.0       0.300      5.000000     16.666667\n",
       "450190  325470_SA  Ba           25.0     150.000    390.000000      2.600000\n",
       "461637  325470_SA  Ce           25.0      14.250     86.900000      6.098246\n",
       "476404  325470_SA  Cr           25.0      93.000     35.000000      2.657143\n",
       "504051  325470_SA  Fe           25.0  134500.000  55500.000000      2.423423\n",
       "516729  325470_SA  La           25.0       5.700     43.000000      7.543860\n",
       "543491  325470_SA  Mn           25.0    1970.000    500.000000      3.940000\n",
       "558290  325470_SA  Th           25.0       0.200     13.900000     69.500000\n",
       "584261  325470_SA  Nb           25.0       3.200      8.500000      2.656250\n",
       "588733  325470_SA  Ta           25.0       0.270      0.640000      2.370370\n",
       "593473  325470_SA  Be           25.0       0.440      3.200000      7.272727\n",
       "626456  325470_SA   P           25.0     260.000    589.108047      2.265800\n",
       "652693  325470_SA   V           25.0     243.000     56.016133      4.338036\n",
       "661226  325470_SA   Y           25.0      20.600     27.141000      1.317524\n",
       "676475  325470_SA  Al           25.0       3.650  57422.598571  15732.218787\n",
       "693387  325470_SA  Bi           25.0       0.050      1.000000     20.000000\n",
       "706825  325470_SA  Cd           25.0       0.220      0.200000      1.100000\n",
       "718975  325470_SA  Rb           25.0       2.200     68.100000     30.954545\n",
       "748774  325470_SA  Sr           25.0      91.100     48.000000      1.897917\n",
       "762012  325470_SA  Ti           25.0       0.597   2472.000000   4140.703518\n",
       "768897  325470_SA  Zr           25.0      14.400    134.000000      9.305556\n",
       "774278  325470_SA  Ga           25.0      15.650     20.000000      1.277955\n",
       "776055  325470_SA  Ge           25.0       0.200      0.220000      1.100000\n",
       "779599  325470_SA  Hf           25.0       0.600      3.400000      5.666667\n",
       "782784  325470_SA  In           25.0       0.125      0.050000      2.500000\n",
       "785799  325470_SA  Li           25.0       8.500     15.500000      1.823529\n",
       "816039  325470_SA   S           25.0   40500.000    500.000000     81.000000\n",
       "822199  325470_SA  Sc           25.0      39.600      9.600000      4.125000\n",
       "843074  325470_SA  Tl           25.0       0.020      5.000000    250.000000"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dh_ele_med[dh_ele_med['collar_id'] == '325470_SA']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So for the elements with high difference of their median to the typical element median, are they legit outliers or data entry errors?\n",
    "\n",
    "- S is correct, Th is correct\n",
    "- Ti, Al, Na, Mg, K, Ca are off by 10000 (source says ppm but it has to be %, based on the MS61 assay specs from ALS saying that minimum ppm is 100, backed up by source documents below)\n",
    "\n",
    "![nicui](img/nicui.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Fix the percentage errors\n",
    "tmp.loc[(tmp['collar_id'] == '325470_SA') &\n",
    "    (tmp['ele'].isin(['Ti', 'Al', 'Na', 'Mg', 'K', 'Ca'])) &\n",
    "    (tmp['result'] > 0 ), 'fixed_result'] = tmp.loc[(tmp['collar_id'] == '325470_SA') &\n",
    "                                (tmp['ele'].isin(['Ti', 'Al', 'Na', 'Mg', 'K', 'Ca'])) &\n",
    "                                (tmp['result'] > 0 ), 'fixed_result'] * 10000\n",
    "\n",
    "\n",
    "# Carry over the rest\n",
    "tmp.loc[(tmp['collar_id'] == '325470_SA') &\n",
    "    ~(tmp['ele'].isin(['Ti', 'Al', 'Na', 'Mg', 'K', 'Ca'])) &\n",
    "    (tmp['result'] > 0 ), 'fixed_result'] = tmp.loc[(tmp['collar_id'] == '325470_SA') &\n",
    "                               ~ (tmp['ele'].isin(['Ti', 'Al', 'Na', 'Mg', 'K', 'Ca'])) &\n",
    "                                (tmp['result'] > 0 ), 'fixed_result']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method is really good for identifying the errors with percentage <-> ppm <-> ppb conversion errors, but the goal is to hunt for column misalignment errors.\n",
    "\n",
    "To do this, I excluded the typical elements that have the %/ppm conversion errors, and found this drill hole that had 4 elements with a median ratio over 400."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>collar_id</th>\n",
       "      <th>ele</th>\n",
       "      <th>sample_counts</th>\n",
       "      <th>dh_ele_med</th>\n",
       "      <th>ele_medians</th>\n",
       "      <th>median_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12215</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Ag</td>\n",
       "      <td>46.0</td>\n",
       "      <td>0.350000</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>1.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106872</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Cu</td>\n",
       "      <td>46.0</td>\n",
       "      <td>91.900000</td>\n",
       "      <td>39.000000</td>\n",
       "      <td>2.356410</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>156853</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Pb</td>\n",
       "      <td>46.0</td>\n",
       "      <td>35.100000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>3.510000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240928</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Ca</td>\n",
       "      <td>46.0</td>\n",
       "      <td>10640.681004</td>\n",
       "      <td>7457.000000</td>\n",
       "      <td>1.426939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>257185</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>K</td>\n",
       "      <td>46.0</td>\n",
       "      <td>21874.481156</td>\n",
       "      <td>15440.810227</td>\n",
       "      <td>1.416667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>273657</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Mg</td>\n",
       "      <td>46.0</td>\n",
       "      <td>20715.233386</td>\n",
       "      <td>9200.000000</td>\n",
       "      <td>2.251656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290309</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Na</td>\n",
       "      <td>46.0</td>\n",
       "      <td>12870.919881</td>\n",
       "      <td>2522.255193</td>\n",
       "      <td>5.102941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>310393</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Co</td>\n",
       "      <td>46.0</td>\n",
       "      <td>29.850000</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>1.658333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>334390</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Mo</td>\n",
       "      <td>46.0</td>\n",
       "      <td>3.300000</td>\n",
       "      <td>1.100000</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>359926</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Ni</td>\n",
       "      <td>92.0</td>\n",
       "      <td>65.000000</td>\n",
       "      <td>18.400000</td>\n",
       "      <td>3.532609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>383098</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Sn</td>\n",
       "      <td>46.0</td>\n",
       "      <td>99.670000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>49.835000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>392455</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>W</td>\n",
       "      <td>46.0</td>\n",
       "      <td>2.900000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>1.724138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>408187</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>As</td>\n",
       "      <td>46.0</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>2.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>443465</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Ba</td>\n",
       "      <td>46.0</td>\n",
       "      <td>287.500000</td>\n",
       "      <td>390.000000</td>\n",
       "      <td>1.356522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>453706</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Ce</td>\n",
       "      <td>46.0</td>\n",
       "      <td>56.400000</td>\n",
       "      <td>86.900000</td>\n",
       "      <td>1.540780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>468614</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Cr</td>\n",
       "      <td>46.0</td>\n",
       "      <td>105.124836</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>3.003567</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>486016</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Fe</td>\n",
       "      <td>46.0</td>\n",
       "      <td>81030.985521</td>\n",
       "      <td>55500.000000</td>\n",
       "      <td>1.460018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>508512</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>La</td>\n",
       "      <td>46.0</td>\n",
       "      <td>30.000000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>1.433333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>526189</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Mn</td>\n",
       "      <td>46.0</td>\n",
       "      <td>813.197026</td>\n",
       "      <td>500.000000</td>\n",
       "      <td>1.626394</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>548304</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Th</td>\n",
       "      <td>46.0</td>\n",
       "      <td>0.700000</td>\n",
       "      <td>13.900000</td>\n",
       "      <td>19.857143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>563089</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>U</td>\n",
       "      <td>46.0</td>\n",
       "      <td>2.350000</td>\n",
       "      <td>8.700000</td>\n",
       "      <td>3.702128</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>579478</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Nb</td>\n",
       "      <td>46.0</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>8.500000</td>\n",
       "      <td>1.411765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>585754</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Ta</td>\n",
       "      <td>46.0</td>\n",
       "      <td>7.850000</td>\n",
       "      <td>0.640000</td>\n",
       "      <td>12.265625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>594610</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Dy</td>\n",
       "      <td>46.0</td>\n",
       "      <td>5.025000</td>\n",
       "      <td>4.250000</td>\n",
       "      <td>1.182353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>597848</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Er</td>\n",
       "      <td>46.0</td>\n",
       "      <td>2.830000</td>\n",
       "      <td>2.380000</td>\n",
       "      <td>1.189076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>601163</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Eu</td>\n",
       "      <td>46.0</td>\n",
       "      <td>1.265000</td>\n",
       "      <td>1.300000</td>\n",
       "      <td>1.027668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>604459</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>46.0</td>\n",
       "      <td>5.110000</td>\n",
       "      <td>5.080000</td>\n",
       "      <td>1.005906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>607690</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Ho</td>\n",
       "      <td>46.0</td>\n",
       "      <td>0.975000</td>\n",
       "      <td>0.840000</td>\n",
       "      <td>1.160714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>611028</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Nd</td>\n",
       "      <td>46.0</td>\n",
       "      <td>0.029300</td>\n",
       "      <td>33.400000</td>\n",
       "      <td>1139.931741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>617803</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>P</td>\n",
       "      <td>46.0</td>\n",
       "      <td>567.289230</td>\n",
       "      <td>589.108047</td>\n",
       "      <td>1.038462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>628423</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Pr</td>\n",
       "      <td>46.0</td>\n",
       "      <td>83.750000</td>\n",
       "      <td>9.340000</td>\n",
       "      <td>8.966809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>631655</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Sm</td>\n",
       "      <td>46.0</td>\n",
       "      <td>80.600000</td>\n",
       "      <td>6.190000</td>\n",
       "      <td>13.021002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>634898</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Tb</td>\n",
       "      <td>46.0</td>\n",
       "      <td>0.955000</td>\n",
       "      <td>0.730000</td>\n",
       "      <td>1.308219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>638100</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Tm</td>\n",
       "      <td>46.0</td>\n",
       "      <td>217.000000</td>\n",
       "      <td>0.340000</td>\n",
       "      <td>638.235294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>643982</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>V</td>\n",
       "      <td>46.0</td>\n",
       "      <td>31.400000</td>\n",
       "      <td>56.016133</td>\n",
       "      <td>1.783953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>655565</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Y</td>\n",
       "      <td>46.0</td>\n",
       "      <td>302.500000</td>\n",
       "      <td>27.141000</td>\n",
       "      <td>11.145499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>662618</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Yb</td>\n",
       "      <td>46.0</td>\n",
       "      <td>137.200000</td>\n",
       "      <td>2.230000</td>\n",
       "      <td>61.524664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>667619</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Al</td>\n",
       "      <td>46.0</td>\n",
       "      <td>69727.441122</td>\n",
       "      <td>57422.598571</td>\n",
       "      <td>1.214286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>682021</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Bi</td>\n",
       "      <td>46.0</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>698397</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Cd</td>\n",
       "      <td>46.0</td>\n",
       "      <td>0.700000</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>3.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>708491</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Cs</td>\n",
       "      <td>46.0</td>\n",
       "      <td>2.150000</td>\n",
       "      <td>1.810000</td>\n",
       "      <td>1.187845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>714269</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Rb</td>\n",
       "      <td>46.0</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>68.100000</td>\n",
       "      <td>681.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>723161</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Sb</td>\n",
       "      <td>46.0</td>\n",
       "      <td>57.785000</td>\n",
       "      <td>2.500000</td>\n",
       "      <td>23.114000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>734208</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Si</td>\n",
       "      <td>46.0</td>\n",
       "      <td>9349.289454</td>\n",
       "      <td>243081.525804</td>\n",
       "      <td>26.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>741157</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Sr</td>\n",
       "      <td>46.0</td>\n",
       "      <td>0.900000</td>\n",
       "      <td>48.000000</td>\n",
       "      <td>53.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>753284</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Ti</td>\n",
       "      <td>46.0</td>\n",
       "      <td>2487.860440</td>\n",
       "      <td>2472.000000</td>\n",
       "      <td>1.006416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>770333</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Ga</td>\n",
       "      <td>46.0</td>\n",
       "      <td>19.950000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>1.002506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>776808</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Hf</td>\n",
       "      <td>46.0</td>\n",
       "      <td>4.050000</td>\n",
       "      <td>3.400000</td>\n",
       "      <td>1.191176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786808</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Lu</td>\n",
       "      <td>46.0</td>\n",
       "      <td>0.435000</td>\n",
       "      <td>0.330000</td>\n",
       "      <td>1.318182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>803833</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>S</td>\n",
       "      <td>46.0</td>\n",
       "      <td>240000.000000</td>\n",
       "      <td>500.000000</td>\n",
       "      <td>480.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>817666</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Sc</td>\n",
       "      <td>46.0</td>\n",
       "      <td>5.750000</td>\n",
       "      <td>9.600000</td>\n",
       "      <td>1.669565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>837279</th>\n",
       "      <td>206200_SA</td>\n",
       "      <td>Tl</td>\n",
       "      <td>46.0</td>\n",
       "      <td>5.750000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>1.150000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        collar_id ele  sample_counts     dh_ele_med    ele_medians  \\\n",
       "12215   206200_SA  Ag           46.0       0.350000       0.200000   \n",
       "106872  206200_SA  Cu           46.0      91.900000      39.000000   \n",
       "156853  206200_SA  Pb           46.0      35.100000      10.000000   \n",
       "240928  206200_SA  Ca           46.0   10640.681004    7457.000000   \n",
       "257185  206200_SA   K           46.0   21874.481156   15440.810227   \n",
       "273657  206200_SA  Mg           46.0   20715.233386    9200.000000   \n",
       "290309  206200_SA  Na           46.0   12870.919881    2522.255193   \n",
       "310393  206200_SA  Co           46.0      29.850000      18.000000   \n",
       "334390  206200_SA  Mo           46.0       3.300000       1.100000   \n",
       "359926  206200_SA  Ni           92.0      65.000000      18.400000   \n",
       "383098  206200_SA  Sn           46.0      99.670000       2.000000   \n",
       "392455  206200_SA   W           46.0       2.900000       5.000000   \n",
       "408187  206200_SA  As           46.0       2.000000       5.000000   \n",
       "443465  206200_SA  Ba           46.0     287.500000     390.000000   \n",
       "453706  206200_SA  Ce           46.0      56.400000      86.900000   \n",
       "468614  206200_SA  Cr           46.0     105.124836      35.000000   \n",
       "486016  206200_SA  Fe           46.0   81030.985521   55500.000000   \n",
       "508512  206200_SA  La           46.0      30.000000      43.000000   \n",
       "526189  206200_SA  Mn           46.0     813.197026     500.000000   \n",
       "548304  206200_SA  Th           46.0       0.700000      13.900000   \n",
       "563089  206200_SA   U           46.0       2.350000       8.700000   \n",
       "579478  206200_SA  Nb           46.0      12.000000       8.500000   \n",
       "585754  206200_SA  Ta           46.0       7.850000       0.640000   \n",
       "594610  206200_SA  Dy           46.0       5.025000       4.250000   \n",
       "597848  206200_SA  Er           46.0       2.830000       2.380000   \n",
       "601163  206200_SA  Eu           46.0       1.265000       1.300000   \n",
       "604459  206200_SA  Gd           46.0       5.110000       5.080000   \n",
       "607690  206200_SA  Ho           46.0       0.975000       0.840000   \n",
       "611028  206200_SA  Nd           46.0       0.029300      33.400000   \n",
       "617803  206200_SA   P           46.0     567.289230     589.108047   \n",
       "628423  206200_SA  Pr           46.0      83.750000       9.340000   \n",
       "631655  206200_SA  Sm           46.0      80.600000       6.190000   \n",
       "634898  206200_SA  Tb           46.0       0.955000       0.730000   \n",
       "638100  206200_SA  Tm           46.0     217.000000       0.340000   \n",
       "643982  206200_SA   V           46.0      31.400000      56.016133   \n",
       "655565  206200_SA   Y           46.0     302.500000      27.141000   \n",
       "662618  206200_SA  Yb           46.0     137.200000       2.230000   \n",
       "667619  206200_SA  Al           46.0   69727.441122   57422.598571   \n",
       "682021  206200_SA  Bi           46.0       0.300000       1.000000   \n",
       "698397  206200_SA  Cd           46.0       0.700000       0.200000   \n",
       "708491  206200_SA  Cs           46.0       2.150000       1.810000   \n",
       "714269  206200_SA  Rb           46.0       0.100000      68.100000   \n",
       "723161  206200_SA  Sb           46.0      57.785000       2.500000   \n",
       "734208  206200_SA  Si           46.0    9349.289454  243081.525804   \n",
       "741157  206200_SA  Sr           46.0       0.900000      48.000000   \n",
       "753284  206200_SA  Ti           46.0    2487.860440    2472.000000   \n",
       "770333  206200_SA  Ga           46.0      19.950000      20.000000   \n",
       "776808  206200_SA  Hf           46.0       4.050000       3.400000   \n",
       "786808  206200_SA  Lu           46.0       0.435000       0.330000   \n",
       "803833  206200_SA   S           46.0  240000.000000     500.000000   \n",
       "817666  206200_SA  Sc           46.0       5.750000       9.600000   \n",
       "837279  206200_SA  Tl           46.0       5.750000       5.000000   \n",
       "\n",
       "        median_ratio  \n",
       "12215       1.750000  \n",
       "106872      2.356410  \n",
       "156853      3.510000  \n",
       "240928      1.426939  \n",
       "257185      1.416667  \n",
       "273657      2.251656  \n",
       "290309      5.102941  \n",
       "310393      1.658333  \n",
       "334390      3.000000  \n",
       "359926      3.532609  \n",
       "383098     49.835000  \n",
       "392455      1.724138  \n",
       "408187      2.500000  \n",
       "443465      1.356522  \n",
       "453706      1.540780  \n",
       "468614      3.003567  \n",
       "486016      1.460018  \n",
       "508512      1.433333  \n",
       "526189      1.626394  \n",
       "548304     19.857143  \n",
       "563089      3.702128  \n",
       "579478      1.411765  \n",
       "585754     12.265625  \n",
       "594610      1.182353  \n",
       "597848      1.189076  \n",
       "601163      1.027668  \n",
       "604459      1.005906  \n",
       "607690      1.160714  \n",
       "611028   1139.931741  \n",
       "617803      1.038462  \n",
       "628423      8.966809  \n",
       "631655     13.021002  \n",
       "634898      1.308219  \n",
       "638100    638.235294  \n",
       "643982      1.783953  \n",
       "655565     11.145499  \n",
       "662618     61.524664  \n",
       "667619      1.214286  \n",
       "682021      3.333333  \n",
       "698397      3.500000  \n",
       "708491      1.187845  \n",
       "714269    681.000000  \n",
       "723161     23.114000  \n",
       "734208     26.000000  \n",
       "741157     53.333333  \n",
       "753284      1.006416  \n",
       "770333      1.002506  \n",
       "776808      1.191176  \n",
       "786808      1.318182  \n",
       "803833    480.000000  \n",
       "817666      1.669565  \n",
       "837279      1.150000  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dh_ele_med[dh_ele_med['collar_id'] == '206200_SA']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So whats up with this hole?\n",
    "\n",
    "- The S median for this drill hole is 480 times smaller than typical values\n",
    "- The Tm median is 680 times larger than typical values\n",
    "- The Nd median is 1139 times smaller than typical values\n",
    "\n",
    "Investigation shows that it belongs to the ENV09877 Data release, and the data disitrubtion (Very low Nd, Very High S, very High Tm), happens to all holes. When looking at the reference data, theres a lot going on in terms of data entry errors.\n",
    "\n",
    "![9877](img/9877.png)\n",
    "\n",
    "Nd is a unit conversion error - its off by a factor of 1000.\n",
    "\n",
    "Then for <i>some</i> samples, theres a column misalignment error. When the Pd/Pt assays are not present, the elements shift over, causing Pd to have the Pr values and Rb to have the Sb values, etc etc.\n",
    "\n",
    "So using the method of finding drillholes with a high amount of elements with drill hole medians significantly different than the typical median, we were able to identify a report with column misalignment errors, and potentially a lot faster than the method of looking at one elements one by one.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp.loc[(tmp['collar_name'].str.startswith('BLRC')) &\n",
    "        (tmp['ele'] == 'Nd'), 'fixed_result' ] = tmp.loc[(tmp['collar_name'].str.startswith('BLRC')) &\n",
    "                                                (tmp['ele'] == 'Nd'), 'result'] * 1000\n",
    "\n",
    "\n",
    "bad_eles = ['Pd','Pr','Pt','Rb','S','Sb','Sc',\n",
    "            'Si','Sm','Sn', 'Sr', 'Ta', 'Tb',\n",
    "            'Th', 'Ti', 'Tl', 'Tm', 'U',\n",
    "            'V', 'W', 'Y', 'Yb', 'Zn', 'Zr']\n",
    "\n",
    "bad_samples = tmp.loc[(tmp['collar_name'].str.startswith('BLRC')) &\n",
    "        (tmp['ele'] == 'Sn') &\n",
    "        (tmp['result'] > 80 ) &\n",
    "        (tmp['fixed_result'].isnull())]['sample_id']\n",
    "\n",
    "\n",
    "\n",
    "tmp.loc[(tmp['collar_name'].str.startswith('BLRC')) &\n",
    "        (tmp['sample_id'].isin(bad_samples)) &\n",
    "        (tmp['ele'].isin(bad_eles)) &\n",
    "        (tmp['fixed_result'].isnull())] = -777\n",
    "\n",
    "tmp.loc[(tmp['collar_name'].str.startswith('BLRC'))  &\n",
    "        (tmp['fixed_result'].isnull()), 'fixed_result'] = tmp.loc[(tmp['collar_name'].str.startswith('BLRC'))  &\n",
    "                                                            (tmp['result'].isnull()), 'result']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section Four - Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": false
   },
   "source": [
    "I spent 5 solid days looking through the geochemistry database, hunting down the mistakes. While general mistakes can be quick to spot when just sorting by the assay result values, it takes a lot of time to understand the extent of errors within a data release.\n",
    "\n",
    "How much of the geochem database have we actually checked? And how many errors have we found?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Records Checked:  613615\n",
      "Total number of assays:  25874780\n",
      "% Checked:  2.3714791004986324\n"
     ]
    }
   ],
   "source": [
    "print('Number of Records Checked: ',(~tmp['fixed_result'].isnull()).sum())\n",
    "print('Total number of assays: ', len(tmp))\n",
    "print('% Checked: ', ((~tmp['fixed_result'].isnull()).sum() / len(tmp) * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So only 2.37% of the database has been inspected for errors after 5 days of verifying records.\n",
    "\n",
    "\n",
    "### Root Causes\n",
    "\n",
    "What are the root causes of the data entry errors found so far?\n",
    "\n",
    "-<b> Unit Conversion Errors:</b> eg When the assay has its units still listed as % but its already been converted into ppm\n",
    "\n",
    "-<b> Column Misalignment:</b> Often caused by some repeat assays or a blank column within the reference document, the column headers become misaligned, causing results of the elements to be aligned to the wrong element\n",
    "    \n",
    "    \n",
    "-<b> Misspelling of Elements: </b>This was only spotted once, but occurred when the Pd assay had the last letter cut off, causing the assays to fall under the P element\n",
    "\n",
    "\n",
    "### Strategies to Detect Errors\n",
    "\n",
    "There are four methods that I've employed to hunt down errors. Each of them have their pros and cons.\n",
    "\n",
    "- Sort an Element from high to low\n",
    "    - Pros: Finds the most obvious errors\n",
    "    - Cons: Lots of false positives from high assay results that come from mine sites\n",
    "\n",
    "- Distance to Deposit/Occurrence\n",
    "    - Pros: Very efficient at finding hard to detect errors\n",
    "    - Cons: Takes a while to calculate, not applicable to every error\n",
    "\n",
    "- Sample Variance\n",
    "    - Pros: Easy to detect unit conversion errors\n",
    "    - Cons: Picks up a lot of false positives of where the sample was assayed again using a different technique\n",
    "\n",
    "- Drillhole Element Median Ratios\n",
    "    - Pros: Really good at detecting errors that have been caused by column misalignment errors\n",
    "    - Cons: Cant use it for the trace elements that have their median value as 0\n",
    "\n",
    "It is simply not feasible to check all the errors (I haven't even looked at surface sampling!), so there is always a trade off between time and effectiveness - dont waste time fixing assays for Silicon if you are working on some Gold analysis. \n",
    "\n",
    "My suggestion is that if you are only working with one element, employ the first two techniques to get rid of as many obvious errors that can completely mess up whatever machine learning/ data science approach you are using. Then start looking at Sample Variances and Drillhole Element Medians to detect more errors.\n",
    "\n",
    "\n",
    "### Save the dataset\n",
    "\n",
    "Only the assays have changed, so we keep the original sample and collar dataframes, and get the updated assay date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "assays_v2 = tmp[['sample_id', 'ele', 'result', 'old_unit', 'method', 'fixed_result', 'isUDL']].reset_index(drop=True)\n",
    "\n",
    "assays_v2.to_csv('sa_assays.csv', index=False)\n",
    "samples.to_csv('sa_samples.csv', index=False)\n",
    "locs.to_csv('sa_collars.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonus - .....machine learning?\n",
    "\n",
    "\n",
    "Originally when I started the notebook, the original aim was to use machine learning to hunt down the hard to detect errors, and to do it faster. However after 5 days of data cleaning, I still wasn't close to getting rid of the easy to detect errors (eg obvious unit conversion errors and column misalignments), so the business value of a machine learning approach over just data analysis techniques was not worth it - its just faster to manually search for these obvious errors.\n",
    "\n",
    "\n",
    "I still think a ML approach would be valuable once all the low hanging fruit is done, so how would you go about doing a machine learning approach? Well because I've been marking assays as error/not error, we now have a dataset that can be used to training a model as follows:\n",
    "\n",
    "![title](img/ml_process.png)\n",
    "\n",
    "What would be the advantages of machine learning? We can start incorporating lots of different features, which will ultimately allow us to weed out every single error. What are some of the features we could include?\n",
    "\n",
    "- Was the assay done by pXRF? - pXRF can cause some funky results due to miscalibration, but these are generally not data entry errors. So a ML model could potentially figure out that if a result for an element is strange but from XRF assay methods, its probably legit.\n",
    "- Distance to deposit: As has been established, distance to deposit is a good metric for establishing data entry errors\n",
    "- What year was the assay from? -  A lot of the old assays have assay methods that have less precision/accuracy, but are still legitimate assays\n",
    "- What unit was the element assayed? - If you have an assay of Iron or Silicon in ppb, this would be a red flag\n",
    "- What drilling method was used? - Shallow Aircore holes are generally going to have a different element distribution than deep diamond drilling\n",
    "\n",
    "So stay tuned, as I still plan to explore this eventually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks for reading!\n",
    "\n",
    "Michael Rodda\n",
    "\n",
    "\n",
    "![logo](img/logov2.png)\n",
    "<center>www.caldera.technology</center>\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}