notebooks/data-sources/TestGrid/metrics/number_of_flakes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quantify Flakes\n",
    "\n",
    "One of the key perfomance indicators that we would like to create greater visbility into and track over time is overall number and percent of flakes that occur. Individual test runs are flagged a \"flake\" if they are run mulitple times with a mix of passes and failes without any changes to the code being tested. Although they occur for individual test runs, there are a number of aggregate views that developers may want to look at to assess the overall health of thier project or testing platform. For Example:\n",
    "\n",
    "* percent flakes on platform each day\n",
    "* percent flakes by tab each week\n",
    "* percent flakes by grid each month\n",
    "* percent flakes by test overall (this can also be seen as a severity level = overall flake rate of test)\n",
    "\n",
    "In order to provide maxium flexibility for the end-user of this work, instead of creating a number of dataframes to answer each of these specifc questions, we will define a long and narrow data structure (a list of tuples saved as a csv for now) that contains only 5 columns (\"timestamp\", \"tab\",\"grid\",\"test\",\"flake\"). This allows superset (or pandas) to perform the last filter and/or aggreagtion of interest to an end user. Which is to say, there may appear to be a lot of repetion within the final dataset, but each row should be unique, and it should provide the simpelest useability for an end-user.    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T16:40:49.971080Z",
     "start_time": "2021-03-24T16:40:49.156552Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "import gzip\n",
    "import os\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import datetime\n",
    "\n",
    "\n",
    "from ipynb.fs.defs.metric_template import testgrid_labelwise_encoding\n",
    "from ipynb.fs.defs.metric_template import CephCommunication\n",
    "from ipynb.fs.defs.metric_template import save_to_disk, read_from_disk\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "\n",
    "load_dotenv(find_dotenv())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Specify variables\n",
    "METRIC_NAME = \"number_of_flakes\"\n",
    "# Specify the path for input grid data,\n",
    "INPUT_DATA_PATH = \"../../../../data/raw/testgrid_183.json.gz\"\n",
    "\n",
    "# Specify the path for output metric data\n",
    "OUTPUT_DATA_PATH = f\"../../../../data/processed/metrics/{METRIC_NAME}\"\n",
    "\n",
    "# Specify whether or not we are running this as a notebook or part of an automation pipeline.\n",
    "AUTOMATION = os.getenv(\"IN_AUTOMATION\")\n",
    "\n",
    "## CEPH Bucket variables\n",
    "## Create a .env file on your local with the correct configs,\n",
    "s3_endpoint_url = os.getenv(\"S3_ENDPOINT\")\n",
    "s3_access_key = os.getenv(\"S3_ACCESS_KEY\")\n",
    "s3_secret_key = os.getenv(\"S3_SECRET_KEY\")\n",
    "s3_bucket = os.getenv(\"S3_BUCKET\")\n",
    "s3_input_data_path = \"raw_data\"\n",
    "metric_path = f\"ai4ci/testgrid/metrics/{METRIC_NAME}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Import data\n",
    "timestamp = datetime.datetime.today()\n",
    "\n",
    "if AUTOMATION:\n",
    "    filename = f\"testgrid_{timestamp.day}{timestamp.month}.json\"\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    s3_object = cc.s3_resource.Object(s3_bucket, f\"{s3_input_data_path}/{filename}\")\n",
    "    file_content = s3_object.get()[\"Body\"].read().decode(\"utf-8\")\n",
    "    testgrid_data = json.loads(file_content)\n",
    "\n",
    "else:\n",
    "    with gzip.open(INPUT_DATA_PATH, \"rb\") as read_file:\n",
    "        testgrid_data = json.load(read_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculation\n",
    "- In this section, we calculate the metric values from the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T16:41:32.522585Z",
     "start_time": "2021-03-24T16:41:05.153836Z"
    }
   },
   "outputs": [],
   "source": [
    "unrolled_list = testgrid_labelwise_encoding(testgrid_data, 13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T16:41:47.182937Z",
     "start_time": "2021-03-24T16:41:32.524941Z"
    }
   },
   "outputs": [],
   "source": [
    "# Convert to dataframe\n",
    "df = pd.DataFrame(\n",
    "    unrolled_list,\n",
    "    columns=[\"timestamp\", \"tab\", \"grid\", \"test\", \"test_duration\", \"flake\"],\n",
    ")\n",
    "df = df.drop(columns=\"test_duration\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>flake</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-03-15 23:40:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-03-15 00:01:06</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-03-13 20:51:32</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-03-13 07:51:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-03-13 06:43:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timestamp                          tab  \\\n",
       "0 2021-03-15 23:40:20  \"redhat-assisted-installer\"   \n",
       "1 2021-03-15 00:01:06  \"redhat-assisted-installer\"   \n",
       "2 2021-03-13 20:51:32  \"redhat-assisted-installer\"   \n",
       "3 2021-03-13 07:51:20  \"redhat-assisted-installer\"   \n",
       "4 2021-03-13 06:43:20  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                grid     test  flake  \n",
       "0  periodic-ci-openshift-release-master-nightly-4...  Overall  False  \n",
       "1  periodic-ci-openshift-release-master-nightly-4...  Overall  False  \n",
       "2  periodic-ci-openshift-release-master-nightly-4...  Overall  False  \n",
       "3  periodic-ci-openshift-release-master-nightly-4...  Overall  False  \n",
       "4  periodic-ci-openshift-release-master-nightly-4...  Overall  False  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Flake Severity Metric \n",
    "Some tests would be flaky for a lot of builds and thus have a high flake rate or severity. This metric can be leveraged to help prioritize the work needed to resolve the flake issues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T16:41:52.845524Z",
     "start_time": "2021-03-24T16:41:47.185173Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test</th>\n",
       "      <th>flake</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>27192485</td>\n",
       "      <td>27192485</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>15157</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>37799</td>\n",
       "      <td>27142857</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            test     flake\n",
       "count   27192485  27192485\n",
       "unique     15157         2\n",
       "top      Overall     False\n",
       "freq       37799  27142857"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_flake_severity = df[[\"test\", \"flake\"]]\n",
    "df_flake_severity.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T16:42:03.156603Z",
     "start_time": "2021-03-24T16:41:52.847442Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test</th>\n",
       "      <th>flake_severity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Add Secret to Workloads.Add Secret to Workload...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Add Secret to Workloads.Add Secret to Workload...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Alertmanager: Configuration.creates a receiver...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Alertmanager: Configuration.deletes a receiver...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Alertmanager: Configuration.displays the Alert...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15152</th>\n",
       "      <td>user.openshift.io~v1~Group.Kubernetes resource...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15153</th>\n",
       "      <td>user.openshift.io~v1~Group.Kubernetes resource...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15154</th>\n",
       "      <td>user.openshift.io~v1~Group.Kubernetes resource...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15155</th>\n",
       "      <td>user.openshift.io~v1~Group.Kubernetes resource...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15156</th>\n",
       "      <td>user.openshift.io~v1~Group.Kubernetes resource...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>15157 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    test  flake_severity\n",
       "0      Add Secret to Workloads.Add Secret to Workload...             0.0\n",
       "1      Add Secret to Workloads.Add Secret to Workload...             0.0\n",
       "2      Alertmanager: Configuration.creates a receiver...             0.0\n",
       "3      Alertmanager: Configuration.deletes a receiver...             0.0\n",
       "4      Alertmanager: Configuration.displays the Alert...             0.0\n",
       "...                                                  ...             ...\n",
       "15152  user.openshift.io~v1~Group.Kubernetes resource...             0.0\n",
       "15153  user.openshift.io~v1~Group.Kubernetes resource...             0.0\n",
       "15154  user.openshift.io~v1~Group.Kubernetes resource...             0.0\n",
       "15155  user.openshift.io~v1~Group.Kubernetes resource...             0.0\n",
       "15156  user.openshift.io~v1~Group.Kubernetes resource...             0.0\n",
       "\n",
       "[15157 rows x 2 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## The following metric implements flake severity\n",
    "## or flake rate by tests\n",
    "## Moving forward, this will be aggregated in Superset\n",
    "## For the sake of completeness, it is implmented here\n",
    "flake_severity = df.groupby(\"test\").flake.mean().reset_index()\n",
    "flake_severity.rename(columns={\"flake\": \"flake_severity\"}, inplace=True)\n",
    "flake_severity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization \n",
    "- Here, we provide a quick visualization of the computed metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T16:42:04.112236Z",
     "start_time": "2021-03-24T16:42:03.159485Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA5AAAAJiCAYAAACxYS2CAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAyPUlEQVR4nO3dbZzU9X3v//fuongBnGUJ4IIm1jwaiseo6HqRGpMcMKKGiyQ9EWr02HqJOV4kGqs1Bowa7arRGCEPtbGJtjacXFQRMGK8SERPSvQUahETFSU1gKC7UhAFZHf+N2z2H+pFvjsIs8jzeQvmO7vzmekH7Cvzm6WuUqlUAgAAAH9Afa0HAAAAYNsgIAEAACgiIAEAACgiIAEAACgiIAEAACgiIAEAACgiIAHY7jz77LMZP358RowYkYMPPjjXXXdd0dedcMIJ+eEPf7iFp9u6brzxxnzlK1+p9RgAbCMEJADbne985zs55JBDMn/+/IwcObLW49TUpEmT8vWvfz1J8tvf/jbDhg3Lxo0bazwVAD2VgARgu7Ns2bL88R//ca3HqDmhCEB3CUgAtiv/63/9r8ybNy+XXnppRowYkddff73r7D/+4z9y+umn59BDD81BBx2U008/PS+88MJbfp+VK1dm7Nix+c53vpMkWbBgQSZOnJiWlpaMGzcu8+bN+4Oz/NM//VNGjRqVESNGZOTIkbnrrru6zn70ox/l6KOPzkEHHZSTTz45S5cuTZJMmTIlra2tm3yfM844I9/97neTJCtWrMhZZ52VQw89NCNHjsxtt93Wdb8bbrghZ599dr785S/ngAMOyB133JEbbrghX/7yl5Mkxx9/fJLkoIMOyogRI/LLX/4yBx98cH796193fY+2trbst99+aW9v/4PPD4D3HgEJwHbltttuS0tLSyZPnpz58+dnhx126Drr7OzMZz/72Tz44IN58MEH07t371x66aVv+h7PP/98TjjhhBx//PE55ZRTsmLFipx++uk544wz8stf/jIXXHBBzj777HeMrFdffTWXX355/vZv/zbz58/P9OnTM3z48CTJfffdl5tuuilTp07NL37xixx44IE577zzkiRjxozJ3XffnUqlkuSN6H3kkUdyzDHHpLOzM2eccUaGDRuWhx56KLfeemtuvfXWzJ07t+tx77///hx11FF57LHHMnbs2E1m+od/+IckyaOPPpr58+fn4IMPzjHHHLNJ2M6aNSsf+chH0tTU1N2XHoD3AAEJAP+pf//+GT16dHbeeef06dMnZ5xxRh599NFN7vPMM8/kxBNPzFlnnZUJEyYkSWbMmJGPfexj+fjHP576+vocdthh2WefffLzn//8HR+vvr4+Tz/9dNatW5dBgwZ1XVY7ffr0nHbaafngBz+YXr16ZdKkSXnyySezdOnStLS0pK6uLo899liSZM6cOdl///0zePDg/Nu//Vva29tz5plnZscdd8wee+yRY489NnfffXfXY+6///454ogjUl9fn5122ukPviaf+cxnMnv27K5gnTFjRsaNG1f+ogLwntKr1gMAQE/x2muv5corr8zcuXPzH//xH0mStWvXpqOjIw0NDUmSmTNn5v3vf39Gjx7d9XXLli3LPffckwcffLDrto0bN+aQQw5528faZZddct111+Xv/u7v8pWvfCUHHHBALrjggnzwgx/MsmXLcsUVV2xyqWqlUsmKFSsydOjQHHPMMZk1a1YOOuigzJw5syvoli5dmpUrV6alpaXr6zo6Ojb5/W677dat12S//fbLTjvtlHnz5mXgwIH593//94waNapb3wOA9w4BCQD/6e/+7u/y3HPP5Qc/+EEGDhyYJ598Mp/+9Ke73n1LkjPPPDNz587Neeedl+uuuy4NDQ1pbm7O+PHjc/nll3fr8Q4//PAcfvjhWbduXb75zW/mq1/9av7xH/8xzc3NmTRp0tu+0zdmzJicdNJJOe200/L4449n2rRpSZLm5ubsvvvuuffee9/2Mevq6rp99pnPfCZ33XVXBg4cmNGjR6d3797deJYAvJe4hBUA/tPatWvTu3fv9OvXL6tWrcrUqVPfdJ8ddtgh119/fV577bX81V/9VTo7OzNu3Lg8+OCDmTt3bjo6OrJ+/frMmzfvbX8AT5K89NJLue+++/Lqq69mxx13zC677JL6+jf+szxx4sTcfPPNefrpp5Mka9asyU9+8pOur917773Tv3//XHzxxfnoRz+afv36JUn23Xff7Lrrrrn55puzbt26dHR05Kmnnsrjjz9e9PybmppSX1+f559/fpPbx40bl/vuuy933XVXPv3pTxd9LwDemwQkAPynE088MevXr8+hhx6aCRMm5PDDD3/L++24446ZOnVq2tractFFF2Xw4MH59re/nZtuuikf+chH8vGPfzy33HJLOjs73/axOjs7873vfS+HH354Dj744Dz66KO55JJLkiSf/OQnc8opp+Tcc8/NAQcckDFjxuShhx7a5OvHjBmT//t//2/GjBnTdVtDQ0NuvPHG/OpXv8qoUaNy6KGH5uKLL84rr7xS9Px33nnnTJo0KX/+53+elpaWLFiwIMkb72zuvffeqaur2+RyWAC2P3WV378uBwDgLfz1X/91Bg0alC996Uu1HgWAGvIZSADgHf32t7/NT3/609xxxx21HgWAGhOQALAFjRgx4i1v/9u//dtt4nLQb37zm7n11ltz2mmnZY899qj1OADUmEtYAQAAKOKH6AAAAFBEQAIAAFBEQAIAAFDED9F5Gy+/vDadnT3r46EDBvRJW1vZv+UFv2NvqIa9oRr2hu6yM1TD3mx59fV16d9/17c8E5Bvo7Oz0uMCMkmPnImez95QDXtDNewN3WVnqIa9qR2XsAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFBEQAIAAFCkV60HoNyG1zsycGDfWo+xzVi3fmPWrH6t1mMAAMB7hoDchuy4Q0PGnjej1mNsM2Z+Y3zW1HoIAAB4D3EJKwAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEUEJAAAAEW2WkC2trZm5MiRGTZsWJ566qk3nU+dOvVNZwsWLMi4ceMyevTonHTSSWlra9vsMwAAAKqz1QJy1KhRuf322zN06NA3nT3xxBNZsGDBJmednZ05//zzM3ny5MyZMyctLS255pprNusMAACA6m21gGxpaUlzc/Obbt+wYUMuvfTSXHLJJZvcvnDhwvTu3TstLS1JkokTJ+aee+7ZrDMAAACq16vWA1x//fUZN25cdt99901uX758eYYMGdL1+6ampnR2dmbVqlVVnzU2NhbPNWBAn+qfFD3GwIF9az1Cj+B1oBr2hmrYG7rLzlANe1M7NQ3I+fPnZ+HChfnyl79cyzHeUlvbK+nsrNR6jE34g9J9L764ptYj1NzAgX29DnSbvaEa9obusjNUw95sefX1dW/7hlpNA/LRRx/N4sWLM2rUqCTJCy+8kJNPPjlXXnllmpubs2zZsq77tre3p76+Po2NjVWfAQAAUL2a/jMep512Wh5++OE88MADeeCBB7LbbrvllltuyUc/+tHss88+WbduXR577LEkyfTp03PUUUclSdVnAAAAVG+rvQN5+eWX5957781LL72Uv/zLv0xjY2Nmz579tvevr6/PVVddlSlTpmT9+vUZOnRorr766s06AwAAoHp1lUqlZ33Qr4foqZ+BHHvejFqPsc2Y+Y3xro+PzwlQHXtDNewN3WVnqIa92fLe6TOQNb2EFQAAgG2HgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKDIVgvI1tbWjBw5MsOGDctTTz2VJHn55Zdz6qmnZvTo0Rk7dmzOPPPMtLe3d33NggULMm7cuIwePTonnXRS2traNvsMAACA6my1gBw1alRuv/32DB06tOu2urq6nHLKKZkzZ05mzpyZPfbYI9dcc02SpLOzM+eff34mT56cOXPmpKWlZbPPAAAAqN5WC8iWlpY0NzdvcltjY2MOOeSQrt/vv//+WbZsWZJk4cKF6d27d1paWpIkEydOzD333LNZZwAAAFSvx3wGsrOzM9///vczcuTIJMny5cszZMiQrvOmpqZ0dnZm1apVVZ8BAABQvV61HuB3Lrvssuyyyy45/vjjaz1KkmTAgD61HoF3wcCBfWs9Qo/gdaAa9oZq2Bu6y85QDXtTOz0iIFtbW/Ob3/wmN954Y+rr33hTtLm5uety1iRpb29PfX19Ghsbqz7rjra2V9LZWdm8J/Yu8wel+158cU2tR6i5gQP7eh3oNntDNewN3WVnqIa92fLq6+ve9g21ml/Ceu2112bhwoWZNm1adtxxx67b99lnn6xbty6PPfZYkmT69Ok56qijNusMAACA6m21dyAvv/zy3HvvvXnppZfyl3/5l2lsbMw3v/nN3HTTTdlzzz0zceLEJMnuu++eadOmpb6+PldddVWmTJmS9evXZ+jQobn66quTpOozAAAAqldXqVR61nWaPURPvYR17Hkzaj3GNmPmN8a7vCEu86A69oZq2Bu6y85QDXuz5fXoS1gBAADYNghIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAimyVgGxtbc3IkSMzbNiwPPXUU123P/fcc5kwYUJGjx6dCRMmZMmSJVv0DAAAgOptlYAcNWpUbr/99gwdOnST26dMmZLjjjsuc+bMyXHHHZfJkydv0TMAAACqt1UCsqWlJc3NzZvc1tbWlkWLFmXMmDFJkjFjxmTRokVpb2/fImcAAABsnl61euDly5dn8ODBaWhoSJI0NDRk0KBBWb58eSqVyrt+1tTU1K35Bgzo8y4+W2pl4MC+tR6hR/A6UA17QzXsDd1lZ6iGvamdmgVkT9fW9ko6Oyu1HmMT/qB034svrqn1CDU3cGBfrwPdZm+ohr2hu+wM1bA3W159fd3bvqFWs4Bsbm7OihUr0tHRkYaGhnR0dGTlypVpbm5OpVJ5188AAADYPDX7ZzwGDBiQ4cOHZ9asWUmSWbNmZfjw4WlqatoiZwAAAGyeukqlssWv07z88stz77335qWXXkr//v3T2NiY2bNnZ/HixbnwwguzevXq9OvXL62trdlrr72SZIucdUdPvYR17Hkzaj3GNmPmN8a7vCEu86A69oZq2Bu6y85QDXuz5b3TJaxbJSC3RQJy2ycg3+AvWaphb6iGvaG77AzVsDdb3jsFZM0uYQUAAGDbIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAoIiABAAAo0qvkTs8880waGxvzvve9L2vXrs0tt9yS+vr6nHzyydl555239IwAAAD0AEXvQJ577rlZvXp1kqS1tTWPPvpoFixYkMmTJ2/R4QAAAOg5it6BXLp0afbaa69UKpX89Kc/zezZs7PTTjtl1KhRW3o+AAAAeoiigOzdu3deeeWVLF68OM3NzWlqasrGjRuzfv36LT0fAAAAPURRQI4ZMyYnnnhi1q5dm+OPPz5JsmjRouy+++5bdDgAAAB6jqKAvOiii/Lwww+nV69eOfTQQ5MkdXV1+eu//ustOhwAAAA9R9EP0bn88svz0Y9+tCsek+TDH/5wHnjggS02GAAAAD1LUUD+0z/901veftddd72rwwAAANBzveMlrD/60Y+SJB0dHV2//p3nn38+jY2N78oQDz74YK6//vpUKpVUKpWceeaZOfLII/Pcc8/lwgsvzKpVq9LY2JjW1tbsueeeSVL1GQAAANV5x4CcMWNGkuT111/v+nXyxucf3/e+96W1tXWzB6hUKvmrv/qr3H777fnQhz6UX/3qV/nzP//zHHHEEZkyZUqOO+64jB8/PjNmzMjkyZNz2223JUnVZwAAAFTnHQPy7//+75Mk1113Xb70pS9tsSHq6+uzZs2aJMmaNWsyaNCgvPzyy1m0aFG++93vJnnjJ8FedtllaW9vT6VSqeqsqalpiz0HAACA97qin8L6u3/CY9ddd01HR0fuvPPONDQ0ZNy4camvL/oY5duqq6vLN7/5zXzhC1/ILrvskrVr1+bmm2/O8uXLM3jw4DQ0NCRJGhoaMmjQoCxfvjyVSqWqs+4E5IABfTbredEzDBzYt9Yj9AheB6phb6iGvaG77AzVsDe1UxSQp59+er72ta9l7733zrXXXpuf/exn6dWrVxYtWpSLLrposwbYuHFjbrrppnz729/OgQcemP/3//5fvvjFL+aqq67arO+7udraXklnZ6WmM/xX/qB034svrqn1CDU3cGBfrwPdZm+ohr2hu+wM1bA3W159fd3bvqFWFJBLlizJ8OHDkyQzZ87M9OnTs8suu2TMmDGbHZBPPvlkVq5cmQMPPDBJcuCBB2bnnXdO7969s2LFinR0dKShoSEdHR1ZuXJlmpubU6lUqjoDAACgekXXn9bX1+f111/Pr3/96/Tt2zdDhgxJv379snbt2s0eYLfddssLL7yQZ599NkmyePHitLW15QMf+ECGDx+eWbNmJUlmzZqV4cOHp6mpKQMGDKjqDAAAgOoVvQP5sY99LOecc05WrVqVY445JknyzDPPZPDgwZs9wMCBA3PJJZfknHPOSV1dXZLkiiuuSGNjYy655JJceOGF+fa3v51+/fpt8lNfqz0DAACgOnWVSuUPftBvw4YNueOOO9KrV6+MHz8+vXr1yrx58/LSSy/lU5/61NaYc6vrqZ+BHHvejD98R5IkM78x3vXx8TkBqmNvqIa9obvsDNWwN1veZn8Gcscdd8yECRPS2dmZl156KYMGDcohhxzyrg4JAABAz1b0GcjVq1fnvPPOy7777psjjzwySXL//ffnuuuu26LDAQAA0HMUBeSUKVPSp0+fPPDAA9lhhx2SJCNGjMhPfvKTLTocAAAAPUfRJay/+MUvMnfu3Oywww5dP+imqakpbW1tW3Q4AAAAeo6idyD79u2bl19+eZPbli1bloEDB26RoQAAAOh53jEgf/dvKX7uc5/L2WefnX/+539OZ2dn5s+fnwsuuCATJ07cKkMCAABQe+8YkJMnT06SnHrqqTn66KNz6aWXZuPGjbnooosyatSonHjiiVtlSAAAAGrvHT8D+bt/IrKuri4nnniiYAQAANiOvWNAdnZ25p//+Z+7QvKtfOQjH3nXhwIAAKDneceA3LBhQ77yla+8bUDW1dXl/vvv3yKDAQAA0LO8Y0DuvPPOAhEAAIAkhf+MBwAAALxjQL7TZx8BAADYvrxjQM6fP39rzQEAAEAP5xJWAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAighIAAAAivSIgFy/fn2mTJmSI488MmPHjs1Xv/rVJMlzzz2XCRMmZPTo0ZkwYUKWLFnS9TXVngEAAFCdHhGQV199dXr37p05c+Zk5syZOeecc5IkU6ZMyXHHHZc5c+bkuOOOy+TJk7u+ptozAAAAqlPzgFy7dm3uvPPOnHPOOamrq0uSvO9970tbW1sWLVqUMWPGJEnGjBmTRYsWpb29veozAAAAqter1gM8//zzaWxszNSpUzNv3rzsuuuuOeecc7LTTjtl8ODBaWhoSJI0NDRk0KBBWb58eSqVSlVnTU1NNXueAAAA27qaB2RHR0eef/757L333rngggvyr//6r5k0aVKuv/76ms41YECfmj4+746BA/vWeoQewetANewN1bA3dJedoRr2pnZqHpDNzc3p1atX1yWn++23X/r375+ddtopK1asSEdHRxoaGtLR0ZGVK1emubk5lUqlqrPuaGt7JZ2dlS3xlKvmD0r3vfjimlqPUHMDB/b1OtBt9oZq2Bu6y85QDXuz5dXX173tG2o1/wxkU1NTDjnkkDzyyCNJ3vgJqm1tbdlzzz0zfPjwzJo1K0kya9asDB8+PE1NTRkwYEBVZwAAAFSvrlKp1Pxttueffz4XXXRRVq1alV69euWLX/xiPv7xj2fx4sW58MILs3r16vTr1y+tra3Za6+9kqTqs1I99R3IsefNqPUY24yZ3xjvf52K/5WO6tgbqmFv6C47QzXszZb3Tu9A9oiA7IkE5LZPQL7BX7JUw95QDXtDd9kZqmFvtrwefQkrAAAA2wYBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQBEBCQAAQJEeFZBTp07NsGHD8tRTTyVJFixYkHHjxmX06NE56aST0tbW1nXfas8AAACoTo8JyCeeeCILFizI0KFDkySdnZ05//zzM3ny5MyZMyctLS255pprNusMAACA6vWIgNywYUMuvfTSXHLJJV23LVy4ML17905LS0uSZOLEibnnnns26wwAAIDq9ar1AEly/fXXZ9y4cdl99927blu+fHmGDBnS9fumpqZ0dnZm1apVVZ81NjYWzzRgQJ/Ne1L0CAMH9q31CD2C14Fq2BuqYW/oLjtDNexN7dQ8IOfPn5+FCxfmy1/+cq1H2URb2yvp7KzUeoxN+IPSfS++uKbWI9TcwIF9vQ50m72hGvaG7rIzVMPebHn19XVv+4ZazQPy0UcfzeLFizNq1KgkyQsvvJCTTz45J5xwQpYtW9Z1v/b29tTX16exsTHNzc1VnQEAAFC9mn8G8rTTTsvDDz+cBx54IA888EB222233HLLLTnllFOybt26PPbYY0mS6dOn56ijjkqS7LPPPlWdAQAAUL2avwP5durr63PVVVdlypQpWb9+fYYOHZqrr756s84AAACoXl2lUulZH/TrIXrqZyDHnjej1mNsM2Z+Y7zr4+NzAlTH3lANe0N32RmqYW+2vHf6DGTNL2EFAABg2yAgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKCIgAQAAKFLzgHz55Zdz6qmnZvTo0Rk7dmzOPPPMtLe3J0kWLFiQcePGZfTo0TnppJPS1tbW9XXVngEAAFCdmgdkXV1dTjnllMyZMyczZ87MHnvskWuuuSadnZ05//zzM3ny5MyZMyctLS255pprkqTqMwAAAKpX84BsbGzMIYcc0vX7/fffP8uWLcvChQvTu3fvtLS0JEkmTpyYe+65J0mqPgMAAKB6NQ/I39fZ2Znvf//7GTlyZJYvX54hQ4Z0nTU1NaWzszOrVq2q+gwAAIDq9ar1AL/vsssuyy677JLjjz8+P/3pT2s6y4ABfWr6+Lw7Bg7sW+sRegSvA9WwN1TD3tBddoZq2Jva6TEB2dramt/85je58cYbU19fn+bm5ixbtqzrvL29PfX19WlsbKz6rDva2l5JZ2dls5/Xu8kflO578cU1tR6h5gYO7Ot1oNvsDdWwN3SXnaEa9mbLq6+ve9s31HrEJazXXnttFi5cmGnTpmXHHXdMkuyzzz5Zt25dHnvssSTJ9OnTc9RRR23WGQAAANWr+TuQTz/9dG666absueeemThxYpJk9913z7Rp03LVVVdlypQpWb9+fYYOHZqrr746SVJfX1/VGQAAANWrq1QqPes6zR6ip17COva8GbUeY5sx8xvjXd4Ql3lQHXtDNewN3WVnqIa92fJ6/CWsAAAA9HwCEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCICEgAAgCLv2YB87rnnMmHChIwePToTJkzIkiVLaj0SAADANu09G5BTpkzJcccdlzlz5uS4447L5MmTaz0SAADANq1XrQfYEtra2rJo0aJ897vfTZKMGTMml112Wdrb29PU1FT0Perr67bkiFUb1H/nWo+wTemp/3fc2rwOVMPeUA17Q3fZGaphb7asd3p935MBuXz58gwePDgNDQ1JkoaGhgwaNCjLly8vDsj+/XfdkiNW7ZaLj6z1CNuUAQP61HqEHsHrQDXsDdWwN3SXnaEa9qZ23rOXsAIAAPDuek8GZHNzc1asWJGOjo4kSUdHR1auXJnm5uYaTwYAALDtek8G5IABAzJ8+PDMmjUrSTJr1qwMHz68+PJVAAAA3qyuUqlUaj3ElrB48eJceOGFWb16dfr165fW1tbstddetR4LAABgm/WeDUgAAADeXe/JS1gBAAB49wlIAAAAighIAAAAighIAAAAigjIHua5557LhAkTMnr06EyYMCFLlix50306Ojryta99LUcccUQ++clP5oc//OHWH5QepWRvHn744Xz2s5/NPvvsk9bW1q0/JD1Oyd5MmzYtn/rUpzJ27Nh89rOfzdy5c7f+oPQoJXvz4x//OGPHjs348eMzduzY3HbbbVt/UHqUkr35nWeffTb77bef/1Zt50p25oYbbshHPvKRjB8/PuPHj8/Xvva1rT/o9qhCj3LCCSdU7rzzzkqlUqnceeedlRNOOOFN97njjjsqJ510UqWjo6PS1tZWOfzwwyvPP//81h6VHqRkb5YsWVJZtGhR5dprr638zd/8zdYekR6oZG8eeuihyquvvlqpVCqVJ598snLggQdWXnvtta06Jz1Lyd6sWbOm0tnZ2fXrT3ziE5Unn3xyq85Jz1KyN5VKpbJx48bK8ccfXzn33HP9t2o7V7Iz3/rWt+xJDXgHsgdpa2vLokWLMmbMmCTJmDFjsmjRorS3t29yv7vvvjuf+9znUl9fn6amphxxxBG55557ajEyPUDp3nzgAx/I8OHD06tXr1qMSQ9TujeHH354dt555yTJsGHDUqlUsmrVqq09Lj1E6d706dMndXV1SZJ169bl9ddf7/o925/SvUmSm2++OZ/4xCey5557buUp6Um6szNsfQKyB1m+fHkGDx6choaGJElDQ0MGDRqU5cuXv+l+Q4YM6fp9c3NzXnjhha06Kz1H6d7A76tmb+688868//3vz2677ba1xqSH6c7e3H///fnUpz6V//E//kdOOeWUDBs2bGuPSw9Ruje/+tWv8vDDD+cv/uIvajAlPUl3/q6ZPXt2xo4dm5NOOinz58/f2qNulwQkAH/QL3/5y1x//fX5xje+UetR2EaMGjUqs2fPzpw5czJjxow8++yztR6JHuz111/PV7/61Xzta1/rigb4QyZOnJj7778/M2fOzMknn5wvfOELefnll2s91nuea9l6kObm5qxYsSIdHR1paGhIR0dHVq5cmebm5jfdb9myZdl3332TvPkdSbYvpXsDv687ezN//vycf/75+fa3v5299tqrBtPSU1Tz982QIUPy4Q9/OD/72c/sz3aqZG9efPHF/Pu//3tOO+20JMnq1atTqVTyyiuv5LLLLqvV6NRI6d81AwcO7Pr1YYcdlubm5jz99NM5+OCDt/bI2xXvQPYgAwYMyPDhwzNr1qwkyaxZszJ8+PA0NTVtcr+jjjoqP/zhD9PZ2Zn29vbcd999GT16dC1Gpgco3Rv4faV78/jjj+dLX/pSvvWtb+W///f/XotR6UFK92bx4sVdv25vb8+8efPyoQ99aKvOSs9RsjdDhgzJvHnz8sADD+SBBx7IiSeemGOPPVY8bqdK/65ZsWJF16+ffPLJLF26NH/0R3+0VWfdHtVVKpVKrYfg/7d48eJceOGFWb16dfr165fW1tbstddeOfXUU3P22Wfnwx/+cDo6OnLppZfmkUceSZKceuqpmTBhQo0np5ZK9uaxxx7Lueeem1deeSWVSiV9+/bN17/+9Rx++OG1Hp8aKdmbP/uzP8vSpUszePDgrq+76qqrfJ5tO1ayN1dccUUeeeSR9OrVK5VKJZ/73Odywgkn1Hp0aqhkb37fDTfckFdffTUXXHBBjSam1kp25oILLsgTTzyR+vr67LDDDjn77LPz8Y9/vNajv+cJSAAAAIq4hBUAAIAiAhIAAIAiAhIAAIAiAhIAAIAiAhIAAIAiAhIAkvz2t7/NsGHDsnHjxj9433nz5uVjH/vYVphq6xkxYkSef/75Wo8BQA/Xq9YDAMDWNHLkyLz00ktpaGjouu2ee+6pySwbNmzItddem7vvvjtr1qxJ//79M2rUqHzlK1/Z6rPMnz+/69cXXnhhBg8enC996UtbfQ4AejYBCcB258Ybb8yf/umfbnLbb3/7260+x80335yFCxfmhz/8YQYNGpSlS5fmscce26ozbNy4Mb16+X8HACjjElYAeAs//vGPc/TRR2fEiBEZNWpUpk+f/rb3ve2223LMMcfkhRdeyIYNG9La2ppPfOIT+dM//dNMnjw569ate8uv+7d/+7ccccQRGTx4cOrq6rL77rvn05/+dNf5ihUrctZZZ+XQQw/NyJEjc9ttt3Xdvu+++2bVqlVd9120aFEOOeSQvP7660mSH/3oRzn66KNz0EEH5eSTT87SpUu77jts2LDcfvvtOfLII3PkkUd23fab3/wm/+f//J/MnDkzt9xyS0aMGJFJkyblO9/5Ts4666xNZr/88stz+eWXd+s1BWDbJyAB4C0MGDAgN910U/7lX/4lV155Za688so88cQTb7rf1KlTc8cdd+Qf/uEfsttuu+Waa67Jc889lzvvvDP33ntvVq5cmWnTpr3lY+y333753ve+l9tvvz2//vWvU6lUus46OztzxhlnZNiwYXnooYdy66235tZbb83cuXMzePDg7L///rn33nu77j9z5syMHj06O+ywQ+67777cdNNNmTp1an7xi1/kwAMPzHnnnbfJY9933335wQ9+kLvvvnuT2ydMmJCxY8fm5JNPzvz583PjjTdm3LhxmTt3blavXp3kjXctZ8+evUnsArB9EJAAbHf+9//+32lpaUlLS0u+8IUvvOV9PvGJT+T9739/6urqcvDBB+ewww7b5PLSSqWSK6+8Mo888khuu+22NDU1pVKp5Ac/+EEuuuiiNDY2pk+fPjn99NMze/bst3yM008/PaeeempmzpyZP/uzP8vhhx+eO+64I8kb7062t7fnzDPPzI477pg99tgjxx57bFfwjR07NrNmzeqa5e67787YsWOTJNOnT89pp52WD37wg+nVq1cmTZqUJ598cpN3IU877bQ0NjZmp512+oOv16BBg9LS0tL1WdG5c+emf//+2Wefff7g1wLw3uJDDwBsd6ZNm/amz0D+Vz//+c8zbdq0LFmyJJ2dnVm3bl0+9KEPdZ2vWbMmP/jBD3Ldddelb9++SZL29va89tpr+exnP9t1v0qlks7Ozrd8jIaGhnz+85/P5z//+axbty4//vGPc9FFF2XffffN0qVLs3LlyrS0tHTdv6Ojo+v3Rx55ZC677LKsXLkyS5YsSX19fdfZsmXLcsUVV6S1tXWTOVasWJGhQ4cmSZqbm7vzkuUzn/lMvv/97+fYY4/NXXfdlfHjx3fr6wF4bxCQAPBfbNiwIWeffXZaW1szatSo7LDDDvnCF76wySWm/fr1y9VXX50vfvGLmTp1ag488MD0798/O+20U2bPnp3Bgwd36zF32mmnfP7zn8+3vvWtPPPMM2lubs7uu+++yWWqv++//bf/lsMOOyx33313nn322RxzzDGpq6tL8kYcTpo0KePGjXvbx/vdfUvPjjjiiFxyySV56qmn8rOf/Sznn39+t54fAO8NLmEFgP9iw4YN2bBhQ5qamtKrV6/8/Oc/zyOPPPKm+x1yyCG55pprctZZZ+Xxxx9PfX19Pve5z+WKK65IW1tbkjd+4M3cuXPf8nG+973vZd68eVm3bl02btyYO+64I2vXrs3ee++dfffdN7vuumtuvvnmrFu3Lh0dHXnqqafy+OOPd3392LFjM2PGjMyZM6fr8tUkmThxYm6++eY8/fTTSd54t/QnP/lJ8fMfMGDAm34qbe/evTN69Oicd955+fCHP5whQ4YUfz8A3jsEJAD8F3369MnFF1+cL37xiznooIMya9asjBw58i3ve9hhh+WKK67IpEmT8sQTT+T888/PBz7wgRx77LE54IAD8hd/8Rd57rnn3vJrd95557S2tuawww7LoYcemttvvz033HBD9thjjzQ0NOTGG2/Mr371q4waNSqHHnpoLr744rzyyitdXz9y5MgsWbIk73vf+/Inf/InXbd/8pOfzCmnnJJzzz03BxxwQMaMGZOHHnqo+Pn/z//5P/PMM8+86TOin/70p/PUU0+5fBVgO1ZX+f3rcQAA3sayZcty9NFH55FHHkmfPn1qPQ4ANeAdSADgD+rs7Mx3v/vdHHPMMeIRYDvmh+gAAO/o1VdfzWGHHZYhQ4bkO9/5Tq3HAaCGXMIKAABAEZewAgAAUERAAgAAUERAAgAAUERAAgAAUERAAgAAUERAAgAAUOT/A4lPpRLH7/iAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x720 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.set(rc={\"figure.figsize\": (15, 10)})\n",
    "flake_severity.hist()\n",
    "plt.ylabel(\"Tests\")\n",
    "plt.xlabel(\"Flake Severity\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above graph we can conclude that most of the tests have very low(~0) flake severity. One might look at this graph and assume that almost all of the tests have flake severity as 0. So, to get more clarity we have plotted another graph and here we've gone from about 8,000 tests to investigate for probable flakiness to less than 100. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T16:42:04.471269Z",
     "start_time": "2021-03-24T16:42:04.114874Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3sAAAJiCAYAAAB+Y9sEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtEElEQVR4nO3de5iVdb3//9fMwAAFbISAxkO29Uo25QlFxQxzg0kSiNlW2B6y8oR9w/KUpoSJp6ZMy0MXmmW6c+fWzAOHxDyU5C7SK7zMMM94AAMFD3hAYGb9/vDac8XP08wAs4YPj8dfw1r3ve73zPq4xud132tNTaVSqQQAAICi1FZ7AAAAANY9sQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQdAp/bEE09k3LhxGTJkSHbddddceOGFrdrvsMMOy/XXX7+ep+tY06ZNy+mnn17tMQDYQIg9ADq1K664IrvttlvmzZuXESNGVHucqpo4cWLOOeecJMmzzz6bQYMGZfXq1VWeCoDOSuwB0KktWrQoH/vYx6o9RtWJOgDaSuwB0Gl98YtfzNy5czN16tQMGTIkq1atarnv5ZdfzjHHHJNhw4Zll112yTHHHJN//OMf7/g4S5YsydixY3PFFVckSe6///5MmDAhQ4cOzX777Ze5c+e+7yy//vWvM3LkyAwZMiQjRozILbfc0nLfr371q+y7777ZZZddcsQRR2ThwoVJkjPOOCONjY1rPM6xxx6bK6+8MkmyePHiTJo0KcOGDcuIESNy9dVXt2x38cUX57jjjstJJ52UnXbaKTfeeGMuvvjinHTSSUmSQw89NEmyyy67ZMiQIfnzn/+cXXfdNQ8//HDLYyxdujQ77LBDli1b9r7fHwDlEXsAdFpXX311hg4dmilTpmTevHnp2rVry33Nzc054IADctddd+Wuu+5Kt27dMnXq1Lc9xjPPPJPDDjsshx56aI488sgsXrw4xxxzTI499tj8+c9/zimnnJLjjjvuPYPo9ddfz9lnn52f/OQnmTdvXq699toMHjw4SXL77bfnsssuyyWXXJI//vGP2XnnnXPiiScmScaMGZNZs2alUqkkeStQ77nnnowePTrNzc059thjM2jQoNx999256qqrctVVV2XOnDktx73jjjvy2c9+Nvfdd1/Gjh27xky/+MUvkiT33ntv5s2bl1133TWjR49eI0JnzJiR3XffPX379m3rjx6AAog9ADZIm2yySUaNGpUePXqkZ8+eOfbYY3Pvvfeusc1jjz2Www8/PJMmTcr48eOTJDfffHP23HPPfPrTn05tbW322GOPbLvttvn973//nserra3No48+mhUrVmTAgAEtl5Zee+21Ofroo7P11lunS5cumThxYh566KEsXLgwQ4cOTU1NTe67774kyezZs7Pjjjtm4MCB+etf/5ply5bla1/7Wurr67PFFlvkoIMOyqxZs1qOueOOO2bvvfdObW1tunfv/r4/k89//vOZOXNmS1zefPPN2W+//Vr/QwWgKF2qPQAAtMcbb7yR8847L3PmzMnLL7+cJHnttdfS1NSUurq6JMn06dPzkY98JKNGjWrZb9GiRbn11ltz1113tdy2evXq7Lbbbu96rA984AO58MIL87Of/Synn356dtppp5xyyinZeuuts2jRopx77rlrXK5ZqVSyePHibLbZZhk9enRmzJiRXXbZJdOnT2+Jr4ULF2bJkiUZOnRoy35NTU1r/PvDH/5wm34mO+ywQ7p37565c+emf//+efrppzNy5Mg2PQYA5RB7AGyQfvazn+XJJ5/Mddddl/79++ehhx7K/vvv33JWK0m+9rWvZc6cOTnxxBNz4YUXpq6uLg0NDRk3blzOPvvsNh1v+PDhGT58eFasWJEf/vCH+fa3v53//u//TkNDQyZOnPiuZ9DGjBmTr3zlKzn66KPzwAMP5NJLL02SNDQ0ZPPNN89tt932rsesqalp832f//znc8stt6R///4ZNWpUunXr1obvEoCSuIwTgA3Sa6+9lm7duqV379556aWXcskll7xtm65du+ZHP/pR3njjjXzzm99Mc3Nz9ttvv9x1112ZM2dOmpqa8uabb2bu3Lnv+uEuSfLCCy/k9ttvz+uvv576+vp84AMfSG3tW79CJ0yYkMsvvzyPPvpokmT58uX5zW9+07Lvxz/+8WyyySaZPHlyPvWpT6V3795Jku233z4f/OAHc/nll2fFihVpamrKI488kgceeKBV33/fvn1TW1ubZ555Zo3b99tvv9x+++255ZZbsv/++7fqsQAok9gDYIN0+OGH580338ywYcMyfvz4DB8+/B23q6+vzyWXXJKlS5fmtNNOy8CBA/PjH/84l112WXbfffd8+tOfzk9/+tM0Nze/67Gam5vz85//PMOHD8+uu+6ae++9N9/5zneSJJ/5zGdy5JFH5oQTTshOO+2UMWPG5O67715j/zFjxuR///d/M2bMmJbb6urqMm3atPz973/PyJEjM2zYsEyePDmvvvpqq77/Hj16ZOLEifnP//zPDB06NPfff3+St84YfvzjH09NTc0al4QCsPGpqfzz9S4AwAbvW9/6VgYMGJDjjz++2qMAUEXeswcABXn22Wfz29/+NjfeeGO1RwGgyjok9l588cV885vfzNNPP536+vpsueWWmTp1avr27ZtBgwZlm222aXnvw/e+970MGjSoI8YCgDUMGTLkHW//yU9+skFcEvnDH/4wV111VY4++uhsscUW1R4HgCrrkMs4X3rppTz88MMtH2vd2NiYl19+Oeeee24GDRqUv/zlL/ngBz+4vscAAADYaHTIB7T06dNnjb9ftOOOO2bRokUdcWgAAICNUoe/Z6+5uTm//OUvM2LEiJbbDjvssDQ1NWXPPffMpEmTUl9f39FjAQAAFKXDP43zzDPPzOLFi3PJJZektrY2zz33XBoaGvLqq6/m5JNPzjbbbOPTwwAAANZSh57Za2xszFNPPZVp06a1fCBLQ0NDkqRnz5458MADc+WVV7b5cV988bU0N3euvyDRr1/PHHH2be3a96eT98nSpa37O0uUpV+/np572sy6oT2sG9rKmqE9rJv1r7a2Jpts8s6ff9JhsXfBBRfkwQcfzOWXX95ymebLL7+cbt26pXv37lm9enVmz56dwYMHt/mxm5srnS72kmTJi2+0e9/O+P3QMTz3tId1Q3tYN7SVNUN7WDfV0yGx9+ijj+ayyy7LRz/60UyYMCFJsvnmm+fII4/MlClTUlNTk9WrV2fIkCH5+te/3hEjAQAAFK1DYu9jH/tYHn744Xe8b/r06R0xAgAAwEalQ/70AgAAAB1L7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABRI7AEAABSoS0cc5MUXX8w3v/nNPP3006mvr8+WW26ZqVOnpm/fvrn//vszZcqUvPnmm9lss83y/e9/P/369euIsQAAAIrVIWf2ampqcuSRR2b27NmZPn16tthii5x//vlpbm7OySefnClTpmT27NkZOnRozj///I4YCQAAoGgdEnt9+vTJbrvt1vLvHXfcMYsWLcqDDz6Ybt26ZejQoUmSCRMm5NZbb+2IkQAAAIrWIZdx/rPm5ub88pe/zIgRI/Lcc89l0003bbmvb9++aW5uzksvvZQ+ffq0+jH79eu5Hiatrv79e1V7BKrEc097WDe0h3VDW1kztId1Uz0dHntnnXVWPvCBD+TQQw/Nb3/723XymEuXvprm5so6eax1ZW0X9fPPL19Hk7Ah6d+/l+eeNrNuaA/rhrayZmgP62b9q62tedeTXx0ae42NjXnqqacybdq01NbWpqGhIYsWLWq5f9myZamtrW3TWT0AAADersP+9MIFF1yQBx98MJdeemnq6+uTJNtuu21WrFiR++67L0ly7bXX5rOf/WxHjQQAAFCsDjmz9+ijj+ayyy7LRz/60UyYMCFJsvnmm+fSSy/N9773vZxxxhlr/OkFAAAA1k6HxN7HPvaxPPzww+9430477ZTp06d3xBgAAAAbjQ67jBMAAICOI/YAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAK1KWjDtTY2JjZs2dn4cKFmT59erbZZpskyYgRI1JfX59u3bolSU466aQMHz68o8YCAAAoUofF3siRI/PFL34xhxxyyNvuu+iii1riDwAAgLXXYbE3dOjQjjoUAADARq/DYu+9nHTSSalUKtl5551zwgknpHfv3m3av1+/nutpsurp379XtUegSjz3tId1Q3tYN7SVNUN7WDfVU/XYu+aaa9LQ0JCVK1fmnHPOydSpU3P++ee36TGWLn01zc2V9TRh+6zton7++eXraBI2JP379/Lc02bWDe1h3dBW1gztYd2sf7W1Ne968qvqn8bZ0NCQJKmvr8/BBx+cv/zlL1WeCAAAYMNX1dh7/fXXs3z5W6VfqVQya9asDB48uJojAQAAFKHDLuM8++yzc9ttt+WFF17Il7/85fTp0yfTpk3LpEmT0tTUlObm5my99dY544wzOmokAACAYnVY7E2ePDmTJ09+2+033XRTR40AAACw0aj6e/YAAABY98QeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgbq0ZqPHHnssffr0yYc+9KG89tpr+elPf5ra2tocccQR6dGjx/qeEQAAgDZq1Zm9E044Ia+88kqSpLGxMffee2/uv//+TJkyZb0OBwAAQPu06szewoULs9VWW6VSqeS3v/1tZs6cme7du2fkyJHrez4AAADaoVWx161bt7z66qt5/PHH09DQkL59+2b16tV588031/d8G6WVq5rSv3+vdu274s3VWf7KG+t4IgAAYEPTqtgbM2ZMDj/88Lz22ms59NBDkyTz58/P5ptvvl6H21jVd63L2BNvbte+038wLsvX8TwAAMCGp1Wxd9ppp+UPf/hDunTpkmHDhiVJampq8q1vfWu9DgcAAED7tOoDWs4+++x86lOfagm9JNluu+1y5513rrfBAAAAaL9Wxd6vf/3rd7z9lltuWafDAAAAsG6852Wcv/rVr5IkTU1NLV//n2eeeSZ9+vRZb4MBAADQfu8Zezff/NaHhKxatarl6+St9+t96EMfSmNj4/qdDgAAgHZ5z9j7r//6ryTJhRdemOOPP75DBgIAAGDtteo9e//3ZxeSty7pvOGGG3LTTTelubl5vQ4HAABA+7Qq9o455pg89dRTSZILLrggP/vZz3LllVfmu9/97nodDgAAgPZpVewtWLAggwcPTpJMnz49P/nJT3LVVVdl1qxZ63U4AAAA2qdVf1S9trY2q1atypNPPplevXpl0003TXNzc8ulnQAAAHQurYq9PffcM1//+tfz0ksvZfTo0UmSxx57LAMHDlyvwwEAANA+rYq9c845JzfeeGO6dOmScePGJUlefPHFTJo0ab0OBwAAQPu0Kvbq6+szfvz4NDc354UXXsiAAQOy2267re/ZAAAAaKdWfUDLK6+8khNPPDHbb7999tlnnyTJHXfckQsvvHC9DgcAAED7tCr2zjjjjPTs2TN33nlnunbtmiQZMmRIfvOb36zX4QAAAGifVl3G+cc//jFz5sxJ165dU1NTkyTp27dvli5dul6HAwAAoH1adWavV69eefHFF9e4bdGiRenfv/96GQoAAIC1856xN2PGjCTJgQcemOOOOy5/+tOf0tzcnHnz5uWUU07JhAkTOmRIAAAA2uY9Y2/KlClJkqOOOir77rtvpk6dmtWrV+e0007LyJEjc/jhh3fIkAAAALTNe75nr1KpJElqampy+OGHizsAAIANxHvGXnNzc/70pz+1RN872X333df5UAAAAKyd94y9lStX5vTTT3/X2Kupqckdd9yxXgYDAACg/d4z9nr06CHmAAAANkCt+tMLAAAAbFjeM/be6716AAAAdF7vGXvz5s3rqDkAAABYh1zGCQAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUCCxBwAAUKAOib3GxsaMGDEigwYNyiOPPNJy+5NPPpnx48dn1KhRGT9+fBYsWNAR4wAAABSvQ2Jv5MiRueaaa7LZZputcfsZZ5yRgw8+OLNnz87BBx+cKVOmdMQ4AAAAxeuQ2Bs6dGgaGhrWuG3p0qWZP39+xowZkyQZM2ZM5s+fn2XLlnXESAAAAEWr2nv2nnvuuQwcODB1dXVJkrq6ugwYMCDPPfdctUYCAAAoRpdqD7Au9OvXs9ojdCr9+/eq9gisBc8f7WHd0B7WDW1lzdAe1k31VC32Ghoasnjx4jQ1NaWuri5NTU1ZsmTJ2y73bI2lS19Nc3NlPUzZftVc1M8/v7xqx2bt9O/fy/NHm1k3tId1Q1tZM7SHdbP+1dbWvOvJr6pdxtmvX78MHjw4M2bMSJLMmDEjgwcPTt++fas1EgAAQDE65Mze2Wefndtuuy0vvPBCvvzlL6dPnz6ZOXNmvvOd7+TUU0/Nj3/84/Tu3TuNjY0dMQ4AAEDxOiT2Jk+enMmTJ7/t9q233jrXX399R4wAAACwUanaZZwAAACsP2IPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQF2qPUCSjBgxIvX19enWrVuS5KSTTsrw4cOrPBUAAMCGq1PEXpJcdNFF2Wabbao9BgAAQBFcxgkAAFCgTnNm76STTkqlUsnOO++cE044Ib179672SAAAABusThF711xzTRoaGrJy5cqcc845mTp1as4///xW79+vX8/1ON2GZeWqpvTv36vd+9Z3rVvHE9FW7X3+2LhZN2v3Graxvv5ZN7SVNUN7WDfV0ylir6GhIUlSX1+fgw8+OMcee2yb9l+69NU0N1fWx2jtVq1FXd+1LmNPvLld+07/wbg8//zydTwRbdG/fy/PAW1m3bylf/9eXv/awLqhrawZ2sO6Wf9qa2ve9eRX1d+z9/rrr2f58rcWQKVSyaxZszJ48OAqTwUAALBhq/qZvaVLl2bSpElpampKc3Nztt5665xxxhnVHgsAAGCDVvXY22KLLXLTTTdVewwAAICiVP0yTgAAANY9sQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFAgsQcAAFCgLtUegHL06t0j3bu1b0mteHN1lr/yxjqeCAAANl5ij3Wme7cuGXvize3ad/oPxmX5Op4HAAA2Zi7jBAAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKJDYAwAAKFCXag8ASbJyVVP69+/Vrn1XvLk6y195Yx1P9P569e6R7t3a/5/Qmyub0q2+7m23t+bnsDbf89rMXa2fNQAAbSf26BTqu9Zl7Ik3t2vf6T8Yl+XreJ7W6N6tS7tnTt6auxrf89rMXa2fNQAAbecyTgAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAKJPQAAgAJ1qfYAdB4rVzWlf/9e1R6jzdZm7jdXNqVbfd06nggAAKpP7NGivmtdxp54c7v3n/6DcetwmtZbm7mn/2DcWu0LAACdlcs4AQAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACtSl2gMAbbdyVVP69++1wR33zZVN6VZft0Htu+LN1Vn+yhvt2hfeT6/ePdK9W/t/FVufvJ+1WWPv9NrZ2t8BXnfprDa2112xBxug+q51GXvize3ad/oPxlXluP937LWZu1r7Lm/XnvD+unfrstb/TVmfvJe1WWNedynRxva66zJOAACAAok9AACAAok9AACAAok9AACAAok9AACAAok9AACAAok9AACAAok9AACAAok9AACAAok9AACAAok9AACAAok9AACAAok9AACAAok9AACAAok9AACAAok9AACAAnWK2HvyySczfvz4jBo1KuPHj8+CBQuqPRIAAMAGrVPE3hlnnJGDDz44s2fPzsEHH5wpU6ZUeyQAAIANWpdqD7B06dLMnz8/V155ZZJkzJgxOeuss7Js2bL07du3VY9RW1uzPkdstwGb9Nio9q3msTfEfat5bN9z23TW15h/tiHM2BE2xOd5bf+bWpu5rZuNg9ddqq2zPZ/VfN1dH95rnppKpVLpwFne5sEHH8wpp5ySmTNnttw2evTofP/7388nPvGJKk4GAACw4eoUl3ECAACwblU99hoaGrJ48eI0NTUlSZqamrJkyZI0NDRUeTIAAIANV9Vjr1+/fhk8eHBmzJiRJJkxY0YGDx7c6vfrAQAA8HZVf89ekjz++OM59dRT88orr6R3795pbGzMVlttVe2xAAAANlidIvYAAABYt6p+GScAAADrntgDAAAokNgDAAAokNgDAAAokNhbC08++WTGjx+fUaNGZfz48VmwYMHbtmlqasqZZ56ZvffeO5/5zGdy/fXXd/ygdCqtWTd/+MMfcsABB2TbbbdNY2Njxw9Jp9OadXPppZfmc5/7XMaOHZsDDjggc+bM6fhB6VRas25uuOGGjB07NuPGjcvYsWNz9dVXd/ygdCqtWTf/54knnsgOO+zgd9VGrjVr5uKLL87uu++ecePGZdy4cTnzzDM7ftCNUYV2O+ywwyo33XRTpVKpVG666abKYYcd9rZtbrzxxspXvvKVSlNTU2Xp0qWV4cOHV5555pmOHpVOpDXrZsGCBZX58+dXLrjggsp3v/vdjh6RTqg16+buu++uvP7665VKpVJ56KGHKjvvvHPljTfe6NA56Vxas26WL19eaW5ubvl6r732qjz00EMdOiedS2vWTaVSqaxevbpy6KGHVk444QS/qzZyrVkzF110kXVSBc7stdPSpUszf/78jBkzJkkyZsyYzJ8/P8uWLVtju1mzZuXAAw9MbW1t+vbtm7333ju33nprNUamE2jtutlyyy0zePDgdOnSpRpj0sm0dt0MHz48PXr0SJIMGjQolUolL730UkePSyfR2nXTs2fP1NTUJElWrFiRVatWtfybjU9r102SXH755dlrr73y0Y9+tIOnpDNpy5qh44m9dnruuecycODA1NXVJUnq6uoyYMCAPPfcc2/bbtNNN235d0NDQ/7xj3906Kx0Hq1dN/DP2rNubrrppnzkIx/Jhz/84Y4ak06mLevmjjvuyOc+97n8+7//e4488sgMGjSoo8elk2jtuvn73/+eP/zhD/nSl75UhSnpTNryWjNz5syMHTs2X/nKVzJv3ryOHnWjJPYACvPnP/85P/rRj/KDH/yg2qOwgRg5cmRmzpyZ2bNn5+abb84TTzxR7ZHoxFatWpVvf/vbOfPMM1v+Bx/ez4QJE3LHHXdk+vTpOeKII/LVr341L774YrXHKp5rxNqpoaEhixcvTlNTU+rq6tLU1JQlS5akoaHhbdstWrQo22+/fZK3n+lj49LadQP/rC3rZt68eTn55JPz4x//OFtttVUVpqWzaM/rzaabbprtttsuv/vd76yfjVRr1s3zzz+fp59+OkcffXSS5JVXXkmlUsmrr76as846q1qjUyWtfa3p379/y9d77LFHGhoa8uijj2bXXXft6JE3Ks7stVO/fv0yePDgzJgxI0kyY8aMDB48OH379l1ju89+9rO5/vrr09zcnGXLluX222/PqFGjqjEynUBr1w38s9aumwceeCDHH398LrroonziE5+oxqh0Iq1dN48//njL18uWLcvcuXOzzTbbdOisdB6tWTebbrpp5s6dmzvvvDN33nlnDj/88Bx00EFCbyPV2teaxYsXt3z90EMPZeHChfnXf/3XDp11Y1RTqVQq1R5iQ/X444/n1FNPzSuvvJLevXunsbExW221VY466qgcd9xx2W677dLU1JSpU6fmnnvuSZIcddRRGT9+fJUnp5pas27uu+++nHDCCXn11VdTqVTSq1evnHPOORk+fHi1x6dKWrNuvvCFL2ThwoUZOHBgy37f+973vP9qI9aadXPuuefmnnvuSZcuXVKpVHLggQfmsMMOq/boVFFr1s0/u/jii/P666/nlFNOqdLEVFtr1swpp5ySv/3tb6mtrU3Xrl1z3HHH5dOf/nS1Ry+e2AMAACiQyzgBAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYAAAAKJPYA2OA8++yzGTRoUFavXv2+286dOzd77rlnB0zVcYYMGZJnnnmm2mMA0Ml1qfYAAPBuRowYkRdeeCF1dXUtt916661VmWXlypW54IILMmvWrCxfvjybbLJJRo4cmdNPP73DZ5k3b17L16eeemoGDhyY448/vsPnAKBzE3sAdGrTpk3LJz/5yTVue/bZZzt8jssvvzwPPvhgrr/++gwYMCALFy7Mfffd16EzrF69Ol26+NUNQOu4jBOADd4NN9yQfffdN0OGDMnIkSNz7bXXvuu2V199dUaPHp1//OMfWblyZRobG7PXXnvlk5/8ZKZMmZIVK1a8435//etfs/fee2fgwIGpqanJ5ptvnv3337/l/sWLF2fSpEkZNmxYRowYkauvvrrl9u233z4vvfRSy7bz58/PbrvtllWrViVJfvWrX2XffffNLrvskiOOOCILFy5s2XbQoEG55pprss8++2SfffZpue2pp57K//zP/2T69On56U9/miFDhmTixIm54oorMmnSpDVmP/vss3P22We36WcKwIZP7AGwwevXr18uu+yy/OUvf8l5552X8847L3/729/ett0ll1ySG2+8Mb/4xS/y4Q9/OOeff36efPLJ3HTTTbntttuyZMmSXHrppe94jB122CE///nPc8011+Thhx9OpVJpua+5uTnHHntsBg0alLvvvjtXXXVVrrrqqsyZMycDBw7MjjvumNtuu61l++nTp2fUqFHp2rVrbr/99lx22WW55JJL8sc//jE777xzTjzxxDWOffvtt+e6667LrFmz1rh9/PjxGTt2bI444ojMmzcv06ZNy3777Zc5c+bklVdeSfLW2cCZM2euEaYAbBzEHgCd2v/7f/8vQ4cOzdChQ/PVr371HbfZa6+98pGPfCQ1NTXZdddds8cee6xxiWWlUsl5552Xe+65J1dffXX69u2bSqWS6667Lqeddlr69OmTnj175phjjsnMmTPf8RjHHHNMjjrqqEyfPj1f+MIXMnz48Nx4441J3jrrt2zZsnzta19LfX19tthiixx00EEtcTZ27NjMmDGjZZZZs2Zl7NixSZJrr702Rx99dLbeeut06dIlEydOzEMPPbTG2b2jjz46ffr0Sffu3d/35zVgwIAMHTq05b2Nc+bMySabbJJtt932ffcFoCwu/AegU7v00kvf9p69/7/f//73ufTSS7NgwYI0NzdnxYoV2WabbVruX758ea677rpceOGF6dWrV5Jk2bJleeONN3LAAQe0bFepVNLc3PyOx6irq8shhxySQw45JCtWrMgNN9yQ0047Ldtvv30WLlyYJUuWZOjQoS3bNzU1tfx7n332yVlnnZUlS5ZkwYIFqa2tbblv0aJFOffcc9PY2LjGHIsXL85mm22WJGloaGjLjyyf//zn88tf/jIHHXRQbrnllowbN65N+wNQBrEHwAZt5cqVOe6449LY2JiRI0ema9eu+epXv7rGZZa9e/fO97///XzjG9/IJZdckp133jmbbLJJunfvnpkzZ2bgwIFtOmb37t1zyCGH5KKLLspjjz2WhoaGbL755mtcqvnP/uVf/iV77LFHZs2alSeeeCKjR49OTU1NkrdCbuLEidlvv/3e9Xj/t21r79t7773zne98J4888kh+97vf5eSTT27T9wdAGVzGCcAGbeXKlVm5cmX69u2bLl265Pe//33uueeet22322675fzzz8+kSZPywAMPpLa2NgceeGDOPffcLF26NMlbH6YyZ86cdzzOz3/+88ydOzcrVqzI6tWrc+ONN+a1117Lxz/+8Wy//fb54Ac/mMsvvzwrVqxIU1NTHnnkkTzwwAMt+48dOzY333xzZs+e3XIJZ5JMmDAhl19+eR599NEkb52F/M1vftPq779fv35v+3TSbt26ZdSoUTnxxBOz3XbbZdNNN2314wFQDrEHwAatZ8+emTx5cr7xjW9kl112yYwZMzJixIh33HaPPfbIueeem4kTJ+Zvf/tbTj755Gy55ZY56KCDstNOO+VLX/pSnnzyyXfct0ePHmlsbMwee+yRYcOG5ZprrsnFF1+cLbbYInV1dZk2bVr+/ve/Z+TIkRk2bFgmT56cV199tWX/ESNGZMGCBfnQhz6Uf/u3f2u5/TOf+UyOPPLInHDCCdlpp50yZsyY3H333a3+/v/jP/4jjz322Nve07j//vvnkUcecQknwEaspvLP17kAAEVYtGhR9t1339xzzz3p2bNntccBoAqc2QOAwjQ3N+fKK6/M6NGjhR7ARswHtABAQV5//fXsscce2XTTTXPFFVdUexwAqshlnAAAAAVyGScAAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECB/j/TMb40afvvlwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x720 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.set(rc={\"figure.figsize\": (15, 10)})\n",
    "flake_severity.hist(bins=50)\n",
    "plt.ylabel(\"Tests\")\n",
    "plt.xlabel(\"Flake Severity\")\n",
    "plt.ylim((0, 25))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above graph we can see that there are other values of flake severity for some tests. However it is a small bunch of tests that show a different set of flake severity. Let's see some tests that have a high flake severity score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top 5 tests with the highest flake score :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T16:42:04.486646Z",
     "start_time": "2021-03-24T16:42:04.473750Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test</th>\n",
       "      <th>flake_severity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5793</th>\n",
       "      <td>openshift-tests.[sig-arch] Monitor cluster whi...</td>\n",
       "      <td>0.536148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>230</th>\n",
       "      <td>Cluster upgrade.[sig-imageregistry] Image regi...</td>\n",
       "      <td>0.410038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>234</th>\n",
       "      <td>Cluster upgrade.[sig-network-edge] Application...</td>\n",
       "      <td>0.404112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>236</th>\n",
       "      <td>Cluster upgrade.[sig-network-edge] Cluster fro...</td>\n",
       "      <td>0.386168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>Cluster upgrade.[sig-api-machinery] Kubernetes...</td>\n",
       "      <td>0.279356</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   test  flake_severity\n",
       "5793  openshift-tests.[sig-arch] Monitor cluster whi...        0.536148\n",
       "230   Cluster upgrade.[sig-imageregistry] Image regi...        0.410038\n",
       "234   Cluster upgrade.[sig-network-edge] Application...        0.404112\n",
       "236   Cluster upgrade.[sig-network-edge] Cluster fro...        0.386168\n",
       "203   Cluster upgrade.[sig-api-machinery] Kubernetes...        0.279356"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flake_severity.nlargest(5, \"flake_severity\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T16:42:15.855179Z",
     "start_time": "2021-03-24T16:42:15.721886Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.001825063064298831"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Overall flake percentage\n",
    "df.flake.sum() / df.flake.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save results to Ceph or locally\n",
    "* Use the following helper function to save the data frame in a parquet format on the Ceph bucket if we are running in automation, and locally if not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = f\"{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\"\n",
    "\n",
    "if AUTOMATION == \"True\":\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    cc.upload_to_ceph(df, metric_path, filename)\n",
    "else:\n",
    "    save_to_disk(df, OUTPUT_DATA_PATH, filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>flake</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-03-15 23:40:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-03-15 00:01:06</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-03-13 20:51:32</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-03-13 07:51:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-03-13 06:43:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27192480</th>\n",
       "      <td>2021-03-14 00:01:00</td>\n",
       "      <td>\"redhat-single-node\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>openshift-tests.[sig-arch] Monitor cluster whi...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27192481</th>\n",
       "      <td>2021-03-13 00:01:07</td>\n",
       "      <td>\"redhat-single-node\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>openshift-tests.[sig-arch] Monitor cluster whi...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27192482</th>\n",
       "      <td>2021-03-12 04:22:20</td>\n",
       "      <td>\"redhat-single-node\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>openshift-tests.[sig-arch] Monitor cluster whi...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27192483</th>\n",
       "      <td>2021-03-11 00:01:18</td>\n",
       "      <td>\"redhat-single-node\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>openshift-tests.[sig-arch] Monitor cluster whi...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27192484</th>\n",
       "      <td>2021-03-10 00:01:03</td>\n",
       "      <td>\"redhat-single-node\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>openshift-tests.[sig-arch] Monitor cluster whi...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>27192485 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   timestamp                          tab  \\\n",
       "0        2021-03-15 23:40:20  \"redhat-assisted-installer\"   \n",
       "1        2021-03-15 00:01:06  \"redhat-assisted-installer\"   \n",
       "2        2021-03-13 20:51:32  \"redhat-assisted-installer\"   \n",
       "3        2021-03-13 07:51:20  \"redhat-assisted-installer\"   \n",
       "4        2021-03-13 06:43:20  \"redhat-assisted-installer\"   \n",
       "...                      ...                          ...   \n",
       "27192480 2021-03-14 00:01:00         \"redhat-single-node\"   \n",
       "27192481 2021-03-13 00:01:07         \"redhat-single-node\"   \n",
       "27192482 2021-03-12 04:22:20         \"redhat-single-node\"   \n",
       "27192483 2021-03-11 00:01:18         \"redhat-single-node\"   \n",
       "27192484 2021-03-10 00:01:03         \"redhat-single-node\"   \n",
       "\n",
       "                                                       grid  \\\n",
       "0         periodic-ci-openshift-release-master-nightly-4...   \n",
       "1         periodic-ci-openshift-release-master-nightly-4...   \n",
       "2         periodic-ci-openshift-release-master-nightly-4...   \n",
       "3         periodic-ci-openshift-release-master-nightly-4...   \n",
       "4         periodic-ci-openshift-release-master-nightly-4...   \n",
       "...                                                     ...   \n",
       "27192480  periodic-ci-openshift-release-master-nightly-4...   \n",
       "27192481  periodic-ci-openshift-release-master-nightly-4...   \n",
       "27192482  periodic-ci-openshift-release-master-nightly-4...   \n",
       "27192483  periodic-ci-openshift-release-master-nightly-4...   \n",
       "27192484  periodic-ci-openshift-release-master-nightly-4...   \n",
       "\n",
       "                                                       test  flake  \n",
       "0                                                   Overall  False  \n",
       "1                                                   Overall  False  \n",
       "2                                                   Overall  False  \n",
       "3                                                   Overall  False  \n",
       "4                                                   Overall  False  \n",
       "...                                                     ...    ...  \n",
       "27192480  openshift-tests.[sig-arch] Monitor cluster whi...   True  \n",
       "27192481  openshift-tests.[sig-arch] Monitor cluster whi...   True  \n",
       "27192482  openshift-tests.[sig-arch] Monitor cluster whi...   True  \n",
       "27192483  openshift-tests.[sig-arch] Monitor cluster whi...  False  \n",
       "27192484  openshift-tests.[sig-arch] Monitor cluster whi...  False  \n",
       "\n",
       "[27192485 rows x 5 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Sanity check to see if the dataset is the same\n",
    "if AUTOMATION == \"True\":\n",
    "    sanity_check = cc.read_from_ceph(metric_path, filename)\n",
    "else:\n",
    "    sanity_check = read_from_disk(OUTPUT_DATA_PATH, filename)\n",
    "\n",
    "sanity_check"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "This notebook computed number of flakes and the flake severity metric. The dataframe saved on ceph can be used to generate aggregated views and visualizations."
   ]
  }
 ],
 "metadata": {
  "finalized": {
   "timestamp": 1616083168568,
   "trusted": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}