RootInteractive/tutorial/bokehDraw/compression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f4412757",
   "metadata": {},
   "source": [
    "# Lossy compression\n",
    "\n",
    "Lossy and lossles compression is very important feature i the RootInteractive.\n",
    "\n",
    "Depending on the layout of the data, a different compression factor can be achieved. Huge reduction factors can be achieved, partly due to **the entropy of the data** (e.g. Gaussian distribution), partly due to the **repetitions in the case of flattened arrays.**\n",
    "\n",
    "For example, in the real use case of dEdx simulation (clusters per track), the common track properties are very well compressed, the charge properties also have a small entry factor. Normally, the factor **O(10-20)- 5-10% is reached** - depending on the repetition and the entropy of input data.\n",
    "\n",
    "\n",
    "In the code below you can find how to parametrize lossy ans lossless comprsssion of the data.\n",
    "\n",
    "Example use case for Q vectors along the track:\n",
    "\n",
    "```\n",
    "compressCDSPipe\n",
    "Compresses 1 dNprimdx .* [('relative', 16), 'code', 'zip']\n",
    "Compression factor 1502730 33602297 0.04472104987346549 1 dNprimdx\n",
    "Compress 2 qVector .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 3637456 27522312 0.13216389669588804 2 qVector\n",
    "Compress 3 region .* [('relative', 16), 'code', 'zip']\n",
    "Compression factor 579220 27522277 0.02104549707133607 3 region\n",
    "Compress 4 qMean .* [('relative', 16), 'code', 'zip']\n",
    "Compress factor 1485573 33602294 0.04421046372607775 4 qMean\n",
    "Compress 5 nTotVector .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 3336317 27522315 0.1212222518345568 5 nTotVector\n",
    "Compress 6 nPrimMean .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 1502830 33602298 0.04472402452951283 6 nPrimMean\n",
    "Compress 7 qStd .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 1474259 33602293 0.04387376182928945 7 qStd\n",
    "Compress 8 nTotStd .* [('relative', 16), 'code', 'zip']\n",
    "Compress factor 1488320 33602296 0.044292211460788274 8 nTotStd\n",
    "Compress 9 nTotMean .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 1499257 33602297 0.04461769384396549 9 nTotMean\n",
    "Compress 10 TransGEM .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 1461148 33602297 0.04348357494727221 10 TransGEM\n",
    "Compress 11 nPrimStd .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 1487282 33602297 0.044261319397301914 11 nPrimStd\n",
    "Compress 12 padLength .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 595667 27522314 0.021643056612172945 12 padLength\n",
    "Compress 13 nPrimVector .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 2779472 27522316 0.10098975682133728 13 nPrimVector\n",
    "Compress 14 lognPrimStd .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 1485884 33602300 0.0442197111507248 14 lognPrimStd\n",
    "Compress 15 SatOn .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 150691 22962324 0.006562532607762176 15 SatOn\n",
    "Compress 16 nSecSatur .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 1725854 33602298 0.05136118964244648 16 nSecSatur\n",
    "Compress 17 logqStd .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 1470552 33602296 0.04376343806982713 17 logqStd\n",
    "Compress 18 lognTotStd .* [('relative', 16), 'code', 'zip']\n",
    "Compress Factor 1484058 33602299 0.04416537094679147 18 lognTotStd\n",
    "Compress 19 lognSecSatur .* [('relative', 16), 'code', 'zip']\n",
    "Compress factor 1474271 33602301 0.043874108502271914 19 lognSecSatur\n",
    "Compress 20 region.factor() .* [('relative', 16), 'code', 'zip']\n",
    "Compress factor 553452 6080146 0.09102610364948473 20 region.factor()\n",
    "Compress 21 SatOn.factor() .* [('relative', 16), 'code', 'zip']\n",
    "Compress factor 197180 6080146 0.03243014230250392 21 SatOn.factor()\n",
    "Compress _all 31371473 609564013 0.051465428291285954 21\n",
    "\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bfaaede",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bokeh.io import output_notebook, show\n",
    "from bokeh.plotting import output_file\n",
    "from RootInteractive.InteractiveDrawing.bokeh.bokehDrawSA import bokehDrawSA\n",
    "from RootInteractive.InteractiveDrawing.bokeh.bokehTools import bokehDrawArray\n",
    "from RootInteractive.Tools.pandaTools import initMetadata\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import math\n",
    "import logging\n",
    "output_notebook()\n",
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:100% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c28b52b",
   "metadata": {},
   "source": [
    "## Generate data\n",
    "* A and B from normal distribution\n",
    "* C from uniform \\[0, 1\\]\n",
    "* D Bernoulli distribution\n",
    "\n",
    "* Add derived variables - two from normal distribution, one approximately exponential depending on A,B,C,D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e7e4e75",
   "metadata": {},
   "outputs": [],
   "source": [
    "npoints = 1000000\n",
    "A = np.random.randn(npoints)\n",
    "B = np.random.randn(npoints)\n",
    "unifC = np.random.random_sample(npoints)\n",
    "boolD = np.random.random_sample(npoints) > .47\n",
    "derivedE = A+boolD*(A*.15-B*.4+.1)+.1*np.random.randn(npoints)\n",
    "derivedF = np.random.exponential(1/((derivedE**2)+(np.sin(2*math.pi*unifC)+1.4)))\n",
    "derivedG = 100+15*A+2*np.random.randn(npoints)\n",
    "df = pd.DataFrame({\"A\":A,\"B\":B,\"unifC\":unifC,\"boolD\":boolD,\"derivedE\":derivedE,\"derivedF\":derivedF, \"derivedG\":derivedG})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "249ebfc7",
   "metadata": {},
   "source": [
    "## Make figures and selection widgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a1afd7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameterArray = [\n",
    "    {\"name\": \"size\", \"value\":7, \"range\":[0, 30]},\n",
    "    {\"name\": \"legendFontSize\", \"value\":\"13px\", \"options\":[\"9px\", \"11px\", \"13px\", \"15px\"]},\n",
    "    {\"name\": \"legendVisible\", \"value\":True},\n",
    "    {\"name\": \"nPointRender\", \"range\":[0, 5000], \"value\": 1000},\n",
    "]\n",
    "figureArray = [\n",
    "    [['derivedG'], ['derivedE'], {\"colorZvar\": \"B\"}],\n",
    "    [['derivedE'], ['A','B']],\n",
    "    [['unifC'], ['derivedF'], { \"colorZvar\": \"derivedG\"}],\n",
    "    [['derivedF'], ['derivedG'], {\"colorZvar\": \"derivedE\", \"errY\": \"10*A\"} ],\n",
    "    [['A'], ['B'], {\"colorZvar\": \"derivedF\"}],\n",
    "    {\"size\":\"size\", \"legend_options\": {\"label_text_font_size\": \"legendFontSize\", \"visible\":\"legendVisible\"}}\n",
    "]\n",
    "layout = {\n",
    "    \"A\": [\n",
    "        [0, 1, 2, {'y_visible': 1, 'x_visible':1, 'plot_height': 300}],\n",
    "        {'plot_height': 100, 'sizing_mode': 'scale_width', 'y_visible' : 2}\n",
    "        ],\n",
    "    \"B\": [\n",
    "        [3, 4, {'y_visible': 3, 'x_visible':1, 'plot_height': 300}],\n",
    "        {'plot_height': 100, 'sizing_mode': 'scale_width', 'y_visible' : 2}\n",
    "        ]\n",
    "}\n",
    "widgetParams=[\n",
    "    ['range', ['A']],\n",
    "    ['range', ['B']],\n",
    "    ['range', ['unifC']],\n",
    "    ['multiSelect', ['boolD']],\n",
    "    ['range', ['derivedE']],\n",
    "    ['spinnerRange', ['derivedF']],\n",
    "    ['range',[\"derivedG\"]],\n",
    "    ['toggle',['legendVisible'], {\"name\":\"legendVisible\"}],\n",
    "    ['select',['legendFontSize'], {\"name\":\"legendSize\"}],\n",
    "    ['slider',['size'], {\"name\":\"markerSize\"}],\n",
    "    ['slider',['nPointRender'], {\"name\":\"nPoint\"}]\n",
    "]\n",
    "widgetLayoutDesc={\n",
    "    \"Selection\": [[0, 1, 2], [3, 4], [5, 6], {'sizing_mode': 'scale_width'}],\n",
    "    \"Graphics\": [[\"legendVisible\", \"nPoint\"],[\"legendSize\", \"markerSize\"]]\n",
    "    }   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57e7afa2",
   "metadata": {},
   "source": [
    "* Optimization\n",
    "    * Compress the data\n",
    "        * bokehDrawArray (and bokehDrawSA) take an arrayCompression parameter, which is a list of (regex, pipeline) pairs, where regex is the regular expression used to match column names\n",
    "          and pipeline is a list of operations to be used on the column. Supported values are \"relative\", \"delta\", \"zip\" and \"base64\" \n",
    "        * Example: \n",
    "            ``arrayCompressionParam = [\n",
    "            (\".conv.Sigma.*\",[(\"relative\",7), \"code\", \"zip\"]), \n",
    "            (\".delta.\",[(\"relative\",10), \"code\", \"zip\"]), \n",
    "            (\".i2.\",[(\"relative\",7), \"code\", \"zip\"\"]), \n",
    "            (\".*\",[(\"relative\",8), \"code\", \"zip\"])]``\n",
    "            * Variables will be compressed in the given order. Once a variable was compressed, it will not be overwritten by another compression.\n",
    "            * Tuple paramters: `(\".conv.Sigma.*\",[(\"relative\",7), \"code\", \"zip\"])`\n",
    "                * first parameter is a regex expression to match the column names to be compressed\n",
    "                * second parameter is a list of operation to be used on the column\n",
    "                    * most relevant for the user is the first parameter of the list which defines the quantization\n",
    "                        * \"absolute\": precision to be used in absolute units of the given variable, e.g. 0.0001\n",
    "                        * \"relative\": precision to be used in units of bits, e.g. 10\n",
    "                    * \"code\", \"zip\"\n",
    "                        * lossless compression\n",
    "                        * code - factor the column into \"codes\" and \"factors\" - two columns\n",
    "                        * at the time of writing this tutorial \"code\" - factoring the columns - resulted in suboptial compression because of a bug that will be fixed soon - factors aren't encoded properly\n",
    "                        * zip - compress using gzip\n",
    "                    * \"base64\"\n",
    "                        * base64 encoding - as of the current version it's automatically used where appropriate, there should be no need to use this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe1c29fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "arrayCompression = [\n",
    "    (\"unif.*\", [(\"delta\", .01), \"zip\"]),\n",
    "    (\"bool.*\", [\"zip\"]),\n",
    "    (\".*\", [(\"relative\", 16), \"zip\"]),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bd4a238",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_file(\"test_compression.html\")\n",
    "bokehDrawSA.fromArray(df, None, figureArray, widgetParams, layout=layout,\n",
    "                            widgetLayout=widgetLayoutDesc, nPointRender=\"nPointRender\", parameterArray=parameterArray, arrayCompression=arrayCompression, useNotebook=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29bd95d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_file(\"test_nocompression.html\")\n",
    "bokehDrawSA.fromArray(df, None, figureArray, widgetParams, layout=layout,\n",
    "                            widgetLayout=widgetLayoutDesc, nPointRender=\"nPointRender\", parameterArray=parameterArray, useNotebook=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ee0dfb2",
   "metadata": {},
   "source": [
    "## Option \"code\" in compressArray\n",
    "* to be optimized  in some cases it improve in some cases not - to be fixed in next release"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c99c06fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "arrayCompression = [\n",
    "    (\"unif.*\", [(\"delta\", .01), \"code\", \"zip\"]),\n",
    "    (\"bool.*\", [\"zip\"]),\n",
    "    (\".*\", [(\"relative\", 16), \"code\",\"zip\"]),\n",
    "]\n",
    "output_file(\"test_compression_code.html\")\n",
    "bokehDrawSA.fromArray(df, None, figureArray, widgetParams, layout=layout,\n",
    "                            widgetLayout=widgetLayoutDesc, nPointRender=\"nPointRender\", parameterArray=parameterArray, arrayCompression=arrayCompression, useNotebook=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2622a5db",
   "metadata": {},
   "source": [
    "## IN this particular case factor 34 % compression accheved\n",
    "```\n",
    "-rw-r--r-- 1 miranov alice 27494089 Dec  6 20:19 test_compression.html\n",
    "-rw-r--r-- 1 miranov alice 80092088 Dec  6 20:19 test_nocompression.html\n",
    "-rw-r--r-- 1 miranov alice 27494247 Dec  6 20:19 test_compression_code.html\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f61ee7fb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}