docs/events_generation_walk-through.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3267aeeb",
   "metadata": {},
   "source": [
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/elastic/geneve/main?labpath=docs/events_generation_walk-through.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "813e2182",
   "metadata": {},
   "source": [
    "# Rule-based events generation walk-through\n",
    "\n",
    "You can access an interactive version of this document by clicking on the \"launch binder\" badge above. You will be able to edit the `In [ ]` cells below by clicking in the grey area and executing them by pressing `Ctrl+Enter`.\n",
    "\n",
    "## Preliminaries\n",
    "\n",
    "The API for generating events is exposed by module `geneve.events_emitter`, the `SourceEvents` class provides the front-end for the most common use cases.\n",
    "\n",
    "Here the module imports needed for the rest of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b99e3b39",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipynb  # needed only in Jupiter notebooks\n",
    "\n",
    "from geneve.events_emitter import SourceEvents\n",
    "from geneve.utils import load_schema, load_rules\n",
    "\n",
    "from itertools import islice\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "163376b2",
   "metadata": {},
   "source": [
    "## Schema\n",
    "\n",
    "Generating documents requires a specification of the fields types, such as `long` or `ip`. This is the duty of the _schema_.\n",
    "\n",
    "An example of schema defining fields `x`, `y`, and `z` as float numbers is `{'x': {'type': 'float'}, 'y': {'type': 'float'}, 'z': {'type': 'float'}}`. Fields not defined in the schema are considered of type `keyword`, a kind of string.\n",
    "\n",
    "The default schema is just `{}`, therefore every field is considered of type `keyword`. To change that, assign to `SourceEvents.schema` as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "09c0fa46",
   "metadata": {},
   "outputs": [],
   "source": [
    "SourceEvents.schema = {'x': {'type': 'float'}, 'y': {'type': 'float'}, 'z': {'type': 'float'}}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9c4befb",
   "metadata": {},
   "source": [
    "From now on every document generator will use the assigned schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "27150fbd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'x': {'type': 'float'}, 'y': {'type': 'float'}, 'z': {'type': 'float'}}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = SourceEvents()\n",
    "a.schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cfa62106",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'x': {'type': 'float'}, 'y': {'type': 'float'}, 'z': {'type': 'float'}}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = SourceEvents()\n",
    "b.schema"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efb9fc4c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It's also possible to specify the schema on a per-case basis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e1b62866",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'x': {'type': 'float'}}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = SourceEvents(schema={'x': {'type': 'float'}})\n",
    "c.schema"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4ba83d4",
   "metadata": {},
   "source": [
    "The rest of the notebook assumes the rich and standardized [ECS](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) 8.2.0 as default schema, as loaded below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2bdab68f",
   "metadata": {},
   "outputs": [],
   "source": [
    "SourceEvents.schema = load_schema('./etc/ecs-8.2.0.tar.gz', 'generated/ecs/ecs_flat.yml')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f0bfedc",
   "metadata": {},
   "source": [
    "## Generate documents from queries\n",
    "\n",
    "In the simplest form, documents can be generated as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "64c04a1e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Event(meta=None, doc={'process': {'name': 'Okb.exe'}, '@timestamp': '2023-02-06T11:54:08.391+01:00'})]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(SourceEvents.from_query('process.name : *.exe'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0285c889",
   "metadata": {},
   "source": [
    "Which is equivalent to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5211efa3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Event(meta=None, doc={'process': {'name': 'zDCtQRQAWVhZQ.exe'}, '@timestamp': '2023-02-06T11:54:08.397+01:00'})]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "se = SourceEvents()\n",
    "se.add_query('process.name : *.exe')\n",
    "next(se)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8acc87aa",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this second form, multiple queries can be added and the generated documents shall match either of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "df5ff1a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Event(meta=None, doc={'process': {'name': 'UlX.exe'}, '@timestamp': '2023-02-06T11:54:08.408+01:00'})]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "se = SourceEvents()\n",
    "se.add_query('process.name : *.exe')\n",
    "se.add_query('source.ip : 10.0.0.0/8')\n",
    "next(se)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56b7fdaf",
   "metadata": {},
   "source": [
    "The query language is guessed automatically. While the above are Kuery based, what follows uses EQL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7df95cc7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Event(meta=None, doc={'event': {'category': ['process']}, 'process': {'name': 'eXomVOjfNESLV.exe'}, '@timestamp': '2023-02-06T11:54:08.423+01:00'})]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "se = SourceEvents()\n",
    "se.add_query('process where process.name : \"*.exe\"')\n",
    "next(se)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d09b6fa4",
   "metadata": {},
   "source": [
    "Currently only Kuery and EQL are supported though others will be added.\n",
    "\n",
    "Note how `next(se)` returns an array of `Event(..)` objects. It's an array because the query could result in the generation of multiple events, think at the `sequence` EQL queries.\n",
    "\n",
    "The `Event(..)` object is used instead of a simple document so to wrap also meta data passed at the time of query insertion, ex. `add_query(.., meta=<your meta data>)`.\n",
    "\n",
    "A way to unpack the documents is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "88f96f3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'event': {'category': ['process']},\n",
       "  'process': {'name': 'yO.exe'},\n",
       "  '@timestamp': '2023-02-06T11:54:08.429+01:00'}]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[event.doc for event in next(se)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb3dfa0c",
   "metadata": {},
   "source": [
    "## Generate documents from rules\n",
    "\n",
    "Similarily to the query cases, it's possible to generate documents from one or more rules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d36a18ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Event(meta=None, doc={'destination': {'port': 26}, 'event': {'category': ['network_traffic'], 'dataset': 'zeek.smtp'}, 'network': {'transport': 'tcp'}, '@timestamp': '2023-02-06T11:54:09.161+01:00'})]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use the local copy of detection-rules 8.2.0\n",
    "rules = load_rules('./etc/detection-rules-8.2.0.tar.gz', (\n",
    "    'rules/network/command_and_control_port_26_activity.toml',\n",
    "    'rules/network/command_and_control_telnet_port_activity.toml',\n",
    "))\n",
    "\n",
    "# load the rules\n",
    "se = SourceEvents()\n",
    "for rule in rules:\n",
    "    se.add_rule(rule)\n",
    "\n",
    "# generate one document\n",
    "next(se)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c95cfcc",
   "metadata": {},
   "source": [
    "## Generate documents over and over\n",
    "\n",
    "Adding queries and rules to a `SourceEvents` object triggers some operations like parsing the query, collecting the field constraints, trying to generate documents for the first time.\n",
    "\n",
    "When it's important to avoid unnecessary computations, the result of such initial operations can be reused by preserving the `SourceEvents` object between the calls to `next`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "471d7196",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[Event(meta=None, doc={'process': {'name': 'UW.dll'}, '@timestamp': '2023-02-06T11:54:09.170+01:00'})],\n",
       " [Event(meta=None, doc={'process': {'name': 'QcPNdamyOhK.dll'}, '@timestamp': '2023-02-06T11:54:09.170+01:00'})],\n",
       " [Event(meta=None, doc={'process': {'name': 'IUOqztL.dll'}, '@timestamp': '2023-02-06T11:54:09.170+01:00'})],\n",
       " [Event(meta=None, doc={'process': {'name': 'LXiXjZzvOtUOmZe.dll'}, '@timestamp': '2023-02-06T11:54:09.170+01:00'})],\n",
       " [Event(meta=None, doc={'process': {'name': 'PzWTf.exe'}, '@timestamp': '2023-02-06T11:54:09.170+01:00'})]]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "se = SourceEvents.from_query('process.name : (*.exe or *.dll)')\n",
    "\n",
    "[next(se) for n in range(5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1973fc1",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Mappings of the generated documents\n",
    "\n",
    "The `SourceEvents` object can build mappings describing all the fields used in the loaded queries and rules. For this task it's employed the same schema used for the documents generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9084932b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'properties': {'@timestamp': {'type': 'date'},\n",
       "  'network': {'properties': {'transport': {'type': 'keyword'}}},\n",
       "  'destination': {'properties': {'port': {'type': 'long'}}},\n",
       "  'event': {'properties': {'category': {'type': 'keyword'},\n",
       "    'dataset': {'type': 'keyword'}}}}}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use the local copy of detection-rules 8.2.0\n",
    "rules = load_rules('./etc/detection-rules-8.2.0.tar.gz', (\n",
    "    'rules/network/command_and_control_port_26_activity.toml',\n",
    "    'rules/network/command_and_control_telnet_port_activity.toml',\n",
    "))\n",
    "\n",
    "# load the rules\n",
    "se = SourceEvents()\n",
    "for rule in rules:\n",
    "    se.add_rule(rule)\n",
    "\n",
    "# generate the mappings\n",
    "se.mappings()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a13b6ef3",
   "metadata": {},
   "source": [
    "## Query validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0acb36c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unsolvable constraints: destination.port (empty solution space, 1025 <= x <= 1023)\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    SourceEvents.from_query('destination.port < 1024 and (destination.port > 512 or destination.port > 1024)')\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "437ee16e",
   "metadata": {},
   "source": [
    "## Using as iterator\n",
    "\n",
    "The `SourceEvents` class implements the iterator protocol, as the usage of `next` for generating new documents anticipated. Because documents can be generated indefinitely, the iterator is infinite and therefore some care is needed.\n",
    "\n",
    "Uses like `list(se)` or `for docs in se: print(docs)` are troublesome, the first would sooner or later exhaust all the resources, the second would never terminate spontaneously. You then need to ask yourself how many documents you need or what other conditions will break the loop.\n",
    "\n",
    "As example, this prints 10 documents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "40b51f95",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Event(meta=None, doc={'process': {'name': 'aAiUtZuFzS.exe'}, '@timestamp': '2023-02-06T11:54:09.198+01:00'})]\n",
      "[Event(meta=None, doc={'process': {'name': 'exGSt.exe'}, '@timestamp': '2023-02-06T11:54:09.199+01:00'})]\n",
      "[Event(meta=None, doc={'process': {'name': 'fHegHLwRFdRU.exe'}, '@timestamp': '2023-02-06T11:54:09.199+01:00'})]\n",
      "[Event(meta=None, doc={'process': {'name': 'NQUfbtQnvOfc.exe'}, '@timestamp': '2023-02-06T11:54:09.199+01:00'})]\n",
      "[Event(meta=None, doc={'process': {'name': 'RNMkiiucklGs.exe'}, '@timestamp': '2023-02-06T11:54:09.199+01:00'})]\n",
      "[Event(meta=None, doc={'process': {'name': 'CjwPWZufhZj.exe'}, '@timestamp': '2023-02-06T11:54:09.199+01:00'})]\n",
      "[Event(meta=None, doc={'process': {'name': 'BaZ.dll'}, '@timestamp': '2023-02-06T11:54:09.199+01:00'})]\n",
      "[Event(meta=None, doc={'process': {'name': 'BpuTSHUHPLzr.dll'}, '@timestamp': '2023-02-06T11:54:09.199+01:00'})]\n",
      "[Event(meta=None, doc={'process': {'name': 'XzxmYFKtCcN.exe'}, '@timestamp': '2023-02-06T11:54:09.199+01:00'})]\n",
      "[Event(meta=None, doc={'process': {'name': 'AxvQuDFINxxM.exe'}, '@timestamp': '2023-02-06T11:54:09.199+01:00'})]\n"
     ]
    }
   ],
   "source": [
    "se = SourceEvents.from_query('process.name : (*.exe or *.dll)')\n",
    "\n",
    "for docs in islice(se, 10):\n",
    "    print(docs)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}