misp-playbooks/pb_bulk_delete_events-with_output.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ca1fbfb5",
   "metadata": {
    "tags": [
     "playbook-introduction"
    ]
   },
   "source": [
    "# Bulk delete MISP events\n",
    "\n",
    "## Introduction\n",
    "\n",
    "- UUID: **7d3fd7fe-fe4b-48a3-8e7e-8bd4e4bf2f0d**\n",
    "- Started from [issue 29](https://github.com/MISP/misp-playbooks/issues/29)\n",
    "- State: **Published** : demo version with **output**\n",
    "- Purpose: A playbook to assist MISP users in doing bulk deletes of MISP events.\n",
    "    - Deletes are done for events created by **organisations**, for events before or after specific **dates**, **published** or unpublished events or for events with specific **tags**.\n",
    "    - A summary of the actions is printed and published on Mattermost.\n",
    "- Tags: [ \"delete\", \"clean\", \"curation\"]\n",
    "- External resources: **Mattermost**\n",
    "- Target audience: **CTI**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a512a2c1",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "# Playbook\n",
    "\n",
    "- **Bulk delete MISP events**\n",
    "    - Introduction\n",
    "- **Preparation**\n",
    "    - PR:1 Initialise environment\n",
    "    - PR:2 Load helper functions\n",
    "- **Identify events**\n",
    "    - SR:1 Execution flow\n",
    "    - SR:2 Search options\n",
    "    - SR:3 Remove from blocklist?\n",
    "- **Delete events**\n",
    "    - DL:1 Delete events created by an organisation\n",
    "    - DL:2 Delete events created or changed after or before a specific date\n",
    "    - DL:3 Delete published or not published events\n",
    "    - DL:4 Delete events with (or without) specific tags\n",
    "- **Closure**\n",
    "    - EN:1 Create the summary of the playbook \n",
    "    - EN:2 Send a summary to Mattermost\n",
    "    - EN:3 End of the playbook \n",
    "- External references\n",
    "- Technical details"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dc24913",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Preparation\n",
    "\n",
    "## PR:1 Initialise environment\n",
    "\n",
    "This section **initialises the playbook environment** and loads the required Python libraries. \n",
    "\n",
    "The credentials for MISP (**API key**) and other services are loaded from the file `keys.py` in the directory **vault**. A [PyMISP](https://github.com/MISP/PyMISP) object is created to interact with MISP and the active MISP server is displayed. By printing out the server name you know that it's possible to connect to MISP. In case of a problem PyMISP will indicate the error with `PyMISPError: Unable to connect to MISP`.\n",
    "\n",
    "The contents of the `keys.py` file should contain at least :\n",
    "\n",
    "```\n",
    "misp_url=\"<MISP URL>\"                  # The URL to our MISP server\n",
    "misp_key=\"<MISP API KEY>\"              # The MISP API key\n",
    "misp_verifycert=<True or False>        # Ignore certificate errors\n",
    "mattermost_playbook_user=\"<MATTERMOST USER>\"\n",
    "mattermost_hook=\"<MATTERMOST WEBHOOK>\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "5f94c336",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The version of PyMISP recommended by the MISP instance (2.4.176) is newer than the one you're using now (2.4.173). Please upgrade PyMISP.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The \u001b[92mPython libraries\u001b[90m are loaded and the \u001b[92mcredentials\u001b[90m are read from the keys file.\n",
      "I will use the MISP server \u001b[92mhttps://misp.demo.cudeso.be/\u001b[90m for this playbook.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Initialise Python environment\n",
    "import urllib3\n",
    "import sys\n",
    "import json\n",
    "from pyfaup.faup import Faup\n",
    "from prettytable import PrettyTable, MARKDOWN\n",
    "from IPython.display import Image, display, display_markdown, HTML\n",
    "from datetime import date\n",
    "import requests\n",
    "import uuid\n",
    "from uuid import uuid4\n",
    "from pymisp import *\n",
    "from pymisp.tools import GenericObjectGenerator\n",
    "import re\n",
    "import time\n",
    "from datetime import datetime\n",
    "\n",
    "# Load the credentials\n",
    "sys.path.insert(0, \"../vault/\")\n",
    "from keys import *\n",
    "if misp_verifycert is False:\n",
    "    import urllib3\n",
    "    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)\n",
    "print(\"The \\033[92mPython libraries\\033[90m are loaded and the \\033[92mcredentials\\033[90m are read from the keys file.\")\n",
    "\n",
    "# Create the PyMISP object\n",
    "misp = PyMISP(misp_url, misp_key, misp_verifycert)\n",
    "print(\"I will use the MISP server \\033[92m{}\\033[90m for this playbook.\\n\\n\".format(misp_url))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79acf8c2-0863-4c3e-a309-91b5a7e632a3",
   "metadata": {},
   "source": [
    "## PR:2 Load helper functions\n",
    "\n",
    "The next cell contains **helper functions** that are used in this playbook. \n",
    "\n",
    "Instead of distributing helper functions as separate Python files this playbook includes all the required code as one code cell. This makes portability of playbooks between instances easier. The downside is that functions defined in this playbook need to be defined again in other playbooks, which is not optimal for code re-use. For this iteration of playbooks it is chosen to include the code in the playbook (more portability), but you can easily create one \"helper\" file that contains all the helper code and then import that file in each playbook (for example by adding to the previous cell `from helpers import *`). Note that the graphical workflow image is included as an external image. A missing image would not influence the further progress of the playbook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b57b5779-181f-4271-871b-f56282bbbb4e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def table_print_before_delete(event_list, return_str=False):\n",
    "    '''\n",
    "    Table to print with the events from event_list that are about to be deleted\n",
    "    '''    \n",
    "    delete_events_table = PrettyTable()\n",
    "    delete_events_table.field_names = [\"List ID\", \"Event ID\", \"Org\", \"Info\", \"Date\", \"Published\"]\n",
    "    delete_events_table.align[\"ID\"] = \"l\"\n",
    "    delete_events_table.align[\"Org\"] = \"l\"\n",
    "    delete_events_table.align[\"Info\"] = \"l\"    \n",
    "    delete_events_table._max_width = {\"Org\": 30, \"Info\": 30}\n",
    "    position = 0\n",
    "    for event in event_list:\n",
    "        delete_events_table.add_row([position, event.id, event.orgc.name, event.info, event.date, event.published])\n",
    "        position += 1\n",
    "    if not return_str:\n",
    "        print(delete_events_table.get_string(sortby=\"List ID\"))\n",
    "    else:\n",
    "        return delete_events_table\n",
    "\n",
    "\n",
    "\n",
    "def delete_events(event_list):\n",
    "    '''\n",
    "    Do the delete of the events in event_list\n",
    "    '''    \n",
    "    print(\"Starting with delete ...\")\n",
    "    for event in event_list:\n",
    "        print(\" Delete \\033[91m{}\\033[90m (ID: {}, UUID: {})\".format(event.info, event.id, event.uuid))\n",
    "        result = misp.delete_event(event.uuid)\n",
    "        if not \"errors\" in result:\n",
    "            if remove_from_event_block_list:\n",
    "                result = misp.delete_event_blocklist(event.uuid)\n",
    "    print(\"Delete finished\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0277c63-c3d0-4077-a99d-f3c4980dae11",
   "metadata": {},
   "source": [
    "# Identify events\n",
    "\n",
    "## SR:1 Execution flow\n",
    "\n",
    "The next sections use a similar approach:\n",
    "\n",
    "1. First **search** for the events;\n",
    "2. **List** and **review** the results;\n",
    "3. Do the actual **delete**.\n",
    "\n",
    "## SR:2 Search options\n",
    "\n",
    "First **search** (or identify) the events that you'd like to delete. The following sections provide different ways to search for these events. You can \n",
    "- Delete events created by an **organisation**\n",
    "- Delete events before or after a specific **date**\n",
    "- Delete **published** or **not-published** events\n",
    "- Delete events with or without specific **tags**\n",
    "\n",
    "## SR:3 Remove from blocklist?\n",
    "\n",
    "When an event is deleted in MISP, it's unique identifier (the UUID) is put on the [MISP event blocklist](https://www.circl.lu/doc/misp/administration/#event-blocklist). Blocklisting an event prevents the event from being added again on the instance, for example if you synchronise with external communities or import events from external sources. It's also a feature often overlooked when setting up synchronised MISP servers. During the configuration phase it can confuse administrators that some events are added, and others are skipped for no apparent reason. As a rule of thumb, if you want to delete the event, and never receive an update again, then put it on the blocklist. If you are configuring and testing the synchronisation process then remove the event from the blocklist after deleting it. \n",
    "\n",
    "In this playbook you can control this behaviour with `remove_from_event_block_list`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "95a78b32-d094-40da-8069-e870414ce6a7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# After deleting the event, also remove it from the blocklist\n",
    "remove_from_event_block_list = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79195b20-70bd-46e9-819f-9e54f42b82fc",
   "metadata": {},
   "source": [
    "# Delete events\n",
    "\n",
    "## DL:1 Delete events created by an organisation\n",
    "\n",
    "This section allows you to delete events that are **created** by a specific **organisation**.\n",
    "\n",
    "### Search\n",
    "\n",
    "You can search for the organisation based on their **name**, organisation **ID** or **UUID**. The playbook attempts to guess the type of input you provided in the variable `org`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "445eb751-ad0b-4107-9435-b6549434b6e2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found organsation: \u001b[92mabuse.ch\u001b[90m (ID: 2, UUID: 9b086132-8588-49ed-97fd-8578a777822c)\n",
      "Searching for events ...\n",
      " \u001b[92m2\u001b[90m events found.\n"
     ]
    }
   ],
   "source": [
    "# Provide an organisation name, ID or UUID\n",
    "#org = 16\n",
    "org = \"abuse.ch\"\n",
    "#org = \"55f6ea5e-2c60-40e5-964f-47a8950d210f\"\n",
    "\n",
    "\n",
    "# Code block to search for events\n",
    "org_id = False\n",
    "if isinstance(org, int):\n",
    "    result = misp.get_organisation(org, pythonify=True)\n",
    "    if not \"errors\" in result:\n",
    "        org_id = result.id\n",
    "elif isinstance(org, str):\n",
    "    try:\n",
    "        uuid_obj = uuid.UUID(org, version=4)\n",
    "        result = misp.get_organisation(str(org), pythonify=True)\n",
    "        if not \"errors\" in result:\n",
    "            org_id = result.id\n",
    "    except ValueError:  \n",
    "        organisations = misp.organisations(scope=\"all\", pythonify=True)\n",
    "        for org_search in organisations:\n",
    "            if org_search.name.lower().strip() == org.lower().strip():\n",
    "                org_id = org_search.id\n",
    "                break\n",
    "\n",
    "if org_id:\n",
    "    org_delete_events = misp.get_organisation(org_id, pythonify=True)\n",
    "    print(\"Found organsation: \\033[92m{}\\033[90m (ID: {}, UUID: {})\".format(org_delete_events.name, org_delete_events.id, org_delete_events.uuid))\n",
    "    print(\"Searching for events ...\")\n",
    "    event_list = misp.search(\"events\", org=org_delete_events.id, pythonify=True)\n",
    "    if not \"errors\" in event_list and len(event_list) > 0:\n",
    "        print(\" \\033[92m{}\\033[90m events found.\".format(len(event_list)))\n",
    "    else:\n",
    "        print(\" No \\033[91mevents\\033[90m found for this organisation.\")\n",
    "else:\n",
    "    print(\"Unable to find organisation with identifier \\033[91m{}\\033[90m\".format(org))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da797034-07a0-48f8-9108-6fa217f93036",
   "metadata": {},
   "source": [
    "### List and review\n",
    "\n",
    "Print the **list** of events stored in `event_list`.\n",
    "\n",
    "**Remove** events from the list with `pop`. For example `event_list.pop(2)` removes the event with **List ID** two (we start counting at zero, so it's actually the *third* element) from the event list. Do not confuse List ID with the Event ID.\n",
    "\n",
    "You can execute these cells multiple times to review and edit the results, before doing the actual delete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a0683c5d-2257-4576-abf1-19b06e4b0ebb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+----------+----------+-------------------------------+------------+-----------+\n",
      "| List ID | Event ID | Org      | Info                          |    Date    | Published |\n",
      "+---------+----------+----------+-------------------------------+------------+-----------+\n",
      "|    0    |   3157   | abuse.ch | ThreatFox IOCs for 2022-10-02 | 2022-10-02 |    True   |\n",
      "|    1    |   3288   | abuse.ch | ThreatFox IOCs for 2022-05-17 | 2022-05-17 |    True   |\n",
      "+---------+----------+----------+-------------------------------+------------+-----------+\n"
     ]
    }
   ],
   "source": [
    "# Print out the list of events before deleting them\n",
    "table_print_before_delete(event_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ec9a4867-69f5-43f7-99c0-4f52d40526c3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<MISPEvent(info=ThreatFox IOCs for 2022-05-17)\n",
      "There are \u001b[92m1\u001b[90m events found.\n"
     ]
    }
   ],
   "source": [
    "# Example on how to remove one element from the list\n",
    "print(event_list.pop(1))\n",
    "print(\"There are \\033[92m{}\\033[90m events found.\".format(len(event_list)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8990c2dc-e202-4f3c-82c9-1b8cd86ef91f",
   "metadata": {},
   "source": [
    "### Delete\n",
    "\n",
    "The next cell deletes the events from your MISP instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b6f0831e-e04c-42be-9548-3f7aff8fbef8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting with delete ...\n",
      " Delete \u001b[91mThreatFox IOCs for 2022-10-02\u001b[90m (ID: 3157, UUID: c247795b-f71a-493e-8f1d-60facce9f31f)\n",
      "Delete finished\n"
     ]
    }
   ],
   "source": [
    "delete_events(event_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00467a89-9af7-402f-a05c-ef7a187ab690",
   "metadata": {},
   "source": [
    "## DL:2 Delete events created or changed after or before a specific date\n",
    "\n",
    "This section allows you to delete events created, changed or published before or after a specific **date**.\n",
    "\n",
    "### Search\n",
    "\n",
    "There are multiple ways to identify events based on their date. If you are not familiar with the different dates and timestamps that are part of MISP events then it's best to check out the [Using timestamps in MISP](https://github.com/MISP/misp-playbooks/blob/main/misp-playbooks/pb_using_timestamps_in_MISP-with_output.ipynb) playbook before continuing.\n",
    "\n",
    "- You can search for recently **published** events with the `publish_timestamp`. \n",
    "    - Use `publish_timestamp=\"10d\"` (publish timestamp of 10d or less) for a specific date or `publish_timestamp=[\"3d\",\"10d\"]` (publish timestamp between 3 and 10 days ago) for a time range.\n",
    "- Search for recently **changed** events with `timestamp`. \n",
    "    - The syntax is similar as with published events. Both `timestamp=\"3d\"` and `timestamp=[\"3d\",\"10d\"]` are valid.\n",
    "- Use the **event date** field with `date_from` and `date_to`. This field expects a YYYY-MM-DD format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "461fe012-54e2-4efd-8ae9-d99825cb4825",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching for events ...\n",
      " \u001b[92m4\u001b[90m events found.\n"
     ]
    }
   ],
   "source": [
    "print(\"Searching for events ...\")\n",
    "event_list = misp.search(\"events\", date_from=\"2020-12-13\", date_to=\"2020-12-14\", pythonify=True)\n",
    "#event_list = misp.search(\"events\", timestamp=\"3d\", pythonify=True)\n",
    "#event_list = misp.search(\"events\", publish_timestamp=[\"3d\",\"10d\"], pythonify=True)\n",
    "\n",
    "# Search for the events\n",
    "if not \"errors\" in event_list and len(event_list) > 0:\n",
    "    print(\" \\033[92m{}\\033[90m events found.\".format(len(event_list)))\n",
    "else:\n",
    "    print(\" No \\033[91mevents\\033[90m found for this organisation.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "016eb94a-f012-425d-93f1-5223cd5043f1",
   "metadata": {},
   "source": [
    "### List and review\n",
    "\n",
    "Print the **list** of events stored in `event_list`.\n",
    "\n",
    "**Remove** events from the list with `pop`. For example `event_list.pop(2)` removes the event with **List ID** two (we start counting at zero, so it's actually the *third* element) from the event list. Do not confuse List ID with the Event ID.\n",
    "\n",
    "You can execute these cells multiple times to review and edit the results, before doing the actual delete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4bb26f6a-905e-474f-a4c0-c907fe3479a8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+----------+--------+--------------------------------+------------+-----------+\n",
      "| List ID | Event ID | Org    | Info                           |    Date    | Published |\n",
      "+---------+----------+--------+--------------------------------+------------+-----------+\n",
      "|    0    |   2386   | CIRCL  | OSINT - UNC2452 / SUNBURST     | 2020-12-14 |    True   |\n",
      "|         |          |        | @vxunderground OSINT related   |            |           |\n",
      "|         |          |        | findings                       |            |           |\n",
      "|    1    |   2710   | CUDESO | Highly Evasive Attacker        | 2020-12-14 |    True   |\n",
      "|         |          |        | Leverages SolarWinds Supply    |            |           |\n",
      "|         |          |        | Chain to Compromise Multiple   |            |           |\n",
      "|         |          |        | Global Victims With SUNBURST   |            |           |\n",
      "|         |          |        | Backdoor                       |            |           |\n",
      "|    2    |   2884   | CUDESO | xHunt Campaign: Newly          | 2020-12-13 |    True   |\n",
      "|         |          |        | Discovered Backdoors Using     |            |           |\n",
      "|         |          |        | Deleted Email Drafts and DNS   |            |           |\n",
      "|         |          |        | Tunneling for Command and      |            |           |\n",
      "|         |          |        | Control                        |            |           |\n",
      "|    3    |   2892   | CUDESO | njRAT Spreading Through Active | 2020-12-13 |    True   |\n",
      "|         |          |        | Pastebin Command and Control   |            |           |\n",
      "|         |          |        | Tunnel                         |            |           |\n",
      "+---------+----------+--------+--------------------------------+------------+-----------+\n"
     ]
    }
   ],
   "source": [
    "# Print out the list of events before deleting them\n",
    "table_print_before_delete(event_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c4d9e00f-803d-429d-b01d-3b1d81015711",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<MISPEvent(info=OSINT - UNC2452 / SUNBURST @vxunderground OSINT related findings)\n",
      "There are \u001b[92m3\u001b[90m events found.\n"
     ]
    }
   ],
   "source": [
    "# Example on how to remove one element from the list\n",
    "print(event_list.pop(0))\n",
    "print(\"There are \\033[92m{}\\033[90m events found.\".format(len(event_list)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19beedcd-72fb-48c7-948d-67daa4e932a2",
   "metadata": {},
   "source": [
    "### Delete\n",
    "\n",
    "The next cell deletes the events from your MISP instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "07371fc1-98e7-4a54-b819-b980b555e81d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting with delete ...\n",
      " Delete \u001b[91mHighly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims With SUNBURST Backdoor\u001b[90m (ID: 2710, UUID: 5fd7bbc9-4f68-4057-a6fa-e430c0a8ab16)\n",
      " Delete \u001b[91mxHunt Campaign: Newly Discovered Backdoors Using Deleted Email Drafts and DNS Tunneling for Command and Control\u001b[90m (ID: 2884, UUID: 5fd62412-4cec-450a-a2bc-e52ec0a8ab16)\n",
      " Delete \u001b[91mnjRAT Spreading Through Active Pastebin Command and Control Tunnel\u001b[90m (ID: 2892, UUID: 5fd61f03-15fc-4acd-8a68-e42fc0a8ab16)\n",
      "Delete finished\n"
     ]
    }
   ],
   "source": [
    "delete_events(event_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aae99b1b-fd2a-41fc-99bb-80a7ba3983c8",
   "metadata": {},
   "source": [
    "## DL:3 Delete published or not published events\n",
    "\n",
    "### Search\n",
    "\n",
    "You can combine the previous activities by limiting the search to published or not published events. If you ommit the `published` option from the search query (as is the case for the previous cells), then the playbook ignores the published state of an event.\n",
    "\n",
    "- `published=0` only events that are **not published**\n",
    "- `published=1` only events that are **published**\n",
    "- Do **not add** the `published` option in the search if you want to have both published and not published events.\n",
    "\n",
    "### List and review\n",
    "\n",
    "In this section there's no *list* or *review* activity. Scroll back to the previous cells and add the `published` option to the search query. You can also use the `published` state in the next section as well.\n",
    "\n",
    "As an example, the query `event_list = misp.search(\"events\", date_from=\"2023-07-04\", date_to=\"2023-07-06\", published=1, pythonify=True)` returns the published events between 4 Jul and 6 Jul."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbf7201c-05de-4882-bd1a-251971cfca54",
   "metadata": {},
   "source": [
    "## DL:4 Delete events with (or without) specific tags\n",
    "\n",
    "**Tags** in MISP allow you to specify **contextual** information. You can also use them to identify events you'd like to delete. \n",
    "\n",
    "### Search\n",
    "\n",
    "In the next example we search for events that are not tlp:red ([TLP](https://www.first.org/tlp/)), have the workflow state incomplete and require a review for false positive. The search uses the **build_complex_query** feature of PyMISP to combine logical ORs, ANds and NOTs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e80dc35a-591d-4b92-a94b-7c25497ee39c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching for events ...\n",
      " \u001b[92m3\u001b[90m events found.\n"
     ]
    }
   ],
   "source": [
    "or_parameters = [ ]\n",
    "and_parameters = [ \"workflow:state=\\\"incomplete\\\"\", \"workflow:todo=\\\"review-for-false-positive\\\"\" ]\n",
    "not_parameters = [ \"tlp:red\"]\n",
    "\n",
    "tags = misp.build_complex_query(or_parameters=or_parameters, and_parameters=and_parameters, not_parameters=not_parameters)\n",
    "\n",
    "print(\"Searching for events ...\")\n",
    "event_list = misp.search(\"events\", tags=tags, pythonify=True)\n",
    "# Search for the events\n",
    "if not \"errors\" in event_list and len(event_list) > 0:\n",
    "    print(\" \\033[92m{}\\033[90m events found.\".format(len(event_list)))\n",
    "else:\n",
    "    print(\" No \\033[91mevents\\033[90m found for this organisation.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60b057b1-fe67-4f73-af3e-ecce5b66da8c",
   "metadata": {},
   "source": [
    "### List and review\n",
    "\n",
    "Print the **list** of events stored in `event_list`.\n",
    "\n",
    "**Remove** events from the list with `pop`. For example `event_list.pop(2)` removes the event with **List ID** two (we start counting at zero, so it's actually the *third* element) from the event list. Do not confuse List ID with the Event ID.\n",
    "\n",
    "You can execute these cells multiple times to review and edit the results, before doing the actual delete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "93057975-034a-48f7-a486-6e00be5748b7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+----------+------+--------------------------------+------------+-----------+\n",
      "| List ID | Event ID | Org  | Info                           |    Date    | Published |\n",
      "+---------+----------+------+--------------------------------+------------+-----------+\n",
      "|    0    |   2655   | DEMO | Demo Event Phishing            | 2023-05-05 |    True   |\n",
      "|    1    |   2989   | DEMO | Domain reputation              | 2023-06-01 |   False   |\n",
      "|         |          |      | investigation for              |            |           |\n",
      "|         |          |      | officeaddons.com               |            |           |\n",
      "|    2    |   3100   | DEMO | Malware triage for             | 2023-11-01 |   False   |\n",
      "|         |          |      | certutil.exe                   |            |           |\n",
      "+---------+----------+------+--------------------------------+------------+-----------+\n"
     ]
    }
   ],
   "source": [
    "# Print out the list of events before deleting them\n",
    "table_print_before_delete(event_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "bdbfc8a2-f746-4651-a9dc-89c5ebf9c6a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are \u001b[92m3\u001b[90m events found.\n"
     ]
    }
   ],
   "source": [
    "# Example on how to remove one element from the list\n",
    "#print(event_list.pop(1))\n",
    "print(\"There are \\033[92m{}\\033[90m events found.\".format(len(event_list)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5e10015-8d2c-4af4-8225-18f6f6a58296",
   "metadata": {},
   "source": [
    "### Delete\n",
    "\n",
    "The next cell deletes the events from your MISP instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "5c7024bc-bac3-4980-8c1e-ab4704749d2a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting with delete ...\n",
      " Delete \u001b[91mDemo Event Phishing\u001b[90m (ID: 2655, UUID: fb4a934f-4694-406c-a26d-d148e881f875)\n",
      " Delete \u001b[91mDomain reputation investigation for officeaddons.com\u001b[90m (ID: 2989, UUID: c3a10866-19c5-497c-b78d-a0d44f472363)\n",
      " Delete \u001b[91mMalware triage for certutil.exe\u001b[90m (ID: 3100, UUID: 0c6dc44c-9370-4376-a479-31d5976b65d8)\n",
      "Delete finished\n"
     ]
    }
   ],
   "source": [
    "delete_events(event_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70dea5ca",
   "metadata": {},
   "source": [
    "# Closure\n",
    "\n",
    "In this **closure** or end step we create a **summary** of the actions that were performed by the playbook. The summary is printed and can also be send to a chat channel. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75b87ba6-19cc-4127-9299-480f708a61ff",
   "metadata": {},
   "source": [
    "## EN:1 Create the summary of the playbook \n",
    "\n",
    "The next section creates a summary and stores the output in the variable `summary` in Markdown format. It also stores an intro text in the variable `intro`. These variables can later be used when sending information to Mattermost or TheHive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "12561137-e759-440b-9d21-dc6def901cbc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The \u001b[92msummary\u001b[90m of the playbook is available.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "summary = \"# MISP Playbook summary\\nBulk delete events \\n\\n\"\n",
    "\n",
    "current_date = datetime.now()\n",
    "formatted_date = current_date.strftime(\"%Y-%m-%d\")\n",
    "summary += \"## Action\\n\\n\"\n",
    "summary += \"- Date: **{}**\\n\".format(formatted_date)\n",
    "summary += \"- Events affected: **{}**\\n\".format(len(event_list))\n",
    "if remove_from_event_block_list:\n",
    "    summary += \"- Remove events from **blocklist** after delete.\\n\"\n",
    "else:\n",
    "    summary += \"- Keep events in the **blocklist** after delete.\\n\"\n",
    "summary += \"\\n\\n\"\n",
    "summary += \"## Events\\n\\n\"\n",
    "summary_table = table_print_before_delete(event_list, True)\n",
    "summary_table.set_style(MARKDOWN)\n",
    "summary += summary_table.get_string(sortby=\"List ID\")\n",
    "summary += \"\\n\\n\"\n",
    "\n",
    "print(\"The \\033[92msummary\\033[90m of the playbook is available.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eba96062-deaa-4fb9-8302-7b3ef829201e",
   "metadata": {},
   "source": [
    "## EN:2 Send a summary to Mattermost\n",
    "\n",
    "Now you can send the summary to Mattermost. You can send the summary in two ways by selecting one of the options for the variable `send_to_mattermost_option` in the next cell.\n",
    "\n",
    "- The default option where the entire summary is in the **chat**, or\n",
    "- a short intro and the summary in a **card**\n",
    "\n",
    "For this playbook we rely on a webhook in Mattermost. You can add a webhook by choosing the gear icon in Mattermost, then choose Integrations and then **Incoming Webhooks**. Set a channel for the webhook and lock the webhook to this channel with *\"Lock to this channel\"*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "8c455bc3-35cc-436d-8237-95c81b57208f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "send_to_mattermost_option = \"via a chat message\"\n",
    "#send_to_mattermost_option = \"via a chat message with card\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "ca49f33e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary is \u001b[92msent to Mattermost.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "message = False\n",
    "if send_to_mattermost_option == \"via a chat message\":\n",
    "    message = {\"username\": mattermost_playbook_user, \"text\": summary}\n",
    "elif send_to_mattermost_option == \"via a chat message with card\":\n",
    "    message = {\"username\": mattermost_playbook_user, \"text\": intro, \"props\": {\"card\": summary}}\n",
    "\n",
    "if message:\n",
    "    r = requests.post(mattermost_hook, data=json.dumps(message))\n",
    "    r.raise_for_status()\n",
    "if message and r.status_code == 200:\n",
    "    print(\"Summary is \\033[92msent to Mattermost.\\n\")\n",
    "else:\n",
    "    print(\"\\033[91mFailed to sent summary\\033[90m to Mattermost.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "498b8eff-7d1b-47b7-90c4-e13cc738b36c",
   "metadata": {},
   "source": [
    "## EN:3 End of the playbook "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "4c03bb30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m End of the playbook\n"
     ]
    }
   ],
   "source": [
    "print(\"\\033[92m End of the playbook\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b17af9f2",
   "metadata": {},
   "source": [
    "## External references <a name=\"extreferences\"></a>\n",
    "\n",
    "- [The MISP Project](https://www.misp-project.org/)\n",
    "- [Mattermost](https://mattermost.com/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1c813f6",
   "metadata": {},
   "source": [
    "## Technical details \n",
    "\n",
    "### Documentation\n",
    "\n",
    "This playbook requires these Python **libraries** to exist in the environment where the playbook is executed. You can install them with `pip install <library>`.\n",
    "\n",
    "```\n",
    "pyfaup\n",
    "chardet\n",
    "PrettyTable\n",
    "ipywidgets\n",
    "```\n",
    "\n",
    "### Colour codes\n",
    "\n",
    "The output from Python displays some text in different colours. These are the colour codes\n",
    "\n",
    "```\n",
    "Red = '\\033[91m'\n",
    "Green = '\\033[92m'\n",
    "Blue = '\\033[94m'\n",
    "Cyan = '\\033[96m'\n",
    "White = '\\033[97m'\n",
    "Yellow = '\\033[93m'\n",
    "Magenta = '\\033[95m'\n",
    "Grey = '\\033[90m'\n",
    "Black = '\\033[90m'\n",
    "Default = '\\033[99m'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14dce812-e432-4c18-8358-f8b66e5b94a5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Attachments",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}