docs/notebooks/Base64Unpack.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Title: msticpy - Base64 Decoder\n",
    "## Description:\n",
    "This module allows you to extract base64 encoded content from a string or columns of a Pandas DataFrame.\n",
    "The library returns the following information:\n",
    "- decoded string (if decodable to utf-8 or utf-16)\n",
    "- hashes of the decoded segment (MD5, SHA1, SHA256)\n",
    "- string of printable byte values (e.g. for submission to a disassembler)\n",
    "- the detected decoded file type (limited)\n",
    "\n",
    "If the results of the decoding contain further encoded strings these will be decoded recursively. If the encoded string appears to be a zip, gzip or tar archive, the contents will be decompressed after decoding. In the case of zip and tar, the contents of the archive will also be checked for base64 encoded content and decoded/decompressed if possible.\n",
    "\n",
    "You must have msticpy installed to run this notebook:\n",
    "```\n",
    "%pip install --upgrade msticpy\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Title:-msticpy---Base64-Decoder\" data-toc-modified-id=\"Title:-msticpy---Base64-Decoder-1\">Title: msticpy - Base64 Decoder</a></span><ul class=\"toc-item\"><li><span><a href=\"#Description:\" data-toc-modified-id=\"Description:-1.1\">Description:</a></span></li><li><span><a href=\"#Decoding-Base64-String\" data-toc-modified-id=\"Decoding-Base64-String-1.2\">Decoding Base64 String</a></span></li><li><span><a href=\"#Using-a-DataFrame-as-Input\" data-toc-modified-id=\"Using-a-DataFrame-as-Input-1.3\">Using a DataFrame as Input</a></span><ul class=\"toc-item\"><li><span><a href=\"#Notes\" data-toc-modified-id=\"Notes-1.3.1\">Notes</a></span></li></ul></li><li><span><a href=\"#Interpreting-the-DataFrame-output.\" data-toc-modified-id=\"Interpreting-the-DataFrame-output.-1.4\">Interpreting the DataFrame output.</a></span><ul class=\"toc-item\"><li><span><a href=\"#SourceIndex-column-allows-you-to-merge-the-results-with-the-input-DataFrame\" data-toc-modified-id=\"SourceIndex-column-allows-you-to-merge-the-results-with-the-input-DataFrame-1.4.1\">SourceIndex column allows you to merge the results with the input DataFrame</a></span></li></ul></li><li><span><a href=\"#Decoding-Nested-Base64/Archives\" data-toc-modified-id=\"Decoding-Nested-Base64/Archives-1.5\">Decoding Nested Base64/Archives</a></span></li><li><span><a href=\"#IPython-magic\" data-toc-modified-id=\"IPython-magic-1.6\">IPython magic</a></span></li><li><span><a href=\"#Pandas-Extension\" data-toc-modified-id=\"Pandas-Extension-1.7\">Pandas Extension</a></span></li><li><span><a href=\"#To-Do-Items\" data-toc-modified-id=\"To-Do-Items-1.8\">To-Do Items</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T01:06:47.111019Z",
     "start_time": "2020-02-06T01:06:45.189863Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<hr><h4>Starting Notebook initialization...</h4> <br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "msticpy version installed: 1.7.0 latest published: 1.7.5<br>A newer version of msticpy - 1.7.5 is available.<br>Upgrade with 'pip install --upgrade msticpy'<br> <br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Processing imports.... <br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Imported: pd (pandas), IPython.get_ipython, IPython.display.display, IPython.display.HTML, IPython.display.Markdown, widgets (ipywidgets), pathlib.Path, plt (matplotlib.pyplot), matplotlib.MatplotlibDeprecationWarning, np (numpy), sns (seaborn), msticpy, msticpy.data.QueryProvider, msticpy.vis.foliummap.FoliumMap, msticpy.common.utility.md, msticpy.common.utility.md_warn, msticpy.common.wsconfig.WorkspaceConfig, msticpy.init.pivot.Pivot, msticpy.datamodel.entities, msticpy.init.nbmagics, msticpy.vis.mp_pandas_plot, msticpy.init.mp_pandas_accessors <br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Checking configuration.... <br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Azure CLI credentials not detected. (see <a href='https://github.com/Azure/Azure-Sentinel-Notebooks/wiki/Caching-credentials-with-Azure-CLI'>Caching credentials with Azure CLI</>) <br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Setting notebook options.... <br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "This product includes GeoLite2 data created by MaxMind, available from\n",
       "<a href=\"https://www.maxmind.com\">https://www.maxmind.com</a>.\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "This library uses services provided by ipstack.\n",
       "<a href=\"https://ipstack.com\">https://ipstack.com</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Using Open PageRank. See https://www.domcop.com/openpagerank/what-is-openpagerank<br> <br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " <br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h4>Notebook initialization complete</h4> <br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Imports\n",
    "import sys\n",
    "MIN_REQ_PYTHON = (3,6)\n",
    "if sys.version_info < MIN_REQ_PYTHON:\n",
    "    print('Check the Kernel->Change Kernel menu and ensure that Python 3.6')\n",
    "    print('or later is selected as the active kernel.')\n",
    "    sys.exit(\"Python %s.%s or later is required.\\n\" % MIN_REQ_PYTHON)\n",
    "\n",
    "\n",
    "from IPython.display import display\n",
    "import pandas as pd\n",
    "\n",
    "# Import Base64 module\n",
    "import msticpy\n",
    "msticpy.init_notebook(globals())\n",
    "from msticpy.transform import base64unpack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T01:06:47.128628Z",
     "start_time": "2020-02-06T01:06:47.112010Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CommandLine</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>.\\ftp  -s:C:\\RECYCLER\\xxppyy.exe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>.\\reg  not /domain:everything that /sid:shines is /krbtgt:golden !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>cmd  /c \"systeminfo &amp;&amp; systeminfo\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>.\\rundll32  /C 42424.exe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>.\\rundll32  /C c:\\users\\MSTICAdmin\\42424.exe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                          CommandLine\n",
       "0                                    .\\ftp  -s:C:\\RECYCLER\\xxppyy.exe\n",
       "1  .\\reg  not /domain:everything that /sid:shines is /krbtgt:golden !\n",
       "2                                  cmd  /c \"systeminfo && systeminfo\"\n",
       "3                                            .\\rundll32  /C 42424.exe\n",
       "4                        .\\rundll32  /C c:\\users\\MSTICAdmin\\42424.exe"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load test data\n",
    "process_tree = pd.read_csv('data/process_tree.csv',\n",
    "                           parse_dates=[\"TimeGenerated\"],\n",
    "                           infer_datetime_format=True)\n",
    "process_tree[['CommandLine']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='decoding_string'></a>[Contents](#contents)\n",
    "## Decoding Base64 String\n",
    "\n",
    "Base64 decode an input string.\n",
    "\n",
    "```\n",
    "    Base64 decode an input string.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    input_string : str, optional\n",
    "        single string to decode (the default is None)\n",
    "    trace : bool, optional\n",
    "        Show additional status (the default is None)\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    Tuple[str, Optional[List[BinaryRecord]]]\n",
    "        Decoded string and additional metadata\n",
    "\n",
    "    Notes\n",
    "    -----\n",
    "    Items that decode to utf-8 or utf-16 strings will be returned as decoded\n",
    "    strings replaced in the original string. If the encoded string is a\n",
    "    known binary type it will identify the file type and return the hashes\n",
    "    of the file. If any binary types are known archives (zip, tar, gzip) it\n",
    "    will unpack the contents of the archive.\n",
    "    For any binary it will return the decoded file as a byte array, and as a\n",
    "    printable list of byte values. If the input is a string the function\n",
    "    returns:\n",
    "\n",
    "    - decoded string: this is the input string with any decoded sections\n",
    "      replaced by the results of the decoding\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T01:06:47.142805Z",
     "start_time": "2020-02-06T01:06:47.129618Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'.\\\\powershell  -enc JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4AHAAcgBlAHMAcwBpAG8AbgAnACkAIAAkAHQA'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get a commandline from our data set\n",
    "cmdline = process_tree['CommandLine'].loc[39]\n",
    "cmdline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T01:06:47.161188Z",
     "start_time": "2020-02-06T01:06:47.143800Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(\".\\\\powershell  -enc <decoded type='string' name='[None]' index='1' depth='1'>$\\x00t\\x00 \\x00=\\x00 \\x00'\\x00d\\x00i\\x00r\\x00'\\x00;\\x00\\r\\x00\\n\\x00&\\x00 \\x00(\\x00'\\x00I\\x00n\\x00v\\x00o\\x00k\\x00e\\x00'\\x00+\\x00'\\x00-\\x00E\\x00x\\x00p\\x00r\\x00e\\x00s\\x00s\\x00i\\x00o\\x00n\\x00'\\x00)\\x00 \\x00$\\x00t\\x00</decoded>\",    reference  \\\n",
      "0  (, 1., 1)   \n",
      "\n",
      "                                                                                       original_string  \\\n",
      "0  JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4AHAAcgBlAHMAcwBpAG8A...   \n",
      "\n",
      "  file_name file_type  \\\n",
      "0   unknown      None   \n",
      "\n",
      "                                                                                           input_bytes  \\\n",
      "0  b\"$\\x00t\\x00 \\x00=\\x00 \\x00'\\x00d\\x00i\\x00r\\x00'\\x00;\\x00\\r\\x00\\n\\x00&\\x00 \\x00(\\x00'\\x00I\\x00n\\...   \n",
      "\n",
      "                                                                           decoded_string  \\\n",
      "0  $\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000;\u0000\\r\u0000\\n\u0000&\u0000 \u0000(\u0000'\u0000I\u0000n\u0000v\u0000o\u0000k\u0000e\u0000'\u0000+\u0000'\u0000-\u0000E\u0000x\u0000p\u0000r\u0000e\u0000s\u0000s\u0000i\u0000o\u0000n\u0000'\u0000)\u0000 \u0000$\u0000t\u0000   \n",
      "\n",
      "  encoding_type  \\\n",
      "0         utf-8   \n",
      "\n",
      "                                                                                           file_hashes  \\\n",
      "0  {'md5': '6cd1486db221e532cc2011c9beeb4ffc', 'sha1': '6e485467d7e06502046b7c84a8ef067cfe1512ad', ...   \n",
      "\n",
      "                                md5                                      sha1  \\\n",
      "0  6cd1486db221e532cc2011c9beeb4ffc  6e485467d7e06502046b7c84a8ef067cfe1512ad   \n",
      "\n",
      "                                                             sha256  \\\n",
      "0  d3291dab1ae552b91e6b50d7460ceaa39f6f92b2cda4335dd77e28d25c62ce34   \n",
      "\n",
      "                                                                                       printable_bytes  \n",
      "0  24 00 74 00 20 00 3d 00 20 00 27 00 64 00 69 00 72 00 27 00 3b 00 0d 00 0a 00 26 00 20 00 28 00 ...  )\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:390: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results = df_results.append(\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:431: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results.append(child_records, ignore_index=True, sort=False),\n"
     ]
    }
   ],
   "source": [
    "# Decode the string\n",
    "base64_dec_str = base64unpack.unpack(input_string=cmdline)\n",
    "\n",
    "# Print decoded string\n",
    "print(base64_dec_str)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='dataframeinput'></a>[Contents](#toc)\n",
    "## Using a DataFrame as Input\n",
    "You can replace to base64.unpack_df() to pass a DataFrame as an argument.\n",
    "Use the ```column``` parameter to specify which column to process.\n",
    "\n",
    "In the case of DataFrame input, the output DataFrame contains these additional columns:\n",
    " - src_index - the index of the row in the input dataframe from which the data came.\n",
    " - full_decoded_string - the full decoded string with any decoded replacements. This is only really useful for top-level items, since nested items will only show the 'full' string representing the child fragment.\n",
    "\n",
    "```\n",
    "Base64 decode strings taken from a pandas dataframe.\n",
    "\n",
    "Parameters\n",
    "----------\n",
    "data : pd.DataFrame\n",
    "    dataframe containing column to decode\n",
    "column : str\n",
    "    Name of dataframe text column\n",
    "trace : bool, optional\n",
    "    Show additional status (the default is None)\n",
    "\n",
    "Returns\n",
    "-------\n",
    "pd.DataFrame\n",
    "    Decoded string and additional metadata in dataframe\n",
    "```\n",
    "\n",
    "### Notes\n",
    "\n",
    "Items that decode to utf-8 or utf-16 strings will be returned as decoded\n",
    "strings replaced in the original string. If the encoded string is a\n",
    "known binary type it will identify the file type and return the hashes\n",
    "of the file. If any binary types are known archives (zip, tar, gzip) it\n",
    "will unpack the contents of the archive.\n",
    "For any binary it will return the decoded file as a byte array, and as a\n",
    "printable list of byte values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T01:06:47.222153Z",
     "start_time": "2020-02-06T01:06:47.162187Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:390: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results = df_results.append(\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:431: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results.append(child_records, ignore_index=True, sort=False),\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:390: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results = df_results.append(\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:431: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results.append(child_records, ignore_index=True, sort=False),\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:390: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results = df_results.append(\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:431: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results.append(child_records, ignore_index=True, sort=False),\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:390: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results = df_results.append(\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:431: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results.append(child_records, ignore_index=True, sort=False),\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reference</th>\n",
       "      <th>original_string</th>\n",
       "      <th>file_name</th>\n",
       "      <th>file_type</th>\n",
       "      <th>input_bytes</th>\n",
       "      <th>decoded_string</th>\n",
       "      <th>encoding_type</th>\n",
       "      <th>file_hashes</th>\n",
       "      <th>md5</th>\n",
       "      <th>sha1</th>\n",
       "      <th>sha256</th>\n",
       "      <th>printable_bytes</th>\n",
       "      <th>src_index</th>\n",
       "      <th>CommandLine</th>\n",
       "      <th>full_decoded_string</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4AHAAcgBlAHMAcwBpAG8A...</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b\"$\\x00t\\x00 \\x00=\\x00 \\x00'\\x00d\\x00i\\x00r\\x00'\\x00;\\x00\\r\\x00\\n\\x00&amp;\\x00 \\x00(\\x00'\\x00I\\x00n\\...</td>\n",
       "      <td>$\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000;\u0000\\r\u0000\\n\u0000&amp;\u0000 \u0000(\u0000'\u0000I\u0000n\u0000v\u0000o\u0000k\u0000e\u0000'\u0000+\u0000'\u0000-\u0000E\u0000x\u0000p\u0000r\u0000e\u0000s\u0000s\u0000i\u0000o\u0000n\u0000'\u0000)\u0000 \u0000$\u0000t\u0000</td>\n",
       "      <td>utf-8</td>\n",
       "      <td>{'md5': '6cd1486db221e532cc2011c9beeb4ffc', 'sha1': '6e485467d7e06502046b7c84a8ef067cfe1512ad', ...</td>\n",
       "      <td>6cd1486db221e532cc2011c9beeb4ffc</td>\n",
       "      <td>6e485467d7e06502046b7c84a8ef067cfe1512ad</td>\n",
       "      <td>d3291dab1ae552b91e6b50d7460ceaa39f6f92b2cda4335dd77e28d25c62ce34</td>\n",
       "      <td>24 00 74 00 20 00 3d 00 20 00 27 00 64 00 69 00 72 00 27 00 3b 00 0d 00 0a 00 26 00 20 00 28 00 ...</td>\n",
       "      <td>39</td>\n",
       "      <td>.\\powershell  -enc JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4A...</td>\n",
       "      <td>.\\powershell  -enc &lt;decoded type='string' name='[None]' index='1' depth='1'&gt;$\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'</td>\n",
       "      <td>None</td>\n",
       "      <td>binary</td>\n",
       "      <td>{'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...</td>\n",
       "      <td>9a45b2520e930dc9186f6d93a7798a13</td>\n",
       "      <td>f526c90fa0744e3a63d84421ff25e3f5a3d697cb</td>\n",
       "      <td>c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239</td>\n",
       "      <td>69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a</td>\n",
       "      <td>40</td>\n",
       "      <td>cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &gt;&gt; blah.ps1\"</td>\n",
       "      <td>cmd  /c \"echo # &lt;decoded value='binary'  name='[None]' type='None' index='1' depth='1'&gt;69 a6 9a ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'</td>\n",
       "      <td>None</td>\n",
       "      <td>binary</td>\n",
       "      <td>{'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...</td>\n",
       "      <td>9a45b2520e930dc9186f6d93a7798a13</td>\n",
       "      <td>f526c90fa0744e3a63d84421ff25e3f5a3d697cb</td>\n",
       "      <td>c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239</td>\n",
       "      <td>69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a</td>\n",
       "      <td>41</td>\n",
       "      <td>cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &gt;&gt; blah.ps1\"</td>\n",
       "      <td>cmd  /c \"echo # &lt;decoded value='binary'  name='[None]' type='None' index='1' depth='1'&gt;69 a6 9a ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>81ed03caf6901e444c72ac67d192fb9c</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b'\\xf3W\\x9d\\xd3w\\x1a\\x7f\\xaft\\xd5\\xee8\\xe1\\xce\\xf6i\\xce\\xbbw_v}\\xbf\\\\'</td>\n",
       "      <td>None</td>\n",
       "      <td>binary</td>\n",
       "      <td>{'md5': '1c8cc6299bd654bbcd85710968d6a87c', 'sha1': '55377391141f59a2ff5ae4765d9f0b4438adfd73', ...</td>\n",
       "      <td>1c8cc6299bd654bbcd85710968d6a87c</td>\n",
       "      <td>55377391141f59a2ff5ae4765d9f0b4438adfd73</td>\n",
       "      <td>fd80ceba7cfb49d296886c10d9a3497d63c89a589587cda7d818cb4644842660</td>\n",
       "      <td>f3 57 9d d3 77 1a 7f af 74 d5 ee 38 e1 ce f6 69 ce bb 77 5f 76 7d bf 5c</td>\n",
       "      <td>44</td>\n",
       "      <td>implant.exe  81ed03caf6901e444c72ac67d192fb9c</td>\n",
       "      <td>implant.exe  &lt;decoded value='binary'  name='[None]' type='None' index='1' depth='1'&gt;f3 57 9d d3 ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   reference  \\\n",
       "0  (, 1., 1)   \n",
       "1  (, 1., 1)   \n",
       "2  (, 1., 1)   \n",
       "3  (, 1., 1)   \n",
       "\n",
       "                                                                                       original_string  \\\n",
       "0  JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4AHAAcgBlAHMAcwBpAG8A...   \n",
       "1                                                                     aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa   \n",
       "2                                                                     aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa   \n",
       "3                                                                     81ed03caf6901e444c72ac67d192fb9c   \n",
       "\n",
       "  file_name file_type  \\\n",
       "0   unknown      None   \n",
       "1   unknown      None   \n",
       "2   unknown      None   \n",
       "3   unknown      None   \n",
       "\n",
       "                                                                                           input_bytes  \\\n",
       "0  b\"$\\x00t\\x00 \\x00=\\x00 \\x00'\\x00d\\x00i\\x00r\\x00'\\x00;\\x00\\r\\x00\\n\\x00&\\x00 \\x00(\\x00'\\x00I\\x00n\\...   \n",
       "1                          b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'   \n",
       "2                          b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'   \n",
       "3                               b'\\xf3W\\x9d\\xd3w\\x1a\\x7f\\xaft\\xd5\\xee8\\xe1\\xce\\xf6i\\xce\\xbbw_v}\\xbf\\\\'   \n",
       "\n",
       "                                                                           decoded_string  \\\n",
       "0  $\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000;\u0000\\r\u0000\\n\u0000&\u0000 \u0000(\u0000'\u0000I\u0000n\u0000v\u0000o\u0000k\u0000e\u0000'\u0000+\u0000'\u0000-\u0000E\u0000x\u0000p\u0000r\u0000e\u0000s\u0000s\u0000i\u0000o\u0000n\u0000'\u0000)\u0000 \u0000$\u0000t\u0000   \n",
       "1                                                                                    None   \n",
       "2                                                                                    None   \n",
       "3                                                                                    None   \n",
       "\n",
       "  encoding_type  \\\n",
       "0         utf-8   \n",
       "1        binary   \n",
       "2        binary   \n",
       "3        binary   \n",
       "\n",
       "                                                                                           file_hashes  \\\n",
       "0  {'md5': '6cd1486db221e532cc2011c9beeb4ffc', 'sha1': '6e485467d7e06502046b7c84a8ef067cfe1512ad', ...   \n",
       "1  {'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...   \n",
       "2  {'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...   \n",
       "3  {'md5': '1c8cc6299bd654bbcd85710968d6a87c', 'sha1': '55377391141f59a2ff5ae4765d9f0b4438adfd73', ...   \n",
       "\n",
       "                                md5                                      sha1  \\\n",
       "0  6cd1486db221e532cc2011c9beeb4ffc  6e485467d7e06502046b7c84a8ef067cfe1512ad   \n",
       "1  9a45b2520e930dc9186f6d93a7798a13  f526c90fa0744e3a63d84421ff25e3f5a3d697cb   \n",
       "2  9a45b2520e930dc9186f6d93a7798a13  f526c90fa0744e3a63d84421ff25e3f5a3d697cb   \n",
       "3  1c8cc6299bd654bbcd85710968d6a87c  55377391141f59a2ff5ae4765d9f0b4438adfd73   \n",
       "\n",
       "                                                             sha256  \\\n",
       "0  d3291dab1ae552b91e6b50d7460ceaa39f6f92b2cda4335dd77e28d25c62ce34   \n",
       "1  c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239   \n",
       "2  c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239   \n",
       "3  fd80ceba7cfb49d296886c10d9a3497d63c89a589587cda7d818cb4644842660   \n",
       "\n",
       "                                                                                       printable_bytes  \\\n",
       "0  24 00 74 00 20 00 3d 00 20 00 27 00 64 00 69 00 72 00 27 00 3b 00 0d 00 0a 00 26 00 20 00 28 00 ...   \n",
       "1                              69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a   \n",
       "2                              69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a   \n",
       "3                              f3 57 9d d3 77 1a 7f af 74 d5 ee 38 e1 ce f6 69 ce bb 77 5f 76 7d bf 5c   \n",
       "\n",
       "   src_index  \\\n",
       "0         39   \n",
       "1         40   \n",
       "2         41   \n",
       "3         44   \n",
       "\n",
       "                                                                                           CommandLine  \\\n",
       "0  .\\powershell  -enc JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4A...   \n",
       "1                                        cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> blah.ps1\"   \n",
       "2                                        cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> blah.ps1\"   \n",
       "3                                                        implant.exe  81ed03caf6901e444c72ac67d192fb9c   \n",
       "\n",
       "                                                                                   full_decoded_string  \n",
       "0  .\\powershell  -enc <decoded type='string' name='[None]' index='1' depth='1'>$\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000...  \n",
       "1  cmd  /c \"echo # <decoded value='binary'  name='[None]' type='None' index='1' depth='1'>69 a6 9a ...  \n",
       "2  cmd  /c \"echo # <decoded value='binary'  name='[None]' type='None' index='1' depth='1'>69 a6 9a ...  \n",
       "3  implant.exe  <decoded value='binary'  name='[None]' type='None' index='1' depth='1'>f3 57 9d d3 ...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# specify the data and column parameters\n",
    "dec_df = base64unpack.unpack_df(data=process_tree, column='CommandLine')\n",
    "dec_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='dataframe_colums'></a>[Contents](#contents)\n",
    "## Interpreting the DataFrame output.\n",
    "For simple strings the Base64 decoded output is straightforward. However for nested encodings this can get a little complex and difficult to represent in a tabular format.\n",
    "\n",
    "**Columns**\n",
    " - reference - The index of the row item in dotted notation in depth.seq pairs (e.g. 1.2.2.3 would be the 3 item at depth 3 that is a child of the 2nd item found at depth 1). This may not always be an accurate notation - it is mainly use to allow you to associate an individual row with the reference value contained in the full_decoded_string column of the topmost item).\n",
    " - original_string - the original string before decoding.\n",
    " - file_name - filename, if any (only if this is an item in zip or tar file).\n",
    " - file_type - a guess at the file type (this is currently elementary and only includes a few file types).\n",
    " - input_bytes - the decoded bytes as a Python bytes string.\n",
    " - decoded_string - the decoded string if it can be decoded as a UTF-8 or UTF-16 string. Note: binary sequences may often successfully decode as UTF-16 strings but, in these cases, the decodings are meaningless.\n",
    " - encoding_type - encoding type (UTF-8 or UTF-16) if a decoding was possible, otherwise 'binary'.\n",
    " - file_hashes - collection of file hashes for any decoded item.\n",
    " - md5 - md5 hash as a separate column.\n",
    " - sha1 - sha1 hash as a separate column.\n",
    " - sha256 - sha256 hash as a separate column.\n",
    " - printable_bytes - printable version of input_bytes as a string of \\xNN values\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='mergeresults'></a>[Contents](#contents)\n",
    "### SourceIndex column allows you to merge the results with the input DataFrame\n",
    "Where an input row has multiple decoded elements (e.g. a nested encoding or a zip or other archive file), the output of this merge will result in duplicate rows from the input (one per element match). The DataFrame index from the source is preserved in the `src_index` column.\n",
    "\n",
    "Note: you may need to force the type of the `src_index` column to be the same type as the original DataFrame in order to merge. In the example below case we are matching with the default numeric index so we force the type to be numeric. In cases where you are using an index of a different dtype you will need to convert the `src_index` (dtype=object) to match the type of your index column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T01:06:47.258155Z",
     "start_time": "2020-02-06T01:06:47.223152Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TenantId</th>\n",
       "      <th>Account</th>\n",
       "      <th>EventID</th>\n",
       "      <th>TimeGenerated</th>\n",
       "      <th>Computer</th>\n",
       "      <th>SubjectUserSid</th>\n",
       "      <th>SubjectUserName</th>\n",
       "      <th>SubjectDomainName</th>\n",
       "      <th>SubjectLogonId</th>\n",
       "      <th>NewProcessId</th>\n",
       "      <th>NewProcessName</th>\n",
       "      <th>TokenElevationType</th>\n",
       "      <th>ProcessId</th>\n",
       "      <th>CommandLine_x</th>\n",
       "      <th>ParentProcessName</th>\n",
       "      <th>TargetLogonId</th>\n",
       "      <th>SourceComputerId</th>\n",
       "      <th>TimeCreatedUtc</th>\n",
       "      <th>NodeRole</th>\n",
       "      <th>Level</th>\n",
       "      <th>ProcessId1</th>\n",
       "      <th>NewProcessId1</th>\n",
       "      <th>reference</th>\n",
       "      <th>original_string</th>\n",
       "      <th>file_name</th>\n",
       "      <th>file_type</th>\n",
       "      <th>input_bytes</th>\n",
       "      <th>decoded_string</th>\n",
       "      <th>encoding_type</th>\n",
       "      <th>file_hashes</th>\n",
       "      <th>md5</th>\n",
       "      <th>sha1</th>\n",
       "      <th>sha256</th>\n",
       "      <th>printable_bytes</th>\n",
       "      <th>src_index</th>\n",
       "      <th>CommandLine_y</th>\n",
       "      <th>full_decoded_string</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SourceIndex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>802d39e1-9d70-404d-832c-2de5e2478eda</td>\n",
       "      <td>MSTICAlertsWin1\\MSTICAdmin</td>\n",
       "      <td>4688</td>\n",
       "      <td>2019-01-15 05:15:13.567</td>\n",
       "      <td>MSTICAlertsWin1</td>\n",
       "      <td>S-1-5-21-996632719-2361334927-4038480536-500</td>\n",
       "      <td>MSTICAdmin</td>\n",
       "      <td>MSTICAlertsWin1</td>\n",
       "      <td>0xfaac27</td>\n",
       "      <td>0x1684</td>\n",
       "      <td>C:\\Diagnostics\\UserTmp\\powershell.exe</td>\n",
       "      <td>%%1936</td>\n",
       "      <td>0xbc8</td>\n",
       "      <td>.\\powershell  -enc JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4A...</td>\n",
       "      <td>C:\\Windows\\System32\\cmd.exe</td>\n",
       "      <td>0x0</td>\n",
       "      <td>46fe7078-61bb-4bed-9430-7ac01d91c273</td>\n",
       "      <td>2019-01-15 05:15:13.567</td>\n",
       "      <td>sibling</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4AHAAcgBlAHMAcwBpAG8A...</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b\"$\\x00t\\x00 \\x00=\\x00 \\x00'\\x00d\\x00i\\x00r\\x00'\\x00;\\x00\\r\\x00\\n\\x00&amp;\\x00 \\x00(\\x00'\\x00I\\x00n\\...</td>\n",
       "      <td>$\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000;\u0000\\r\u0000\\n\u0000&amp;\u0000 \u0000(\u0000'\u0000I\u0000n\u0000v\u0000o\u0000k\u0000e\u0000'\u0000+\u0000'\u0000-\u0000E\u0000x\u0000p\u0000r\u0000e\u0000s\u0000s\u0000i\u0000o\u0000n\u0000'\u0000)\u0000 \u0000$\u0000t\u0000</td>\n",
       "      <td>utf-8</td>\n",
       "      <td>{'md5': '6cd1486db221e532cc2011c9beeb4ffc', 'sha1': '6e485467d7e06502046b7c84a8ef067cfe1512ad', ...</td>\n",
       "      <td>6cd1486db221e532cc2011c9beeb4ffc</td>\n",
       "      <td>6e485467d7e06502046b7c84a8ef067cfe1512ad</td>\n",
       "      <td>d3291dab1ae552b91e6b50d7460ceaa39f6f92b2cda4335dd77e28d25c62ce34</td>\n",
       "      <td>24 00 74 00 20 00 3d 00 20 00 27 00 64 00 69 00 72 00 27 00 3b 00 0d 00 0a 00 26 00 20 00 28 00 ...</td>\n",
       "      <td>39.0</td>\n",
       "      <td>.\\powershell  -enc JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4A...</td>\n",
       "      <td>.\\powershell  -enc &lt;decoded type='string' name='[None]' index='1' depth='1'&gt;$\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>802d39e1-9d70-404d-832c-2de5e2478eda</td>\n",
       "      <td>MSTICAlertsWin1\\MSTICAdmin</td>\n",
       "      <td>4688</td>\n",
       "      <td>2019-01-15 05:15:13.683</td>\n",
       "      <td>MSTICAlertsWin1</td>\n",
       "      <td>S-1-5-21-996632719-2361334927-4038480536-500</td>\n",
       "      <td>MSTICAdmin</td>\n",
       "      <td>MSTICAlertsWin1</td>\n",
       "      <td>0xfaac27</td>\n",
       "      <td>0x16b8</td>\n",
       "      <td>C:\\Diagnostics\\UserTmp\\cmd.exe</td>\n",
       "      <td>%%1936</td>\n",
       "      <td>0xbc8</td>\n",
       "      <td>cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &gt;&gt; blah.ps1\"</td>\n",
       "      <td>C:\\Windows\\System32\\cmd.exe</td>\n",
       "      <td>0x0</td>\n",
       "      <td>46fe7078-61bb-4bed-9430-7ac01d91c273</td>\n",
       "      <td>2019-01-15 05:15:13.683</td>\n",
       "      <td>sibling</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'</td>\n",
       "      <td>None</td>\n",
       "      <td>binary</td>\n",
       "      <td>{'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...</td>\n",
       "      <td>9a45b2520e930dc9186f6d93a7798a13</td>\n",
       "      <td>f526c90fa0744e3a63d84421ff25e3f5a3d697cb</td>\n",
       "      <td>c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239</td>\n",
       "      <td>69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a</td>\n",
       "      <td>40.0</td>\n",
       "      <td>cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &gt;&gt; blah.ps1\"</td>\n",
       "      <td>cmd  /c \"echo # &lt;decoded value='binary'  name='[None]' type='None' index='1' depth='1'&gt;69 a6 9a ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>802d39e1-9d70-404d-832c-2de5e2478eda</td>\n",
       "      <td>MSTICAlertsWin1\\MSTICAdmin</td>\n",
       "      <td>4688</td>\n",
       "      <td>2019-01-15 05:15:13.793</td>\n",
       "      <td>MSTICAlertsWin1</td>\n",
       "      <td>S-1-5-21-996632719-2361334927-4038480536-500</td>\n",
       "      <td>MSTICAdmin</td>\n",
       "      <td>MSTICAlertsWin1</td>\n",
       "      <td>0xfaac27</td>\n",
       "      <td>0x16ec</td>\n",
       "      <td>C:\\Diagnostics\\UserTmp\\cmd.exe</td>\n",
       "      <td>%%1936</td>\n",
       "      <td>0xbc8</td>\n",
       "      <td>cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &gt;&gt; blah.ps1\"</td>\n",
       "      <td>C:\\Windows\\System32\\cmd.exe</td>\n",
       "      <td>0x0</td>\n",
       "      <td>46fe7078-61bb-4bed-9430-7ac01d91c273</td>\n",
       "      <td>2019-01-15 05:15:13.793</td>\n",
       "      <td>sibling</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'</td>\n",
       "      <td>None</td>\n",
       "      <td>binary</td>\n",
       "      <td>{'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...</td>\n",
       "      <td>9a45b2520e930dc9186f6d93a7798a13</td>\n",
       "      <td>f526c90fa0744e3a63d84421ff25e3f5a3d697cb</td>\n",
       "      <td>c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239</td>\n",
       "      <td>69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a</td>\n",
       "      <td>41.0</td>\n",
       "      <td>cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &gt;&gt; blah.ps1\"</td>\n",
       "      <td>cmd  /c \"echo # &lt;decoded value='binary'  name='[None]' type='None' index='1' depth='1'&gt;69 a6 9a ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>802d39e1-9d70-404d-832c-2de5e2478eda</td>\n",
       "      <td>MSTICAlertsWin1\\MSTICAdmin</td>\n",
       "      <td>4688</td>\n",
       "      <td>2019-01-15 05:15:12.003</td>\n",
       "      <td>MSTICAlertsWin1</td>\n",
       "      <td>S-1-5-21-996632719-2361334927-4038480536-500</td>\n",
       "      <td>MSTICAdmin</td>\n",
       "      <td>MSTICAlertsWin1</td>\n",
       "      <td>0xfaac27</td>\n",
       "      <td>0x1250</td>\n",
       "      <td>C:\\Diagnostics\\UserTmp\\implant.exe</td>\n",
       "      <td>%%1936</td>\n",
       "      <td>0xbc8</td>\n",
       "      <td>implant.exe  81ed03caf6901e444c72ac67d192fb9c</td>\n",
       "      <td>C:\\Windows\\System32\\cmd.exe</td>\n",
       "      <td>0x0</td>\n",
       "      <td>46fe7078-61bb-4bed-9430-7ac01d91c273</td>\n",
       "      <td>2019-01-15 05:15:12.003</td>\n",
       "      <td>sibling</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>81ed03caf6901e444c72ac67d192fb9c</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b'\\xf3W\\x9d\\xd3w\\x1a\\x7f\\xaft\\xd5\\xee8\\xe1\\xce\\xf6i\\xce\\xbbw_v}\\xbf\\\\'</td>\n",
       "      <td>None</td>\n",
       "      <td>binary</td>\n",
       "      <td>{'md5': '1c8cc6299bd654bbcd85710968d6a87c', 'sha1': '55377391141f59a2ff5ae4765d9f0b4438adfd73', ...</td>\n",
       "      <td>1c8cc6299bd654bbcd85710968d6a87c</td>\n",
       "      <td>55377391141f59a2ff5ae4765d9f0b4438adfd73</td>\n",
       "      <td>fd80ceba7cfb49d296886c10d9a3497d63c89a589587cda7d818cb4644842660</td>\n",
       "      <td>f3 57 9d d3 77 1a 7f af 74 d5 ee 38 e1 ce f6 69 ce bb 77 5f 76 7d bf 5c</td>\n",
       "      <td>44.0</td>\n",
       "      <td>implant.exe  81ed03caf6901e444c72ac67d192fb9c</td>\n",
       "      <td>implant.exe  &lt;decoded value='binary'  name='[None]' type='None' index='1' depth='1'&gt;f3 57 9d d3 ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         TenantId                     Account  \\\n",
       "SourceIndex                                                                     \n",
       "39           802d39e1-9d70-404d-832c-2de5e2478eda  MSTICAlertsWin1\\MSTICAdmin   \n",
       "40           802d39e1-9d70-404d-832c-2de5e2478eda  MSTICAlertsWin1\\MSTICAdmin   \n",
       "41           802d39e1-9d70-404d-832c-2de5e2478eda  MSTICAlertsWin1\\MSTICAdmin   \n",
       "44           802d39e1-9d70-404d-832c-2de5e2478eda  MSTICAlertsWin1\\MSTICAdmin   \n",
       "\n",
       "             EventID           TimeGenerated         Computer  \\\n",
       "SourceIndex                                                     \n",
       "39              4688 2019-01-15 05:15:13.567  MSTICAlertsWin1   \n",
       "40              4688 2019-01-15 05:15:13.683  MSTICAlertsWin1   \n",
       "41              4688 2019-01-15 05:15:13.793  MSTICAlertsWin1   \n",
       "44              4688 2019-01-15 05:15:12.003  MSTICAlertsWin1   \n",
       "\n",
       "                                           SubjectUserSid SubjectUserName  \\\n",
       "SourceIndex                                                                 \n",
       "39           S-1-5-21-996632719-2361334927-4038480536-500      MSTICAdmin   \n",
       "40           S-1-5-21-996632719-2361334927-4038480536-500      MSTICAdmin   \n",
       "41           S-1-5-21-996632719-2361334927-4038480536-500      MSTICAdmin   \n",
       "44           S-1-5-21-996632719-2361334927-4038480536-500      MSTICAdmin   \n",
       "\n",
       "            SubjectDomainName SubjectLogonId NewProcessId  \\\n",
       "SourceIndex                                                 \n",
       "39            MSTICAlertsWin1       0xfaac27       0x1684   \n",
       "40            MSTICAlertsWin1       0xfaac27       0x16b8   \n",
       "41            MSTICAlertsWin1       0xfaac27       0x16ec   \n",
       "44            MSTICAlertsWin1       0xfaac27       0x1250   \n",
       "\n",
       "                                    NewProcessName TokenElevationType  \\\n",
       "SourceIndex                                                             \n",
       "39           C:\\Diagnostics\\UserTmp\\powershell.exe             %%1936   \n",
       "40                  C:\\Diagnostics\\UserTmp\\cmd.exe             %%1936   \n",
       "41                  C:\\Diagnostics\\UserTmp\\cmd.exe             %%1936   \n",
       "44              C:\\Diagnostics\\UserTmp\\implant.exe             %%1936   \n",
       "\n",
       "            ProcessId  \\\n",
       "SourceIndex             \n",
       "39              0xbc8   \n",
       "40              0xbc8   \n",
       "41              0xbc8   \n",
       "44              0xbc8   \n",
       "\n",
       "                                                                                                   CommandLine_x  \\\n",
       "SourceIndex                                                                                                        \n",
       "39           .\\powershell  -enc JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4A...   \n",
       "40                                                 cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> blah.ps1\"   \n",
       "41                                                 cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> blah.ps1\"   \n",
       "44                                                                 implant.exe  81ed03caf6901e444c72ac67d192fb9c   \n",
       "\n",
       "                       ParentProcessName TargetLogonId  \\\n",
       "SourceIndex                                              \n",
       "39           C:\\Windows\\System32\\cmd.exe           0x0   \n",
       "40           C:\\Windows\\System32\\cmd.exe           0x0   \n",
       "41           C:\\Windows\\System32\\cmd.exe           0x0   \n",
       "44           C:\\Windows\\System32\\cmd.exe           0x0   \n",
       "\n",
       "                                 SourceComputerId           TimeCreatedUtc  \\\n",
       "SourceIndex                                                                  \n",
       "39           46fe7078-61bb-4bed-9430-7ac01d91c273  2019-01-15 05:15:13.567   \n",
       "40           46fe7078-61bb-4bed-9430-7ac01d91c273  2019-01-15 05:15:13.683   \n",
       "41           46fe7078-61bb-4bed-9430-7ac01d91c273  2019-01-15 05:15:13.793   \n",
       "44           46fe7078-61bb-4bed-9430-7ac01d91c273  2019-01-15 05:15:12.003   \n",
       "\n",
       "            NodeRole  Level  ProcessId1  NewProcessId1  reference  \\\n",
       "SourceIndex                                                         \n",
       "39           sibling      1         NaN            NaN  (, 1., 1)   \n",
       "40           sibling      1         NaN            NaN  (, 1., 1)   \n",
       "41           sibling      1         NaN            NaN  (, 1., 1)   \n",
       "44           sibling      1         NaN            NaN  (, 1., 1)   \n",
       "\n",
       "                                                                                                 original_string  \\\n",
       "SourceIndex                                                                                                        \n",
       "39           JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4AHAAcgBlAHMAcwBpAG8A...   \n",
       "40                                                                              aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa   \n",
       "41                                                                              aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa   \n",
       "44                                                                              81ed03caf6901e444c72ac67d192fb9c   \n",
       "\n",
       "            file_name file_type  \\\n",
       "SourceIndex                       \n",
       "39            unknown      None   \n",
       "40            unknown      None   \n",
       "41            unknown      None   \n",
       "44            unknown      None   \n",
       "\n",
       "                                                                                                     input_bytes  \\\n",
       "SourceIndex                                                                                                        \n",
       "39           b\"$\\x00t\\x00 \\x00=\\x00 \\x00'\\x00d\\x00i\\x00r\\x00'\\x00;\\x00\\r\\x00\\n\\x00&\\x00 \\x00(\\x00'\\x00I\\x00n\\...   \n",
       "40                                   b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'   \n",
       "41                                   b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'   \n",
       "44                                        b'\\xf3W\\x9d\\xd3w\\x1a\\x7f\\xaft\\xd5\\xee8\\xe1\\xce\\xf6i\\xce\\xbbw_v}\\xbf\\\\'   \n",
       "\n",
       "                                                                                     decoded_string  \\\n",
       "SourceIndex                                                                                           \n",
       "39           $\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000;\u0000\\r\u0000\\n\u0000&\u0000 \u0000(\u0000'\u0000I\u0000n\u0000v\u0000o\u0000k\u0000e\u0000'\u0000+\u0000'\u0000-\u0000E\u0000x\u0000p\u0000r\u0000e\u0000s\u0000s\u0000i\u0000o\u0000n\u0000'\u0000)\u0000 \u0000$\u0000t\u0000   \n",
       "40                                                                                             None   \n",
       "41                                                                                             None   \n",
       "44                                                                                             None   \n",
       "\n",
       "            encoding_type  \\\n",
       "SourceIndex                 \n",
       "39                  utf-8   \n",
       "40                 binary   \n",
       "41                 binary   \n",
       "44                 binary   \n",
       "\n",
       "                                                                                                     file_hashes  \\\n",
       "SourceIndex                                                                                                        \n",
       "39           {'md5': '6cd1486db221e532cc2011c9beeb4ffc', 'sha1': '6e485467d7e06502046b7c84a8ef067cfe1512ad', ...   \n",
       "40           {'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...   \n",
       "41           {'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...   \n",
       "44           {'md5': '1c8cc6299bd654bbcd85710968d6a87c', 'sha1': '55377391141f59a2ff5ae4765d9f0b4438adfd73', ...   \n",
       "\n",
       "                                          md5  \\\n",
       "SourceIndex                                     \n",
       "39           6cd1486db221e532cc2011c9beeb4ffc   \n",
       "40           9a45b2520e930dc9186f6d93a7798a13   \n",
       "41           9a45b2520e930dc9186f6d93a7798a13   \n",
       "44           1c8cc6299bd654bbcd85710968d6a87c   \n",
       "\n",
       "                                                 sha1  \\\n",
       "SourceIndex                                             \n",
       "39           6e485467d7e06502046b7c84a8ef067cfe1512ad   \n",
       "40           f526c90fa0744e3a63d84421ff25e3f5a3d697cb   \n",
       "41           f526c90fa0744e3a63d84421ff25e3f5a3d697cb   \n",
       "44           55377391141f59a2ff5ae4765d9f0b4438adfd73   \n",
       "\n",
       "                                                                       sha256  \\\n",
       "SourceIndex                                                                     \n",
       "39           d3291dab1ae552b91e6b50d7460ceaa39f6f92b2cda4335dd77e28d25c62ce34   \n",
       "40           c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239   \n",
       "41           c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239   \n",
       "44           fd80ceba7cfb49d296886c10d9a3497d63c89a589587cda7d818cb4644842660   \n",
       "\n",
       "                                                                                                 printable_bytes  \\\n",
       "SourceIndex                                                                                                        \n",
       "39           24 00 74 00 20 00 3d 00 20 00 27 00 64 00 69 00 72 00 27 00 3b 00 0d 00 0a 00 26 00 20 00 28 00 ...   \n",
       "40                                       69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a   \n",
       "41                                       69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a   \n",
       "44                                       f3 57 9d d3 77 1a 7f af 74 d5 ee 38 e1 ce f6 69 ce bb 77 5f 76 7d bf 5c   \n",
       "\n",
       "             src_index  \\\n",
       "SourceIndex              \n",
       "39                39.0   \n",
       "40                40.0   \n",
       "41                41.0   \n",
       "44                44.0   \n",
       "\n",
       "                                                                                                   CommandLine_y  \\\n",
       "SourceIndex                                                                                                        \n",
       "39           .\\powershell  -enc JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4A...   \n",
       "40                                                 cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> blah.ps1\"   \n",
       "41                                                 cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> blah.ps1\"   \n",
       "44                                                                 implant.exe  81ed03caf6901e444c72ac67d192fb9c   \n",
       "\n",
       "                                                                                             full_decoded_string  \n",
       "SourceIndex                                                                                                       \n",
       "39           .\\powershell  -enc <decoded type='string' name='[None]' index='1' depth='1'>$\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000...  \n",
       "40           cmd  /c \"echo # <decoded value='binary'  name='[None]' type='None' index='1' depth='1'>69 a6 9a ...  \n",
       "41           cmd  /c \"echo # <decoded value='binary'  name='[None]' type='None' index='1' depth='1'>69 a6 9a ...  \n",
       "44           implant.exe  <decoded value='binary'  name='[None]' type='None' index='1' depth='1'>f3 57 9d d3 ...  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set the type of the SourceIndex column. \n",
    "dec_df['SourceIndex'] = pd.to_numeric(dec_df['src_index'])\n",
    "merged_df = (process_tree\n",
    "             .merge(right=dec_df, how='left', left_index=True, right_on='SourceIndex')\n",
    "             .drop(columns=['Unnamed: 0'])\n",
    "             .set_index('SourceIndex'))\n",
    "\n",
    "# Show the result of the merge (only those rows that have a value in original_string)\n",
    "merged_df.dropna(subset=['original_string'])\n",
    "\n",
    "# Note the output of unpack_items() may have multiple rows (for nested encodings) \n",
    "# In this case merged DF will have duplicate rows from the source."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='nested_encodings'></a>[Contents](#contents)\n",
    "## Decoding Nested Base64/Archives\n",
    "The module will try to follow nested encodings. It uses the following algorithm:\n",
    "1. Search for a pattern in the input that looks like a Base64 encoded string\n",
    "2. If not a known undecodable_string, try to decode the matched pattern.\n",
    "   - If the base 64 string matches a known archive type (zip, tar, gzip) also decompress or unpack\n",
    "     - For multi-item archives (zip, tar) process each contained item recursively (i.e. go to item 1. with \n",
    "      child item as input)\n",
    "   - For anything that decodes to a UTF-8 or UTF-16 string replace the input pattern with the decoded string\n",
    "   - Recurse over resultant output (i.e. submit decoded/replaced string to 1.)\n",
    "3. If decoding fails, add to list of undecodable_strings (prevents infinite looping over something that looks like a base64 string but isn't)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T01:06:47.275122Z",
     "start_time": "2020-02-06T01:06:47.260131Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "powershell.exe  -nop -w hidden -encodedcommand \n",
      "<decoded value='multiple binary' type='multiple' index='1' depth='1'>\n",
      "<decoded value='binary'  name='[zip] Filename: PeDll.dll' type='zip' index='1.1' depth='2'>3e 7b f4 bf 50 41 33 30 01 23 9d 3f 8d 4c d4 01 b0 5e 08 d0 3f c4 0c 01 a0 71 00 50 08 21 9c a6 12 1a 66 81 4b 3f a9 a6 d3 9e 53 60 80 22 01 03 00 00 80 00 00 00 00 00 00 00 00 18 c0 83 f6 fc 01 60 2d aa aa aa aa aa aa aa aa aa aa aa 0a aa aa 1a 1a 80 a1 2d aa aa aa aa aa aa aa aa aa 2a a2 11 fa c8 00 e8 7f 01 60 fd 07 c0 ff 05 80 ff 07 c0 ff 05 40 ff 01 46 00 b3 03 40 28 87 91 69 76 00 c8 20 a3 03 00 20 62 13</decoded>\n",
      "<decoded type='string' name='[zip] Filename: b64inzip.foo' index='1.2' depth='2'>Base64 encoded string in zip file</decoded>\n",
      "<decoded type='string' name='[zip] Filename: PlainTextInZip.dll' index='1.3' depth='2'>Unencoded text file in zip</decoded></decoded>\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:390: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results = df_results.append(\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:431: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results.append(child_records, ignore_index=True, sort=False),\n"
     ]
    }
   ],
   "source": [
    "encoded_cmd = '''\n",
    "powershell.exe  -nop -w hidden -encodedcommand \n",
    "UEsDBBQAAAAIAGBXkk3LfdszdwAAAIoAAAAJAAAAUGVEbGwuZGxss6v+sj/A0diA\n",
    "UXmufa/PFcYNcRwX7I/wMC4oZAjgUJyzTEgqrdHbfuWyy/OCExqUGJkZGBoYoEDi\n",
    "QPO3P4wJuqsQgGvVKimphoUIIa1Fgr9OMLyoZ0z4y37gP2vDfxDp8J/RjWEzs4NG\n",
    "+8TMMoYTCouZGRSShAFQSwMEFAAAAAAAYYJrThx8YzUhAAAAIQAAAAwAAABiNjRp\n",
    "bnppcC5mb29CYXNlNjQgZW5jb2RlZCBzdHJpbmcgaW4gemlwIGZpbGVQSwMEFAAA\n",
    "AAAAi4JrTvMfsJUaAAAAGgAAABIAAABQbGFpblRleHRJblppcC5kbGxVbmVuY29k\n",
    "ZWQgdGV4dCBmaWxlIGluIHppcFBLAQIUABQAAAAIAGBXkk3LfdszdwAAAIoAAAAJ\n",
    "AAAAAAAAAAAAIAAAAAAAAABQZURsbC5kbGxQSwECFAAUAAAAAABhgmtOHHxjNSEA\n",
    "AAAhAAAADAAAAAAAAAABACAAAACeAAAAYjY0aW56aXAuZm9vUEsBAhQAFAAAAAAA\n",
    "i4JrTvMfsJUaAAAAGgAAABIAAAAAAAAAAQAgAAAA6QAAAFBsYWluVGV4dEluWmlw\n",
    "LmRsbFBLBQYAAAAAAwADALEAAAAzAQAAAAA='''\n",
    "\n",
    "import re\n",
    "dec_string, dec_df = base64unpack.unpack(input_string=encoded_cmd)\n",
    "print(dec_string.replace('<decoded', '\\n<decoded'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IPython magic\n",
    "\n",
    "You can use the line magic `%b64` or cell magic `%%b64` to decode text directly in a cell\n",
    "\n",
    "The b64 magic supports the following options:\n",
    "\n",
    "```\n",
    "--out OUT, -o OUT  The variable to return the results in the variable `OUT`\n",
    "                   Note: the output is a tuple of decoded string and pandas DataFrame\n",
    "--pretty, -p       Print formatted version of output (if you `print` the output)\n",
    "--clean, -c        Print decoded string with no formatting\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T01:06:47.290113Z",
     "start_time": "2020-02-06T01:06:47.276122Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:390: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results = df_results.append(\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:431: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results.append(child_records, ignore_index=True, sort=False),\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<?xml version=\"1.0\" encoding=\"utf-8\"?>\\n<decoded_string>\\n powershell.exe  -nop -w hidden -encodedcommand\\n <decoded depth=\"1\" index=\"1\" type=\"multiple\" value=\"multiple binary\">\\n  <decoded depth=\"2\" index=\"1.1\" name=\"[zip] Filename: PeDll.dll\" type=\"zip\" value=\"binary\">\\n   3e 7b f4 bf 50 41 33 30 01 23 9d 3f 8d 4c d4 01 b0 5e 08 d0 3f c4 0c 01 a0 71 00 50 08 21 9c a6 12 1a 66 81 4b 3f a9 a6 d3 9e 53 60 80 22 01 03 00 00 80 00 00 00 00 00 00 00 00 18 c0 83 f6 fc 01 60 2d aa aa aa aa aa aa aa aa aa aa aa 0a aa aa 1a 1a 80 a1 2d aa aa aa aa aa aa aa aa aa 2a a2 11 fa c8 00 e8 7f 01 60 fd 07 c0 ff 05 80 ff 07 c0 ff 05 40 ff 01 46 00 b3 03 40 28 87 91 69 76 00 c8 20 a3 03 00 20 62 13\\n  </decoded>\\n  <decoded depth=\"2\" index=\"1.2\" name=\"[zip] Filename: b64inzip.foo\" type=\"string\">\\n   Base64 encoded string in zip file\\n  </decoded>\\n  <decoded depth=\"2\" index=\"1.3\" name=\"[zip] Filename: PlainTextInZip.dll\" type=\"string\">\\n   Unencoded text file in zip\\n  </decoded>\\n </decoded>\\n</decoded_string>'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%b64 --pretty --out dec_xml\n",
    "powershell.exe  -nop -w hidden -encodedcommand \n",
    "UEsDBBQAAAAIAGBXkk3LfdszdwAAAIoAAAAJAAAAUGVEbGwuZGxss6v+sj/A0diA\n",
    "UXmufa/PFcYNcRwX7I/wMC4oZAjgUJyzTEgqrdHbfuWyy/OCExqUGJkZGBoYoEDi\n",
    "QPO3P4wJuqsQgGvVKimphoUIIa1Fgr9OMLyoZ0z4y37gP2vDfxDp8J/RjWEzs4NG\n",
    "+8TMMoYTCouZGRSShAFQSwMEFAAAAAAAYYJrThx8YzUhAAAAIQAAAAwAAABiNjRp\n",
    "bnppcC5mb29CYXNlNjQgZW5jb2RlZCBzdHJpbmcgaW4gemlwIGZpbGVQSwMEFAAA\n",
    "AAAAi4JrTvMfsJUaAAAAGgAAABIAAABQbGFpblRleHRJblppcC5kbGxVbmVuY29k\n",
    "ZWQgdGV4dCBmaWxlIGluIHppcFBLAQIUABQAAAAIAGBXkk3LfdszdwAAAIoAAAAJ\n",
    "AAAAAAAAAAAAIAAAAAAAAABQZURsbC5kbGxQSwECFAAUAAAAAABhgmtOHHxjNSEA\n",
    "AAAhAAAADAAAAAAAAAABACAAAACeAAAAYjY0aW56aXAuZm9vUEsBAhQAFAAAAAAA\n",
    "i4JrTvMfsJUaAAAAGgAAABIAAAAAAAAAAQAgAAAA6QAAAFBsYWluVGV4dEluWmlw\n",
    "LmRsbFBLBQYAAAAAAwADALEAAAAzAQAAAAA="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T01:06:47.294164Z",
     "start_time": "2020-02-06T01:06:47.291113Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
      "<decoded_string>\n",
      " powershell.exe  -nop -w hidden -encodedcommand\n",
      " <decoded depth=\"1\" index=\"1\" type=\"multiple\" value=\"multiple binary\">\n",
      "  <decoded depth=\"2\" index=\"1.1\" name=\"[zip] Filename: PeDll.dll\" type=\"zip\" value=\"binary\">\n",
      "   3e 7b f4 bf 50 41 33 30 01 23 9d 3f 8d 4c d4 01 b0 5e 08 d0 3f c4 0c 01 a0 71 00 50 08 21 9c a6 12 1a 66 81 4b 3f a9 a6 d3 9e 53 60 80 22 01 03 00 00 80 00 00 00 00 00 00 00 00 18 c0 83 f6 fc 01 60 2d aa aa aa aa aa aa aa aa aa aa aa 0a aa aa 1a 1a 80 a1 2d aa aa aa aa aa aa aa aa aa 2a a2 11 fa c8 00 e8 7f 01 60 fd 07 c0 ff 05 80 ff 07 c0 ff 05 40 ff 01 46 00 b3 03 40 28 87 91 69 76 00 c8 20 a3 03 00 20 62 13\n",
      "  </decoded>\n",
      "  <decoded depth=\"2\" index=\"1.2\" name=\"[zip] Filename: b64inzip.foo\" type=\"string\">\n",
      "   Base64 encoded string in zip file\n",
      "  </decoded>\n",
      "  <decoded depth=\"2\" index=\"1.3\" name=\"[zip] Filename: PlainTextInZip.dll\" type=\"string\">\n",
      "   Unencoded text file in zip\n",
      "  </decoded>\n",
      " </decoded>\n",
      "</decoded_string>\n"
     ]
    }
   ],
   "source": [
    "print(dec_xml[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Extension\n",
    "\n",
    "The decoding functionality is also available in a pandas extension `mp`.\n",
    "This supports a method `b64extract()`. \n",
    "\n",
    "This supports the same syntax\n",
    "as `unpack_df` (described earlier)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T01:06:47.360891Z",
     "start_time": "2020-02-06T01:06:47.295155Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:390: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results = df_results.append(\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:431: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results.append(child_records, ignore_index=True, sort=False),\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:390: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results = df_results.append(\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:431: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results.append(child_records, ignore_index=True, sort=False),\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:390: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results = df_results.append(\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:431: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results.append(child_records, ignore_index=True, sort=False),\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:390: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results = df_results.append(\n",
      "e:\\src\\msticpy\\msticpy\\transform\\base64unpack.py:431: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df_results.append(child_records, ignore_index=True, sort=False),\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reference</th>\n",
       "      <th>original_string</th>\n",
       "      <th>file_name</th>\n",
       "      <th>file_type</th>\n",
       "      <th>input_bytes</th>\n",
       "      <th>decoded_string</th>\n",
       "      <th>encoding_type</th>\n",
       "      <th>file_hashes</th>\n",
       "      <th>md5</th>\n",
       "      <th>sha1</th>\n",
       "      <th>sha256</th>\n",
       "      <th>printable_bytes</th>\n",
       "      <th>src_index</th>\n",
       "      <th>CommandLine</th>\n",
       "      <th>full_decoded_string</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4AHAAcgBlAHMAcwBpAG8A...</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b\"$\\x00t\\x00 \\x00=\\x00 \\x00'\\x00d\\x00i\\x00r\\x00'\\x00;\\x00\\r\\x00\\n\\x00&amp;\\x00 \\x00(\\x00'\\x00I\\x00n\\...</td>\n",
       "      <td>$\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000;\u0000\\r\u0000\\n\u0000&amp;\u0000 \u0000(\u0000'\u0000I\u0000n\u0000v\u0000o\u0000k\u0000e\u0000'\u0000+\u0000'\u0000-\u0000E\u0000x\u0000p\u0000r\u0000e\u0000s\u0000s\u0000i\u0000o\u0000n\u0000'\u0000)\u0000 \u0000$\u0000t\u0000</td>\n",
       "      <td>utf-8</td>\n",
       "      <td>{'md5': '6cd1486db221e532cc2011c9beeb4ffc', 'sha1': '6e485467d7e06502046b7c84a8ef067cfe1512ad', ...</td>\n",
       "      <td>6cd1486db221e532cc2011c9beeb4ffc</td>\n",
       "      <td>6e485467d7e06502046b7c84a8ef067cfe1512ad</td>\n",
       "      <td>d3291dab1ae552b91e6b50d7460ceaa39f6f92b2cda4335dd77e28d25c62ce34</td>\n",
       "      <td>24 00 74 00 20 00 3d 00 20 00 27 00 64 00 69 00 72 00 27 00 3b 00 0d 00 0a 00 26 00 20 00 28 00 ...</td>\n",
       "      <td>39</td>\n",
       "      <td>.\\powershell  -enc JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4A...</td>\n",
       "      <td>.\\powershell  -enc &lt;decoded type='string' name='[None]' index='1' depth='1'&gt;$\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'</td>\n",
       "      <td>None</td>\n",
       "      <td>binary</td>\n",
       "      <td>{'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...</td>\n",
       "      <td>9a45b2520e930dc9186f6d93a7798a13</td>\n",
       "      <td>f526c90fa0744e3a63d84421ff25e3f5a3d697cb</td>\n",
       "      <td>c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239</td>\n",
       "      <td>69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a</td>\n",
       "      <td>40</td>\n",
       "      <td>cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &gt;&gt; blah.ps1\"</td>\n",
       "      <td>cmd  /c \"echo # &lt;decoded value='binary'  name='[None]' type='None' index='1' depth='1'&gt;69 a6 9a ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'</td>\n",
       "      <td>None</td>\n",
       "      <td>binary</td>\n",
       "      <td>{'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...</td>\n",
       "      <td>9a45b2520e930dc9186f6d93a7798a13</td>\n",
       "      <td>f526c90fa0744e3a63d84421ff25e3f5a3d697cb</td>\n",
       "      <td>c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239</td>\n",
       "      <td>69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a</td>\n",
       "      <td>41</td>\n",
       "      <td>cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa &gt;&gt; blah.ps1\"</td>\n",
       "      <td>cmd  /c \"echo # &lt;decoded value='binary'  name='[None]' type='None' index='1' depth='1'&gt;69 a6 9a ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(, 1., 1)</td>\n",
       "      <td>81ed03caf6901e444c72ac67d192fb9c</td>\n",
       "      <td>unknown</td>\n",
       "      <td>None</td>\n",
       "      <td>b'\\xf3W\\x9d\\xd3w\\x1a\\x7f\\xaft\\xd5\\xee8\\xe1\\xce\\xf6i\\xce\\xbbw_v}\\xbf\\\\'</td>\n",
       "      <td>None</td>\n",
       "      <td>binary</td>\n",
       "      <td>{'md5': '1c8cc6299bd654bbcd85710968d6a87c', 'sha1': '55377391141f59a2ff5ae4765d9f0b4438adfd73', ...</td>\n",
       "      <td>1c8cc6299bd654bbcd85710968d6a87c</td>\n",
       "      <td>55377391141f59a2ff5ae4765d9f0b4438adfd73</td>\n",
       "      <td>fd80ceba7cfb49d296886c10d9a3497d63c89a589587cda7d818cb4644842660</td>\n",
       "      <td>f3 57 9d d3 77 1a 7f af 74 d5 ee 38 e1 ce f6 69 ce bb 77 5f 76 7d bf 5c</td>\n",
       "      <td>44</td>\n",
       "      <td>implant.exe  81ed03caf6901e444c72ac67d192fb9c</td>\n",
       "      <td>implant.exe  &lt;decoded value='binary'  name='[None]' type='None' index='1' depth='1'&gt;f3 57 9d d3 ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   reference  \\\n",
       "0  (, 1., 1)   \n",
       "1  (, 1., 1)   \n",
       "2  (, 1., 1)   \n",
       "3  (, 1., 1)   \n",
       "\n",
       "                                                                                       original_string  \\\n",
       "0  JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4AHAAcgBlAHMAcwBpAG8A...   \n",
       "1                                                                     aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa   \n",
       "2                                                                     aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa   \n",
       "3                                                                     81ed03caf6901e444c72ac67d192fb9c   \n",
       "\n",
       "  file_name file_type  \\\n",
       "0   unknown      None   \n",
       "1   unknown      None   \n",
       "2   unknown      None   \n",
       "3   unknown      None   \n",
       "\n",
       "                                                                                           input_bytes  \\\n",
       "0  b\"$\\x00t\\x00 \\x00=\\x00 \\x00'\\x00d\\x00i\\x00r\\x00'\\x00;\\x00\\r\\x00\\n\\x00&\\x00 \\x00(\\x00'\\x00I\\x00n\\...   \n",
       "1                          b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'   \n",
       "2                          b'i\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9ai\\xa6\\x9a'   \n",
       "3                               b'\\xf3W\\x9d\\xd3w\\x1a\\x7f\\xaft\\xd5\\xee8\\xe1\\xce\\xf6i\\xce\\xbbw_v}\\xbf\\\\'   \n",
       "\n",
       "                                                                           decoded_string  \\\n",
       "0  $\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000;\u0000\\r\u0000\\n\u0000&\u0000 \u0000(\u0000'\u0000I\u0000n\u0000v\u0000o\u0000k\u0000e\u0000'\u0000+\u0000'\u0000-\u0000E\u0000x\u0000p\u0000r\u0000e\u0000s\u0000s\u0000i\u0000o\u0000n\u0000'\u0000)\u0000 \u0000$\u0000t\u0000   \n",
       "1                                                                                    None   \n",
       "2                                                                                    None   \n",
       "3                                                                                    None   \n",
       "\n",
       "  encoding_type  \\\n",
       "0         utf-8   \n",
       "1        binary   \n",
       "2        binary   \n",
       "3        binary   \n",
       "\n",
       "                                                                                           file_hashes  \\\n",
       "0  {'md5': '6cd1486db221e532cc2011c9beeb4ffc', 'sha1': '6e485467d7e06502046b7c84a8ef067cfe1512ad', ...   \n",
       "1  {'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...   \n",
       "2  {'md5': '9a45b2520e930dc9186f6d93a7798a13', 'sha1': 'f526c90fa0744e3a63d84421ff25e3f5a3d697cb', ...   \n",
       "3  {'md5': '1c8cc6299bd654bbcd85710968d6a87c', 'sha1': '55377391141f59a2ff5ae4765d9f0b4438adfd73', ...   \n",
       "\n",
       "                                md5                                      sha1  \\\n",
       "0  6cd1486db221e532cc2011c9beeb4ffc  6e485467d7e06502046b7c84a8ef067cfe1512ad   \n",
       "1  9a45b2520e930dc9186f6d93a7798a13  f526c90fa0744e3a63d84421ff25e3f5a3d697cb   \n",
       "2  9a45b2520e930dc9186f6d93a7798a13  f526c90fa0744e3a63d84421ff25e3f5a3d697cb   \n",
       "3  1c8cc6299bd654bbcd85710968d6a87c  55377391141f59a2ff5ae4765d9f0b4438adfd73   \n",
       "\n",
       "                                                             sha256  \\\n",
       "0  d3291dab1ae552b91e6b50d7460ceaa39f6f92b2cda4335dd77e28d25c62ce34   \n",
       "1  c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239   \n",
       "2  c1f6c05bdbe28a58557a9477cd0fa96fbc5e7c54ceb6057ec15eca4c664c4239   \n",
       "3  fd80ceba7cfb49d296886c10d9a3497d63c89a589587cda7d818cb4644842660   \n",
       "\n",
       "                                                                                       printable_bytes  \\\n",
       "0  24 00 74 00 20 00 3d 00 20 00 27 00 64 00 69 00 72 00 27 00 3b 00 0d 00 0a 00 26 00 20 00 28 00 ...   \n",
       "1                              69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a   \n",
       "2                              69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a 69 a6 9a   \n",
       "3                              f3 57 9d d3 77 1a 7f af 74 d5 ee 38 e1 ce f6 69 ce bb 77 5f 76 7d bf 5c   \n",
       "\n",
       "   src_index  \\\n",
       "0         39   \n",
       "1         40   \n",
       "2         41   \n",
       "3         44   \n",
       "\n",
       "                                                                                           CommandLine  \\\n",
       "0  .\\powershell  -enc JAB0ACAAPQAgACcAZABpAHIAJwA7AA0ACgAmACAAKAAnAEkAbgB2AG8AawBlACcAKwAnAC0ARQB4A...   \n",
       "1                                        cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> blah.ps1\"   \n",
       "2                                        cmd  /c \"echo # aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >> blah.ps1\"   \n",
       "3                                                        implant.exe  81ed03caf6901e444c72ac67d192fb9c   \n",
       "\n",
       "                                                                                   full_decoded_string  \n",
       "0  .\\powershell  -enc <decoded type='string' name='[None]' index='1' depth='1'>$\u0000t\u0000 \u0000=\u0000 \u0000'\u0000d\u0000i\u0000r\u0000'\u0000...  \n",
       "1  cmd  /c \"echo # <decoded value='binary'  name='[None]' type='None' index='1' depth='1'>69 a6 9a ...  \n",
       "2  cmd  /c \"echo # <decoded value='binary'  name='[None]' type='None' index='1' depth='1'>69 a6 9a ...  \n",
       "3  implant.exe  <decoded value='binary'  name='[None]' type='None' index='1' depth='1'>f3 57 9d d3 ...  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "process_tree.mp.b64extract(column='CommandLine')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='todos'></a>[Contents](#contents)\n",
    "## To-Do Items\n",
    "- Use more comprehensive list of binary magic numbers and match on byte values after decoding to get better file typing\n",
    "- Output nested decodings in a more readable output\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "318.996px",
    "width": "320.994px"
   },
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "406.193px",
    "left": "1468.4px",
    "right": "20px",
    "top": "120px",
    "width": "456.572px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}