docs/notebooks/AzureBlobStorage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Azure Blob Service Functions\n",
    "\n",
    "MSTICpy versions > 0.8.9\n",
    "\n",
    "### Description\n",
    "\n",
    "This Notebook provides an example of using the Azure Blob Storage features of MSTICpy in order interact with an Azure Blob Storage account.\n",
    "\n",
    "You must have msticpy installed to run this notebook:\n",
    "```\n",
    "%pip install --upgrade msticpy[azure]\n",
    "```\n",
    "\n",
    "### Installation and imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from msticpy.data.storage import AzureBlobStorage\n",
    "from msticpy.data import data_obfus as mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step is to initalize the AzureBlobStorage client. Each client is specific to a seperate Azure Blob Storage account and you must pass it the account name when initalizing the client. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "abs = AzureBlobStorage(abs_name=\"abscontainer\")\n",
    "abs.connect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step when working with Azure Blob Storage is identifying the container to store blobs in, we can enumerate all the containers in the account with `.containers()`. Note that only containers you have permissons to access will be shown, the Azure Blob Storage features of MSTICpy use the same authentication methods as other Azure service elements and you can specific `auth_methods` when calling `.connect` to select the required authentication method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attempting to sign-in with environment variable credentials...\n",
      "obfuscating columns:\n",
      "name, etag, \n",
      "done\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>last_modified</th>\n",
       "      <th>etag</th>\n",
       "      <th>public_access</th>\n",
       "      <th>has_immutability_policy</th>\n",
       "      <th>deleted</th>\n",
       "      <th>version</th>\n",
       "      <th>has_legal_hold</th>\n",
       "      <th>metadata</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bikjdaerb</td>\n",
       "      <td>2020-11-06 21:53:33+00:00</td>\n",
       "      <td>rbjkmanljijmqpqmpjc</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>epfdkmlmqkmqb</td>\n",
       "      <td>2020-11-19 15:22:38+00:00</td>\n",
       "      <td>ooidklonjqooimbjkpb</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name             last_modified                 etag public_access  \\\n",
       "0      bikjdaerb 2020-11-06 21:53:33+00:00  rbjkmanljijmqpqmpjc          None   \n",
       "1  epfdkmlmqkmqb 2020-11-19 15:22:38+00:00  ooidklonjqooimbjkpb          None   \n",
       "\n",
       "   has_immutability_policy deleted version  has_legal_hold metadata  \n",
       "0                    False    None    None           False     None  \n",
       "1                    False    None    None           False     None  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "containers = abs.containers()\n",
    "display(containers.mp_obf.obfuscate(column_map={\"etag\": \"str\", \"name\":\"str\"}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also create new containers with `.create_container()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>last_modified</th>\n",
       "      <th>etag</th>\n",
       "      <th>public_access</th>\n",
       "      <th>has_immutability_policy</th>\n",
       "      <th>deleted</th>\n",
       "      <th>version</th>\n",
       "      <th>has_legal_hold</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>documenationcontainer</td>\n",
       "      <td>2020-11-24 17:40:53+00:00</td>\n",
       "      <td>\"0x8D890A0179F6EAA\"</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    name             last_modified                 etag  \\\n",
       "0  documenationcontainer 2020-11-24 17:40:53+00:00  \"0x8D890A0179F6EAA\"   \n",
       "\n",
       "  public_access  has_immutability_policy deleted version  has_legal_hold  \n",
       "0          None                    False    None    None           False  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "abs.create_container(container_name=\"documenationcontainer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have a container we can write blobs to the container. Blob objects can be of any type but here we are passing a simple string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Upload complete\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "abs.upload_to_blob(blob=\"Here is some test data\", container_name=\"documenationcontainer\", blob_name=\"test-blob\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can list blobs within a container in the same way we listed containers, here you can see the blob we just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>container</th>\n",
       "      <th>snapshot</th>\n",
       "      <th>version_id</th>\n",
       "      <th>is_current_version</th>\n",
       "      <th>blob_type</th>\n",
       "      <th>encrypted_metadata</th>\n",
       "      <th>last_modified</th>\n",
       "      <th>etag</th>\n",
       "      <th>size</th>\n",
       "      <th>...</th>\n",
       "      <th>remaining_retention_days</th>\n",
       "      <th>creation_time</th>\n",
       "      <th>archive_status</th>\n",
       "      <th>encryption_key_sha256</th>\n",
       "      <th>encryption_scope</th>\n",
       "      <th>request_server_encrypted</th>\n",
       "      <th>object_replication_source_properties</th>\n",
       "      <th>object_replication_destination_policy</th>\n",
       "      <th>tag_count</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>test-blob</td>\n",
       "      <td>documenationcontainer</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>BlobType.BlockBlob</td>\n",
       "      <td>None</td>\n",
       "      <td>2020-11-24 17:47:41+00:00</td>\n",
       "      <td>0x8D890A10A6DC6A3</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>2020-11-24 17:47:41+00:00</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 34 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        name              container snapshot version_id is_current_version  \\\n",
       "0  test-blob  documenationcontainer     None       None               None   \n",
       "\n",
       "            blob_type encrypted_metadata             last_modified  \\\n",
       "0  BlobType.BlockBlob               None 2020-11-24 17:47:41+00:00   \n",
       "\n",
       "                etag  size  ... remaining_retention_days  \\\n",
       "0  0x8D890A10A6DC6A3    22  ...                     None   \n",
       "\n",
       "              creation_time archive_status encryption_key_sha256  \\\n",
       "0 2020-11-24 17:47:41+00:00           None                  None   \n",
       "\n",
       "   encryption_scope request_server_encrypted  \\\n",
       "0              None                     None   \n",
       "\n",
       "  object_replication_source_properties object_replication_destination_policy  \\\n",
       "0                                   []                                  None   \n",
       "\n",
       "  tag_count  tags  \n",
       "0      None  None  \n",
       "\n",
       "[1 rows x 34 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "abs.blobs(container_name=\"documenationcontainer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also return the contents of a blob like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'Here is some test data'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "abs.get_blob(container_name=\"documenationcontainer\", blob_name=\"test-blob\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When sharing blobs its often necessary to access a SAS token that allow a user to access a specific blob for a specific amount of time, without having to have explicity permissions on the Azure Blob Storage account.\n",
    "In the example below we obsfucate the retruned SAS token but it will take the form of a full URL to the blob, with the required SAS token appeneded. By default tokens generated allow access for 7 days but this can be adjusted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'irrjnoieonelkirmqkerfibemcjbclillbjcncpiooqjklelmqebrambdkjaaqpqirrjnoieonelkirmqkerfibemcjbclillbjcncpiooqjklelmqebrambdkjaaqpqirrjnoieonelkirmqkerfibemcjbclillbjcncpiooqjklelmqebrambdkjaaqpqirrjnoieonelkirmqkerfibemcjbclillbjcncpiooqjklelmqebrambdkjaaqpqirrjnoieonelkirmqkerfibemcjbclillbjcncpiooqjklelmqebrambdkjaaqpq'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sast = abs.get_sas_token(container_name=\"documenationcontainer\", blob_name=\"test-blob\")\n",
    "mask.hash_string(sast)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we can delete a blob, below we delete the blob we just created and then list all blobs in the container, which returns nothing as it is now empty:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "abs.delete_blob(container_name=\"documenationcontainer\", blob_name=\"test-blob\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "abs.blobs(container_name=\"documenationcontainer\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}