Beginning Machine and Deep Learning with Zeek logs/03 - beginning torch.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: Nik Alleyne <br>\n",
    "Author Blog: **https://www.securitynik.com** <br>\n",
    "Author GitHub: **github.com/securitynik** <br>\n",
    "\n",
    "Author Books: [  <br>\n",
    "\n",
    "                \"https://www.amazon.ca/Learning-Practicing-Leveraging-Practical-Detection/dp/1731254458/\", \n",
    "                \n",
    "                \"https://www.amazon.ca/Learning-Practicing-Mastering-Network-Forensics/dp/1775383024/\" \n",
    "            ] \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why this series?\n",
    "When teaching the SANS SEC595: Applied Data Science and Machine Learning for Cybersecurity Professionals \n",
    "**https://www.sans.org/cyber-security-courses/applied-data-science-machine-learning/** I am always asked,\n",
    "\"Will you be sharing your demo notebooks?\" or \"Can we get a copy of your demo notebooks?\" or ... well you get the point.\n",
    "My answer is always no. Not that I do not want to share, (sharing is caring :-D) , but the demo notebooks \n",
    "by themselves, would not make sense or add real value. Hence, this series! \n",
    "\n",
    "This is my supplemental work, similar to what I would do in the demos but with a lot more details and references."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 03 - Beginning PyTorch"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The series includes the following: <br>\n",
    "01 - Beginning Numpy <br>\n",
    "02 - Beginning Tensorflow  <br>\n",
    "03 - Beginning PyTorch <br>\n",
    "04 - Beginning Pandas <br>\n",
    "05 - Beginning Matplotlib <br>\n",
    "06 - Beginning Data Scaling <br>\n",
    "07 - Beginning Principal Component Analysis (PCA) <br>\n",
    "08 - Beginning Machine Learning Anomaly Detection - Isolation Forest and Local Outlier Factor <br>\n",
    "09 - Beginning Unsupervised Machine Learning - Clustering - K-means and DBSCAN <br>\n",
    "10 - Beginning Supervise Learning - Machine Learning - Logistic Regression, Decision Trees and Metrics <br>\n",
    "11 - Beginning Linear Regression - Machine Learning <br>\n",
    "12 - Beginning Deep Learning - Anomaly Detection with AutoEncoders, Tensorflow <br>\n",
    "13 - Beginning Deep Learning - Anomaly Detection with AutoEncoders, PyTroch <br>\n",
    "14 - Beginning Deep Learning - Linear Regression, Tensorflow <br>\n",
    "15 - Beginning Deep Learning - Linear Regression, PyTorch <br>\n",
    "16 - Beginning Deep Learning - Classification, Tensorflow <br>\n",
    "17 - Beginning Deep Learning - Classification, Pytorch <br>\n",
    "18 - Beginning Deep Learning - Classification - regression - MIMO - Functional API Tensorflow <br> \n",
    "19 - Beginning Deep Learning - Convolution Networks - Tensorflow <br>\n",
    "20 - Beginning Deep Learning - Convolution Networks - PyTorch <br>\n",
    "21 - Beginning Regularization - Early Stopping, Dropout, L2 (Ridge), L1 (Lasso) <br>\n",
    "22 - Beginning Model TFServing <br>\n",
    "\n",
    "But conn.log is not the only log file within Zeek. Let's build some models for DNS and HTTP logs. <br>\n",
    "I choose unsupervised, because there are no labels coming with these data. <br>\n",
    "\n",
    "23 - Continuing Anomaly Learning - Zeek DNS Log - Machine Learning <br>\n",
    "24 - Continuing Unsupervised Learning - Zeek HTTP Log - Machine Learning <br>\n",
    "\n",
    "This was a specific ask by someone in one of my class. <br>\n",
    "25 - Beginning - Reading Executables and Building a Neural Network to make predictions on suspicious vs suspicious  <br><br>\n",
    "\n",
    "With 25 notebooks in this series, it is quite possible there are things I could have or should have done differently.  <br>\n",
    "If you find any thing, you think fits those criteria, drop me a line. <br>\n",
    "\n",
    "If you find this series beneficial, I would greatly appreciate your feedback.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the torch library\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Similar to Tensorflow - see \n",
    "#       02 - Beginning Tensorflow  \n",
    "# PyTorch is a deep learning library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First up, PyTorch is a deep learning framework\n",
    "# While Numpy as seen in notebook \n",
    "#   01 - Beginning Numpy\n",
    "# cannot be used with GPU, PyTorch can\n",
    "# The system this notebook is being developed on does not have a GPU\n",
    "# Confirming the devices currently available and we see CUDA is not available\n",
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([10])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setup an integer Tensor array with 1 item\n",
    "x = torch.tensor([10])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the datetype of x\n",
    "x.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Tensor"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Confirm the type is of tensor\n",
    "type(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Because no GPU is available, by default everything is placed on the CPU\n",
    "x.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([10, 20, 30])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setup the tensor with multiple integer items\n",
    "x = torch.tensor([10, 20, 30])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([10., 20., 30.]), torch.float32)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setup the tensor with multiple float items\n",
    "x = torch.tensor([10., 20., 30.])\n",
    "x, x.dtype\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([10., 20., 30.])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Alternatively, cast the multiple item tensor from integer to float\n",
    "x = torch.tensor([10, 20, 30], dtype=torch.float32)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[10., 20., 30.]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add a new dimension to the tensor\n",
    "# Make it a 2 dimension tensor\n",
    "# Notice the additional \"[\" and \"]\"\n",
    "x = torch.tensor([[10, 20, 30]], dtype=torch.float32)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[10., 20., 30.]]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also add a new dimension by unsqueezing axis 1\n",
    "# This also moves the x above from 2 to 3 dimensions\n",
    "x.unsqueeze(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[10., 20., 30.]]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Alternatively, we can manually create a tensor of 3 dimensions\n",
    "x = torch.tensor([[[10, 20, 30]]], dtype=torch.float32)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[10.],\n",
       "        [20.],\n",
       "        [30.]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reshape the x tensor\n",
    "# In this case, (-1, 1) means any amount of rows but only one column\n",
    "# For this scenario, since x is 1 row and 3 columns, this transitions it to 3 rows and 1 column\n",
    "# Notice the transition from 1 to 2 dimensions\n",
    "x = torch.tensor([[[10, 20, 30]]], dtype=torch.float32).reshape(-1,1)\n",
    "x\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([10., 20., 30.]), torch.Size([3]))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Like the unsqueezing above, we can squeeze also\n",
    "# In this case, we move x from a 3x1 shape to a vector of shape 3\n",
    "# Notice above we have 3 rows now we have 1\n",
    "x.squeeze(dim=1), x.squeeze(dim=1).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[10., 20., 30.]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reshape the x tensor\n",
    "# In this case, (-1, 1) means any amount of rows but only one column\n",
    "# For this scenario, since x is 1 row and 3 columns, this keeps it to 3 rows and 1 column\n",
    "# Notice the transition from 1 to 2 dimensions\n",
    "x = torch.tensor([[[10, 20, 30]]], dtype=torch.float32).reshape(1,-1)\n",
    "x\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[1., 2., 3., 4., 5.]]), tensor([[6., 7., 8., 9., 0.]]))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create two tensors to stack\n",
    "# Notice, in this case, unlike Numpy and Tensorflow, I created these tensors with 2 dimensions\n",
    "#   01 - Beginning Numpy\n",
    "#   02 - Beginning Tensorflow\n",
    "x = torch.tensor([[1, 2, 3, 4, 5]], dtype=torch.float32)\n",
    "y = torch.tensor([[6, 7, 8, 9, 0]], dtype=torch.float32)\n",
    "\n",
    "x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2., 3., 4., 5.],\n",
       "        [6., 7., 8., 9., 0.]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Stack x and y vertically. Going down the columns, i.e. axis=0\n",
    "# Note this needs to be a list/array of items\n",
    "# This is very helpful, if you would like to stack two datasets to create 1\n",
    "# Maybe you have new samples/observations\n",
    "z = torch.cat((x, y), dim=0)\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Similar to numpy vstack, we can use vstack in PyTorch\n",
    "#      01 - Beginning Numpy\n",
    "z = torch.vstack((x, y))\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2., 3., 4., 5., 6., 7., 8., 9., 0.]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Stack x and y across the columns, i.e. axis=1\n",
    "# Note this needs to be a tuple of items\n",
    "# This is helpful when you want to add new features to your dataset\n",
    "z = torch.cat((x, y), dim=1)\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2., 3., 4., 5., 6., 7., 8., 9., 0.]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We could also use torch.hstack here\n",
    "# Horizontal stack\n",
    "# This allows us to stack across axis=1\n",
    "z = torch.hstack((x, y))\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([5]),)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the offset within x where the value equals 4\n",
    "x = torch.tensor([10, 9, 8, 7, 6, 5, 4], dtype=torch.float32)\n",
    "z = torch.where((x == 5))\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(5.)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Confirming the return positioned \n",
    "x[5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate a 4*4 tensor of ones\n",
    "torch.ones((4,4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a 6x6 tensor with all zeros\n",
    "x = torch.zeros((6,6))\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Update the item at position row 0, column 0 with 2\n",
    "# Counting for both the rows and columns start 0\n",
    "# As a result, even though this tensor is 6x6, you \n",
    "# will be going from 0 to 5 for the indexes\n",
    "x[0,0] = 2\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  2.,   0.,   0.,   0.,   0.,   0.],\n",
       "        [  0.,   0.,   0.,   0.,   0.,   0.],\n",
       "        [  0.,   0.,   0.,   0.,   0.,   0.],\n",
       "        [  0.,   0.,   0.,   0.,   0., 100.],\n",
       "        [  0.,   0.,   0.,   0.,   0.,   0.],\n",
       "        [  0.,   0.,   0.,   0.,   0.,   0.]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Update the items at position row 3, column 5 with 100\n",
    "x[3,5] = 100\n",
    "x\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  2.,   0.,   0.,   0.,   0.,   0.],\n",
       "        [  0.,   0.,   0.,   0.,   0.,   0.],\n",
       "        [  0.,   0.,   0.,   0.,   0.,   0.],\n",
       "        [  0.,   0.,   0.,   0.,   0., 100.],\n",
       "        [  0.,   0.,   0.,   0.,   0.,   0.],\n",
       "        [  0.,  23.,  23.,  23.,   0.,   0.]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Maybe instead, manipulate columns 1, 2 and 3 of the last row. \n",
    "# Remember, the last row is row 5\n",
    "# We can do x[5, 1:4] = 23 \n",
    "# or\n",
    "x[-1, 1:4] = 23\n",
    "\n",
    "# you might notice below, while we specified 1:4, we only were able to update up to index position 3 \n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  2.,   0.,   0.,   0.,   0.,   0.],\n",
       "        [  0., -10., -10., -10., -10., -10.],\n",
       "        [  0.,   0.,   0.,   0.,   0.,   0.],\n",
       "        [  0.,   0.,   0.,   0.,   0., 100.],\n",
       "        [  0.,   0.,   0.,   0.,   0.,   0.],\n",
       "        [  0.,  23.,  23.,  23.,   0.,   0.]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# One more. Change the values from the last to the second column\n",
    "# Giving them a value of -10\n",
    "x[1, -5:] = -10\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(100.)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the max value of all the items in the tensor\n",
    "torch.max(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.return_types.max(\n",
       "values=tensor([  2.,  23.,  23.,  23.,   0., 100.]),\n",
       "indices=tensor([0, 5, 5, 5, 0, 3]))"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the max value in the tensor going down the columns \n",
    "# Going across axis=0\n",
    "# Notice torch returns the values as well as its index\n",
    "torch.max(x, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.return_types.max(\n",
       "values=tensor([  2.,   0.,   0., 100.,   0.,  23.]),\n",
       "indices=tensor([0, 0, 0, 5, 0, 1]))"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the max value in the tensor across each row\n",
    "# Notice torch returns the values as well as its index\n",
    "torch.max(x, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[10,  9,  8,  7,  5]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create an vector to be transposed\n",
    "x = torch.tensor([[10, 9, 8, 7, 5]])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[10],\n",
       "        [ 9],\n",
       "        [ 8],\n",
       "        [ 7],\n",
       "        [ 5]])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use the full transpose function to change from a row vector to a column vector\n",
    "x.transpose(-2, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[10],\n",
       "        [ 9],\n",
       "        [ 8],\n",
       "        [ 7],\n",
       "        [ 5]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Maybe you like the shorter way to transpose\n",
    "# simply use .T on the array\n",
    "x.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 0., 0., 0.],\n",
       "        [0., 1., 0., 0.],\n",
       "        [0., 0., 1., 0.],\n",
       "        [0., 0., 0., 1.]])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a 4 * 4 eye tensor\n",
    "# Notice all the ones on the diagonal\n",
    "# This is helpful also when you think about one-hot encoding\n",
    "x = torch.eye(n=4, m=4, dtype=torch.float32)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[6, 4, 3],\n",
       "        [9, 8, 0]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preparing to do some math\n",
    "# Define a tensor with axes 0 and 1\n",
    "x = torch.tensor([[6,4,3], [9, 8, 0]])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(30)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the sum of x\n",
    "x_sum = torch.sum(x)\n",
    "x_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([13, 17])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Rather than get the sum of the entire tensor, \n",
    "# Get the sum going across  the rows, i.e axis=1\n",
    "x_sum = torch.sum(x, axis=1)\n",
    "x_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([15, 12,  3])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Similarly get the sum  going down the columns, i.e axis=0\n",
    "x_sum = torch.sum(x, axis=0)\n",
    "x_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([4.3333, 5.6667])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the average going down the columns, axes=1\n",
    "# Notice I had to cast x's datatype from integer to float\n",
    "# if not, the returned average is an integer rather than a float\n",
    "x_avg = torch.mean(x.type(torch.float32), dim=1)\n",
    "x_avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([7.5000, 6.0000, 1.5000])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the average of going across the rows, axes=1\n",
    "# Notice I had to cast x's datatype from integer to float\n",
    "# If not, the returned average is an integer rather than a float\n",
    "x_avg = torch.mean(x.type(torch.float32), dim=0)\n",
    "x_avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(18)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate a random number between 10 and 20\n",
    "# Because I would like a scalar output, the size is empty\n",
    "# Scalar being a single value or rank 0 tensor\n",
    "torch.randint(low=10, high=20, size=())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Run: 0 \t value: 17\n",
      "Run: 1 \t value: 17\n",
      "Run: 2 \t value: 17\n",
      "Run: 3 \t value: 17\n",
      "Run: 4 \t value: 17\n"
     ]
    }
   ],
   "source": [
    "# You may have instances you wish to generate the same random number\n",
    "# Maybe for demonstration purposes. Like in this notebook :-) \n",
    "# In this case, first set the random seed\n",
    "\n",
    "for idx, value in enumerate(range(5)):\n",
    "    # Set the random seed\n",
    "    torch.random.manual_seed(10)\n",
    "\n",
    "    # Generate the number\n",
    "    print(f'Run: {idx} \\t value: {torch.randint(low=10, high=20, size=())}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(1), tensor([ 2, 10,  5,  2,  3], dtype=torch.int32))"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In looking for max values, you might instead want the index of that value\n",
    "# This is beneficial when dealing with Softmax activation functions in your neural networks output layer\n",
    "# The output below has '1' which means the max value is at index position 1. \n",
    "# Counting from 0, we see 10 in position 1\n",
    "x = torch.tensor([2, 10,5,2,3], dtype=torch.int32)\n",
    "torch.argmax(x), x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[2, 3, 4]], dtype=torch.int32),\n",
       " tensor([[5],\n",
       "         [4],\n",
       "         [3]], dtype=torch.int32))"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Multiply two tensor\n",
    "# First create the matrices \n",
    "x = torch.tensor([[2,3,4]], dtype=torch.int32)\n",
    "y = torch.tensor([[5],[4],[3]], dtype=torch.int32)\n",
    "x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1, 3]), torch.Size([3, 1]))"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remember, if we are going to perform the dot product on two tensors\n",
    "# we need to ensure the inner dimension are the same.\n",
    "# Let's confirm the shape of these two tensors\n",
    "# We get 3,3 for the inner dimensions\n",
    "x.shape, y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[34]], dtype=torch.int32)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the dot product of the two vectors\n",
    "# Note the need to go across dimension 1\n",
    "torch.tensordot(a=x, b=y, dims=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[10, 15, 20],\n",
       "        [ 8, 12, 16],\n",
       "        [ 6,  9, 12]], dtype=torch.int32)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Alternatively, perform a pairwise or Hadamar product\n",
    "torch.multiply(input=x, other=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2, 3, 4]], dtype=torch.int32)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the cumulative sum of the vector\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2, 5, 9]])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the cumulative sum\n",
    "# Notice how the first value in x remains the same, then the first and second are added\n",
    "# Then the second and third are added\n",
    "torch.cumsum(x, dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2, 3, 4]])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Do the same for dimension 0. \n",
    "# Note no changes to x\n",
    "torch.cumsum(x, dim=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[[[2, 3, 4, 5, 6]]]]), 'Dimensions:', 4)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create x with 4 dimensions\n",
    "x = torch.tensor([[[[2,3,4,5,6]]]])\n",
    "x, 'Dimensions:', x.ndim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([2, 3, 4, 5, 6]), tensor([2, 3, 4, 5, 6]))"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Flatten x to a vector\n",
    "# Using two different methods\n",
    "# This is very helpful when moving from, for example a convolution network to a dense network\n",
    "#   19 - Beginning Deep Learning, - Convolution Networks - Tensorflow\n",
    "#   20 - Beginning Deep Learning, - Convolution Networks, PyTorch\n",
    "x.flatten(), torch.flatten(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[[[2, 3, 4, 5, 6]]]]), tensor([2, 3, 4, 5, 6]))"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Alternatively, we could have used torch.ravel() to get a 1D array\n",
    "x, torch.ravel(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2, 3, 4, 5, 6])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Maybe Just squeeze\n",
    "x.squeeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[3, 4, 5],\n",
       "         [6, 7, 8]]),\n",
       " torch.Size([2, 3]))"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a 2*3 tensor\n",
    "x = torch.tensor([[3,4,5], [6,7,8]])\n",
    "x, x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[3, 4],\n",
       "        [6, 7]])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Delete the last column from x\n",
    "# Basically keep column 0 and 1\n",
    "t = x\n",
    "t[..., 0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[3],\n",
       "        [6]])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Delete the last two column from x\n",
    "# Basically keep column 0\n",
    "# https://stackoverflow.com/questions/62372762/delete-an-element-from-torch-tensor\n",
    "t = x\n",
    "t[..., 0:1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[3, 4, 5]])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Delete the last row from x\n",
    "# Basically keep the first row. Index 0\n",
    "t = x\n",
    "t[0:1, ...]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[40, 10, 18],\n",
       "        [ 8,  5, 18]])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# One more before moving on, broadcasting\n",
    "# Taking a 2x3 tensor and multiple by a 1*3 vector\n",
    "# https://numpy.org/doc/stable/user/basics.broadcasting.html\n",
    "torch.multiply(torch.tensor([[10,2,3], [2,1,3]]), torch.tensor([4,5,6]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[40, 10, 18],\n",
       "        [ 8,  5, 18]])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The above is the same as multiplying \n",
    "# [[10,2,3],   * [[4,5,6]\n",
    "# [2,1,3]]        [4,5,6]]\n",
    "torch.multiply(torch.tensor([[10,2,3], [2,1,3]]), torch.tensor([[4,5,6], [4,5,6]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Maybe we want to save our Torch tensor\n",
    "torch.save(obj=x, f='x-tensor.pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x-tensor.pt\n"
     ]
    }
   ],
   "source": [
    "# Verify the file has been saved\n",
    "!dir /b x-tensor.pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[3, 4, 5],\n",
       "        [6, 7, 8]])"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load the saved tensor\n",
    "torch.load(f='x-tensor.pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Thats' it"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional References and good reads/videos <br>\n",
    "https://pytorch.org/docs/stable/index.html <br>\n",
    "https://pytorch.org/docs/stable/tensors.html <br>\n",
    "https://medium.com/@schartz/the-shape-of-tensor-bab75001d7bc <br>\n",
    "https://pytorch.org/docs/stable/generated/torch.cat.html <br>\n",
    "https://pytorch.org/docs/stable/generated/torch.eye.html <br>\n",
    "https://pytorch.org/docs/stable/generated/torch.randint.html <br>\n",
    "https://www.tensorflow.org/guide/tensor <br>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "main-env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "dcb100998b7ef434a33dbfac35f576e40d01f04c34d9ff6f6aad819b8a88c169"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}