C1- Linear Algebra for Machine Learning and Data Science/Week3/C1_W3_Lab_2_matrix_multiplication.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "48446b1e",
   "metadata": {},
   "source": [
    "# Matrix Multiplication"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b6934ea",
   "metadata": {},
   "source": [
    "In this lab you will use `NumPy` functions to perform matrix multiplication and see how it can be used in the Machine Learning applications. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4439a0e9",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "\n",
    "- [ 1 - Definition of Matrix Multiplication](#1)\n",
    "- [ 2 - Matrix Multiplication using Python](#2)\n",
    "- [ 3 - Matrix Convention and Broadcasting](#3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2743d75",
   "metadata": {},
   "source": [
    "## Packages\n",
    "\n",
    "Load the `NumPy` package to access its functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b463e5b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f9be5bb",
   "metadata": {},
   "source": [
    "<a name='1'></a>\n",
    "## 1 - Definition of Matrix Multiplication\n",
    "\n",
    "If $A$ is an $m \\times n$ matrix and $B$ is an $n \\times p$ matrix, the matrix product $C = AB$ (denoted without multiplication signs or dots) is defined to be the $m \\times p$ matrix such that \n",
    "$c_{ij}=a_{i1}b_{1j}+a_{i2}b_{2j}+\\ldots+a_{in}b_{nj}=\\sum_{k=1}^{n} a_{ik}b_{kj}, \\tag{4}$\n",
    "\n",
    "where $a_{ik}$ are the elements of matrix $A$, $b_{kj}$ are the elements of matrix $B$, and $i = 1, \\ldots, m$, $k=1, \\ldots, n$, $j = 1, \\ldots, p$. In other words, $c_{ij}$ is the dot product of the $i$-th row of $A$ and the $j$-th column of $B$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecd63af9",
   "metadata": {},
   "source": [
    "<a name='2'></a>\n",
    "## 2 - Matrix Multiplication using Python\n",
    "\n",
    "Like with the dot product, there are a few ways to perform matrix multiplication in Python. As discussed in the previous lab, the calculations are more efficient in the vectorized form. Let's discuss the most commonly used functions in the vectorized form. First, define two matrices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8b0f59f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrix A (3 by 3):\n",
      " [[4 9 9]\n",
      " [9 1 6]\n",
      " [9 2 3]]\n",
      "Matrix B (3 by 2):\n",
      " [[2 2]\n",
      " [5 7]\n",
      " [4 4]]\n"
     ]
    }
   ],
   "source": [
    "A = np.array([[4, 9, 9], [9, 1, 6], [9, 2, 3]])\n",
    "print(\"Matrix A (3 by 3):\\n\", A)\n",
    "\n",
    "B = np.array([[2, 2], [5, 7], [4, 4]])\n",
    "print(\"Matrix B (3 by 2):\\n\", B)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdf047c9",
   "metadata": {},
   "source": [
    "You can multiply matrices $A$ and $B$ using `NumPy` package function `np.matmul()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "43452598",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 89, 107],\n",
       "       [ 47,  49],\n",
       "       [ 40,  44]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.matmul(A, B)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7be5d42a",
   "metadata": {},
   "source": [
    "Which will output $3 \\times 2$ matrix as a `np.array`. Python operator `@` will also work here giving the same result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bb36ba42",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 89, 107],\n",
       "       [ 47,  49],\n",
       "       [ 40,  44]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A @ B"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0186638b",
   "metadata": {},
   "source": [
    "<a name='3'></a>\n",
    "## 3 - Matrix Convention and Broadcasting\n",
    "\n",
    "Mathematically, matrix multiplication is defined only if number of the columns of matrix $A$ is equal to the number of the rows of matrix $B$ (you can check again the definition in the secition [1](#1) and see that otherwise the dot products between rows and columns will not be defined). \n",
    "\n",
    "Thus, in the example above ([2](#2)), changing the order of matrices when performing the multiplication $BA$ will not work as the above rule does not hold anymore. You can check it by running the cells below - both of them will give errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3ecc05e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 3 is different from 2)\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    np.matmul(B, A)\n",
    "except ValueError as err:\n",
    "    print(err)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ea9c6d13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 3 is different from 2)\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    B @ A\n",
    "except ValueError as err:\n",
    "    print(err)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05d9a674",
   "metadata": {},
   "source": [
    "So when using matrix multiplication you will need to be very careful about the dimensions - the number of the columns in the first matrix should match the number of the rows in the second matrix. This is very important for your future understanding of Neural Networks and how they work. \n",
    "\n",
    "However, for multiplying of the vectors, `NumPy` has a shortcut. You can define two vectors $x$ and $y$ of the same size (which one can understand as two $3 \\times 1$ matrices). If you check the shape of the vector $x$, you can see that :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fab77ce6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of vector x: (3,)\n",
      "Number of dimensions of vector x: 1\n",
      "Shape of vector x, reshaped to a matrix: (3, 1)\n",
      "Number of dimensions of vector x, reshaped to a matrix: 2\n"
     ]
    }
   ],
   "source": [
    "x = np.array([1, -2, -5])\n",
    "y = np.array([4, 3, -1])\n",
    "\n",
    "print(\"Shape of vector x:\", x.shape)\n",
    "print(\"Number of dimensions of vector x:\", x.ndim)\n",
    "print(\"Shape of vector x, reshaped to a matrix:\", x.reshape((3, 1)).shape)\n",
    "print(\"Number of dimensions of vector x, reshaped to a matrix:\", x.reshape((3, 1)).ndim)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bd337df",
   "metadata": {},
   "source": [
    "Following the matrix convention, multiplication of matrices $3 \\times 1$ and $3 \\times 1$ is not defined. For matrix multiplication you would expect an error in the following cell, but let's check the output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f655677c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.matmul(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fc01d74",
   "metadata": {},
   "source": [
    "You can see that there is no error and that the result is actually a dot product $x \\cdot y\\,$! So, vector $x$ was automatically transposed into the vector $1 \\times 3$ and matrix multiplication $x^Ty$ was calculated. While this is very convenient, you need to keep in mind such functionality in Python and pay attention to not use it in a wrong way. The following cell will return an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d92006f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 3 is different from 1)\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    np.matmul(x.reshape((3, 1)), y.reshape((3, 1)))\n",
    "except ValueError as err:\n",
    "    print(err)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ace12c7d",
   "metadata": {},
   "source": [
    "You might have a question in you mind: does `np.dot()` function also work for matrix multiplication? Let's try it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f296e528",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 89, 107],\n",
       "       [ 47,  49],\n",
       "       [ 40,  44]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(A, B)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dbbdc0f",
   "metadata": {},
   "source": [
    "Yes, it works! What actually happens is what is called **broadcasting** in Python: `NumPy` broadcasts this dot product operation to all rows and all columns, you get the resultant product matrix. Broadcasting also works in other cases, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "68ded501",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2,  7,  7],\n",
       "       [ 7, -1,  4],\n",
       "       [ 7,  0,  1]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A - 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eec1d0d2",
   "metadata": {},
   "source": [
    "Mathematically, subtraction of the $3 \\times 3$ matrix $A$ and a scalar is not defined, but Python broadcasts the scalar, creating a $3 \\times 3$ `np.array` and performing subtraction element by element. A practical example of matrix multiplication can be seen in a linear regression model. You will implement it in this week's assignment!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86605d6f",
   "metadata": {},
   "source": [
    "Congratulations on finishing this lab!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76db64ac",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}