Step-5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LRCN Approach implementation\n",
    "\n",
    "The LRCN Approach is implemented by combining Convolution and LSTM layers in a single model. The CNN model can be used to extracts the spatial features from the frames in the video, and for this purpose, a pre-trained model can be used, that can be fine-tuned for the problem. And the LSTM model can then use the features extracted by CNN, to predict the action being performed in the video. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5.1: Construct the Model\n",
    "\n",
    "To implement our LRCN architecture, we will use time-distributed **`Conv2D`** layers which will be followed by **`MaxPooling2D`** and **`Dropout`** layers. The feature extracted from the **`Conv2D`** layers will be then flattened using the  **`Flatten`** layer and will be fed to a **`LSTM`** layer. The **`Dense`** layer with softmax activation will then use the output from the **`LSTM`** layer to predict the action being performed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_LRCN_model():\n",
    "    '''\n",
    "    This function will construct the required LRCN model.\n",
    "    Returns:\n",
    "        model: It is the required constructed LRCN model.\n",
    "    '''\n",
    "\n",
    "    # We will use a Sequential model for model construction.\n",
    "    model = Sequential()\n",
    "    \n",
    "    # Define the Model Architecture.\n",
    "    ########################################################################################################################\n",
    "    \n",
    "    model.add(TimeDistributed(Conv2D(16, (3, 3), padding='same',activation = 'relu'),\n",
    "                              input_shape = (SEQUENCE_LENGTH, IMAGE_HEIGHT, IMAGE_WIDTH, 3)))\n",
    "    \n",
    "    model.add(TimeDistributed(MaxPooling2D((4, 4)))) \n",
    "    model.add(TimeDistributed(Dropout(0.25)))\n",
    "    \n",
    "    model.add(TimeDistributed(Conv2D(32, (3, 3), padding='same',activation = 'relu')))\n",
    "    model.add(TimeDistributed(MaxPooling2D((4, 4))))\n",
    "    model.add(TimeDistributed(Dropout(0.25)))\n",
    "    \n",
    "    model.add(TimeDistributed(Conv2D(64, (3, 3), padding='same',activation = 'relu')))\n",
    "    model.add(TimeDistributed(MaxPooling2D((2, 2))))\n",
    "    model.add(TimeDistributed(Dropout(0.25)))\n",
    "    \n",
    "    model.add(TimeDistributed(Conv2D(64, (3, 3), padding='same',activation = 'relu')))\n",
    "    model.add(TimeDistributed(MaxPooling2D((2, 2))))\n",
    "    #model.add(TimeDistributed(Dropout(0.25)))\n",
    "                                      \n",
    "    model.add(TimeDistributed(Flatten()))\n",
    "                                      \n",
    "    model.add(LSTM(32))\n",
    "                                      \n",
    "    model.add(Dense(len(CLASSES_LIST), activation = 'softmax'))\n",
    "\n",
    "    ########################################################################################################################\n",
    "\n",
    "    # Display the models summary.\n",
    "    model.summary()\n",
    "    \n",
    "    # Return the constructed LRCN model.\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the function **`create_LRCN_model()`** to construct the required `LRCN` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct the required LRCN model.\n",
    "LRCN_model = create_LRCN_model()\n",
    "\n",
    "# Display the success message.\n",
    "print(\"Model Created Successfully!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the model structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the structure of the contructed LRCN model.\n",
    "plot_model(LRCN_model, to_file = 'LRCN_model_structure_plot.png', show_shapes = True, show_layer_names = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2: Compile & Train the model\n",
    "\n",
    "After checking the structure, compile and start training the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an Instance of Early Stopping Callback.\n",
    "early_stopping_callback = EarlyStopping(monitor = 'val_loss', patience = 15, mode = 'min', restore_best_weights = True)\n",
    " \n",
    "# Compile the model and specify loss function, optimizer and metrics to the model.\n",
    "LRCN_model.compile(loss = 'categorical_crossentropy', optimizer = 'Adam', metrics = [\"accuracy\"])\n",
    "\n",
    "# Start training the model.\n",
    "LRCN_model_training_history = LRCN_model.fit(x = features_train, y = labels_train, epochs = 70, batch_size = 4 ,\n",
    "                                             shuffle = True, validation_split = 0.2, callbacks = [early_stopping_callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate the trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the trained model.\n",
    "model_evaluation_history = LRCN_model.evaluate(features_test, labels_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the Model\n",
    "\n",
    "After that, we will save the model for future uses using the same technique we had used for the previous model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the loss and accuracy from model_evaluation_history.\n",
    "model_evaluation_loss, model_evaluation_accuracy = model_evaluation_history\n",
    "\n",
    "# Define the string date format.\n",
    "# Get the current Date and Time in a DateTime Object.\n",
    "# Convert the DateTime object to string according to the style mentioned in date_time_format string.\n",
    "date_time_format = '%Y_%m_%d__%H_%M_%S'\n",
    "current_date_time_dt = dt.datetime.now()\n",
    "current_date_time_string = dt.datetime.strftime(current_date_time_dt, date_time_format)\n",
    "    \n",
    "# Define a useful name for our model to make it easy for us while navigating through multiple saved models.\n",
    "model_file_name = f'LRCN_model___Date_Time_{current_date_time_string}___Loss_{model_evaluation_loss}___Accuracy_{model_evaluation_accuracy}.h5'\n",
    "\n",
    "# Save the Model.\n",
    "LRCN_model.save(model_file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5.3: Plot Model’s Loss & Accuracy Curves\n",
    "\n",
    "Use the function **`plot_metric()`** we had created above to visualize the training and validation metrics of this model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize the training and validation loss metrices.\n",
    "plot_metric(LRCN_model_training_history, 'loss', 'val_loss', 'Total Loss vs Total Validation Loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize the training and validation accuracy metrices.\n",
    "plot_metric(LRCN_model_training_history, 'accuracy', 'val_accuracy', 'Total Accuracy vs Total Validation Accuracy')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.10 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.8.10"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}