Chapter05/5.12 BiDirectional RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BiDirectional RNN\n",
    "\n",
    "In Bidirectional RNN, we have two different layers of hidden units. Both of these layers\n",
    "connect from the input to the output layer. In one layer the hidden states are shared from\n",
    "left to right and in another layer, it is shared from right to left.\n",
    "\n",
    "\n",
    "But what does this mean? To put it simply, one layer moves forward through time from the\n",
    "start of the sequence while another layer moves backward through time from the end of the\n",
    "sequence.\n",
    "\n",
    "\n",
    "As shown in the following figure, we have two hidden layers: forward hidden layer and\n",
    "backward layer.\n",
    "\n",
    "* In forward hidden layer, hidden state values are shared from past time steps i.e\n",
    "$h_o$ is shared to $h_1$, $h_1$ is shared $h_2$ to and so on.\n",
    "In backward hidden layer, hidden start values are shared from the future time\n",
    "steps i.e $z_3$ to $z_2$, $z_2$ to $z_1$ and so on. \n",
    "\n",
    "\n",
    "![image to be added](images/10.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the use of Bidirectional RNN? In certain cases, reading the input sequence from\n",
    "both sides is very useful. So a Bidirectional RNN consists of two RNN, one reading the\n",
    "sentence forward and the other reading the sentence backward.\n",
    "\n",
    "For instance, consider the following sentence:  Archie lived for 13 years in _____. So he is good at speaking Chinese\n",
    "\n",
    "\n",
    "If we use an RNN to predict the blank in the above sentence it would be ambiguous. As we\n",
    "know that RNN can make predictions based on only the set of words it has seen so far. In\n",
    "the above sentence, to predict the blank, RNN has seen only the words, Archie, lived, for, 13,\n",
    "years, in. But these words do not provide much context and does not give any clarity to\n",
    "predict the correct word. It just says Archie lived for 13 years in. With this information alone\n",
    "we cannot predict the next word correctly.\n",
    "\n",
    "But if we read the words following the blank as well i.e So, he, is, good, at, speaking, Chinese.\n",
    "Then we can say that Archie lived for 13 years in China since it is given that he is good at\n",
    "speaking Chinese. So if we use directional RNN to predict the blank, it will predict\n",
    "correctly, since it reads the sentence in both forward and backward directions before making predictions\n",
    "Bidirectional RNN has been used in various applications such as POS tagging where it is\n",
    "vital to know the word before and after the target word, language translation, predicting\n",
    "protein structure, dependency parsing, and so on. However, bidirectional RNN is\n",
    "not suitable for online settings where we don't know the future. \n",
    "\n",
    "The forward propagation steps in bidirectional RNN is given as:\n",
    "\n",
    "Forward hidden layer:\n",
    "$$h_{t}=\\sigma\\left(U_{h} x_{t}+W_{h} h_{t-1}\\right)$$\n",
    "\n",
    "Backward hidden layer:\n",
    "$$z_{t}=\\sigma\\left(U_{z} x_{t}+W_{z} z_{t+1}\\right)$$ \n",
    "\n",
    "Output:\n",
    "$$\\hat{y}_{t}=\\operatorname{softmax}\\left(V_{h} h_{t}+V_{z} z_{t}\\right)$$\n",
    "\n",
    "Implementing Bidirectional RNN is simple with TensorFlow.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Thus we learned how works, in the next section, we will see how to implement Bidirectional RNN is TensorFlow.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}