summation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "summation.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lAu96tIy_MxN",
        "colab_type": "text"
      },
      "source": [
        "# Sequence to sequence model for mathematical sum using keras LSTM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "te_7xtws_Phe",
        "colab_type": "text"
      },
      "source": [
        "* Sequence to Sequence : Input is in form of sequence and output is also in the form of sequence\n",
        "\n",
        "## There are different kinds of seq2seq models. Some of them are:\n",
        "* one-to-one: The network produces one output for each input time step\n",
        "* one-to-many: The network produces multiple outputs for a single input. eg: image captioning\n",
        "* many-to-one: The network produces one output after taking inputs at multiple time steps. eg: forecasting\n",
        "* many-to-many: The network will provide multiple outputs for multiple input time steps. eg: translation\n",
        "\n",
        "#### We are going to use many to many model. \n",
        "\n",
        "The many to many sequence model uses something called encoder-decoder model.\n",
        "\n",
        "![alt text](https://media.geeksforgeeks.org/wp-content/uploads/seq2seq.png)\n",
        "###### image source: https://media.geeksforgeeks.org/wp-content/uploads/seq2seq.png \n",
        "\n",
        "* In this simple model,it can be simplified that encodes encodes the input and store the information in a thought vector and then decoder decodes the sequence data to provide the output.\n",
        "\n",
        "* In our model, the integers will be converted to strings and then added using the neural networks.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wuInhY40C4hC",
        "colab_type": "text"
      },
      "source": [
        "#Program Flow\n",
        "\n",
        "Generate input data to our model: \n",
        "* X: [[3, 10], [2, 5], [2, 8]]   # pair of 2 numbers which are less than 10\n",
        "* Y: [13, 7, 10]                 # sum of each pair of x \n",
        "\n",
        "converting them to string: \n",
        "* X: ['  3+10', '   2+5', '   2+8']\n",
        "* Y: ['13', ' 7', '10']\n",
        "\n",
        "Integer encoding them as machines cant understand human language:\n",
        "* X: [[11, 11, 3, 10, 1, 0], [11, 11, 11, 2, 10, 5], [11, 11, 11, 2, 10, 8]]\n",
        "* Y: [[1, 3], [11, 7], [1, 0]]\n",
        "\n",
        "One-hot encoding them or binary encoding them\n",
        "* X: shape = 6 X 12\n",
        "*[[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], \n",
        "* [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], \n",
        "* [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], \n",
        "* [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0], \n",
        "* [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \n",
        "* [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]\n",
        "\n",
        "* Y: shaep = 2 X 12\n",
        "* [[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \n",
        "* -[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0]] \n",
        "\n",
        "Creating a seq2seq model using keras lstm layers and fitting model on X and Y\n",
        "\n",
        "Finally, predicting the values or summation\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XU-N2_-146Et",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Importing essential libraries\n",
        "\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "from math import ceil, log10\n",
        "from random import randint,seed"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WWAhASNU5DUp",
        "colab_type": "code",
        "outputId": "56f2ba37-17b1-4704-dd6d-877cbb410e24",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "#function to generate random pair of numbers and their sum\n",
        "\n",
        "def random_number_generator(n_examples, n_numbers, largest):\n",
        "  x,y = list(), list()\n",
        "  for i in range(n_examples):\n",
        "    in_pattern = [randint(1,largest) for _ in range(n_numbers)]\n",
        "    out_pattern = sum(in_pattern)\n",
        "    x.append(in_pattern)\n",
        "    y.append(out_pattern)\n",
        "\n",
        "  return x, y\n",
        "\n",
        "seed(1)\n",
        "n_examples = 1000\n",
        "n_numbers = 2\n",
        "largest = 10\n",
        "\n",
        "x,y = random_number_generator(n_examples, n_numbers, largest)\n",
        "print(x[:3])\n",
        "print(y[:3])"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[3, 10], [2, 5], [2, 8]]\n",
            "[13, 7, 10]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Oe5E9Gg46TH2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Function to convert numbers to string\n",
        "\n",
        "def string_convert(x_list, y_list, n_numbers, largest): \n",
        "  x_string = list()\n",
        "  y_string = list()\n",
        "\n",
        "  max_input_length = n_numbers * ceil(log10((largest+1)) + n_numbers - 1)   # using ceil and log10 funciton from math library to get the length of sequence\n",
        "  \n",
        "  for word in x_list:\n",
        "    x_str = '+'.join([str(i) for i in word])\n",
        "    x_strr = ''.join([' ' for _ in range(max_input_length - len(x_str))]) + x_str\n",
        "    x_string.append(x_strr)\n",
        "  \n",
        "  max_output_length = ceil(log10(n_numbers * (largest + 1)))\n",
        "  \n",
        "  for word in y_list:\n",
        "    y_str = str(word) \n",
        "    y_strr = ''.join([' ' for _ in range(max_output_length - len(y_str))]) + y_str\n",
        "    y_string.append(y_strr)\n",
        "\n",
        "  return x_string, y_string"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nGFsc4RP98cr",
        "colab_type": "code",
        "outputId": "390fe87d-40c1-49c7-9735-78ea95cb77b2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "x,y = string_convert(x, y, n_numbers, largest)\n",
        "print(x[:3])\n",
        "print(y[:3])"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['  3+10', '   2+5', '   2+8']\n",
            "['13', ' 7', '10']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DTFGR_xhl-vc",
        "colab_type": "code",
        "outputId": "4ff730b8-7477-475f-a559-6547e467a395",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# Integer encoding\n",
        "\n",
        "# The number in the generated data is not greater than 10 that means the vocabulary will look like alphabet below:\n",
        "alphabet = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '+', ' ']\n",
        "\n",
        "def integer_encoding(x,y,alphabet):\n",
        "  char_to_int = dict((c,i) for i,c in enumerate(alphabet))     # dictionary of word and its index in alphabet. eg: {'0':0, '1':1, ...}\n",
        "                                                               # which means the index of string '0' is 0\n",
        "  x_encode = list()                                            # this is used to get the index of character for integer encoding  \n",
        "  for number in x:\n",
        "    integer = [char_to_int[word] for word in number]\n",
        "    x_encode.append(integer)\n",
        "\n",
        "  y_encode = list()\n",
        "  for number in y:\n",
        "    integer = [char_to_int[word] for word in number]\n",
        "    y_encode.append(integer)\n",
        "\n",
        "  return x_encode,y_encode\n",
        "\n",
        "x_enc,y_enc = integer_encoding(x,y, alphabet)\n",
        "print(x_enc[:3])\n",
        "print(y_enc[:3])"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[11, 11, 3, 10, 1, 0], [11, 11, 11, 2, 10, 5], [11, 11, 11, 2, 10, 8]]\n",
            "[[1, 3], [11, 7], [1, 0]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nsmOwoo4ka6O",
        "colab_type": "code",
        "outputId": "b5183436-73f3-4312-dc93-f6964f5db4d9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "#binary encoding or one_hot_encoding\n",
        "\n",
        "def one_hot_encoding(x,y,vocab_size):\n",
        "  x_enc = list()\n",
        "  for seq in x:\n",
        "    pattern = list()\n",
        "    for index in seq:\n",
        "      vector = [0 for _ in range(vocab_size)]\n",
        "      vector[index] = 1                         # iterating through each character in sequence and binary encoding it\n",
        "      pattern.append(vector)\n",
        "    x_enc.append(pattern)\n",
        "\n",
        "  y_enc = list()\n",
        "  for seq in y:\n",
        "    pattern = list()\n",
        "    for index in seq:\n",
        "      vector = [0 for _ in range(vocab_size)]\n",
        "      vector[index] = 1\n",
        "      pattern.append(vector)\n",
        "    y_enc.append(pattern)\n",
        "\n",
        "  return x_enc, y_enc\n",
        "\n",
        "x_onehot, y_onehot = one_hot_encoding(x_enc,y_enc, len(alphabet))\n",
        "print(x_onehot[:3])\n",
        "print(y_onehot[:3])"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0], [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0], [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0]], [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0], [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0]]]\n",
            "[[[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0]], [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0]], [[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RvUaCIUpwReQ",
        "colab_type": "code",
        "outputId": "1dc0e869-fdb6-463b-e84b-20d40384b336",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "# Converting the list of one hot encoded data to array\n",
        "\n",
        "x_onehot, y_onehot = np.array(x_onehot), np.array(y_onehot)\n",
        "print(x_onehot.shape)\n",
        "print(y_onehot.shape)\n",
        "\n",
        "# \n",
        "t = x_onehot.shape[1]                     # t = length of input sequence\n",
        "o = y_onehot.shape[1]                     # o = length of output sequence\n",
        "s = len(alphabet)                         # s = vocabulary size\n",
        "print('length of input sequence: ', t)\n",
        "print('length of output sequence: ', o)\n",
        "print('Vocab size :', s)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(1000, 6, 12)\n",
            "(1000, 2, 12)\n",
            "length of input sequence:  6\n",
            "length of output sequence:  2\n",
            "Vocab size : 12\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8ATaWnq8m7HN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Putting all the above functions in one funciton to generate random set of data at any point of time\n",
        "\n",
        "def generate_data(n_samples, n_numbers, largest, alphabet):\n",
        "  #generating number pair\n",
        "  x, y = random_number_generator(n_examples, n_numbers, largest)\n",
        "  #converting to string\n",
        "  x,y = string_convert(x, y, n_numbers, largest)\n",
        "  #integer encoding\n",
        "  x_enc,y_enc = integer_encoding(x,y, alphabet)\n",
        "  #onehot encoding\n",
        "  x_onehot, y_onehot = one_hot_encoding(x_enc,y_enc, len(alphabet))\n",
        "  #converting to array\n",
        "  x_onehot, y_onehot = np.array(x_onehot), np.array(y_onehot)\n",
        "\n",
        "  return x_onehot, y_onehot\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XYnIKCI-nqqw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# inversion of integer encoding\n",
        "\n",
        "# This is reverse of integer encoding. Here, we will use the index of the list to take out its word at that given index \n",
        "# eg: {1:'1', ... 3:'3'}, if we have 3 integer index, we can convert it into string value of 3\n",
        "\n",
        "def invert(seq, alphabet):\n",
        "  int_to_char = dict((i,c) for i,c in enumerate(alphabet))\n",
        "  strings = list()\n",
        "  for pattern in seq:\n",
        "    string = int_to_char[np.argmax(pattern)]                # using argmax to avoid getting index in float\n",
        "    strings.append(string)\n",
        "  return ''.join(strings)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tl8akrDKrMqM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# building model\n",
        "\n",
        "from tensorflow.keras.layers import Input, LSTM, RepeatVector, TimeDistributed, Dense\n",
        "from tensorflow.keras.models import Model\n",
        "\n",
        "i = Input(shape=(t,s))                                  # t = length of input sequence,  s = vocabulary size\n",
        "x = LSTM(100)(i)                                        # Encoder layer lstm\n",
        "x = RepeatVector(2)(x)                                  # The output of encoder lstm layer is 1D whereas the decoder lstm layer accepts 3D input\n",
        "                                                        # -so repeat vector repeats the input layer to match with the decoder lstm layer input\n",
        "x = LSTM(50, return_sequences=True)(x)                  # Decoder layer lstm\n",
        "x = TimeDistributed(Dense(s, activation='softmax'))(x)  # Dense layer can give only one output and in our example we need 2 output vectors(sum of 20+20 = 40)\n",
        "                                                        # - so we use the same Dense layer twice using TimeDistributed layer\n",
        "\n",
        "model = Model(i,x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WGaz1SCwtfw9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qKSeC5NhuNWY",
        "colab_type": "code",
        "outputId": "9fb18f52-1182-4a14-d115-4260aa9cd96e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        }
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         [(None, 6, 12)]           0         \n",
            "_________________________________________________________________\n",
            "lstm (LSTM)                  (None, 100)               45200     \n",
            "_________________________________________________________________\n",
            "repeat_vector (RepeatVector) (None, 2, 100)            0         \n",
            "_________________________________________________________________\n",
            "lstm_1 (LSTM)                (None, 2, 50)             30200     \n",
            "_________________________________________________________________\n",
            "time_distributed (TimeDistri (None, 2, 12)             612       \n",
            "=================================================================\n",
            "Total params: 76,012\n",
            "Trainable params: 76,012\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7k4PfEYpuW2r",
        "colab_type": "code",
        "outputId": "67a125f4-d969-4621-9f7e-61a01b8889be",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "r = model.fit(x_onehot, y_onehot, epochs=100, validation_split=0.2)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/100\n",
            "25/25 [==============================] - 1s 29ms/step - loss: 2.3901 - accuracy: 0.3675 - val_loss: 2.2519 - val_accuracy: 0.3725\n",
            "Epoch 2/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 2.0434 - accuracy: 0.3756 - val_loss: 1.7549 - val_accuracy: 0.3725\n",
            "Epoch 3/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 1.6076 - accuracy: 0.3787 - val_loss: 1.5895 - val_accuracy: 0.3675\n",
            "Epoch 4/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 1.5493 - accuracy: 0.3731 - val_loss: 1.5549 - val_accuracy: 0.3675\n",
            "Epoch 5/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 1.5217 - accuracy: 0.3819 - val_loss: 1.5483 - val_accuracy: 0.3700\n",
            "Epoch 6/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 1.5154 - accuracy: 0.3862 - val_loss: 1.5390 - val_accuracy: 0.3700\n",
            "Epoch 7/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 1.5046 - accuracy: 0.3856 - val_loss: 1.5208 - val_accuracy: 0.3700\n",
            "Epoch 8/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 1.4796 - accuracy: 0.3800 - val_loss: 1.4707 - val_accuracy: 0.3900\n",
            "Epoch 9/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 1.4222 - accuracy: 0.4606 - val_loss: 1.4287 - val_accuracy: 0.4725\n",
            "Epoch 10/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 1.3857 - accuracy: 0.4712 - val_loss: 1.3939 - val_accuracy: 0.4625\n",
            "Epoch 11/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 1.3339 - accuracy: 0.4931 - val_loss: 1.3452 - val_accuracy: 0.4625\n",
            "Epoch 12/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 1.2914 - accuracy: 0.5281 - val_loss: 1.3212 - val_accuracy: 0.5075\n",
            "Epoch 13/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 1.2494 - accuracy: 0.5275 - val_loss: 1.2639 - val_accuracy: 0.5200\n",
            "Epoch 14/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 1.2257 - accuracy: 0.5500 - val_loss: 1.2249 - val_accuracy: 0.5175\n",
            "Epoch 15/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 1.1920 - accuracy: 0.5587 - val_loss: 1.1986 - val_accuracy: 0.5525\n",
            "Epoch 16/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 1.1373 - accuracy: 0.6094 - val_loss: 1.1496 - val_accuracy: 0.6000\n",
            "Epoch 17/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 1.0924 - accuracy: 0.6225 - val_loss: 1.0907 - val_accuracy: 0.6100\n",
            "Epoch 18/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 1.0401 - accuracy: 0.6756 - val_loss: 1.0648 - val_accuracy: 0.6250\n",
            "Epoch 19/100\n",
            "25/25 [==============================] - 0s 10ms/step - loss: 1.0077 - accuracy: 0.6769 - val_loss: 0.9997 - val_accuracy: 0.6675\n",
            "Epoch 20/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.9508 - accuracy: 0.7144 - val_loss: 0.9593 - val_accuracy: 0.6675\n",
            "Epoch 21/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.8994 - accuracy: 0.7487 - val_loss: 0.9019 - val_accuracy: 0.7325\n",
            "Epoch 22/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.8452 - accuracy: 0.7606 - val_loss: 0.8366 - val_accuracy: 0.7625\n",
            "Epoch 23/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.7999 - accuracy: 0.8000 - val_loss: 0.8028 - val_accuracy: 0.7475\n",
            "Epoch 24/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.7668 - accuracy: 0.7825 - val_loss: 0.7403 - val_accuracy: 0.8225\n",
            "Epoch 25/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.7268 - accuracy: 0.7769 - val_loss: 0.6947 - val_accuracy: 0.8450\n",
            "Epoch 26/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.6605 - accuracy: 0.8675 - val_loss: 0.6498 - val_accuracy: 0.9000\n",
            "Epoch 27/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.6167 - accuracy: 0.8875 - val_loss: 0.6069 - val_accuracy: 0.8600\n",
            "Epoch 28/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.5718 - accuracy: 0.8988 - val_loss: 0.5545 - val_accuracy: 0.9250\n",
            "Epoch 29/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.5367 - accuracy: 0.9194 - val_loss: 0.5205 - val_accuracy: 0.9250\n",
            "Epoch 30/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.5028 - accuracy: 0.9212 - val_loss: 0.5023 - val_accuracy: 0.9250\n",
            "Epoch 31/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.4760 - accuracy: 0.9294 - val_loss: 0.4608 - val_accuracy: 0.9200\n",
            "Epoch 32/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.4417 - accuracy: 0.9337 - val_loss: 0.4252 - val_accuracy: 0.9575\n",
            "Epoch 33/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.4239 - accuracy: 0.9331 - val_loss: 0.4038 - val_accuracy: 0.9500\n",
            "Epoch 34/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.3901 - accuracy: 0.9469 - val_loss: 0.3796 - val_accuracy: 0.9675\n",
            "Epoch 35/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.3728 - accuracy: 0.9488 - val_loss: 0.3667 - val_accuracy: 0.9725\n",
            "Epoch 36/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.3536 - accuracy: 0.9613 - val_loss: 0.3336 - val_accuracy: 0.9725\n",
            "Epoch 37/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.3248 - accuracy: 0.9656 - val_loss: 0.3188 - val_accuracy: 0.9625\n",
            "Epoch 38/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.3086 - accuracy: 0.9619 - val_loss: 0.3044 - val_accuracy: 0.9775\n",
            "Epoch 39/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.2885 - accuracy: 0.9775 - val_loss: 0.2811 - val_accuracy: 0.9700\n",
            "Epoch 40/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.2750 - accuracy: 0.9756 - val_loss: 0.2649 - val_accuracy: 0.9800\n",
            "Epoch 41/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.2626 - accuracy: 0.9756 - val_loss: 0.2651 - val_accuracy: 0.9725\n",
            "Epoch 42/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.2465 - accuracy: 0.9875 - val_loss: 0.2340 - val_accuracy: 0.9875\n",
            "Epoch 43/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.2306 - accuracy: 0.9887 - val_loss: 0.2237 - val_accuracy: 0.9850\n",
            "Epoch 44/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.2178 - accuracy: 0.9962 - val_loss: 0.2187 - val_accuracy: 0.9900\n",
            "Epoch 45/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.2058 - accuracy: 0.9881 - val_loss: 0.1975 - val_accuracy: 0.9850\n",
            "Epoch 46/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.1948 - accuracy: 0.9925 - val_loss: 0.1909 - val_accuracy: 0.9900\n",
            "Epoch 47/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.1887 - accuracy: 0.9925 - val_loss: 0.1837 - val_accuracy: 0.9850\n",
            "Epoch 48/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.1792 - accuracy: 0.9931 - val_loss: 0.1913 - val_accuracy: 0.9950\n",
            "Epoch 49/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.1693 - accuracy: 0.9962 - val_loss: 0.1614 - val_accuracy: 0.9950\n",
            "Epoch 50/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.1567 - accuracy: 0.9900 - val_loss: 0.1536 - val_accuracy: 0.9850\n",
            "Epoch 51/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.1500 - accuracy: 0.9925 - val_loss: 0.1498 - val_accuracy: 0.9950\n",
            "Epoch 52/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.1474 - accuracy: 0.9912 - val_loss: 0.1407 - val_accuracy: 0.9900\n",
            "Epoch 53/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.1357 - accuracy: 0.9962 - val_loss: 0.1339 - val_accuracy: 0.9950\n",
            "Epoch 54/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.1309 - accuracy: 0.9937 - val_loss: 0.1273 - val_accuracy: 0.9950\n",
            "Epoch 55/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.1242 - accuracy: 0.9894 - val_loss: 0.1207 - val_accuracy: 0.9950\n",
            "Epoch 56/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.1172 - accuracy: 0.9931 - val_loss: 0.1177 - val_accuracy: 0.9950\n",
            "Epoch 57/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.1125 - accuracy: 0.9950 - val_loss: 0.1117 - val_accuracy: 0.9950\n",
            "Epoch 58/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.1068 - accuracy: 0.9925 - val_loss: 0.1050 - val_accuracy: 0.9950\n",
            "Epoch 59/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.1026 - accuracy: 0.9962 - val_loss: 0.1011 - val_accuracy: 0.9950\n",
            "Epoch 60/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0977 - accuracy: 0.9956 - val_loss: 0.0984 - val_accuracy: 0.9950\n",
            "Epoch 61/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0964 - accuracy: 0.9956 - val_loss: 0.0925 - val_accuracy: 0.9950\n",
            "Epoch 62/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0899 - accuracy: 0.9962 - val_loss: 0.0883 - val_accuracy: 0.9950\n",
            "Epoch 63/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0857 - accuracy: 0.9962 - val_loss: 0.0870 - val_accuracy: 0.9950\n",
            "Epoch 64/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0826 - accuracy: 0.9962 - val_loss: 0.0820 - val_accuracy: 0.9950\n",
            "Epoch 65/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0783 - accuracy: 0.9962 - val_loss: 0.0785 - val_accuracy: 0.9950\n",
            "Epoch 66/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0754 - accuracy: 0.9962 - val_loss: 0.0757 - val_accuracy: 0.9950\n",
            "Epoch 67/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0728 - accuracy: 0.9962 - val_loss: 0.0714 - val_accuracy: 0.9950\n",
            "Epoch 68/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0702 - accuracy: 0.9962 - val_loss: 0.0693 - val_accuracy: 0.9950\n",
            "Epoch 69/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0674 - accuracy: 0.9962 - val_loss: 0.0711 - val_accuracy: 0.9950\n",
            "Epoch 70/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0659 - accuracy: 0.9962 - val_loss: 0.0680 - val_accuracy: 0.9950\n",
            "Epoch 71/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0626 - accuracy: 0.9962 - val_loss: 0.0625 - val_accuracy: 0.9950\n",
            "Epoch 72/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0599 - accuracy: 0.9962 - val_loss: 0.0605 - val_accuracy: 0.9950\n",
            "Epoch 73/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0577 - accuracy: 0.9962 - val_loss: 0.0579 - val_accuracy: 0.9950\n",
            "Epoch 74/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0557 - accuracy: 0.9962 - val_loss: 0.0563 - val_accuracy: 0.9950\n",
            "Epoch 75/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0536 - accuracy: 0.9962 - val_loss: 0.0539 - val_accuracy: 0.9950\n",
            "Epoch 76/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0519 - accuracy: 0.9962 - val_loss: 0.0521 - val_accuracy: 0.9950\n",
            "Epoch 77/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0500 - accuracy: 0.9962 - val_loss: 0.0507 - val_accuracy: 0.9950\n",
            "Epoch 78/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0487 - accuracy: 0.9962 - val_loss: 0.0480 - val_accuracy: 0.9950\n",
            "Epoch 79/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0471 - accuracy: 0.9962 - val_loss: 0.0465 - val_accuracy: 0.9950\n",
            "Epoch 80/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0455 - accuracy: 0.9962 - val_loss: 0.0461 - val_accuracy: 0.9950\n",
            "Epoch 81/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0441 - accuracy: 0.9981 - val_loss: 0.0434 - val_accuracy: 0.9950\n",
            "Epoch 82/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0425 - accuracy: 0.9962 - val_loss: 0.0429 - val_accuracy: 0.9950\n",
            "Epoch 83/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0412 - accuracy: 0.9987 - val_loss: 0.0408 - val_accuracy: 0.9950\n",
            "Epoch 84/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0401 - accuracy: 0.9962 - val_loss: 0.0400 - val_accuracy: 1.0000\n",
            "Epoch 85/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0387 - accuracy: 0.9975 - val_loss: 0.0385 - val_accuracy: 1.0000\n",
            "Epoch 86/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0373 - accuracy: 0.9981 - val_loss: 0.0373 - val_accuracy: 0.9950\n",
            "Epoch 87/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0364 - accuracy: 0.9962 - val_loss: 0.0368 - val_accuracy: 1.0000\n",
            "Epoch 88/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0353 - accuracy: 1.0000 - val_loss: 0.0350 - val_accuracy: 1.0000\n",
            "Epoch 89/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0342 - accuracy: 1.0000 - val_loss: 0.0341 - val_accuracy: 1.0000\n",
            "Epoch 90/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0333 - accuracy: 1.0000 - val_loss: 0.0332 - val_accuracy: 1.0000\n",
            "Epoch 91/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0321 - accuracy: 0.9994 - val_loss: 0.0321 - val_accuracy: 1.0000\n",
            "Epoch 92/100\n",
            "25/25 [==============================] - 0s 7ms/step - loss: 0.0311 - accuracy: 1.0000 - val_loss: 0.0314 - val_accuracy: 1.0000\n",
            "Epoch 93/100\n",
            "25/25 [==============================] - 0s 7ms/step - loss: 0.0305 - accuracy: 1.0000 - val_loss: 0.0304 - val_accuracy: 1.0000\n",
            "Epoch 94/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0294 - accuracy: 1.0000 - val_loss: 0.0290 - val_accuracy: 1.0000\n",
            "Epoch 95/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0284 - accuracy: 1.0000 - val_loss: 0.0282 - val_accuracy: 1.0000\n",
            "Epoch 96/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0276 - accuracy: 1.0000 - val_loss: 0.0280 - val_accuracy: 1.0000\n",
            "Epoch 97/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0269 - accuracy: 1.0000 - val_loss: 0.0263 - val_accuracy: 1.0000\n",
            "Epoch 98/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0258 - accuracy: 1.0000 - val_loss: 0.0260 - val_accuracy: 1.0000\n",
            "Epoch 99/100\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.0250 - accuracy: 1.0000 - val_loss: 0.0246 - val_accuracy: 1.0000\n",
            "Epoch 100/100\n",
            "25/25 [==============================] - 0s 5ms/step - loss: 0.0243 - accuracy: 1.0000 - val_loss: 0.0244 - val_accuracy: 1.0000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eMgte_30xk9y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Generating random data to test our model\n",
        "x,y = generate_data(n_examples, n_numbers, largest, alphabet)  \n",
        "result = model.predict(x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G3FGniuvz3fl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Testing our model\n",
        "expected = [invert(x, alphabet) for x in y]\n",
        "predicted = [invert(x,alphabet) for x in result]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZuyMnHDE0Hai",
        "colab_type": "code",
        "outputId": "58dca43d-199a-4915-bf74-8c0563681f89",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        }
      },
      "source": [
        "# Getting first 20 numbers and sum prediction\n",
        "for i in range(20):\n",
        "\tprint('Expected=%s, Predicted=%s' % (expected[i], predicted[i]))"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Expected= 6, Predicted= 6\n",
            "Expected=10, Predicted=10\n",
            "Expected=12, Predicted=12\n",
            "Expected= 7, Predicted= 7\n",
            "Expected=11, Predicted=11\n",
            "Expected=11, Predicted=11\n",
            "Expected= 6, Predicted= 6\n",
            "Expected= 8, Predicted= 8\n",
            "Expected= 5, Predicted= 5\n",
            "Expected=12, Predicted=12\n",
            "Expected=16, Predicted=16\n",
            "Expected= 5, Predicted= 5\n",
            "Expected=12, Predicted=12\n",
            "Expected=16, Predicted=16\n",
            "Expected=10, Predicted=10\n",
            "Expected=14, Predicted=14\n",
            "Expected=17, Predicted=17\n",
            "Expected= 6, Predicted= 6\n",
            "Expected=11, Predicted=11\n",
            "Expected=11, Predicted=11\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tv9xs3yW0laX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# saving the model\n",
        "model.save('math_sum.h5') # creates HDF5 file of the model"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}