8_1_get_feature_embeddings.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "8.1.get feature embeddings.ipynb",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nmningmei/Deep_learning_fMRI_EEG/blob/master/8_1_get_feature_embeddings.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_49pV__Aiyij",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from IPython.display import Image"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zyQXea1JbWUY",
        "colab_type": "text"
      },
      "source": [
        "# Part I: What is feature embdding?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fNhuPgLJbb5V",
        "colab_type": "text"
      },
      "source": [
        "1. [An embedding is a translation of a high-dimensional vector into a low-dimensional space. Ideally, an embedding captures some of the semantics of the input by placing semantically similar inputs close together in the embedding space.](https://cloud.google.com/solutions/machine-learning/overview-extracting-and-serving-feature-embeddings-for-machine-learning#what_is_an_embedding)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HCM05ty4kIt-",
        "colab_type": "text"
      },
      "source": [
        "## Ultimate goal of computer vision model embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "keqDFRuifnk2",
        "colab_type": "code",
        "outputId": "5798ca36-51b2-44bb-e18e-c4538e343d7b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 335
        }
      },
      "source": [
        "Image(url = 'https://raw.githubusercontent.com/mangye16/Unsupervised_Embedding_Learning/master/fig/Motivation.png')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://raw.githubusercontent.com/mangye16/Unsupervised_Embedding_Learning/master/fig/Motivation.png\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xMho5uWei3DC",
        "colab_type": "code",
        "outputId": "23bedccd-6819-4ca9-95ab-bd8d6cfa21ef",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 401
        }
      },
      "source": [
        "Image(url = 'https://raw.githubusercontent.com/mangye16/Unsupervised_Embedding_Learning/master/fig/Pipeline.png')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://raw.githubusercontent.com/mangye16/Unsupervised_Embedding_Learning/master/fig/Pipeline.png\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RZO6H8qbjguM",
        "colab_type": "text"
      },
      "source": [
        "[Ye et al., 2019, arXiv](https://arxiv.org/pdf/1904.03436.pdf)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pD5ZEOP2kjJ3",
        "colab_type": "code",
        "outputId": "0a2c128a-c2e6-4df8-fb3b-77df1253747c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 452
        }
      },
      "source": [
        "Image(url = 'https://raw.githubusercontent.com/nmningmei/BOLD5000_autoencoder/master/figures/autoencoder%20phase%204.jpg')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://raw.githubusercontent.com/nmningmei/BOLD5000_autoencoder/master/figures/autoencoder%20phase%204.jpg\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p7VkfMofkNu8",
        "colab_type": "text"
      },
      "source": [
        "## Ultimate goal of word embeddings: looks good, never works.\n",
        "\n",
        "These are the only examples that work."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0GKWNvPvjNWT",
        "colab_type": "code",
        "outputId": "5e60b5a4-a975-4a1d-df3d-eaef77466459",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 568
        }
      },
      "source": [
        "Image(url = 'https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2017/06/06062705/Word-Vectors.png')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2017/06/06062705/Word-Vectors.png\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xa5lMX5hkDgp",
        "colab_type": "text"
      },
      "source": [
        "[Hunter Heidenreich ](http://hunterheidenreich.com/blog/intro-to-word-embeddings/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F7fjG8O9kwrB",
        "colab_type": "text"
      },
      "source": [
        "# Let's start with the difficult one: train a word embedding model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_1dcKwqGj9Nh",
        "colab_type": "code",
        "outputId": "70f66500-5a61-4a05-bfc5-c1e93dea81b1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 653
        }
      },
      "source": [
        "Image(url = 'https://jaxenter.com/wp-content/uploads/2018/08/image-2-768x632.png')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://jaxenter.com/wp-content/uploads/2018/08/image-2-768x632.png\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vUTLtCepk8CN",
        "colab_type": "text"
      },
      "source": [
        "What is showed above is called \"skip-gram\" $\\rightarrow$ mapping 1 word to many words\n",
        "\n",
        "The other way around is called \"Continuous Bag of Words\" (CBOW)\n",
        "\n",
        "[source: Tommaso Teofili August 17, 2018](https://jaxenter.com/deep-learning-search-word2vec-147782.html)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NoKUlSk-msq3",
        "colab_type": "code",
        "outputId": "49f773fb-8217-49a6-b09c-bc4ce20c695a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 292
        }
      },
      "source": [
        "Image(url = 'https://miro.medium.com/max/778/0*Yl7I7bH52zk8m_8R.')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://miro.medium.com/max/778/0*Yl7I7bH52zk8m_8R.\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "89Rfm8pllAbQ",
        "colab_type": "code",
        "outputId": "92860fe6-b254-4e67-e635-8094ded80636",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 781
        }
      },
      "source": [
        "Image(url = 'https://miro.medium.com/max/778/0*CldH-gf1GuWjqNjt.')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://miro.medium.com/max/778/0*CldH-gf1GuWjqNjt.\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9s2eopGomfyG",
        "colab_type": "text"
      },
      "source": [
        "From [Introduction to Word Vectors](https://medium.com/@jayeshbahire/introduction-to-word-vectors-ea1d4e4b84bf)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P7EL2W7bnELZ",
        "colab_type": "text"
      },
      "source": [
        "[Classical Word2Vec - the 2013 paper](https://www.tensorflow.org/tutorials/representation/word2vec)\n",
        "\n",
        "pro:\n",
        "1. basic usage\n",
        "2. local co-occurence\n",
        "3. old\n",
        "4. **It is the reason people use 300-D as the representaion space**\n",
        "\n",
        "con:\n",
        "1. not handle rare words\n",
        "2. trained on wiki corpus"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tQoLgKLonp06",
        "colab_type": "text"
      },
      "source": [
        "[GloVe](https://nlp.stanford.edu/projects/glove/)\n",
        "\n",
        "pro:\n",
        "1. more globle co-oocurrence capturing\n",
        "2. handle rare words\n",
        "\n",
        "con:\n",
        "1. expensive to train\n",
        "2. limited languages\n",
        "3. trained on wiki corpus"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uK7wQTLJoV2Z",
        "colab_type": "text"
      },
      "source": [
        "[FastText](https://fasttext.cc/docs/en/pretrained-vectors.html), now supports 157 languages, and they are easy to download\n",
        "\n",
        "pro:\n",
        "1. easy to train\n",
        "2. more globle co-occurence\n",
        "3. language support\n",
        "\n",
        "con:\n",
        "1. not handle rare words\n",
        "2. trained on wiki"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EpNV0nj4o3nV",
        "colab_type": "text"
      },
      "source": [
        "## Example of how to use a pre-trained word embedding\n",
        "\n",
        "```\n",
        "# for example, load fast test model in memory\n",
        "# fasttext_link: http://dcc.uchile.cl/~jperez/word-embeddings/fasttext-sbwc.vec.gz\n",
        "\n",
        "import gensim # only available in Cajal02 here\n",
        "\n",
        "fasttest_model = gensim.models.keyedvectors.KeyedVectors.load_word2vec_format(fasttext_downloaded_file_name)\n",
        "for word in words:\n",
        "    word_vector_representation = fasttest_model.get_vector(word)\n",
        "\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VoiceoJ3pc2J",
        "colab_type": "text"
      },
      "source": [
        "# Move on to computer vision for images\n",
        "\n",
        "## Videos are just many frames of images, aren't they?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HxRITEAqncVE",
        "colab_type": "code",
        "outputId": "d6de71e0-5649-445b-f830-9d537b1f63e0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 627
        }
      },
      "source": [
        "Image(url = 'https://www.jneurosci.org/content/jneuro/35/27/10005/F1.large.jpg?width=800&height=600&carousel=1')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://www.jneurosci.org/content/jneuro/35/27/10005/F1.large.jpg?width=800&height=600&carousel=1\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YsgnsRCequ5O",
        "colab_type": "text"
      },
      "source": [
        "[Güçlü and Gerven, 2015](https://www.jneurosci.org/content/35/27/10005.full)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cH35ne7bqTpH",
        "colab_type": "text"
      },
      "source": [
        "## Very important note: almost all the pretrained computer vision models are trained by supervised fashion to classify some classes\n",
        "\n",
        "### 1. One of the most common dataset they are classifying on is the [ImageNet](www.image-net.org)\n",
        "\n",
        "### 2. [different computer vision models are better in encoding information to explain variance](http://www.brain-score.org/#leaderboard) in brain responses: DenseNet169 is better in V4 but worse in V1, while MobileNetV2 is better in V1 but worse in V4 and IT. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7sWZNKszr0q_",
        "colab_type": "text"
      },
      "source": [
        "# Part II: live coding section"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BiBt4lOkr4xA",
        "colab_type": "text"
      },
      "source": [
        "require:\n",
        "\n",
        "1. zip file of your stimuli (words in text form, images are zipped, videos are segmented into images, sound waves, sound waves?)\n",
        "2. stimulus files are ready to use\n",
        "3. open a new colab from your Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v-iVSyLAqIaR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}