site/en/hub/tutorials/tf_hub_delf_module.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RUymE2l9GZfO"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "JMyTNwSJGGWg"
      },
      "outputs": [],
      "source": [
        "# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0DmDwGPOGfaQ"
      },
      "source": [
        "# How to match images using DELF and TensorFlow Hub\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/hub/tutorials/tf_hub_delf_module\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/hub/tutorials/tf_hub_delf_module.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/hub/tutorials/tf_hub_delf_module.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/hub/tutorials/tf_hub_delf_module.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://tfhub.dev/google/delf/1\"><img src=\"https://www.tensorflow.org/images/hub_logo_32px.png\" />See TF Hub model</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f3nk38tIKytQ"
      },
      "source": [
        "TensorFlow Hub (TF-Hub) is a platform to share machine learning expertise packaged in reusable resources, notably pre-trained **modules**.\n",
        "\n",
        "In this colab, we will use a module that packages the [DELF](https://github.com/tensorflow/models/tree/master/research/delf) neural network and logic for processing images to identify keypoints and their descriptors. The weights of the neural network were trained on images of landmarks as described in [this paper](https://arxiv.org/abs/1612.06321)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q4DN769E2O_R"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lrKaWOB_cuS3"
      },
      "outputs": [],
      "source": [
        "!pip install scikit-image"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SI7eVflHHxvi"
      },
      "outputs": [],
      "source": [
        "from absl import logging\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from PIL import Image, ImageOps\n",
        "from scipy.spatial import cKDTree\n",
        "from skimage.feature import plot_matches\n",
        "from skimage.measure import ransac\n",
        "from skimage.transform import AffineTransform\n",
        "from six import BytesIO\n",
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "import tensorflow_hub as hub\n",
        "from six.moves.urllib.request import urlopen"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qquo2HiONiDK"
      },
      "source": [
        "## The data\n",
        "\n",
        "In the next cell, we specify the URLs of two images we would like to process with DELF in order to match and compare them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l93ye4WFIqIV"
      },
      "outputs": [],
      "source": [
        "#@title Choose images\n",
        "images = \"Bridge of Sighs\" #@param [\"Bridge of Sighs\", \"Golden Gate\", \"Acropolis\", \"Eiffel tower\"]\n",
        "if images == \"Bridge of Sighs\":\n",
        "  # from: https://commons.wikimedia.org/wiki/File:Bridge_of_Sighs,_Oxford.jpg\n",
        "  # by: N.H. Fischer\n",
        "  IMAGE_1_URL = 'https://upload.wikimedia.org/wikipedia/commons/2/28/Bridge_of_Sighs%2C_Oxford.jpg'\n",
        "  # from https://commons.wikimedia.org/wiki/File:The_Bridge_of_Sighs_and_Sheldonian_Theatre,_Oxford.jpg\n",
        "  # by: Matthew Hoser\n",
        "  IMAGE_2_URL = 'https://upload.wikimedia.org/wikipedia/commons/c/c3/The_Bridge_of_Sighs_and_Sheldonian_Theatre%2C_Oxford.jpg'\n",
        "elif images == \"Golden Gate\":\n",
        "  IMAGE_1_URL = 'https://upload.wikimedia.org/wikipedia/commons/1/1e/Golden_gate2.jpg'\n",
        "  IMAGE_2_URL = 'https://upload.wikimedia.org/wikipedia/commons/3/3e/GoldenGateBridge.jpg'\n",
        "elif images == \"Acropolis\":\n",
        "  IMAGE_1_URL = 'https://upload.wikimedia.org/wikipedia/commons/c/ce/2006_01_21_Ath%C3%A8nes_Parth%C3%A9non.JPG'\n",
        "  IMAGE_2_URL = 'https://upload.wikimedia.org/wikipedia/commons/5/5c/ACROPOLIS_1969_-_panoramio_-_jean_melis.jpg'\n",
        "else:\n",
        "  IMAGE_1_URL = 'https://upload.wikimedia.org/wikipedia/commons/d/d8/Eiffel_Tower%2C_November_15%2C_2011.jpg'\n",
        "  IMAGE_2_URL = 'https://upload.wikimedia.org/wikipedia/commons/a/a8/Eiffel_Tower_from_immediately_beside_it%2C_Paris_May_2008.jpg'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ttlHtcmiN6QF"
      },
      "source": [
        "Download, resize, save and display the images."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "E6RMomGJSfeb"
      },
      "outputs": [],
      "source": [
        "def download_and_resize(name, url, new_width=256, new_height=256):\n",
        "  path = tf.keras.utils.get_file(url.split('/')[-1], url)\n",
        "  image = Image.open(path)\n",
        "  image = ImageOps.fit(image, (new_width, new_height), Image.LANCZOS)\n",
        "  return image"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "reajtO7XSj7Y"
      },
      "outputs": [],
      "source": [
        "image1 = download_and_resize('image_1.jpg', IMAGE_1_URL)\n",
        "image2 = download_and_resize('image_2.jpg', IMAGE_2_URL)\n",
        "\n",
        "plt.subplot(1,2,1)\n",
        "plt.imshow(image1)\n",
        "plt.subplot(1,2,2)\n",
        "plt.imshow(image2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "leKqkoT9OP7r"
      },
      "source": [
        "## Apply the DELF module to the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A3WoT1-SPoTI"
      },
      "source": [
        "The DELF module takes an image as input and will describe noteworthy points with vectors. The following cell contains the core of this colab's logic."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pXr2tUhvp1Ue"
      },
      "outputs": [],
      "source": [
        "delf = hub.load('https://tfhub.dev/google/delf/1').signatures['default']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pvAU_gUHoYcY"
      },
      "outputs": [],
      "source": [
        "def run_delf(image):\n",
        "  np_image = np.array(image)\n",
        "  float_image = tf.image.convert_image_dtype(np_image, tf.float32)\n",
        "\n",
        "  return delf(\n",
        "      image=float_image,\n",
        "      score_threshold=tf.constant(100.0),\n",
        "      image_scales=tf.constant([0.25, 0.3536, 0.5, 0.7071, 1.0, 1.4142, 2.0]),\n",
        "      max_feature_num=tf.constant(1000))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FEzgHAT0UDNP"
      },
      "outputs": [],
      "source": [
        "result1 = run_delf(image1)\n",
        "result2 = run_delf(image2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NByyBA5yOL2b"
      },
      "source": [
        "## Use the locations and description vectors to match the images"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "both",
        "id": "mVaKXT3cMSib"
      },
      "outputs": [],
      "source": [
        "#@title TensorFlow is not needed for this post-processing and visualization\n",
        "def match_images(image1, image2, result1, result2):\n",
        "  distance_threshold = 0.8\n",
        "\n",
        "  # Read features.\n",
        "  num_features_1 = result1['locations'].shape[0]\n",
        "  print(\"Loaded image 1's %d features\" % num_features_1)\n",
        "  \n",
        "  num_features_2 = result2['locations'].shape[0]\n",
        "  print(\"Loaded image 2's %d features\" % num_features_2)\n",
        "\n",
        "  # Find nearest-neighbor matches using a KD tree.\n",
        "  d1_tree = cKDTree(result1['descriptors'])\n",
        "  _, indices = d1_tree.query(\n",
        "      result2['descriptors'],\n",
        "      distance_upper_bound=distance_threshold)\n",
        "\n",
        "  # Select feature locations for putative matches.\n",
        "  locations_2_to_use = np.array([\n",
        "      result2['locations'][i,]\n",
        "      for i in range(num_features_2)\n",
        "      if indices[i] != num_features_1\n",
        "  ])\n",
        "  locations_1_to_use = np.array([\n",
        "      result1['locations'][indices[i],]\n",
        "      for i in range(num_features_2)\n",
        "      if indices[i] != num_features_1\n",
        "  ])\n",
        "\n",
        "  # Perform geometric verification using RANSAC.\n",
        "  _, inliers = ransac(\n",
        "      (locations_1_to_use, locations_2_to_use),\n",
        "      AffineTransform,\n",
        "      min_samples=3,\n",
        "      residual_threshold=20,\n",
        "      max_trials=1000)\n",
        "\n",
        "  print('Found %d inliers' % sum(inliers))\n",
        "\n",
        "  # Visualize correspondences.\n",
        "  _, ax = plt.subplots()\n",
        "  inlier_idxs = np.nonzero(inliers)[0]\n",
        "  plot_matches(\n",
        "      ax,\n",
        "      image1,\n",
        "      image2,\n",
        "      locations_1_to_use,\n",
        "      locations_2_to_use,\n",
        "      np.column_stack((inlier_idxs, inlier_idxs)),\n",
        "      matches_color='b')\n",
        "  ax.axis('off')\n",
        "  ax.set_title('DELF correspondences')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tpEgqOvCYlPY"
      },
      "outputs": [],
      "source": [
        "match_images(image1, image2, result1, result2)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "RUymE2l9GZfO"
      ],
      "name": "tf_hub_delf_module.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}