site/ja/hub/tutorials/action_recognition_with_tf_hub.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x8Q7Un821X1A"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1W4rIAFt1Ui3"
      },
      "outputs": [],
      "source": [
        "# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cDq0CIKc1vO_"
      },
      "source": [
        "# Inflated 3D CNN による行動認識\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>     <a target=\"_blank\" href=\"https://www.tensorflow.org/hub/tutorials/action_recognition_with_tf_hub\">     <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\">     TensorFlow.org で表示</a>\n",
        "</td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/ja/hub/tutorials/action_recognition_with_tf_hub.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">Google Colabで実行</a>\n",
        "</td>\n",
        "  <td><a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/ja/hub/tutorials/action_recognition_with_tf_hub.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\">GitHub でソースを表示</a></td>\n",
        "  <td><a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/ja/hub/tutorials/action_recognition_with_tf_hub.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\">ノートブックをダウンロード</a></td>\n",
        "  <td><a href=\"https://tfhub.dev/deepmind/i3d-kinetics-400/1\"><img src=\"https://www.tensorflow.org/images/hub_logo_32px.png\"> TF Hub モデルを参照</a></td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h6W3FhoP3TxC"
      },
      "source": [
        "この Colab では、[tfhub.dev/deepmind/i3d-kinetics-400/1](https://tfhub.dev/deepmind/i3d-kinetics-400/1) モジュールを用いて動画データの行動の認識を実演します。dぴが何のアクションを検出するその他のモデルは、[こちら](https://tfhub.dev/s?module-type=video-classification)をご覧ください。\n",
        "\n",
        "基礎となるモデルは、Joao Carreira と Andrew Zisserman による論文『[Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://arxiv.org/abs/1705.07750)』に記載されています。この論文は 2017 年 5 月に arXiv に投稿され、CVPR 2017 の学会論文として発表されました。ソースコードは [github](https://github.com/deepmind/kinetics-i3d) で公開されています。\n",
        "\n",
        "『Quo Vadis』では新しい動画分類のためのアーキテクチャ、Inflated 3D Convnet（I3D）が発表されました。このアーキテクチャでこれらのモデルのファインチューニングを行い、UCF101 と HMDB51 のデータセットで最先端の結果が達成されています。Kinetics データセットで事前トレーニングされた I3D のモデルは、CVPR 2017 [Charades Challenge](http://vuchallenge.org/charades.html) でも 1 位を獲得しました。\n",
        "\n",
        "オリジナルのモジュールは [Kinetics-400 データセット](https://www.deepmind.com/open-source/kinetics)でトレーニングされており、約 400 種類の異なる行動を認知しています。これらの行動のラベルは [label map ファイル](https://github.com/deepmind/kinetics-i3d/blob/master/data/label_map.txt)内にあります。\n",
        "\n",
        "この Colab では、これを UCF101 データセットの動画中の動作を認識するために使用します。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R_0xc2jyNGRp"
      },
      "source": [
        "## セットアップ"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mOHMWsFnITdi"
      },
      "outputs": [],
      "source": [
        "!pip install -q imageio\n",
        "!pip install -q opencv-python\n",
        "!pip install -q git+https://github.com/tensorflow/docs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "both",
        "id": "USf0UvkYIlKo"
      },
      "outputs": [],
      "source": [
        "#@title Import the necessary modules\n",
        "# TensorFlow and TF-Hub modules.\n",
        "from absl import logging\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from tensorflow_docs.vis import embed\n",
        "\n",
        "logging.set_verbosity(logging.ERROR)\n",
        "\n",
        "# Some modules to help with reading the UCF101 dataset.\n",
        "import random\n",
        "import re\n",
        "import os\n",
        "import tempfile\n",
        "import ssl\n",
        "import cv2\n",
        "import numpy as np\n",
        "\n",
        "# Some modules to display an animation using imageio.\n",
        "import imageio\n",
        "from IPython import display\n",
        "\n",
        "from urllib import request  # requires python3"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "both",
        "id": "IuMMS3TGdws7"
      },
      "outputs": [],
      "source": [
        "#@title Helper functions for the UCF101 dataset\n",
        "\n",
        "# Utilities to fetch videos from UCF101 dataset\n",
        "UCF_ROOT = \"https://www.crcv.ucf.edu/THUMOS14/UCF101/UCF101/\"\n",
        "_VIDEO_LIST = None\n",
        "_CACHE_DIR = tempfile.mkdtemp()\n",
        "# As of July 2020, crcv.ucf.edu doesn't use a certificate accepted by the\n",
        "# default Colab environment anymore.\n",
        "unverified_context = ssl._create_unverified_context()\n",
        "\n",
        "def list_ucf_videos():\n",
        "  \"\"\"Lists videos available in UCF101 dataset.\"\"\"\n",
        "  global _VIDEO_LIST\n",
        "  if not _VIDEO_LIST:\n",
        "    index = request.urlopen(UCF_ROOT, context=unverified_context).read().decode(\"utf-8\")\n",
        "    videos = re.findall(\"(v_[\\w_]+\\.avi)\", index)\n",
        "    _VIDEO_LIST = sorted(set(videos))\n",
        "  return list(_VIDEO_LIST)\n",
        "\n",
        "def fetch_ucf_video(video):\n",
        "  \"\"\"Fetchs a video and cache into local filesystem.\"\"\"\n",
        "  cache_path = os.path.join(_CACHE_DIR, video)\n",
        "  if not os.path.exists(cache_path):\n",
        "    urlpath = request.urljoin(UCF_ROOT, video)\n",
        "    print(\"Fetching %s => %s\" % (urlpath, cache_path))\n",
        "    data = request.urlopen(urlpath, context=unverified_context).read()\n",
        "    open(cache_path, \"wb\").write(data)\n",
        "  return cache_path\n",
        "\n",
        "# Utilities to open video files using CV2\n",
        "def crop_center_square(frame):\n",
        "  y, x = frame.shape[0:2]\n",
        "  min_dim = min(y, x)\n",
        "  start_x = (x // 2) - (min_dim // 2)\n",
        "  start_y = (y // 2) - (min_dim // 2)\n",
        "  return frame[start_y:start_y+min_dim,start_x:start_x+min_dim]\n",
        "\n",
        "def load_video(path, max_frames=0, resize=(224, 224)):\n",
        "  cap = cv2.VideoCapture(path)\n",
        "  frames = []\n",
        "  try:\n",
        "    while True:\n",
        "      ret, frame = cap.read()\n",
        "      if not ret:\n",
        "        break\n",
        "      frame = crop_center_square(frame)\n",
        "      frame = cv2.resize(frame, resize)\n",
        "      frame = frame[:, :, [2, 1, 0]]\n",
        "      frames.append(frame)\n",
        "      \n",
        "      if len(frames) == max_frames:\n",
        "        break\n",
        "  finally:\n",
        "    cap.release()\n",
        "  return np.array(frames) / 255.0\n",
        "\n",
        "def to_gif(images):\n",
        "  converted_images = np.clip(images * 255, 0, 255).astype(np.uint8)\n",
        "  imageio.mimsave('./animation.gif', converted_images, duration=40)\n",
        "  return embed.embed_file('./animation.gif')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "pIKTs-KneUfz"
      },
      "outputs": [],
      "source": [
        "#@title Get the kinetics-400 labels\n",
        "# Get the kinetics-400 action labels from the GitHub repository.\n",
        "KINETICS_URL = \"https://raw.githubusercontent.com/deepmind/kinetics-i3d/master/data/label_map.txt\"\n",
        "with request.urlopen(KINETICS_URL) as obj:\n",
        "  labels = [line.decode(\"utf-8\").strip() for line in obj.readlines()]\n",
        "print(\"Found %d labels.\" % len(labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GBvmjVICIp3W"
      },
      "source": [
        "# UCF101 データセットを使用する"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V-QcxdhLIfi2"
      },
      "outputs": [],
      "source": [
        "# Get the list of videos in the dataset.\n",
        "ucf_videos = list_ucf_videos()\n",
        "  \n",
        "categories = {}\n",
        "for video in ucf_videos:\n",
        "  category = video[2:-12]\n",
        "  if category not in categories:\n",
        "    categories[category] = []\n",
        "  categories[category].append(video)\n",
        "print(\"Found %d videos in %d categories.\" % (len(ucf_videos), len(categories)))\n",
        "\n",
        "for category, sequences in categories.items():\n",
        "  summary = \", \".join(sequences[:2])\n",
        "  print(\"%-20s %4d videos (%s, ...)\" % (category, len(sequences), summary))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c0ZvVDruN2nU"
      },
      "outputs": [],
      "source": [
        "# Get a sample cricket video.\n",
        "video_path = fetch_ucf_video(\"v_CricketShot_g04_c02.avi\")\n",
        "sample_video = load_video(video_path)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hASLA90YFPTO"
      },
      "outputs": [],
      "source": [
        "sample_video.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "POf5XgffvXlD"
      },
      "outputs": [],
      "source": [
        "i3d = hub.load(\"https://tfhub.dev/deepmind/i3d-kinetics-400/1\").signatures['default']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mDXgaOD1zhMP"
      },
      "source": [
        "id3 モデルを実行して、上位 5 つの行動予測を出力します。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3mTbqA5JGYUx"
      },
      "outputs": [],
      "source": [
        "def predict(sample_video):\n",
        "  # Add a batch axis to the sample video.\n",
        "  model_input = tf.constant(sample_video, dtype=tf.float32)[tf.newaxis, ...]\n",
        "\n",
        "  logits = i3d(model_input)['default'][0]\n",
        "  probabilities = tf.nn.softmax(logits)\n",
        "\n",
        "  print(\"Top 5 actions:\")\n",
        "  for i in np.argsort(probabilities)[::-1][:5]:\n",
        "    print(f\"  {labels[i]:22}: {probabilities[i] * 100:5.2f}%\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ykaXQcGRvK4E"
      },
      "outputs": [],
      "source": [
        "predict(sample_video)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PHsq0lHXCsD4"
      },
      "source": [
        "では、次のリンクから新しい動画を試してみましょう。https://commons.wikimedia.org/wiki/Category:Videos_of_sports\n",
        "\n",
        "Patrick Gillett の[こちらの動画](https://commons.wikimedia.org/wiki/File:End_of_a_jam.ogv)はどうでしょうか。 "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p-mZ9fFPCoNq"
      },
      "outputs": [],
      "source": [
        "!curl -O https://upload.wikimedia.org/wikipedia/commons/8/86/End_of_a_jam.ogv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lpLmE8rjEbAF"
      },
      "outputs": [],
      "source": [
        "video_path = \"End_of_a_jam.ogv\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CHZJ9qTLErhV"
      },
      "outputs": [],
      "source": [
        "sample_video = load_video(video_path)[:100]\n",
        "sample_video.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2ZNLkEZ9Er-c"
      },
      "outputs": [],
      "source": [
        "to_gif(sample_video)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yskHIRbxEtjS"
      },
      "outputs": [],
      "source": [
        "predict(sample_video)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "x8Q7Un821X1A"
      ],
      "name": "action_recognition_with_tf_hub.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}