custom1_DatasetMapper.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 続・detectron2 for まちカドまぞく ～カスタム・データマッパー編～\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/33882378/79108731-e6051e80-7db1-11ea-8159-3cc55c472201.jpg\">\n",
    "\n",
    "標準以外の水増しを行う方法を試してみます。\n",
    "\n",
    "まずは detectron2 で用意されている水増し機能を使用して見たいのですが、そのためにはトレーナーに渡す「データマッパー」というものを\n",
    "作る必要があるようです。\n",
    "\n",
    "---\n",
    "\n",
    "### データセットの準備\n",
    "\n",
    "前回「detectron2 for まちカドまぞく」と同じです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import json\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# VoTT のエクスポートファイルや、画像が格納されているディレクトリ\n",
    "BASE_DIRECTORY = './vott-json-export/'\n",
    "# VoTT のエクスポートファイル名\n",
    "EXPORT_FILENAME = 'Machikado-export.json'\n",
    "# 訓練データに使用する割合\n",
    "TRAIN_RATIO = 0.8\n",
    "# 乱数シード\n",
    "RANDOM_STATE = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "警告: name: 59.jpg - 画像サイズが不一致であるためスキップ image_size:(268, 201), ./vott-json-export/Machikado-export.json: (600, 600)\n"
     ]
    }
   ],
   "source": [
    "from detectron2.data import DatasetCatalog, MetadataCatalog\n",
    "from machikado_util.Machikado_vott import get_cat_names, get_machikado_dicts\n",
    "\n",
    "# vott エクスポートファイルの読み込み\n",
    "CAT_NAME2ID, CAT_ID2NAME = get_cat_names(os.path.join(BASE_DIRECTORY, EXPORT_FILENAME))\n",
    "dataset_dicts = get_machikado_dicts(os.path.join(BASE_DIRECTORY, EXPORT_FILENAME), BASE_DIRECTORY, CAT_NAME2ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### カスタムデータマッパー\n",
    "\n",
    "> オリジナルの DatasetMapper の位置は detectron2/data/dataset_mapper.py\n",
    "> \n",
    "> 以下のエラーが出る場合は、shapely をインストール\n",
    "> ```\n",
    "> ModuleNotFoundError: No module named 'shapely'\n",
    "> ```\n",
    "> ```\n",
    "> pip install shapely\n",
    "> ```\n",
    "\n",
    "* T.RandomContrast なんかは、detectron2/data/transforms/transform_gen.py にありますので、一度確認してみてください。\n",
    "\n",
    "* ポイントは、`T.Random～` という関数がジェレータの役割を担っていて、`get_transform()` で、実際の変形を加える関数を返していることです。\n",
    "* コンストラクタで、ジェネレータを生成し、__call__ で、ジェネレータからトランスフォームを生成しています。\n",
    "<p>トレーナーは訓練データ必要になる度にデータマッパーの def __call__(self, dataset_dict) を呼び出して、\n",
    "データに変更を加えた画像や、セグメンテーション用の座標データを取得しているわけです。</p>\n",
    "* 一見複雑そうですが数があるだけです。\n",
    "* 形状に変形を加える場合は、順番に注意してください。イメージと座標データを同じ順番で変形しないと？な感じになります。\n",
    "* あと、面倒なので今回使用するデータ以外は面倒を見ていません。\n",
    "<p>（イレギュラーなものは、assert しまくっています。）</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "import logging\n",
    "import torch\n",
    "\n",
    "from detectron2.data import transforms as T\n",
    "from detectron2.data import detection_utils as utils\n",
    "\n",
    "class MachikadoDatasetMapper:\n",
    "    def __init__(self, cfg, is_train=True):\n",
    "        assert cfg.MODEL.MASK_ON, '今回はセグメンテーションのみを対象にする'\n",
    "        assert not cfg.MODEL.KEYPOINT_ON, 'キーポイントは扱わない'\n",
    "        assert not cfg.MODEL.LOAD_PROPOSALS, 'pre-computed proposals っていうのがよくわからん・・・・とりあえず無効前提で'\n",
    "        \n",
    "        self.cont_gen = None\n",
    "        self.bright_gen = None\n",
    "        self.crop_gen = None\n",
    "        self.rotate_gen = None\n",
    "        \n",
    "        if is_train:\n",
    "            if cfg.INPUT.CONTRAST.ENABLED:\n",
    "                self.cont_gen = T.RandomContrast(cfg.INPUT.CONTRAST.RANGE[0], cfg.INPUT.CONTRAST.RANGE[1])\n",
    "            if cfg.INPUT.BRIGHTNESS.ENABLED:\n",
    "                self.bright_gen = T.RandomBrightness(cfg.INPUT.BRIGHTNESS.RANGE[0], cfg.INPUT.BRIGHTNESS.RANGE[1])\n",
    "            self.extent_gen = None\n",
    "            if cfg.INPUT.EXTENT.ENABLED:\n",
    "                self.extent_gen = T.RandomExtent(scale_range=(1, 1), shift_range=cfg.INPUT.EXTENT.SHIFT_RANGE)\n",
    "            if cfg.INPUT.CROP.ENABLED:\n",
    "                self.crop_gen = T.RandomCrop(cfg.INPUT.CROP.TYPE, cfg.INPUT.CROP.SIZE)\n",
    "                logging.getLogger(__name__).info('CropGen used in training: ' + str(self.crop_gen))\n",
    "            if cfg.INPUT.ROTATE.ENABLED:\n",
    "                self.rotate_gen = T.RandomRotation(cfg.INPUT.ROTATE.ANGLE, expand=False)\n",
    "        \n",
    "        self.tfm_gens = utils.build_transform_gen(cfg, is_train)\n",
    "        \n",
    "        self.img_format = cfg.INPUT.FORMAT\n",
    "        self.mask_format = cfg.INPUT.MASK_FORMAT\n",
    "        \n",
    "        self.is_train = is_train\n",
    "\n",
    "    def __call__(self, dataset_dict):\n",
    "        assert 'annotations' in dataset_dict, '今回はセグメンテーションのみを対象にする'\n",
    "        assert not 'sem_seg_file_name' in dataset_dict, 'パノプティックセグメンテーションは行わない'\n",
    "        \n",
    "        dataset_dict = copy.deepcopy(dataset_dict)\n",
    "        \n",
    "        image = utils.read_image(dataset_dict['file_name'], format=self.img_format)\n",
    "        utils.check_image_size(dataset_dict, image)\n",
    "        \n",
    "        # 明るさ・コントラスト\n",
    "        if self.cont_gen is not None:\n",
    "            tfm = self.cont_gen.get_transform(image)\n",
    "            image = tfm.apply_image(image)\n",
    "        if self.bright_gen is not None:\n",
    "            tfm = self.bright_gen.get_transform(image)\n",
    "            image = tfm.apply_image(image)\n",
    "        # アフィン\n",
    "        if self.rotate_gen is not None:\n",
    "            rotate_tfm = self.rotate_gen.get_transform(image)\n",
    "            image = rotate_tfm.apply_image(image)\n",
    "        if self.extent_gen is not None:\n",
    "            extent_tfm = self.extent_gen.get_transform(image)\n",
    "            image = extent_tfm.apply_image(image)\n",
    "        if self.crop_gen is not None:\n",
    "            crop_tfm = utils.gen_crop_transform_with_instance(\n",
    "                self.crop_gen.get_crop_size(image.shape[:2]), image.shape[:2], np.random.choice(dataset_dict['annotations']))\n",
    "            image = crop_tfm.apply_image(image)\n",
    "        \n",
    "        image, transforms = T.apply_transform_gens(self.tfm_gens, image)\n",
    "        \n",
    "        if self.crop_gen is not None:\n",
    "            transforms = crop_tfm + transforms\n",
    "        if self.extent_gen is not None:\n",
    "            transforms = extent_tfm + transforms\n",
    "        if self.rotate_gen is not None:\n",
    "            transforms = rotate_tfm + transforms\n",
    "            \n",
    "        # テストの場合はアノテーションがいらないので削除して終了\n",
    "        if not self.is_train:\n",
    "            dataset_dict.pop('annotations', None)\n",
    "            dataset_dict.pop('sem_seg_file_name', None)\n",
    "            return dataset_dict\n",
    "\n",
    "        image_shape = image.shape[:2]  # h, w\n",
    "        dataset_dict['image'] = torch.as_tensor(np.ascontiguousarray(image.transpose(2, 0, 1)))\n",
    "\n",
    "        annos = [utils.transform_instance_annotations(obj, transforms, image_shape, keypoint_hflip_indices=None)\n",
    "                 for obj in dataset_dict.pop('annotations')\n",
    "                 if obj.get(\"iscrowd\", 0) == 0]\n",
    "\n",
    "        instances = utils.annotations_to_instances(annos, image_shape, mask_format=self.mask_format)\n",
    "\n",
    "        # マスクからバウンディングボックスを作成\n",
    "        if self.crop_gen and instances.has(\"gt_masks\"):\n",
    "            instances.gt_boxes = instances.gt_masks.get_bounding_boxes()\n",
    "\n",
    "        dataset_dict[\"instances\"] = utils.filter_empty_instances(instances)\n",
    "\n",
    "        return dataset_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### カスタマイズした設定\n",
    "\n",
    "カスタムした分増えた設定をどうしようかと考えたが、cfg にくっつければ良いのでは？と思ったので追加する関数を作った。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from detectron2.config import CfgNode as CN\n",
    "\n",
    "# カスタムした分の設定を追加する\n",
    "def append_custom_cfg(cfg):\n",
    "    cfg.INPUT.ROTATE = CN()\n",
    "    cfg.INPUT.CONTRAST = CN()\n",
    "    cfg.INPUT.BRIGHTNESS = CN()\n",
    "    cfg.INPUT.EXTENT = CN()\n",
    "\n",
    "    cfg.INPUT.ROTATE.ENABLED = True\n",
    "    cfg.INPUT.ROTATE.ANGLE = [-20, 20]\n",
    "    cfg.INPUT.CONTRAST.ENABLED = True\n",
    "    cfg.INPUT.CONTRAST.RANGE = (0.5, 1.5)\n",
    "    cfg.INPUT.BRIGHTNESS.ENABLED = True\n",
    "    cfg.INPUT.BRIGHTNESS.RANGE = (0.8, 1.2)\n",
    "    cfg.INPUT.EXTENT.ENABLED = True\n",
    "    cfg.INPUT.EXTENT.SHIFT_RANGE = (0.2, 0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 画像の入力サイズを変更しています。\n",
    "* これによって精度が向上しました。（mAP 70% -> 80%）\n",
    "<p>（アニメ画像なのでそもそも標準の解像度はいらないんですよね・・・）</p>\n",
    "* また、画像が小さくなったので、バッチ処理枚数が増えます。（バッチ枚数が精度向上に貢献している？）\n",
    "<p>（バッチ数については、うちの GPU じゃこれ以上検証無理かな？）</p>\n",
    "* ちなみに、MIN、MAX の意味は、 detectron2/data/transforms/transform_gen.py の ResizeShortestEdge を参照してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from detectron2.config import get_cfg\n",
    "\n",
    "cfg = get_cfg()\n",
    "append_custom_cfg(cfg)\n",
    "\n",
    "cfg.MODEL.MASK_ON = True\n",
    "\n",
    "# インプットサイズを変更しています！\n",
    "cfg.INPUT.MIN_SIZE_TRAIN = 400\n",
    "cfg.INPUT.MAX_SIZE_TRAIN = 640\n",
    "cfg.INPUT.MIN_SIZE_TEST = 400\n",
    "cfg.INPUT.MAX_SIZE_TEST = 640\n",
    "\n",
    "cfg.INPUT.CROP.ENABLED = True\n",
    "cfg.INPUT.CROP.SIZE = [0.8, 0.8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### テスト表示\n",
    "\n",
    "画像サイズを縮小していますが十分でしょう。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from machikado_util.plot_dataset_mapper import plot_dataset_mapper\n",
    "\n",
    "# random.seed(RANDOM_STATE)\n",
    "mapper = MachikadoDatasetMapper(cfg, is_train=True)\n",
    "plot_dataset_mapper(dataset_dicts, mapper)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}