Machikado_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# detectron2 for まちカドまぞく ～訓練編～\n",
    "\n",
    "<img src=https://user-images.githubusercontent.com/33882378/79190134-739e4800-7e5e-11ea-98df-9b1a52276964.jpg>\n",
    "\n",
    "detectron2 で独自データセット学習する方法\n",
    "\n",
    "参考にしたホームページ: https://demura.net/deeplearning/16807.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import json\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 独自のデータセットを読み込んで、データセットを用意する\n",
    "\n",
    "基本的に元になっているデータ形式をゴリゴリ自分で読み込んで、detectron2 の形式に変換していく\n",
    "\n",
    "### VoTT Export 形式からの読み込み"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# VoTT のエクスポートファイルや、画像が格納されているディレクトリ\n",
    "BASE_DIRECTORY = './vott-json-export/'\n",
    "# VoTT のエクスポートファイル名\n",
    "EXPORT_FILENAME = 'Machikado-export.json'\n",
    "# 訓練データに使用する割合\n",
    "TRAIN_RATIO = 0.8\n",
    "# 乱数シード\n",
    "RANDOM_STATE = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* vott の場合は \"tags\" 格納されているカテゴリ名が格納されているのでそれを読み出せば良い"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([(0, 'Shamiko'),\n",
       "             (1, 'Gosenzo'),\n",
       "             (2, 'Lilith'),\n",
       "             (3, 'Momo'),\n",
       "             (4, 'Mikan'),\n",
       "             (5, 'Mob')])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import OrderedDict\n",
    "\n",
    "# エクスポートファイルからカテゴリ名を調べる\n",
    "with open(os.path.join(BASE_DIRECTORY, EXPORT_FILENAME), 'r') as f:\n",
    "    json_data = json.load(f)\n",
    "    \n",
    "CAT_NAME2ID = OrderedDict()\n",
    "CAT_ID2NAME = OrderedDict()\n",
    "\n",
    "for i, node in enumerate(json_data['tags']):\n",
    "    CAT_NAME2ID[node['name']] = i\n",
    "    CAT_ID2NAME[i] = node['name']\n",
    "\n",
    "CAT_ID2NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* DatasetCatalog へ設定する関数は、以下のコードのように**所定の形式の辞書のリスト**を返す引数の無い関数を作れば良い\n",
    "* vott のアノテーションデータは \"assets\" に全て格納されているので、\"assets\" をゴリゴリ読み込んでいく。\n",
    "* マスクに必要な座標データは、\"regions\" に格納されている。\"regions\" は複数の領域データを含んでいる可能性があるので全て列挙する。\n",
    "領域データがない場合もあるのでそれも処理しておく。\n",
    "<p>（VoTT で閲覧のみで、アノテーションされていないデータは領域データが無い）</p>\n",
    "* 領域データの \"tags\" にカテゴリ名が格納されているので、１つ前のセルで読み込んだタグ情報(CAT_NAME2ID)を使用して整数のIDを振っていく。\n",
    "<p>（tags は複数のカテゴリ名が格納されている可能性がある。VoTT で複数のタグをチェック出来るので注意）</p>\n",
    "* 実際の画像とアノテーションデータ内の画像サイズが不一致な場合はスキップする。\n",
    "<p>（ファイルがVoTT で閲覧後（目のアイコンマークがついた状態）で該当ファイルを差し替えても、元の画像サイズが使用されるためマスク座標のスケールが合わなくなる。）</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "from detectron2.structures import BoxMode\n",
    "\n",
    "# machikado用にアレンジした読み込み関数\n",
    "def get_machikado_dicts():\n",
    "    with open(os.path.join(BASE_DIRECTORY, EXPORT_FILENAME), 'r') as f:\n",
    "        json_data = json.load(f, object_pairs_hook=OrderedDict) # データ順を固定しておく\n",
    "    \n",
    "    assets = json_data['assets']\n",
    "\n",
    "    dataset_dicts = []\n",
    "    for item in assets.values():\n",
    "        asset = item['asset']\n",
    "        regions = item['regions']\n",
    "\n",
    "        if len(regions) == 0:\n",
    "            print('警告: name: {} - 領域データが空だったのでスキップ'.format(asset['name']))\n",
    "            continue\n",
    "        \n",
    "        # 画像サイズを取得し確認する\n",
    "        # （VoTT でアノテーション中画像を差し替えると画像のサイズが古い画像のままになるので修正する）\n",
    "        file_name = os.path.join(BASE_DIRECTORY, asset['name'])\n",
    "        im = Image.open(file_name)\n",
    "        w, h = im.size\n",
    "        \n",
    "        if asset['size']['height'] != h or asset['size']['width'] != w:\n",
    "            print('警告: name: {} - 画像サイズが不一致であるためスキップ image_size:({}, {}), {}: ({}, {})'.format(\n",
    "                asset['name'], asset['size']['width'], asset['size']['height'], EXPORT_FILENAME, w, h))\n",
    "            continue\n",
    "        \n",
    "        record = {}\n",
    "        record['file_name'] = file_name\n",
    "        record['height'] = h\n",
    "        record['width'] = w\n",
    "        \n",
    "        objs = []\n",
    "        for region in regions:\n",
    "            points = region['points']\n",
    "            assert len(points), '座標データが無い！'\n",
    "\n",
    "            if len(region['tags']) > 1:\n",
    "                print('警告: name: {} - 複数のタグを確認！ tags: {}'.format(asset['name'], region['tags']))\n",
    "\n",
    "            poly = []\n",
    "            for pt in points:\n",
    "                poly += [pt['x'], pt['y']]\n",
    "\n",
    "            bbox = region['boundingBox']\n",
    "\n",
    "            obj = {\n",
    "                'bbox': [bbox['left'], bbox['top'], bbox['width'], bbox['height']],\n",
    "                'bbox_mode': BoxMode.XYWH_ABS, # XYWH_REL はまだサポートされていないらしい\n",
    "                'segmentation': [poly],\n",
    "                'category_id': CAT_NAME2ID[region['tags'][0]],\n",
    "                'iscrowd': 0\n",
    "            }\n",
    "            objs.append(obj)\n",
    "\n",
    "        record['annotations'] = objs\n",
    "        dataset_dicts.append(record)\n",
    "        \n",
    "    return dataset_dicts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DatasetCatalogを用意する\n",
    "\n",
    "* 初めから訓練、テストが分かれてれば良いが、一緒のフォルダでも分割後にラムダ式で指定すれば良い。\n",
    "<p>（まあ最初から分けた方が後々楽そうではあるので VoTT のデータを分けて保存出来れば良いと思うが、VoTT の標準ではないんですよね・・・）</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "警告: name: 59.jpg - 画像サイズが不一致であるためスキップ image_size:(268, 201), Machikado-export.json: (600, 600)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Metadata(name='test', thing_classes=['Shamiko', 'Gosenzo', 'Lilith', 'Momo', 'Mikan', 'Mob'])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from detectron2.data import DatasetCatalog, MetadataCatalog\n",
    "\n",
    "dataset_dicts = get_machikado_dicts()\n",
    "\n",
    "# 訓練用、テスト用に分ける\n",
    "random.seed(RANDOM_STATE)\n",
    "random.shuffle(dataset_dicts)\n",
    "\n",
    "split_idx = int(len(dataset_dicts) * TRAIN_RATIO) + 1\n",
    "\n",
    "# 登録\n",
    "DatasetCatalog.clear()\n",
    "DatasetCatalog.register('train', lambda : dataset_dicts[:split_idx])\n",
    "DatasetCatalog.register('test', lambda : dataset_dicts[split_idx:])\n",
    "\n",
    "MetadataCatalog.get('train').set(thing_classes=list(CAT_NAME2ID.keys()))\n",
    "MetadataCatalog.get('test').set(thing_classes=list(CAT_NAME2ID.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 学習\n",
    "\n",
    "* detectron2/engine/defaults.py 399 行目コメントアウトでウザイ モデル表示をしないように出来る。\n",
    "```python\n",
    "    @classmethod\n",
    "    def build_model(cls, cfg):\n",
    "        \"\"\"\n",
    "        Returns:\n",
    "            torch.nn.Module:\n",
    "\n",
    "        It now calls :func:`detectron2.modeling.build_model`.\n",
    "        Overwrite it if you'd like a different model.\n",
    "        \"\"\"\n",
    "        model = build_model(cfg)\n",
    "        logger = logging.getLogger(__name__)\n",
    "#        logger.info(\"Model:\\n{}\".format(model)) # モデル表示を抑制\n",
    "        return model\n",
    "```\n",
    "\n",
    "\n",
    "* モデルを変更したい場合は https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md ここに色々あるので試すと良い。\n",
    "* `cfg.MODEL.WEIGHTS = ` の部分は使用したい重みファイルに変える。\n",
    "<p>(ダウンロードしてきた重みファイルへのパスを設定する)</p>\n",
    "* merge_from_file はきちんと推論の時も揃えないとダメ！！\n",
    "\n",
    "* 訓練でに以下のエラーが出る場合\n",
    "```python\n",
    "ModuleNotFoundError: No module named 'shapely'\n",
    "```\n",
    "\n",
    "```\n",
    "pip install shapely\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from detectron2.config import get_cfg\n",
    "\n",
    "cfg = get_cfg()\n",
    "cfg.OUTPUT_DIR = './output'\n",
    "cfg.CUDA = 'cuda:0'\n",
    "\n",
    "# cfg.merge_from_file(\"../configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml\")\n",
    "# cfg.MODEL.WEIGHTS = './coco_models/model_final_f10217.pkl'\n",
    "# cfg.SOLVER.IMS_PER_BATCH = 2\n",
    "\n",
    "# 重いけど、これ精度良いです。\n",
    "cfg.merge_from_file('../configs/COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x.yaml')\n",
    "cfg.MODEL.WEIGHTS = './coco_models/model_final_2d9806.pkl'\n",
    "cfg.SOLVER.IMS_PER_BATCH = 1 # GTX2070 ではこれが限界\n",
    "\n",
    "cfg.DATASETS.TRAIN = ('train',)\n",
    "cfg.DATASETS.TEST = ()   # no metrics implemented for this dataset\n",
    "cfg.DATALOADER.NUM_WORKERS = 2\n",
    "cfg.SOLVER.BASE_LR = 0.00025\n",
    "cfg.SOLVER.MAX_ITER = 1500    # 300 iterations seems good enough, but you can certainly train longer <- とあるが、まあデータセットによるよね・・・\n",
    "cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128   # faster, and good enough for this toy dataset\n",
    "cfg.MODEL.ROI_HEADS.NUM_CLASSES = len(CAT_ID2NAME) \n",
    "\n",
    "# ランダムクロップを有効にする\n",
    "cfg.INPUT.CROP.ENABLED = True\n",
    "cfg.INPUT.CROP.SIZE = [0.9, 0.9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorBoard を見たければ、detectron2_Machikado まで cd で移動して・・・\n",
    "\n",
    "```\n",
    "tensorboard --logdir output\n",
    "```\n",
    "\n",
    "docker な人は以下の様にする\n",
    "（事前に6006ポートをポートフォワーディングしておくことと、--bind_all が必要）\n",
    "\n",
    "```\n",
    "tensorboard --logdir output --bind_all\n",
    "```\n",
    "今回の学習結果はこの様な感じでした。\n",
    "\n",
    "<img src=https://user-images.githubusercontent.com/33882378/79212522-bbcf6180-7e82-11ea-96c2-359c690130fd.jpg>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[04/13 23:04:09 d2.data.build]: \u001b[0mRemoved 0 images with no usable annotations. 92 images left.\n",
      "\u001b[32m[04/13 23:04:09 d2.data.build]: \u001b[0mDistribution of instances among all 6 categories:\n",
      "\u001b[36m|  category  | #instances   |  category  | #instances   |  category  | #instances   |\n",
      "|:----------:|:-------------|:----------:|:-------------|:----------:|:-------------|\n",
      "|  Shamiko   | 70           |  Gosenzo   | 28           |   Lilith   | 12           |\n",
      "|    Momo    | 41           |   Mikan    | 15           |    Mob     | 12           |\n",
      "|            |              |            |              |            |              |\n",
      "|   total    | 178          |            |              |            |              |\u001b[0m\n",
      "\u001b[32m[04/13 23:04:09 d2.data.common]: \u001b[0mSerializing 92 elements to byte tensors and concatenating them all ...\n",
      "\u001b[32m[04/13 23:04:09 d2.data.common]: \u001b[0mSerialized dataset takes 0.23 MiB\n",
      "\u001b[32m[04/13 23:04:09 d2.data.dataset_mapper]: \u001b[0mCropGen used in training: RandomCrop(crop_type='relative_range', crop_size=[0.9, 0.9])\n",
      "\u001b[32m[04/13 23:04:09 d2.data.detection_utils]: \u001b[0mTransformGens used in training: [ResizeShortestEdge(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style='choice'), RandomFlip()]\n",
      "\u001b[32m[04/13 23:04:09 d2.data.build]: \u001b[0mUsing training sampler TrainingSampler\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'roi_heads.box_predictor.cls_score.weight' has shape (81, 1024) in the checkpoint but (7, 1024) in the model! Skipped.\n",
      "'roi_heads.box_predictor.cls_score.bias' has shape (81,) in the checkpoint but (7,) in the model! Skipped.\n",
      "'roi_heads.box_predictor.bbox_pred.weight' has shape (320, 1024) in the checkpoint but (24, 1024) in the model! Skipped.\n",
      "'roi_heads.box_predictor.bbox_pred.bias' has shape (320,) in the checkpoint but (24,) in the model! Skipped.\n",
      "'roi_heads.mask_head.predictor.weight' has shape (80, 256, 1, 1) in the checkpoint but (6, 256, 1, 1) in the model! Skipped.\n",
      "'roi_heads.mask_head.predictor.bias' has shape (80,) in the checkpoint but (6,) in the model! Skipped.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[04/13 23:04:10 d2.engine.train_loop]: \u001b[0mStarting training from iteration 0\n",
      "\u001b[32m[04/13 23:04:20 d2.utils.events]: \u001b[0m eta: 0:12:04  iter: 19  total_loss: 3.101  loss_cls: 1.862  loss_box_reg: 0.479  loss_mask: 0.690  loss_rpn_cls: 0.006  loss_rpn_loc: 0.018  time: 0.4878  data_time: 0.0088  lr: 0.000005  max_mem: 3675M\n",
      "\u001b[32m[04/13 23:04:29 d2.utils.events]: \u001b[0m eta: 0:11:42  iter: 39  total_loss: 2.989  loss_cls: 1.735  loss_box_reg: 0.505  loss_mask: 0.688  loss_rpn_cls: 0.002  loss_rpn_loc: 0.008  time: 0.4769  data_time: 0.0018  lr: 0.000010  max_mem: 3771M\n",
      "\u001b[32m[04/13 23:04:39 d2.utils.events]: \u001b[0m eta: 0:11:27  iter: 59  total_loss: 3.125  loss_cls: 1.474  loss_box_reg: 0.906  loss_mask: 0.683  loss_rpn_cls: 0.003  loss_rpn_loc: 0.011  time: 0.4704  data_time: 0.0019  lr: 0.000015  max_mem: 3771M\n",
      "\u001b[32m[04/13 23:04:48 d2.utils.events]: \u001b[0m eta: 0:11:15  iter: 79  total_loss: 2.536  loss_cls: 1.129  loss_box_reg: 0.693  loss_mask: 0.679  loss_rpn_cls: 0.002  loss_rpn_loc: 0.011  time: 0.4716  data_time: 0.0019  lr: 0.000020  max_mem: 3771M\n",
      "\u001b[32m[04/13 23:04:58 d2.utils.events]: \u001b[0m eta: 0:11:06  iter: 99  total_loss: 2.158  loss_cls: 0.826  loss_box_reg: 0.684  loss_mask: 0.659  loss_rpn_cls: 0.004  loss_rpn_loc: 0.020  time: 0.4725  data_time: 0.0019  lr: 0.000025  max_mem: 3771M\n",
      "\u001b[32m[04/13 23:05:08 d2.utils.events]: \u001b[0m eta: 0:11:02  iter: 119  total_loss: 2.378  loss_cls: 0.829  loss_box_reg: 0.883  loss_mask: 0.654  loss_rpn_cls: 0.004  loss_rpn_loc: 0.016  time: 0.4777  data_time: 0.0017  lr: 0.000030  max_mem: 3771M\n",
      "\u001b[32m[04/13 23:05:18 d2.utils.events]: \u001b[0m eta: 0:10:58  iter: 139  total_loss: 1.779  loss_cls: 0.551  loss_box_reg: 0.547  loss_mask: 0.621  loss_rpn_cls: 0.001  loss_rpn_loc: 0.014  time: 0.4794  data_time: 0.0016  lr: 0.000035  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:05:27 d2.utils.events]: \u001b[0m eta: 0:10:47  iter: 159  total_loss: 1.692  loss_cls: 0.470  loss_box_reg: 0.569  loss_mask: 0.613  loss_rpn_cls: 0.003  loss_rpn_loc: 0.011  time: 0.4785  data_time: 0.0016  lr: 0.000040  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:05:37 d2.utils.events]: \u001b[0m eta: 0:10:40  iter: 179  total_loss: 2.227  loss_cls: 0.700  loss_box_reg: 0.900  loss_mask: 0.584  loss_rpn_cls: 0.002  loss_rpn_loc: 0.013  time: 0.4811  data_time: 0.0016  lr: 0.000045  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:05:47 d2.utils.events]: \u001b[0m eta: 0:10:40  iter: 199  total_loss: 1.717  loss_cls: 0.499  loss_box_reg: 0.680  loss_mask: 0.517  loss_rpn_cls: 0.002  loss_rpn_loc: 0.012  time: 0.4837  data_time: 0.0017  lr: 0.000050  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:05:57 d2.utils.events]: \u001b[0m eta: 0:10:30  iter: 219  total_loss: 1.858  loss_cls: 0.566  loss_box_reg: 0.720  loss_mask: 0.529  loss_rpn_cls: 0.001  loss_rpn_loc: 0.011  time: 0.4834  data_time: 0.0017  lr: 0.000055  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:06:07 d2.utils.events]: \u001b[0m eta: 0:10:17  iter: 239  total_loss: 1.899  loss_cls: 0.519  loss_box_reg: 0.887  loss_mask: 0.459  loss_rpn_cls: 0.001  loss_rpn_loc: 0.014  time: 0.4828  data_time: 0.0016  lr: 0.000060  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:06:17 d2.utils.events]: \u001b[0m eta: 0:10:10  iter: 259  total_loss: 1.660  loss_cls: 0.466  loss_box_reg: 0.776  loss_mask: 0.489  loss_rpn_cls: 0.000  loss_rpn_loc: 0.010  time: 0.4840  data_time: 0.0017  lr: 0.000065  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:06:27 d2.utils.events]: \u001b[0m eta: 0:10:04  iter: 279  total_loss: 1.659  loss_cls: 0.470  loss_box_reg: 0.826  loss_mask: 0.391  loss_rpn_cls: 0.000  loss_rpn_loc: 0.012  time: 0.4860  data_time: 0.0017  lr: 0.000070  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:06:37 d2.utils.events]: \u001b[0m eta: 0:09:54  iter: 299  total_loss: 1.446  loss_cls: 0.409  loss_box_reg: 0.764  loss_mask: 0.344  loss_rpn_cls: 0.000  loss_rpn_loc: 0.013  time: 0.4865  data_time: 0.0018  lr: 0.000075  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:06:47 d2.utils.events]: \u001b[0m eta: 0:09:45  iter: 319  total_loss: 1.557  loss_cls: 0.433  loss_box_reg: 0.797  loss_mask: 0.369  loss_rpn_cls: 0.001  loss_rpn_loc: 0.019  time: 0.4864  data_time: 0.0017  lr: 0.000080  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:06:57 d2.utils.events]: \u001b[0m eta: 0:09:35  iter: 339  total_loss: 1.329  loss_cls: 0.378  loss_box_reg: 0.614  loss_mask: 0.326  loss_rpn_cls: 0.000  loss_rpn_loc: 0.011  time: 0.4868  data_time: 0.0017  lr: 0.000085  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:07:06 d2.utils.events]: \u001b[0m eta: 0:09:25  iter: 359  total_loss: 1.423  loss_cls: 0.394  loss_box_reg: 0.724  loss_mask: 0.330  loss_rpn_cls: 0.000  loss_rpn_loc: 0.019  time: 0.4868  data_time: 0.0018  lr: 0.000090  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:07:16 d2.utils.events]: \u001b[0m eta: 0:09:14  iter: 379  total_loss: 1.185  loss_cls: 0.334  loss_box_reg: 0.568  loss_mask: 0.250  loss_rpn_cls: 0.000  loss_rpn_loc: 0.013  time: 0.4864  data_time: 0.0017  lr: 0.000095  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:07:26 d2.utils.events]: \u001b[0m eta: 0:09:05  iter: 399  total_loss: 1.422  loss_cls: 0.467  loss_box_reg: 0.656  loss_mask: 0.267  loss_rpn_cls: 0.000  loss_rpn_loc: 0.017  time: 0.4872  data_time: 0.0018  lr: 0.000100  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:07:36 d2.utils.events]: \u001b[0m eta: 0:08:56  iter: 419  total_loss: 1.390  loss_cls: 0.438  loss_box_reg: 0.608  loss_mask: 0.230  loss_rpn_cls: 0.000  loss_rpn_loc: 0.016  time: 0.4878  data_time: 0.0020  lr: 0.000105  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:07:46 d2.utils.events]: \u001b[0m eta: 0:08:46  iter: 439  total_loss: 1.137  loss_cls: 0.328  loss_box_reg: 0.381  loss_mask: 0.220  loss_rpn_cls: 0.000  loss_rpn_loc: 0.021  time: 0.4887  data_time: 0.0021  lr: 0.000110  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:07:56 d2.utils.events]: \u001b[0m eta: 0:08:35  iter: 459  total_loss: 0.990  loss_cls: 0.329  loss_box_reg: 0.400  loss_mask: 0.186  loss_rpn_cls: 0.000  loss_rpn_loc: 0.015  time: 0.4875  data_time: 0.0019  lr: 0.000115  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:08:06 d2.utils.events]: \u001b[0m eta: 0:08:26  iter: 479  total_loss: 0.799  loss_cls: 0.254  loss_box_reg: 0.346  loss_mask: 0.146  loss_rpn_cls: 0.000  loss_rpn_loc: 0.010  time: 0.4881  data_time: 0.0019  lr: 0.000120  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:08:16 d2.utils.events]: \u001b[0m eta: 0:08:16  iter: 499  total_loss: 0.816  loss_cls: 0.298  loss_box_reg: 0.296  loss_mask: 0.189  loss_rpn_cls: 0.000  loss_rpn_loc: 0.020  time: 0.4882  data_time: 0.0018  lr: 0.000125  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:08:25 d2.utils.events]: \u001b[0m eta: 0:08:06  iter: 519  total_loss: 0.934  loss_cls: 0.291  loss_box_reg: 0.345  loss_mask: 0.177  loss_rpn_cls: 0.000  loss_rpn_loc: 0.022  time: 0.4883  data_time: 0.0017  lr: 0.000130  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:08:36 d2.utils.events]: \u001b[0m eta: 0:07:57  iter: 539  total_loss: 1.076  loss_cls: 0.340  loss_box_reg: 0.453  loss_mask: 0.179  loss_rpn_cls: 0.000  loss_rpn_loc: 0.012  time: 0.4888  data_time: 0.0019  lr: 0.000135  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:08:45 d2.utils.events]: \u001b[0m eta: 0:07:46  iter: 559  total_loss: 0.936  loss_cls: 0.347  loss_box_reg: 0.394  loss_mask: 0.168  loss_rpn_cls: 0.000  loss_rpn_loc: 0.017  time: 0.4888  data_time: 0.0019  lr: 0.000140  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:08:55 d2.utils.events]: \u001b[0m eta: 0:07:36  iter: 579  total_loss: 0.799  loss_cls: 0.317  loss_box_reg: 0.318  loss_mask: 0.180  loss_rpn_cls: 0.000  loss_rpn_loc: 0.015  time: 0.4884  data_time: 0.0019  lr: 0.000145  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:09:04 d2.utils.events]: \u001b[0m eta: 0:07:25  iter: 599  total_loss: 0.579  loss_cls: 0.191  loss_box_reg: 0.221  loss_mask: 0.158  loss_rpn_cls: 0.000  loss_rpn_loc: 0.009  time: 0.4878  data_time: 0.0020  lr: 0.000150  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:09:15 d2.utils.events]: \u001b[0m eta: 0:07:15  iter: 619  total_loss: 0.579  loss_cls: 0.199  loss_box_reg: 0.234  loss_mask: 0.133  loss_rpn_cls: 0.000  loss_rpn_loc: 0.024  time: 0.4883  data_time: 0.0019  lr: 0.000155  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:09:24 d2.utils.events]: \u001b[0m eta: 0:07:05  iter: 639  total_loss: 0.667  loss_cls: 0.234  loss_box_reg: 0.283  loss_mask: 0.121  loss_rpn_cls: 0.000  loss_rpn_loc: 0.010  time: 0.4879  data_time: 0.0019  lr: 0.000160  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:09:34 d2.utils.events]: \u001b[0m eta: 0:06:56  iter: 659  total_loss: 0.660  loss_cls: 0.272  loss_box_reg: 0.331  loss_mask: 0.144  loss_rpn_cls: 0.000  loss_rpn_loc: 0.016  time: 0.4880  data_time: 0.0020  lr: 0.000165  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:09:44 d2.utils.events]: \u001b[0m eta: 0:06:46  iter: 679  total_loss: 0.611  loss_cls: 0.148  loss_box_reg: 0.273  loss_mask: 0.146  loss_rpn_cls: 0.000  loss_rpn_loc: 0.012  time: 0.4881  data_time: 0.0019  lr: 0.000170  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:09:54 d2.utils.events]: \u001b[0m eta: 0:06:36  iter: 699  total_loss: 0.733  loss_cls: 0.242  loss_box_reg: 0.301  loss_mask: 0.156  loss_rpn_cls: 0.000  loss_rpn_loc: 0.013  time: 0.4882  data_time: 0.0020  lr: 0.000175  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:10:03 d2.utils.events]: \u001b[0m eta: 0:06:26  iter: 719  total_loss: 0.534  loss_cls: 0.149  loss_box_reg: 0.233  loss_mask: 0.117  loss_rpn_cls: 0.000  loss_rpn_loc: 0.012  time: 0.4880  data_time: 0.0020  lr: 0.000180  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:10:13 d2.utils.events]: \u001b[0m eta: 0:06:16  iter: 739  total_loss: 0.607  loss_cls: 0.202  loss_box_reg: 0.190  loss_mask: 0.126  loss_rpn_cls: 0.000  loss_rpn_loc: 0.016  time: 0.4883  data_time: 0.0020  lr: 0.000185  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:10:23 d2.utils.events]: \u001b[0m eta: 0:06:07  iter: 759  total_loss: 0.459  loss_cls: 0.141  loss_box_reg: 0.215  loss_mask: 0.103  loss_rpn_cls: 0.000  loss_rpn_loc: 0.008  time: 0.4886  data_time: 0.0020  lr: 0.000190  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:10:34 d2.utils.events]: \u001b[0m eta: 0:05:57  iter: 779  total_loss: 0.627  loss_cls: 0.148  loss_box_reg: 0.265  loss_mask: 0.128  loss_rpn_cls: 0.000  loss_rpn_loc: 0.014  time: 0.4889  data_time: 0.0020  lr: 0.000195  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:10:43 d2.utils.events]: \u001b[0m eta: 0:05:47  iter: 799  total_loss: 0.552  loss_cls: 0.151  loss_box_reg: 0.210  loss_mask: 0.144  loss_rpn_cls: 0.000  loss_rpn_loc: 0.016  time: 0.4887  data_time: 0.0020  lr: 0.000200  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:10:53 d2.utils.events]: \u001b[0m eta: 0:05:37  iter: 819  total_loss: 0.523  loss_cls: 0.149  loss_box_reg: 0.188  loss_mask: 0.131  loss_rpn_cls: 0.000  loss_rpn_loc: 0.011  time: 0.4882  data_time: 0.0020  lr: 0.000205  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:11:02 d2.utils.events]: \u001b[0m eta: 0:05:27  iter: 839  total_loss: 0.517  loss_cls: 0.162  loss_box_reg: 0.190  loss_mask: 0.111  loss_rpn_cls: 0.000  loss_rpn_loc: 0.013  time: 0.4882  data_time: 0.0020  lr: 0.000210  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:11:12 d2.utils.events]: \u001b[0m eta: 0:05:17  iter: 859  total_loss: 0.440  loss_cls: 0.134  loss_box_reg: 0.201  loss_mask: 0.101  loss_rpn_cls: 0.000  loss_rpn_loc: 0.018  time: 0.4884  data_time: 0.0019  lr: 0.000215  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:11:22 d2.utils.events]: \u001b[0m eta: 0:05:07  iter: 879  total_loss: 0.440  loss_cls: 0.101  loss_box_reg: 0.203  loss_mask: 0.113  loss_rpn_cls: 0.000  loss_rpn_loc: 0.011  time: 0.4885  data_time: 0.0019  lr: 0.000220  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:11:32 d2.utils.events]: \u001b[0m eta: 0:04:57  iter: 899  total_loss: 0.373  loss_cls: 0.123  loss_box_reg: 0.132  loss_mask: 0.090  loss_rpn_cls: 0.000  loss_rpn_loc: 0.010  time: 0.4885  data_time: 0.0020  lr: 0.000225  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:11:41 d2.utils.events]: \u001b[0m eta: 0:04:47  iter: 919  total_loss: 0.368  loss_cls: 0.103  loss_box_reg: 0.180  loss_mask: 0.107  loss_rpn_cls: 0.000  loss_rpn_loc: 0.006  time: 0.4875  data_time: 0.0019  lr: 0.000230  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:11:51 d2.utils.events]: \u001b[0m eta: 0:04:37  iter: 939  total_loss: 0.465  loss_cls: 0.127  loss_box_reg: 0.188  loss_mask: 0.106  loss_rpn_cls: 0.000  loss_rpn_loc: 0.012  time: 0.4875  data_time: 0.0020  lr: 0.000235  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:12:01 d2.utils.events]: \u001b[0m eta: 0:04:27  iter: 959  total_loss: 0.420  loss_cls: 0.095  loss_box_reg: 0.193  loss_mask: 0.091  loss_rpn_cls: 0.000  loss_rpn_loc: 0.007  time: 0.4873  data_time: 0.0019  lr: 0.000240  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:12:11 d2.utils.events]: \u001b[0m eta: 0:04:17  iter: 979  total_loss: 0.378  loss_cls: 0.077  loss_box_reg: 0.215  loss_mask: 0.095  loss_rpn_cls: 0.000  loss_rpn_loc: 0.008  time: 0.4877  data_time: 0.0020  lr: 0.000245  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:12:20 d2.utils.events]: \u001b[0m eta: 0:04:07  iter: 999  total_loss: 0.324  loss_cls: 0.072  loss_box_reg: 0.155  loss_mask: 0.080  loss_rpn_cls: 0.000  loss_rpn_loc: 0.012  time: 0.4877  data_time: 0.0020  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:12:30 d2.utils.events]: \u001b[0m eta: 0:03:58  iter: 1019  total_loss: 0.488  loss_cls: 0.145  loss_box_reg: 0.210  loss_mask: 0.107  loss_rpn_cls: 0.000  loss_rpn_loc: 0.013  time: 0.4879  data_time: 0.0020  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:12:41 d2.utils.events]: \u001b[0m eta: 0:03:48  iter: 1039  total_loss: 0.313  loss_cls: 0.044  loss_box_reg: 0.132  loss_mask: 0.086  loss_rpn_cls: 0.000  loss_rpn_loc: 0.011  time: 0.4881  data_time: 0.0020  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:12:50 d2.utils.events]: \u001b[0m eta: 0:03:38  iter: 1059  total_loss: 0.371  loss_cls: 0.076  loss_box_reg: 0.203  loss_mask: 0.075  loss_rpn_cls: 0.000  loss_rpn_loc: 0.012  time: 0.4882  data_time: 0.0020  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:13:00 d2.utils.events]: \u001b[0m eta: 0:03:28  iter: 1079  total_loss: 0.393  loss_cls: 0.079  loss_box_reg: 0.196  loss_mask: 0.108  loss_rpn_cls: 0.000  loss_rpn_loc: 0.012  time: 0.4879  data_time: 0.0018  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:13:10 d2.utils.events]: \u001b[0m eta: 0:03:19  iter: 1099  total_loss: 0.379  loss_cls: 0.085  loss_box_reg: 0.173  loss_mask: 0.082  loss_rpn_cls: 0.000  loss_rpn_loc: 0.008  time: 0.4880  data_time: 0.0017  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:13:20 d2.utils.events]: \u001b[0m eta: 0:03:08  iter: 1119  total_loss: 0.397  loss_cls: 0.078  loss_box_reg: 0.179  loss_mask: 0.085  loss_rpn_cls: 0.000  loss_rpn_loc: 0.012  time: 0.4879  data_time: 0.0016  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:13:29 d2.utils.events]: \u001b[0m eta: 0:02:58  iter: 1139  total_loss: 0.341  loss_cls: 0.060  loss_box_reg: 0.181  loss_mask: 0.095  loss_rpn_cls: 0.000  loss_rpn_loc: 0.010  time: 0.4879  data_time: 0.0017  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:13:39 d2.utils.events]: \u001b[0m eta: 0:02:49  iter: 1159  total_loss: 0.338  loss_cls: 0.075  loss_box_reg: 0.155  loss_mask: 0.073  loss_rpn_cls: 0.000  loss_rpn_loc: 0.014  time: 0.4879  data_time: 0.0019  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:13:49 d2.utils.events]: \u001b[0m eta: 0:02:38  iter: 1179  total_loss: 0.296  loss_cls: 0.046  loss_box_reg: 0.147  loss_mask: 0.085  loss_rpn_cls: 0.000  loss_rpn_loc: 0.007  time: 0.4876  data_time: 0.0019  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:13:58 d2.utils.events]: \u001b[0m eta: 0:02:28  iter: 1199  total_loss: 0.318  loss_cls: 0.074  loss_box_reg: 0.164  loss_mask: 0.090  loss_rpn_cls: 0.000  loss_rpn_loc: 0.010  time: 0.4876  data_time: 0.0018  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:14:08 d2.utils.events]: \u001b[0m eta: 0:02:18  iter: 1219  total_loss: 0.329  loss_cls: 0.064  loss_box_reg: 0.138  loss_mask: 0.089  loss_rpn_cls: 0.000  loss_rpn_loc: 0.010  time: 0.4875  data_time: 0.0018  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:14:18 d2.utils.events]: \u001b[0m eta: 0:02:09  iter: 1239  total_loss: 0.367  loss_cls: 0.080  loss_box_reg: 0.168  loss_mask: 0.083  loss_rpn_cls: 0.000  loss_rpn_loc: 0.008  time: 0.4879  data_time: 0.0018  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:14:29 d2.utils.events]: \u001b[0m eta: 0:01:59  iter: 1259  total_loss: 0.284  loss_cls: 0.048  loss_box_reg: 0.134  loss_mask: 0.079  loss_rpn_cls: 0.000  loss_rpn_loc: 0.008  time: 0.4883  data_time: 0.0018  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:14:38 d2.utils.events]: \u001b[0m eta: 0:01:49  iter: 1279  total_loss: 0.303  loss_cls: 0.058  loss_box_reg: 0.123  loss_mask: 0.086  loss_rpn_cls: 0.000  loss_rpn_loc: 0.013  time: 0.4882  data_time: 0.0019  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:14:49 d2.utils.events]: \u001b[0m eta: 0:01:39  iter: 1299  total_loss: 0.284  loss_cls: 0.050  loss_box_reg: 0.125  loss_mask: 0.074  loss_rpn_cls: 0.000  loss_rpn_loc: 0.015  time: 0.4885  data_time: 0.0021  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:14:59 d2.utils.events]: \u001b[0m eta: 0:01:29  iter: 1319  total_loss: 0.319  loss_cls: 0.055  loss_box_reg: 0.159  loss_mask: 0.076  loss_rpn_cls: 0.000  loss_rpn_loc: 0.006  time: 0.4886  data_time: 0.0020  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:15:08 d2.utils.events]: \u001b[0m eta: 0:01:19  iter: 1339  total_loss: 0.270  loss_cls: 0.054  loss_box_reg: 0.123  loss_mask: 0.081  loss_rpn_cls: 0.000  loss_rpn_loc: 0.009  time: 0.4884  data_time: 0.0020  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:15:18 d2.utils.events]: \u001b[0m eta: 0:01:09  iter: 1359  total_loss: 0.314  loss_cls: 0.058  loss_box_reg: 0.114  loss_mask: 0.103  loss_rpn_cls: 0.000  loss_rpn_loc: 0.009  time: 0.4886  data_time: 0.0021  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:15:28 d2.utils.events]: \u001b[0m eta: 0:00:59  iter: 1379  total_loss: 0.286  loss_cls: 0.047  loss_box_reg: 0.132  loss_mask: 0.076  loss_rpn_cls: 0.000  loss_rpn_loc: 0.008  time: 0.4887  data_time: 0.0019  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:15:38 d2.utils.events]: \u001b[0m eta: 0:00:50  iter: 1399  total_loss: 0.316  loss_cls: 0.067  loss_box_reg: 0.145  loss_mask: 0.091  loss_rpn_cls: 0.000  loss_rpn_loc: 0.010  time: 0.4887  data_time: 0.0020  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:15:48 d2.utils.events]: \u001b[0m eta: 0:00:40  iter: 1419  total_loss: 0.277  loss_cls: 0.037  loss_box_reg: 0.112  loss_mask: 0.077  loss_rpn_cls: 0.000  loss_rpn_loc: 0.010  time: 0.4887  data_time: 0.0019  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:15:57 d2.utils.events]: \u001b[0m eta: 0:00:30  iter: 1439  total_loss: 0.315  loss_cls: 0.047  loss_box_reg: 0.148  loss_mask: 0.076  loss_rpn_cls: 0.000  loss_rpn_loc: 0.008  time: 0.4885  data_time: 0.0020  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:16:08 d2.utils.events]: \u001b[0m eta: 0:00:20  iter: 1459  total_loss: 0.244  loss_cls: 0.043  loss_box_reg: 0.113  loss_mask: 0.071  loss_rpn_cls: 0.000  loss_rpn_loc: 0.013  time: 0.4889  data_time: 0.0019  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:16:17 d2.utils.events]: \u001b[0m eta: 0:00:10  iter: 1479  total_loss: 0.267  loss_cls: 0.042  loss_box_reg: 0.121  loss_mask: 0.087  loss_rpn_cls: 0.000  loss_rpn_loc: 0.010  time: 0.4888  data_time: 0.0018  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:16:36 d2.utils.events]: \u001b[0m eta: 0:00:00  iter: 1499  total_loss: 0.274  loss_cls: 0.029  loss_box_reg: 0.122  loss_mask: 0.066  loss_rpn_cls: 0.000  loss_rpn_loc: 0.014  time: 0.4888  data_time: 0.0016  lr: 0.000250  max_mem: 3773M\n",
      "\u001b[32m[04/13 23:16:36 d2.engine.hooks]: \u001b[0mOverall training speed: 1497 iterations in 0:12:12 (0.4892 s / it)\n",
      "\u001b[32m[04/13 23:16:36 d2.engine.hooks]: \u001b[0mTotal training time: 0:12:25 (0:00:13 on hooks)\n"
     ]
    }
   ],
   "source": [
    "# DefaultTrainer はサンプルなので、ガチにやる人は自分で作るらしい・・・\n",
    "from detectron2.engine import DefaultTrainer\n",
    "\n",
    "# 出力先のディレクトリを作る\n",
    "os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)\n",
    "\n",
    "trainer = DefaultTrainer(cfg) \n",
    "trainer.resume_or_load(resume=False) # True で途中から学習できるらしい\n",
    "trainer.train()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}