每日作業/Day_048_HW_(in Colab)_API_Training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 將自己的 google drive 空間 mount 到 colab 環境\n",
    "\n",
    "- 若在 colab 環境，請執行下面程式，點開連結及選取自己的 gmail，將許可碼拷貝貼在格子裡"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/gdrive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 安裝相關套件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install contextlib2\n",
    "!pip install \"git+https://github.com/philferriere/cocoapi.git#egg=pycocotools&subdirectory=PythonAPI\"\n",
    "!pip install toolz --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 設定環境\n",
    "\n",
    "- 由於 colab 每個 notebook 環境皆獨立，必須重新安裝套件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/gdrive/My\\ Drive/models/research\n",
    "!mkdir train eval\n",
    "#%set_env PYTHONPATH=`pwd`:`pwd`/slim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python setup.py install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd slim\n",
    "!python setup.py install\n",
    "%cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 下載資料 (pets) ; 將資料轉換成 tfrecord\n",
    "\n",
    "- 除了 pets, 也可以下載 coco / open images / pascal VOC 資料集並用已經提供的 tfrecord 轉換工具作轉換\n",
    "    - 轉換程式在 `object_detection/dataset_tools/` 裡\n",
    "- 也可以用定義自己的資料集及資料格式，再客製轉換程式將資料轉換成 tfrecord"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 下載資料\n",
    "!wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz\n",
    "!wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/annotations.tar.gz\n",
    "# 解壓縮\n",
    "!tar -xvf images.tar.gz\n",
    "!tar -xvf annotations.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python object_detection/dataset_tools/create_pet_tf_record.py \\\n",
    " --label_map_path=object_detection/data/pet_label_map.pbtxt \\\n",
    " --data_dir=/content/gdrive/My\\ Drive/models/research \\\n",
    " --output_dir=/content/gdrive/My\\ Drive/models/research\n",
    "!ls *.record*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 到 [model zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) 下載 ssd_mobilenet_v1_coco 預訓練模型權重\n",
    "\n",
    "- 此範例以 ssd_mobilenet_v1 作參考，可以試用別的 (如 ssd_mobilenet_v2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 下載模型權重\n",
    "!wget http://storage.googleapis.com/download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz\n",
    "# 解壓縮\n",
    "!tar -xvf ssd_mobilenet_v1_coco_2018_01_28.tar.gz\n",
    "!cp ssd_mobilenet_v1_coco_2018_01_28/model.ckpt.* ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 調整訓練設定及超參數\n",
    "\n",
    "- 設定檔皆在 object_detection/samples/configs/ 裡，選擇對應的模型設定檔\n",
    "- 此範例為了簡化，故用指令對設定檔作調整，實際上可以直接打開檔案作調整\n",
    "- 一些必備的調整如\n",
    "    - `PATH_TO_BE_CONFIGURED` 為調整檔案路徑位置\n",
    "    - `mscoco_label_map.pbtxt` & `mscoco_train.record` & `mscoco_val.record` 為調整資料集\n",
    "- 其他超參數也可以作調整，是需求而定"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp object_detection/samples/configs/ssd_mobilenet_v1_coco.config .\n",
    "# 修改檔案位置\n",
    "!sed -i \"s|PATH_TO_BE_CONFIGURED|\\.|g\" ssd_mobilenet_v1_coco.config\n",
    "# 修改資料集\n",
    "!sed -i \"s|mscoco_label_map.pbtxt|object_detection/data/pet_label_map.pbtxt|g\" ssd_mobilenet_v1_coco.config\n",
    "!sed -i \"s|mscoco_train.record|pet_faces_train.record|g\" ssd_mobilenet_v1_coco.config\n",
    "!sed -i \"s|mscoco_val.record|pet_faces_val.record|g\" ssd_mobilenet_v1_coco.config\n",
    "# 由於 pets 最終只會產生 10 份 tfrecord, 因此這裏稍作修改\n",
    "!sed -i \"s|00100|00010|g\" ssd_mobilenet_v1_coco.config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 執行訓練"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python object_detection/model_main.py \\\n",
    " --logtostderr \\\n",
    " --pipeline_config_path=ssd_mobilenet_v1_coco.config \\\n",
    " --train_dir=train"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}