tf2/Arch_YOLOv3_3_loss_function.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這邊我會假設你已經看過[Darknet-53_structure](https://github.com/shaoeChen/deeplearning/blob/master/tf2/Arch_YOLOv3_1_Darknet-53_structure.ipynb)與[YOLOv3_structure](https://github.com/shaoeChen/deeplearning/blob/master/tf2/Arch_YOLOv3_2_YOLOv3_structure.ipynb)的說明，而且你也已經完全明白模型的架構是怎麼一回事。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面先給出眾多的參考資料，瞭解一個架構不容易，參考的自然也多了，正確的寫出引用、參照是學習過程中的必要："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [YOLO: Real-Time Object Detection](https://pjreddie.com/darknet/yolo/)\n",
    "* [pjreddie/darknet](https://github.com/pjreddie/darknet)\n",
    "* [YOLOv3_論文翻譯連結](https://hackmd.io/@shaoeChen/SyjI6W2zB/https%3A%2F%2Fhackmd.io%2F%40shaoeChen%2FryHg904h9)\n",
    "* [YOLOv3深度解析](https://blog.csdn.net/leviopku/article/details/82660381)\n",
    "* [qqwweee/keras-yolo3](https://github.com/qqwweee/keras-yolo3)\n",
    "* [YunYang1994/tensorflow-yolov3](https://github.com/YunYang1994/tensorflow-yolov3)\n",
    "* [joymyhome_Yolov3 config file中pad的理解](https://blog.csdn.net/joymyhome/article/details/106349084)\n",
    "* [華為雲開發者社區](https://segmentfault.com/a/1190000039009074)\n",
    "* [pylessons_YOLOv3-TF2-mnist](https://pylessons.com/YOLOv3-TF2-mnist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "相關前置資料的處理可以參考另作[Arch_YOLO_dataset_preprocess_1.ipynb](https://github.com/shaoeChen/deeplearning/blob/master/tf2/Arch_YOLO_dataset_preprocess_1.ipynb)、[Arch_YOLO_dataset_preprocess_2_data_for_yolov3](https://github.com/shaoeChen/deeplearning/blob/master/tf2/Arch_YOLO_dataset_preprocess_2_data_for_yolov3.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我的docker上執行的版本為tensorflow 2.1，雖然現在流行人生苦短我用PyTorch，不過我還是先繼續tf + keras。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.1.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "指定使用的gpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "gpus = tf.config.experimental.list_physical_devices(device_type='GPU')\n",
    "tf.config.experimental.set_visible_devices(devices=gpus[0], device_type='GPU')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'),\n",
       " PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU')]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們已經可以成功的從模型中取得粗、中、細的預測資訊，論文中我們知道，每個NxN的grid cell裡面都會有三個預測框的資訊。但不同尺度的預測框大小是不一樣的，這部份是必需要注意的，這從cfg檔中也可以看的出來。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9個框的大小如下：10,13,  16,30,  33,23,  30,61,  62,45,  59,119,  116,90,  156,198,  373,326  \n",
    "\n",
    "粗(13x13)：116,90,  156,198,  373,326  \n",
    "中(26x26)：30,61,  62,45,  59,119  \n",
    "細(52x52)：10,13,  16,30,  33,23  \n",
    "\n",
    "每一組數值代表的就是高跟寬，下面我們也整理一下這九個anchor box"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 10.,  13.],\n",
       "        [ 16.,  30.],\n",
       "        [ 33.,  23.]],\n",
       "\n",
       "       [[ 30.,  61.],\n",
       "        [ 62.,  45.],\n",
       "        [ 59., 119.]],\n",
       "\n",
       "       [[116.,  90.],\n",
       "        [156., 198.],\n",
       "        [373., 326.]]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np \n",
    "# 1.25,1.625, 2.0,3.75, 4.125,2.875, 1.875,3.8125, 3.875,2.8125, 3.6875,7.4375, 3.625,2.8125, 4.875,6.1875, 11.65625,10.1875\n",
    "# [8, 16, 32]\n",
    "anchors='10,13,  16,30,  33,23,  30,61,  62,45,  59,119,  116,90,  156,198,  373,326'\n",
    "anchors = [float(x) for x in anchors.split(',')]\n",
    "anchors = np.array(anchors).reshape((3, 3, 2))\n",
    "anchors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "ANCHORS = anchors.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "STRIDES = [8, 16, 32]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面我註解兩行程式碼，是來自於參考來源的程式碼，猜測這是為了適應不同stride所做的正規化處理：\n",
    "* 『1.25,1.625, 2.0,3.75, 4.125,2.875』這三組乘上8就會是ANCHORS[0]，這也是52x52的output使用的anchor box\n",
    "* 『30,61,  62,45,  59,119』這三組乘上16就會是ANCHORS[1]，這也是26x26的output使用的anchor box\n",
    "* 『116,90,  156,198,  373,326』這三組乘上32就會是ANCHORS[2]，這也是13x13的output使用的anchor box"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "更多關於YOLO的相關資訊建議可以看一下[吳恩達老師的課程](https://hackmd.io/@shaoeChen/HJUZTKMZz/https%3A%2F%2Fhackmd.io%2Fs%2FSJXmp66KG)，連結是在下的讀書筆記，這可以幫助我們快速有一個概略的觀念。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，我們已經有模型的output，三組output的每個grid cell各有三個框的資訊，每個框裡面則有(1(有沒有框到) + 4(框的資訊) + 80(類別資訊))的資訊。如果我們把模型的output視為一種encoder，那就會需要一個decoder來處理模型的輸出。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "框的資訊我們可以參考論文的一張圖："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://hackmd.io/_uploads/BJlQ-JO3q.png\" width=\"25%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "框的資訊裡面包含它的中心點x、y以及寬(w)、高(h)，也就是$t_x, t_y, t_w, t_h$，中心點的部份是利用sigmoid的特性(輸出為0~1之間的值)計算而得，而$C_x, C_y$的部份則是grid cell的索引，上圖為例，$C_x, C_y$為1, 1，這樣我們就知道是第1個row、第一個col的這個格式的中心點的output結果。而$p_w, p_h$則是anchor box prior，也就是我們上面所定義的ANCHORS。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這邊當然要特別注意的就是，你decoder弄成什麼樣子，你的input就會是什麼樣子，因為你要計算loss，兩邊的單位就是要一致就是了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://hackmd.io/_uploads/ByAPjJkli.png\" width=\"50%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上圖左是我們預想中的把150x150的照片切割成一個5x5照片的模樣，但實際上我們的output是一個grid cell的資訊，算是一個點資訊吧。每個grid cell代表的尺度為30x30，也就是150/5=30，這個值(30)也就是stride。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://hackmd.io/_uploads/ByQ8A1yxi.png\" width=\"25%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "承接上例，照片是150x150，有一個物件的中心點剛好在中間，也就是(75, 75)，它的output經過上面論文公式處理之後是：\n",
    "* $b_x=2.5$，其中$2$是來自於它在索引$2$的row，然後$0.5$是因為它在格子的中間，再乘上stride，也就是30，就會得到75，這也是它的實際中心座標\n",
    "* $b_y=2.5$，其中$2$是來自於它在索引$2$的col，然後$0.5$是因為它在格子的中間，再乘上stride，也就是30，就會得到75，這也是它的實際中心座標\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "程式碼的部份保證參考[YunYang1994](https://github.com/YunYang1994/TensorFlow2.0-Examples/blob/master/4-Object_Detection/YOLOV3/core/yolov3.py)，只是加入自己的理解說明。這邊主要處理的就是將模型的輸出轉為我們需要的樣子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder(yolo_output, class_nums: int = 80, anchor_size: int = 0):\n",
    "    \"\"\"整理YOLOv3的output\n",
    "            \n",
    "    yolo output有三種：Mx13x13x255、Mx26x26x255、Mx52x52x255，其中M代表你的batch size，\n",
    "        每個255裡面都有3個anchor boxes、每個anchor box都有1個置信度、4個框資訊以及80個類別預測結果\n",
    "        當然並不一定是80個類別，這取決於你的實際應用\n",
    "    \n",
    "    class_nums: 類別數量\n",
    "    \n",
    "    anchor_size: NxN的N\n",
    "        2: 13x13 <-> stride: 32\n",
    "        1: 26x26 <-> stride: 16\n",
    "        0: 52x52 <-> stride: 8\n",
    "    \n",
    "    裡面關於anchor box數量、類別數量都是可以依你自己專案實際需求來調整的，\n",
    "    不過這邊就不設置過多的參數來做這一塊，避免複雜化我們的說明過程。\n",
    "    \n",
    "    tf.tile，函數如其名，把你的input看成一個瓦片，然後利用tile複製看你要幾片，假設input=3x3\n",
    "        (1, 2)，就是往右多貼一片，輸出為3x6\n",
    "        (2, 1)，就是往下多貼一片，輸出為6x3\n",
    "    \"\"\"\n",
    "    # 因為我們在定義模型的時候並不確定它的batch_size為何，因此會是None\n",
    "    # 如果這邊我們利用tensor.shape來取得靜態維度的話，那就會是None，後續就會報錯\n",
    "    # 因此採用tf.shape取得動態維度，就可以正常執行    \n",
    "    output_shape = tf.shape(yolo_output)    \n",
    "    batch_size = output_shape[0]\n",
    "    grid_cell_nums = output_shape[1]\n",
    "    \n",
    "    # 這邊主要要將這個輸出轉成M x N x N x 3 x (5 + 80)，其中N指的是你的anchor box的數量，是13x13還是26x26之類的\n",
    "    output_reshape = tf.reshape(tensor=yolo_output, shape=(batch_size, grid_cell_nums, grid_cell_nums, 3, 5 + class_nums))\n",
    "    print(f'output_reshape: {output_reshape.shape}')\n",
    "    \n",
    "    # 框的中心點\n",
    "    output_reshape_txty = output_reshape[:, :, :, :, 0: 2]\n",
    "    print(f'output_reshape_txty: {output_reshape_txty.shape}')\n",
    "    # 框的寬高\n",
    "    output_reshape_twth = output_reshape[:, :, :, :, 2: 4]\n",
    "    print(f'output_reshape_twth: {output_reshape_twth.shape}')\n",
    "    # 有沒有東西的置信度\n",
    "    output_reshape_conf = output_reshape[:, :, :, :, 4: 5]\n",
    "    print(f'output_reshape_conf: {output_reshape_conf.shape}')\n",
    "    # 類別\n",
    "    output_reshape_class_prob = output_reshape[:, :, :, :, 5: ]\n",
    "    print(f'output_reshape_class_prob: {output_reshape_class_prob.shape}')\n",
    "    \n",
    "    # 關於tile之後的結果我放在最下面附錄，方便直觀瞭解這邊的操作\n",
    "    # tf.range之後的格式是秩，也就是(13, )，所以加入一個軸讓它變成(13, 1)，再利用tile來做複製    \n",
    "    y = tf.tile(tf.range(grid_cell_nums, dtype=tf.int32)[:, tf.newaxis], [1, grid_cell_nums])\n",
    "    print(f'y: {y.shape}')\n",
    "    # tf.range之後的格式是秩，也就是(13, )，所以加入一個軸讓它變成(1, 13)，再利用tile來做複製    \n",
    "    x = tf.tile(tf.range(grid_cell_nums, dtype=tf.int32)[tf.newaxis, :], [grid_cell_nums, 1])\n",
    "    print(f'x: {x.shape}')\n",
    "    # 再推一個軸，然後把兩個tensor沿著新增的軸疊起來\n",
    "    # 這時候就變成[[0, 0], [1, 0], [2, 0], ...]\n",
    "    # 如果是13x13的tensor，那這時候的維度就會是13x13x2\n",
    "    xy_grid = tf.concat([x[:, :, tf.newaxis], y[:, :, tf.newaxis]], axis=-1)\n",
    "    print(f'xy_grid: {xy_grid.shape}')\n",
    "    # 這邊再推新軸，這時候就會是1x13x13x1x2\n",
    "    # 然後再經過tile處理，這時候的維度就會是batch_size x 13 x 13 x 3 x 2\n",
    "    # 這時候的維度就跟上面output_reshape_txty、output_reshape_twth的維度一樣\n",
    "    xy_grid = tf.tile(xy_grid[tf.newaxis, :, :, tf.newaxis, :], [batch_size, 1, 1, 3, 1])\n",
    "    print(f'xy_grid: {xy_grid.shape}')\n",
    "    # 將型別從int32轉float32\n",
    "    xy_grid = tf.cast(xy_grid, tf.float32) \n",
    "    print(f'xy_grid: {xy_grid.shape}')    \n",
    "    # 這是根據上圖的公式計算中心點\n",
    "    # 因為我們上面已經整理一下大表，維度都一樣，所以只要直接加上xy_grid就可以得到中心點\n",
    "    # 記得上面的說明，這個中心點是grid cell的資訊，所以還要再乘上stride才會是在影像上的實際位置\n",
    "    pred_xy = (tf.sigmoid(output_reshape_txty) + xy_grid) * STRIDES[anchor_size]\n",
    "    print(f'pred_xy: {pred_xy.shape}')\n",
    "    # 這是根據上圖的公式計算寬高\n",
    "    pred_wh = (tf.exp(output_reshape_twth) * ANCHORS[anchor_size])    \n",
    "    print(f'anchor box: {ANCHORS[anchor_size]}')\n",
    "    print(f'pred_wh: {pred_wh.shape}')\n",
    "    # 把得到的結果堆疊起來，這時候的維度為batch size x 13 x 13 x 3 x 4 -> x、y、寬、高\n",
    "    pred_xywh = tf.concat([pred_xy, pred_wh], axis=-1)\n",
    "    print(f'pred_xywh: {pred_xywh.shape}')\n",
    "    # 這個框有沒有東西的置信度\n",
    "    pred_conf = tf.sigmoid(output_reshape_conf)\n",
    "    print(f'pred_conf: {pred_conf.shape}')\n",
    "    # 類別機率計算\n",
    "    pred_prob = tf.sigmoid(output_reshape_class_prob)\n",
    "    print(f'pred_prob: {pred_prob.shape}')\n",
    "    \n",
    "    return tf.concat([pred_xywh, pred_conf, pred_prob], axis=-1)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "弄個隨機張量，順便觀察過程中的維度變化，比較清楚到底我們自己在做些什麼事。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([5, 13, 13, 255])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_yolo_output = tf.random.uniform(shape=(5, 13, 13, 255))\n",
    "test_yolo_output.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output_reshape: (5, 13, 13, 3, 85)\n",
      "output_reshape_txty: (5, 13, 13, 3, 2)\n",
      "output_reshape_twth: (5, 13, 13, 3, 2)\n",
      "output_reshape_conf: (5, 13, 13, 3, 1)\n",
      "output_reshape_class_prob: (5, 13, 13, 3, 80)\n",
      "y: (13, 13)\n",
      "x: (13, 13)\n",
      "xy_grid: (13, 13, 2)\n",
      "xy_grid: (5, 13, 13, 3, 2)\n",
      "xy_grid: (5, 13, 13, 3, 2)\n",
      "pred_xy: (5, 13, 13, 3, 2)\n",
      "anchor box: [[116.  90.]\n",
      " [156. 198.]\n",
      " [373. 326.]]\n",
      "pred_wh: (5, 13, 13, 3, 2)\n",
      "pred_xywh: (5, 13, 13, 3, 4)\n",
      "pred_conf: (5, 13, 13, 3, 1)\n",
      "pred_prob: (5, 13, 13, 3, 80)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TensorShape([5, 13, 13, 3, 85])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder(test_yolo_output, class_nums=80, anchor_size=2).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "現在我們就某種程度的確定好output之後decoder的資料，input的部份就要配合這一部份來處理，這樣才有辦法計算相關的loss。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://i.imgur.com/yXo3xV3.png\" width=\"50%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下來，我們需要去計算IoU，也就是Intersection over union(IoU)，上面圖片取自吳恩達老師深度學習課程，紅框的部份是物體所在的地方，紫框則是模型推論出來的地方，我們希望它們兩個的交集要大於某個閥值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IoU的部份目前來看有很多種的計算，這邊也給出一個[知乎的連結](https://zhuanlan.zhihu.com/p/94799295)，裡面有說明著IoU、GIoU、DIoU、CIoU。然後這個[CSDN的連結_景唯acr](https://blog.csdn.net/weixin_41735859/article/details/89288493)中的說明也不錯。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def xywh_to_lxlyrxry(boxes1, boxes2):\n",
    "    \"\"\"將bounding box中的xywh轉為左上、右下座標\n",
    "    \n",
    "    boxes的維度為batch_size x anchor_size x anchor_size x anchor_nums x xywh \n",
    "        其中anchor_size指的為yolo的不同尺度的output shape，如13x13、26x26、52x52\n",
    "    \"\"\"\n",
    "    # 首先計算出兩個框的左上、右下點座標    \n",
    "    boxes1 = tf.concat([boxes1[..., :2] - boxes1[..., 2:] * 0.5,\n",
    "                        boxes1[..., :2] + boxes1[..., 2:] * 0.5], axis=-1)\n",
    "    boxes2 = tf.concat([boxes2[..., :2] - boxes2[..., 2:] * 0.5,\n",
    "                        boxes2[..., :2] + boxes2[..., 2:] * 0.5], axis=-1)\n",
    "    # 這邊的boxes已經是左上xy、右下xy的座標\n",
    "    # 座標取小、取大，目前還不確定為什麼還要做一次這樣的動作\n",
    "    # 理解上，左上座標一定是比右下座標來的小，反之亦然\n",
    "    boxes1 = tf.concat([tf.minimum(boxes1[..., :2], boxes1[..., 2:]),\n",
    "                        tf.maximum(boxes1[..., :2], boxes1[..., 2:])], axis=-1)\n",
    "    boxes2 = tf.concat([tf.minimum(boxes2[..., :2], boxes2[..., 2:]),\n",
    "                        tf.maximum(boxes2[..., :2], boxes2[..., 2:])], axis=-1)\n",
    "    \n",
    "    return boxes1, boxes2\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bounding_box_iou(boxes1, boxes2):\n",
    "    \"\"\"計算兩個box的IoU\n",
    "    \n",
    "    boxes1, boxes2: batch_size x anchor_size x anchor_size x anchor_nums x xywh    \n",
    "        xy: 中心座標\n",
    "        wh: 寬高        \n",
    "    \"\"\"\n",
    "    boxes1, boxes2 = xywh_to_lxlyrxry(boxes1, boxes2)\n",
    "    # 各自計算面積，四個元素所代表的為：lx, ly, rx, ry\n",
    "    boxes1_area = (boxes1[..., 2] - boxes1[..., 0]) * (boxes1[..., 3] - boxes1[..., 1])\n",
    "    boxes2_area = (boxes2[..., 2] - boxes2[..., 0]) * (boxes2[..., 3] - boxes2[..., 1])\n",
    "    \n",
    "    # 準備做交集面積的計算\n",
    "    # 交集面積點座標，左上取大、右下取小    \n",
    "    left_up = tf.maximum(boxes1[..., :2], boxes2[..., :2])\n",
    "    right_down = tf.minimum(boxes1[..., 2:], boxes2[..., 2:])\n",
    "    \n",
    "    # 計算交集面積的長寬\n",
    "    inter_section = tf.maximum(right_down - left_up, 0.0)\n",
    "    # 計算交集面積\n",
    "    inter_area = inter_section[..., 0] * inter_section[..., 1]\n",
    "    # 計算聯集面積\n",
    "    union_area = boxes1_area + boxes2_area - inter_area\n",
    "    \n",
    "    iou = 1.0 * inter_area / union_area\n",
    "    # 回傳IoU\n",
    "    return iou, union_area"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "現在我們來弄個隨機資料測試看看函數執行是否正常，相關圖示可參考更下面一點的說明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 1, 1, 1, 4), dtype=float32, numpy=array([[[[[ 2.5,  5. ,  5. , 10. ]]]]], dtype=float32)>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "box1 = tf.constant([2.5, 5, 5, 10])\n",
    "box1 = tf.reshape(tensor=box1, shape=(1, 1, 1, 1, 4))\n",
    "box1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 1, 1, 1, 4), dtype=float32, numpy=array([[[[[ 5. ,  7.5, 10. ,  5. ]]]]], dtype=float32)>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "box2 = tf.constant([5, 7.5, 10, 5])\n",
    "box2 = tf.reshape(tensor=box2, shape=(1, 1, 1, 1, 4))\n",
    "box2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(1, 1, 1, 1), dtype=float32, numpy=array([[[[0.33333334]]]], dtype=float32)>,\n",
       " <tf.Tensor: shape=(1, 1, 1, 1), dtype=float32, numpy=array([[[[75.]]]], dtype=float32)>)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iou, union_area = bounding_box_iou(box1, box2)\n",
    "iou, union_area"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "iou回傳的是batch size x anchor size x anchor size x anchor nums，因為上面的隨機範例中取的是一筆資料，所以這邊回傳的就會是這一筆資料中的13x13個grid cell中的每一個anchor boxes的iou"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IoU很明顯的就是用交集、聯集來做為一個計算比例，總面積有多少，有多少重疊，這是一個簡單的說明。而GIoU的話主要是改善當兩個框沒有重疊的時候有問題。因此在GIoU中引入一個C框，這個C框是能夠包住A、B兩框的最小框：  \n",
    "$$\n",
    "GIoU = IoU - \\dfrac{\\vert A_c - U \\vert}{\\vert A_c \\vert}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$IoU$我們知道，$A_c$指的就是C這個包住A、B兩個框的最小面積，然後$U$是A、B兩個框的聯集，用execl每個格子給個1，拉個10x10的格子來試算一下就會有感覺。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://hackmd.io/_uploads/ryGDTCORq.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IoU減去的部份或許可以理解成C框的空白佔比，愈小很明顯的就是框的愈準，暫比愈小，假設A、B兩框直接密合，那C框大小也會等於A、B，計算出來的部份自然為0，因為$A_c-U=0$，最壞的情況就是兩個框完全沒有交集，產生的自然就會是負數(界於-1\\~1之間，對比IoU是0\\~1之間多了負值區間)。這時候計算它的loss，$1 - GIoU$就會變大，目標是讓loss愈小愈好，收斂效果理論上可以更好。這感覺就有點像是ReLU與Leaky ReLU之類的差異。後續有空再來讀相關論文。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bounding_box_giou(boxes1, boxes2):\n",
    "    \"\"\"計算兩個box的GIoU\n",
    "    \n",
    "    boxes1, boxes2: batch_size x anchor_size x anchor_size x anchor_nums x xywh    \n",
    "        xy: 中心座標\n",
    "        wh: 寬高        \n",
    "    \"\"\"\n",
    "    # 首先取得兩個框的iou與聯集面積\n",
    "    iou, union_area = bounding_box_iou(boxes1, boxes2)\n",
    "    \n",
    "    boxes1, boxes2 = xywh_to_lxlyrxry(boxes1, boxes2)\n",
    "    \n",
    "    # C框的點取得，左上取小，右下取大\n",
    "    enclose_left_up = tf.minimum(boxes1[..., :2], boxes2[..., :2])\n",
    "    print(f'enclose_left_up: {enclose_left_up}')\n",
    "    enclose_right_down = tf.maximum(boxes1[..., 2:], boxes2[..., 2:])\n",
    "    print(f'enclose_right_down: {enclose_right_down}')\n",
    "    # 計算C框的長、寬\n",
    "    enclose = tf.maximum(enclose_right_down - enclose_left_up, 0.0)\n",
    "    print(f'enclose: {enclose}')\n",
    "    # 計算C框的面積\n",
    "    enclose_area = enclose[..., 0] * enclose[..., 1]\n",
    "    print(f'enclose_area: {enclose_area}')\n",
    "    \n",
    "    giou = iou - (1.0 * (enclose_area - union_area) / enclose_area)\n",
    "    \n",
    "    return giou"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "enclose_left_up: [[[[[0. 0.]]]]]\n",
      "enclose_right_down: [[[[[10. 10.]]]]]\n",
      "enclose: [[[[[10. 10.]]]]]\n",
      "enclose_area: [[[[100.]]]]\n"
     ]
    }
   ],
   "source": [
    "giou = bounding_box_giou(box1, box2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 1, 1, 1), dtype=float32, numpy=array([[[[0.08333334]]]], dtype=float32)>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "giou"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這結果跟我們上面設計的手算結果是一致的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最後需要的就是計算loss function，框的好不好(IoU)會有loss、置信度會有loss、類別也會有loss，這幾個loss加總起來就是總的loss，想當然我們希望這個loss愈小愈好。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "針對loss function的部份看了一下論文，大概就只有[YOLOv1](https://hackmd.io/@shaoeChen/SyjI6W2zB/https%3A%2F%2Fhackmd.io%2F%40shaoeChen%2FHy6kUMWNI)有特別說明，下圖最上面是框的好不好的loss，中間是置信度的loss、最下面則是類別的loss：  \n",
    "<img src=\"https://hackmd.io/_uploads/Skico6hA9.png\" width=\"50%\"/>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "對於框的好不好的loss部份，程式碼中可以看的出來有一個bbox_loss_scale：  \n",
    "![](https://hackmd.io/_uploads/HkoYKD90c.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很明顯的，框愈大的，這個scale就愈小，愈小的就愈大，這有一種正規化的感覺，關於這部份我個人有幾點想法：\n",
    "1. 因為拉進實際框的考量，所以我們嚐試減少框的大小對loss的影響\n",
    "2. 小框的scale拉大，我們可以想像一下，一張照片裡面可能有多個物件被框到，有的框大，有的框小，框大的好框，框小的不好處理，所以我們希望加大模型對小框的處理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面給出一個完整的loss function，裡面加入亂七八糟的註解說明，當然在實際程式上線的時候並不會這麼做，只是在說明的時候這樣子比較能清楚知道每一段程式碼的用處。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_nan2zero(source_tensor):\n",
    "    \"\"\"把nan的tensor轉0\n",
    "    \n",
    "    如果用nan去跟其它的tensor加總的話會得到nan，因此需要做一個簡單的轉換\n",
    "    \"\"\"\n",
    "    return tf.where(tf.math.is_nan(source_tensor), 0, source_tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def yolo_loss(output_decoder, output_raw, label, bboxes, class_nums=80):\n",
    "    \"\"\"計算loss\n",
    "    這邊我們假設我們的輸入就會是416\n",
    "    \n",
    "    output_decoder: 整理過的output\n",
    "        type: tensor\n",
    "        M x anchor_size x anchor_size x ahcnor_nums x (5 + C)\n",
    "        \n",
    "    output_raw: 原始的output\n",
    "        type: tensor\n",
    "        M x anchor_size x anchor_size x (anchor_nums x (5 + C))\n",
    "        \n",
    "    label: 訓練資料，格式記得要等同於output_decoder，單位的部份很重要\n",
    "        type: numpy\n",
    "        M x anchor_size x anchor_size x ahcnor_nums x (5 + C)\n",
    "        \n",
    "    bboxes: 一張照片裡面的所有物體框資訊，維度為M x max_bounding_nums x 4\n",
    "        type: numpy\n",
    "        M x max_bboxes_nums x 4          \n",
    "        \n",
    "    class_nums: 類別數量\n",
    "    \"\"\"   \n",
    "#     import pdb;pdb.set_trace()\n",
    "    label = tf.convert_to_tensor(label, dtype=tf.float32)\n",
    "    bboxes = tf.convert_to_tensor(bboxes, dtype=tf.float32)\n",
    "#     print('types:', output_decoder.dtype, output_raw.dtype, label.dtype, bboxes.dtype)\n",
    "    # iou閥值，這部份你可以自己看是要參數化還是要做為環境變數    \n",
    "    IOU_LOSS_THRESH = 0.5\n",
    "    \n",
    "    conv_shape = tf.shape(output_raw)\n",
    "    batch_size = conv_shape[0]\n",
    "    \n",
    "    # 13x13, 26x26, 52x52\n",
    "    anchor_size = conv_shape[1]\n",
    "    input_size = 416\n",
    "    \n",
    "    # 這邊假設類別跟論文一樣是80，因此為5 + 80\n",
    "    conv = tf.reshape(output_raw, shape=(batch_size, anchor_size, anchor_size, 3, 5 + class_nums))\n",
    "    # 原始的置信度\n",
    "    conv_raw_conf = conv[..., 4: 5]\n",
    "    # 原始的類別機率\n",
    "    conv_raw_prob = conv[..., 5:]\n",
    "        \n",
    "    # 整理過的框資訊\n",
    "    pred_xywh = output_decoder[..., 0: 4]\n",
    "    # 整理過的置信度\n",
    "    pred_conf = output_decoder[..., 4: 5]\n",
    "    \n",
    "    # 實際的框資訊\n",
    "    label_xywh = label[..., 0: 4]\n",
    "    # 實際的框有沒有東西\n",
    "    respond_bbox = label[..., 4: 5]\n",
    "    # 實際的類別機率\n",
    "    label_prob = label[..., 5:]\n",
    "    \n",
    "    # 首先計算框的好不好的loss\n",
    "    # 計算預測的與實際的框之間的IoU\n",
    "    giou = tf.expand_dims(bounding_box_giou(pred_xywh, label_xywh), axis=-1) \n",
    "    \n",
    "#     import pdb;pdb.set_trace()\n",
    "    \n",
    "    # input size型別轉換\n",
    "    input_size = tf.cast(input_size, tf.float32)\n",
    "    # 框愈大，值愈小，框愈小，直愈大，那框的大小自然對於整體的loss影響就不至於太大，\n",
    "    # 要注意，這邊框的資訊是實際框，而不是anchor prior或是學習到的那個框        \n",
    "    bbox_loss_scale = 2.0 - (1.0 * label_xywh[:, :, :, :, 2:3] * label_xywh[:, :, :, :, 3:4] / (input_size ** 2))\n",
    "    # 很明顯的，如果實際框是有東西的，那respond_bbox就會是1\n",
    "    giou_loss = respond_bbox * bbox_loss_scale * (1 - giou)\n",
    "    giou_loss = convert_nan2zero(giou_loss)\n",
    "    \n",
    "    # 接下來我們要做的就是計算第二部份的loss，也就是置信度的部份    \n",
    "    # 找出每個grid cell中最大IoU的框，比對的是預測的資訊跟你照片裡面所有的框資訊\n",
    "    # pred_xywh加一維之後為M x N x N x 3 x 1 x 4，bboxes為M x 1 x 1 x 1 x max bounding nums x 4\n",
    "    iou, _ = bounding_box_iou(pred_xywh[:, :, :, :, np.newaxis, :], bboxes[:, np.newaxis, np.newaxis, np.newaxis, :, :])\n",
    "    max_iou = tf.expand_dims(tf.reduce_max(iou, axis=-1), axis=-1)\n",
    "#     import pdb;pdb.set_trace()\n",
    "    # 前面(1.0 - respond-bbos)的部份，只要有東西的話這個結果就會是0，自然就不列入考慮\n",
    "    # 是背景的，小於閥值的那一陀就是1，我們就知道這一堆就是背景\n",
    "    respond_bgd = (1.0 - respond_bbox) * tf.cast(max_iou < IOU_LOSS_THRESH, tf.float32)\n",
    "#     import pdb;pdb.set_trace()\n",
    "    conf_focal = tf.pow(respond_bbox - pred_conf, 2)\n",
    "    conf_loss = conf_focal * (\n",
    "        # 有東西的\n",
    "        respond_bbox * tf.nn.sigmoid_cross_entropy_with_logits(labels=respond_bbox, logits=conv_raw_conf)\n",
    "        +\n",
    "        # 背景\n",
    "        respond_bgd * tf.nn.sigmoid_cross_entropy_with_logits(labels=respond_bbox, logits=conv_raw_conf)\n",
    "    )\n",
    "    conf_loss = convert_nan2zero(conf_loss)\n",
    "    \n",
    "    # 最後就是產品類別，這比較簡單\n",
    "    prob_loss = respond_bbox * tf.nn.sigmoid_cross_entropy_with_logits(labels=label_prob, logits=conv_raw_prob)\n",
    "    prob_loss = convert_nan2zero(prob_loss)\n",
    "#     import pdb;pdb.set_trace()\n",
    "    # 把這三個loss做一個維度的收斂，加總取平均，最終得到的就是每個樣本的loss\n",
    "    # axis的概念就是，沿著那個維度做operator，那個維度就消失，\n",
    "    # 所以沿著1、2、3、4，那留下的就是樣本維度\n",
    "    giou_loss = tf.reduce_mean(tf.reduce_sum(giou_loss, axis=[1, 2, 3, 4]))\n",
    "    conf_loss = tf.reduce_mean(tf.reduce_sum(conf_loss, axis=[1, 2, 3, 4]))\n",
    "    prob_loss = tf.reduce_mean(tf.reduce_sum(prob_loss, axis=[1, 2, 3, 4]))\n",
    "    \n",
    "    return giou_loss, conf_loss, prob_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這個loss function很有意思，把相關可以考慮的都列進來了，下一步我們終於可以開始嚐試訓練模型了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 附錄"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面給出經過tf.tile處理之後的結果，較為直觀瞭解"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(13, 13), dtype=int32, numpy=\n",
       "array([[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [ 1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1],\n",
       "       [ 2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2],\n",
       "       [ 3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3],\n",
       "       [ 4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4],\n",
       "       [ 5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5],\n",
       "       [ 6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6],\n",
       "       [ 7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7],\n",
       "       [ 8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8],\n",
       "       [ 9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9],\n",
       "       [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10],\n",
       "       [11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11],\n",
       "       [12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12]], dtype=int32)>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.tile(tf.range(13, dtype=tf.int32)[:, tf.newaxis], [1, 13])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(13, 13), dtype=int32, numpy=\n",
       "array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],\n",
       "       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12]], dtype=int32)>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.tile(tf.range(13, dtype=tf.int32)[tf.newaxis, :], [13, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = tf.tile(tf.range(13, dtype=tf.int32)[:, tf.newaxis], [1, 13])\n",
    "x = tf.tile(tf.range(13, dtype=tf.int32)[tf.newaxis, :], [13, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(13, 2), dtype=int32, numpy=\n",
       "array([[ 0,  0],\n",
       "       [ 1,  0],\n",
       "       [ 2,  0],\n",
       "       [ 3,  0],\n",
       "       [ 4,  0],\n",
       "       [ 5,  0],\n",
       "       [ 6,  0],\n",
       "       [ 7,  0],\n",
       "       [ 8,  0],\n",
       "       [ 9,  0],\n",
       "       [10,  0],\n",
       "       [11,  0],\n",
       "       [12,  0]], dtype=int32)>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_xy = tf.concat([x[:, :, tf.newaxis], y[:, :, tf.newaxis]], axis=-1)\n",
    "grid_xy[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([5, 13, 13, 3, 2])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 假設batch size = 5\n",
    "grid_xy = tf.tile(grid_xy[tf.newaxis, :, :, tf.newaxis, :], [5, 1, 1, 3, 1])\n",
    "grid_xy.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([5, 13, 13, 3, 6])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.concat([tf.concat([grid_xy, grid_xy], axis=-1), grid_xy], axis=-1).shape"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}