week5/5-5 损失函数loss.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5-5 损失函数loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般来说监督学习的目标函数由损失函数和正则化项组成。(Objective = Loss + Regularization)\n",
    "\n",
    "对于keras库来说, 目标函数的正则化项一般是在各层中指定,例如Dense的kernel_regularizer和bias_constraint等参数约束权重的取值范围。\n",
    "\n",
    "损失函数在模型编译的时候。对于回归模型常用的损失函数是mean_squared_error。\n",
    "\n",
    "对于二元分类来说一般使用的损失函数是binary_crossentropy。\n",
    "\n",
    "对于多分类的话可以使用categorical_crossentropy，如果label进行了one-hot编码的话，则需要使用sparse_categorical_crossentropy。\n",
    "\n",
    "如果有需要的话可以自定义损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import layers, models, losses, regularizers, constraints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一 损失函数和正则化项"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                650       \n",
      "=================================================================\n",
      "Total params: 4,810\n",
      "Trainable params: 4,810\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "tf.keras.backend.clear_session()\n",
    "\n",
    "model = models.Sequential()\n",
    "densor1 = layers.Dense(64, input_dim = 64, kernel_regularizer = regularizers.l2(0.01),\n",
    "                       activity_regularizer = regularizers.l1(0.01), \n",
    "                      kernel_constraint = constraints.MaxNorm(max_value = 2, axis = 0))\n",
    "\n",
    "densor2 = layers.Dense(10, kernel_regularizer = regularizers.l1_l2(0.01, 0.01), activation = 'sigmoid')\n",
    "\n",
    "model.add(densor1)\n",
    "model.add(densor2)\n",
    "model.compile(optimizer = 'rmsprop', loss = 'sparse_categorical_crossentropy', metrics = 'AUC')\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二 内置损失函数\n",
    "* mean_squared_error（均方误差损失，用于回归，简写为 mse, 类与函数实现形式分别为 MeanSquaredError 和 MSE）\n",
    "\n",
    "* mean_absolute_error (平均绝对值误差损失，用于回归，简写为 mae, 类与函数实现形式分别为 MeanAbsoluteError 和 MAE)\n",
    "\n",
    "* mean_absolute_percentage_error (平均百分比误差损失，用于回归，简写为 mape, 类与函数实现形式分别为 MeanAbsolutePercentageError 和 MAPE)\n",
    "\n",
    "* Huber(Huber损失，只有类实现形式，用于回归，介于mse和mae之间，对异常值比较鲁棒，相对mse有一定的优势)\n",
    "\n",
    "* binary_crossentropy(二元交叉熵，用于二分类，类实现形式为 BinaryCrossentropy)\n",
    "\n",
    "* categorical_crossentropy(类别交叉熵，用于多分类，要求label为onehot编码，类实现形式为 CategoricalCrossentropy)\n",
    "\n",
    "* sparse_categorical_crossentropy(稀疏类别交叉熵，用于多分类，要求label为序号编码形式，类实现形式为 SparseCategoricalCrossentropy)\n",
    "\n",
    "* hinge(合页损失函数，用于二分类，最著名的应用是作为支持向量机SVM的损失函数，类实现形式为 Hinge)\n",
    "\n",
    "* kld(相对熵损失，也叫KL散度，常用于最大期望算法EM的损失函数，两个概率分布差异的一种信息度量。类与函数实现形式分别为 KLDivergence 或 KLD)\n",
    "\n",
    "* cosine_similarity(余弦相似度，可用于多分类，类实现形式为 CosineSimilarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三 自定义损失函数\n",
    "继承tf.keras.losses.Loss, 重写call方法从而自定义损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FocalLoss(losses.Loss):\n",
    "    def __init__(self, gamma = 2.0, alpha = 0.25):\n",
    "        self.gamma = gamma\n",
    "        self.alpha = alpha\n",
    "        \n",
    "    def call(self, y_true, y_pred):\n",
    "        pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))\n",
    "        pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))\n",
    "        loss = -tf.sum(self.alpha * tf.pow(1. - pt_1. self.gamma) * tf.log(1e-07 + pt_1)) -tf.sum((1 - self.alpha) \\\n",
    "                                                                                                  * tf.pow(pt_0, self.gamma) \\\n",
    "                                                                                                    * tf.log(1. - pt_0 + 1e-07))\n",
    "        return loss"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tensorflow2",
   "language": "python",
   "name": "tensorflow2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}