7-5,FiBiNET模型.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "827f51d5",
   "metadata": {},
   "source": [
    "\n",
    "# 7-5，FiBiNET模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fccfc16",
   "metadata": {},
   "source": [
    "神经网络的结构设计有3个主流的高级技巧：\n",
    "\n",
    "* 1，高低融合 (将高层次特征与低层次特征融合，提升特征维度的丰富性和多样性，像人一样同时考虑整体和细节)\n",
    "* 2，权值共享 (一个权值矩阵参与多个不同的计算，降低参数规模并同时缓解样本稀疏性，像人一样一条知识多处运用)\n",
    "* 3，动态适应 (不同的输入样本使用不同的权值矩阵，动态地进行特征选择并赋予特征重要度解释性，像人一样聚焦重要信息排除干扰信息)\n",
    "\n",
    "技巧应用范例：\n",
    "\n",
    "* 1，高低融合 (DeepWide,UNet,特征金字塔FPN...)\n",
    "* 2，权值共享 (CNN,RNN,FM,DeepFM,BlinearFFM...)\n",
    "* 3，动态适应 (各种Attention机制...)\n",
    "\n",
    "新浪微博广告推荐技术团队2019年发布的CTR预估模型FiBiNET同时巧妙地运用了以上3种技巧，是神经网络结构设计的教科书级的范例。\n",
    "\n",
    "在此介绍给大家。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "582eca22",
   "metadata": {},
   "source": [
    "参考资料：\n",
    "* FiBiNET论文：https://arxiv.org/pdf/1905.09433.pdf\n",
    "* FiBiNET-结合特征重要性和双线性特征交互进行CTR预估：https://zhuanlan.zhihu.com/p/72931811\n",
    "* 代码实现：https://github.com/xue-pai/FuxiCTR/blob/main/fuxictr/pytorch/models/FiBiNET.py \n",
    "* SENet原理：https://zhuanlan.zhihu.com/p/65459972"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7e24de6",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "<font color=\"red\">\n",
    " \n",
    "公众号 **算法美食屋** 回复关键词：**pytorch**， 获取本项目源码和所用数据集百度云盘下载链接。\n",
    "    \n",
    "</font> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8828a0e5",
   "metadata": {},
   "source": [
    "## 一，FiBiNET原理解析"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca95583d",
   "metadata": {},
   "source": [
    "FiBiNET全称为Feature Importance and Bilinear Interaction Network.\n",
    "\n",
    "顾名思义，其主要的创意有2个。\n",
    "\n",
    "第一个是Feature Importance，通过借鉴SENet（Squeeze-and-Excitation）Attention机制实现特征选择和重要度解释。\n",
    "\n",
    "第二个是Bilinear Interaction Network，这是应用权值共享技巧对 FFM(Field-Aware FM)结构进行改进的一种结构。\n",
    "\n",
    "同时，FiBiNET保留了DeepWide的高低融合的网络架构。\n",
    "\n",
    "所以它综合使用了 高低融合、权值共享、动态适应 这3种神经网络结构设计的高级技巧。一个不落，Triple kill!\n",
    "\n",
    "我们重点介绍一下 SENet Attention 和 Bilinear Interaction."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f400a738",
   "metadata": {},
   "source": [
    "### 1, SENet Attention "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bf342b7",
   "metadata": {},
   "source": [
    "SENet 全称为 Squeeze-and-Excitation Network，是一种通过注意力机制计算特征重要度的网络模块。\n",
    "\n",
    "最早是在CV领域引入，通过在ResNet结构上添加SENet Attention模块，赢得了ImageNet 2017竞赛分类任务的冠军。\n",
    "\n",
    "如何计算各个Feature Map(通道)的特征重要度(注意力权重)呢？\n",
    "\n",
    "SENet的思想非常简洁。\n",
    "\n",
    "step1: 通过全局池化将各个Feature Map由一个一个的矩阵汇总成一个一个的标量。此即Squeeze操作。\n",
    "\n",
    "step2：通过一个2层MLP将汇总成得到的一个一个的标量所构成的向量进行变换，得到注意力权重。此即Excitation操作。 \n",
    "细节一点地说，这个2层的MLP的第1层将通道数量缩减成原来的1/3, 第2层再将通道数恢复。并且每层后面都接入了激活函数。\n",
    "\n",
    "step3：用注意力权重乘以原始的Feature Map。这个是Re-Weight操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f634cfe",
   "metadata": {},
   "source": [
    "图片示意如下。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2sfkkfcy3j208w06fgll.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a79f99ba",
   "metadata": {},
   "source": [
    "pytorch代码实现如下，可能比图片更加好懂。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a704b92e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "class SENetAttention(nn.Module):\n",
    "    \"\"\"\n",
    "    Squeeze-and-Excitation Attention\n",
    "    输入shape: [batch_size, num_fields, d_embed]   #num_fields即num_features\n",
    "    输出shape: [batch_size, num_fields, d_embed]\n",
    "    \"\"\"\n",
    "    def __init__(self, num_fields, reduction_ratio=3):\n",
    "        super().__init__()\n",
    "        reduced_size = max(1, int(num_fields / reduction_ratio))\n",
    "        self.excitation = nn.Sequential(nn.Linear(num_fields, reduced_size, bias=False),\n",
    "                                        nn.ReLU(),\n",
    "                                        nn.Linear(reduced_size, num_fields, bias=False),\n",
    "                                        nn.ReLU())\n",
    "\n",
    "    def forward(self, x):\n",
    "        Z = torch.mean(x, dim=-1, out=None) #1,Sequeeze\n",
    "        A = self.excitation(Z) #2,Excitation\n",
    "        V = x * A.unsqueeze(-1) #3,Re-Weight\n",
    "        return V"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "097ab325",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "57638416",
   "metadata": {},
   "source": [
    "### 2, Bilinear Interaction "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f049c68c",
   "metadata": {},
   "source": [
    "Bilinear Interaction实际上是FFM在权值共享思想下的一种改进，也可以称之为Bilinear FFM。\n",
    "\n",
    "我们先说说FFM(Field-Aware FM)，再看看这个Bilinear FFM 怎么改进的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a745eac",
   "metadata": {},
   "source": [
    "FM用隐向量之间的点积来计算特征之间的交叉，并且一个特征用一个隐向量来表示。\n",
    "\n",
    "FFM认为一个特征用一个隐向量来表达太粗糙了，如果这个特征和不同分组(Field)的特征来做交叉，应该用不同的隐向量。\n",
    "\n",
    "举例来说，考虑一个广告点击预测的场景，广告类别 和 用户所在城市、用户职业之间的交叉。\n",
    "\n",
    "在FM中 一个确定的广告类别 比如游戏广告 不论是和用户所在城市，还是用户职业交叉，都用同一个隐向量。\n",
    "\n",
    "但是FFM认为，用户所在城市和用户职业是两类完全不同的特征(不同Field)，描述它们的向量空间应该是完全不相关的，FM用一个相同的隐向量来和它们做点积不合理。\n",
    "\n",
    "所以，FFM引入了Field(域)的概念，和不同Field的特征做交叉，要使用不同的隐向量。\n",
    "\n",
    "实践表明，FFM这个思路是有效的, FFM的作者阮毓钦正是凭借这个方案赢得了2015年kaggle举办的Criteo比赛的冠军。\n",
    "\n",
    "但是FFM有个很大的缺点，就是参数量太多了。\n",
    "\n",
    "对于FM来说，每个特征只有一个隐向量，假设有n个特征，每个隐向量维度为k，全部隐向量参数矩阵的大小 size = n k.\n",
    "\n",
    "但是对于FFM，有过有f个不同的field，每个特征都将有f-1个隐向量，全部隐向量的参数矩阵的大小增大为 size = (f-1) n k. \n",
    "\n",
    "通常的应用场景中，Field的数量有几十几百维，而Feature的数量有数万数百万维。\n",
    "\n",
    "很显然，FFM将隐向量的参数规模扩大了几十几百倍。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b57c8810",
   "metadata": {},
   "source": [
    "FFM的本质思想是在做特征交叉的时候要区分不同的Field，其实现方式是和不同的Field做交叉时用不同的隐向量。\n",
    "\n",
    "有没有办法保留FFM中区分不同Field的特性，并降低参数规模呢？\n",
    "\n",
    "BilinearFFM说，我有办法，权重共享走起来！\n",
    "\n",
    "BilinearFFM不直接针对不同Field设计不同的隐向量，而是引入了Field变换矩阵来区分不同的Field。\n",
    "\n",
    "每个特征还是一个隐向量，但是和不同的Field的特征做交叉时，先乘上这个特征所在Field的变换矩阵，然后再做后面的点积。\n",
    "\n",
    "因此，同属一个Field的特征共享一个Field变换矩阵。这种bilinear_type叫做 field_each.\n",
    "\n",
    "Field变换矩阵的大小是k^2, 这种方式下，全部隐向量的参数大小加上共享变换矩阵的参数大小一共是 size = n k + f k^2 \n",
    "\n",
    "由于k和f远小于n，这种Bilinear方式相比FM增加的参数量可以忽略不计。\n",
    "\n",
    "\n",
    "除了 同属一个Field的特征共享一个Field变换矩阵外，我们还可以更加简单粗暴一点，所有特征共享一个变换矩阵.\n",
    "\n",
    "这种bilinear_type叫做 field_all.这种方式下，size = n k + k^2 \n",
    "\n",
    "\n",
    "我们也可以更加精细一点，相同的Field组合之间的交互共享一个变换矩阵，这种bilinear_type叫做field_interaction. \n",
    "\n",
    "总共有f(f-1)/2种组合，这种方式下， size = n k + k^2 f(f-1)/2\n",
    "\n",
    "以上就是BilinearFFM的基本思想。\n",
    "\n",
    "\n",
    "FiBiNET中用到的Bilinear Interaction相比BilinearFFM, 还有一处小改动，将点积改成了哈达玛积，如下图所示。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65661157",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2sfj6wh0rj209v08daad.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32bac64f",
   "metadata": {},
   "source": [
    "pytorch代码实现如下，整体不难理解。作2点说明。\n",
    "\n",
    "1，Field概念说明\n",
    "\n",
    "在FFM相关的文章中，引入了Field的概念，以和Feature区分，一个Field中可以包括多个Feature. \n",
    "\n",
    "实际上Field就是我们通常理解的特征，包括数值特征和类别特征，但是Feature是数值特征或者类别特征onehot后的特征。一个类别特征对应一个Field，但是对应多个Feature。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2sh8t8j6pj20gc058mx9.jpg)\n",
    "\n",
    "2，combinations函数说明\n",
    "\n",
    "组合函数combinations从num_fields中任取2种作为组合，共有 num_fields*(num_fields-1)中组合方式。\n",
    "\n",
    "所以输出的Field数量变成了 num_fields*(num_fields-1)/2。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09d50922",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "from itertools import combinations\n",
    "class BilinearInteraction(nn.Module):\n",
    "    \"\"\"\n",
    "    双线性FFM\n",
    "    输入shape: [batch_size, num_fields, d_embed] #num_fields即num_features\n",
    "    输出shape: [batch_size, num_fields*(num_fields-1)/2, d_embed]\n",
    "    \"\"\"\n",
    "    def __init__(self, num_fields, d_embed, bilinear_type=\"field_interaction\"):\n",
    "        super().__init__()\n",
    "        self.bilinear_type = bilinear_type\n",
    "        if self.bilinear_type == \"field_all\":\n",
    "            self.bilinear_layer = nn.Linear(d_embed, d_embed, bias=False)\n",
    "        elif self.bilinear_type == \"field_each\":\n",
    "            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)\n",
    "                                                 for i in range(num_fields)])\n",
    "        elif self.bilinear_type == \"field_interaction\":\n",
    "            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)\n",
    "                                                 for i, j in combinations(range(num_fields), 2)])\n",
    "        else:\n",
    "            raise NotImplementedError()\n",
    "\n",
    "    def forward(self, feature_emb):\n",
    "        feature_emb_list = torch.split(feature_emb, 1, dim=1)\n",
    "        if self.bilinear_type == \"field_all\":\n",
    "            bilinear_list = [self.bilinear_layer(v_i) * v_j\n",
    "                             for v_i, v_j in combinations(feature_emb_list, 2)]\n",
    "        elif self.bilinear_type == \"field_each\":\n",
    "            bilinear_list = [self.bilinear_layer[i](feature_emb_list[i]) * feature_emb_list[j]\n",
    "                             for i, j in combinations(range(len(feature_emb_list)), 2)]\n",
    "        elif self.bilinear_type == \"field_interaction\":\n",
    "            bilinear_list = [self.bilinear_layer[i](v[0]) * v[1]\n",
    "                             for i, v in enumerate(combinations(feature_emb_list, 2))]\n",
    "        return torch.cat(bilinear_list, dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efe26681",
   "metadata": {},
   "source": [
    "### 二，FiBiNET的pytorch实现"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1f340fd",
   "metadata": {},
   "source": [
    "下面是FiBiNET的一个pytorch实现。\n",
    "\n",
    "核心代码是SENetAttention模块和BilinearInteraction模块的实现。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2sffi4huoj20g70a4gm4.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93153b47",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "from itertools import combinations\n",
    "\n",
    "class NumEmbedding(nn.Module):\n",
    "    \"\"\"\n",
    "    连续特征用linear层编码\n",
    "    输入shape: [batch_size,num_features, d_in], # d_in 通常是1\n",
    "    输出shape: [batch_size,num_features, d_out]\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, n: int, d_in: int, d_out: int, bias: bool = False) -> None:\n",
    "        super().__init__()\n",
    "        self.weight = nn.Parameter(torch.Tensor(n, d_in, d_out))\n",
    "        self.bias = nn.Parameter(torch.Tensor(n, d_out)) if bias else None\n",
    "        with torch.no_grad():\n",
    "            for i in range(n):\n",
    "                layer = nn.Linear(d_in, d_out)\n",
    "                self.weight[i] = layer.weight.T\n",
    "                if self.bias is not None:\n",
    "                    self.bias[i] = layer.bias\n",
    "\n",
    "    def forward(self, x_num):\n",
    "        assert x_num.ndim == 3\n",
    "        #x = x_num[..., None] * self.weight[None]\n",
    "        #x = x.sum(-2)\n",
    "        x = torch.einsum(\"bfi,fij->bfj\",x_num,self.weight)\n",
    "        if self.bias is not None:\n",
    "            x = x + self.bias[None]\n",
    "        return x\n",
    "    \n",
    "class CatEmbedding(nn.Module):\n",
    "    \"\"\"\n",
    "    离散特征用Embedding层编码\n",
    "    输入shape: [batch_size, num_features], \n",
    "    输出shape: [batch_size, num_features, d_embed]\n",
    "    \"\"\"\n",
    "    def __init__(self, categories, d_embed):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(sum(categories), d_embed)\n",
    "        self.offsets = nn.Parameter(\n",
    "                torch.tensor([0] + categories[:-1]).cumsum(0),requires_grad=False)\n",
    "        \n",
    "        nn.init.xavier_uniform_(self.embedding.weight.data)\n",
    "\n",
    "    def forward(self, x_cat):\n",
    "        \"\"\"\n",
    "        x_cat: Long tensor of size ``(batch_size, features_num)``\n",
    "        \"\"\"\n",
    "        x = x_cat + self.offsets[None]\n",
    "        return self.embedding(x) \n",
    "    \n",
    "class CatLinear(nn.Module):\n",
    "    \"\"\"\n",
    "    离散特征用Embedding实现线性层（等价于先F.onehot再nn.Linear()）\n",
    "    输入shape: [batch_size, num_features ], \n",
    "    输出shape: [batch_size, d_out]\n",
    "    \"\"\"\n",
    "    def __init__(self, categories, d_out=1):\n",
    "        super().__init__()\n",
    "        self.fc = nn.Embedding(sum(categories), d_out)\n",
    "        self.bias = nn.Parameter(torch.zeros((d_out,)))\n",
    "        self.offsets = nn.Parameter(\n",
    "                torch.tensor([0] + categories[:-1]).cumsum(0),requires_grad=False)\n",
    "        nn.init.xavier_uniform_(self.fc.weight.data)\n",
    "\n",
    "    def forward(self, x_cat):\n",
    "        \"\"\"\n",
    "        Long tensor of size ``(batch_size, num_features)``\n",
    "        \"\"\"\n",
    "        x = x_cat + self.offsets[None]\n",
    "        return torch.sum(self.fc(x), dim=1) + self.bias \n",
    "    \n",
    "class SENetAttention(nn.Module):\n",
    "    \"\"\"\n",
    "    Squeeze-and-Excitation Attention\n",
    "    输入shape: [batch_size, num_fields, d_embed]   #num_fields即num_features\n",
    "    输出shape: [batch_size, num_fields, d_embed]\n",
    "    \"\"\"\n",
    "    def __init__(self, num_fields, reduction_ratio=3):\n",
    "        super().__init__()\n",
    "        reduced_size = max(1, int(num_fields / reduction_ratio))\n",
    "        self.excitation = nn.Sequential(nn.Linear(num_fields, reduced_size, bias=False),\n",
    "                                        nn.ReLU(),\n",
    "                                        nn.Linear(reduced_size, num_fields, bias=False),\n",
    "                                        nn.ReLU())\n",
    "\n",
    "    def forward(self, x):\n",
    "        Z = torch.mean(x, dim=-1, out=None) #1,Sequeeze\n",
    "        A = self.excitation(Z) #2,Excitation\n",
    "        V = x * A.unsqueeze(-1) #3,Re-Weight\n",
    "        return V\n",
    "    \n",
    "class BilinearInteraction(nn.Module):\n",
    "    \"\"\"\n",
    "    双线性FFM\n",
    "    输入shape: [batch_size, num_fields, d_embed] #num_fields即num_features\n",
    "    输出shape: [batch_size, num_fields*(num_fields-1)/2, d_embed]\n",
    "    \"\"\"\n",
    "    def __init__(self, num_fields, d_embed, bilinear_type=\"field_interaction\"):\n",
    "        super().__init__()\n",
    "        self.bilinear_type = bilinear_type\n",
    "        if self.bilinear_type == \"field_all\":\n",
    "            self.bilinear_layer = nn.Linear(d_embed, d_embed, bias=False)\n",
    "        elif self.bilinear_type == \"field_each\":\n",
    "            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)\n",
    "                                                 for i in range(num_fields)])\n",
    "        elif self.bilinear_type == \"field_interaction\":\n",
    "            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)\n",
    "                                                 for i, j in combinations(range(num_fields), 2)])\n",
    "        else:\n",
    "            raise NotImplementedError()\n",
    "\n",
    "    def forward(self, feature_emb):\n",
    "        feature_emb_list = torch.split(feature_emb, 1, dim=1)\n",
    "        if self.bilinear_type == \"field_all\":\n",
    "            bilinear_list = [self.bilinear_layer(v_i) * v_j\n",
    "                             for v_i, v_j in combinations(feature_emb_list, 2)]\n",
    "        elif self.bilinear_type == \"field_each\":\n",
    "            bilinear_list = [self.bilinear_layer[i](feature_emb_list[i]) * feature_emb_list[j]\n",
    "                             for i, j in combinations(range(len(feature_emb_list)), 2)]\n",
    "        elif self.bilinear_type == \"field_interaction\":\n",
    "            bilinear_list = [self.bilinear_layer[i](v[0]) * v[1]\n",
    "                             for i, v in enumerate(combinations(feature_emb_list, 2))]\n",
    "        return torch.cat(bilinear_list, dim=1)\n",
    "    \n",
    "\n",
    "#mlp\n",
    "class MultiLayerPerceptron(nn.Module):\n",
    "    def __init__(self, d_in, d_layers, dropout, \n",
    "                 d_out = 1):\n",
    "        super().__init__()\n",
    "        layers = []\n",
    "        for d in d_layers:\n",
    "            layers.append(nn.Linear(d_in, d))\n",
    "            layers.append(nn.BatchNorm1d(d))\n",
    "            layers.append(nn.ReLU())\n",
    "            layers.append(nn.Dropout(p=dropout))\n",
    "            d_in = d\n",
    "        layers.append(nn.Linear(d_layers[-1], d_out))\n",
    "        self.mlp = nn.Sequential(*layers)\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        float tensor of size ``(batch_size, d_in)``\n",
    "        \"\"\"\n",
    "        return self.mlp(x)\n",
    "    \n",
    "\n",
    "#fibinet \n",
    "class FiBiNET(nn.Module):\n",
    "    \n",
    "    def __init__(self,\n",
    "                 d_numerical, \n",
    "                 categories, \n",
    "                 d_embed,\n",
    "                 mlp_layers, \n",
    "                 mlp_dropout,\n",
    "                 reduction_ratio = 3,\n",
    "                 bilinear_type = \"field_interaction\",\n",
    "                 n_classes = 1):\n",
    "        \n",
    "        super().__init__()\n",
    "        \n",
    "        if d_numerical is None:\n",
    "            d_numerical = 0\n",
    "        if categories is None:\n",
    "            categories = []\n",
    "            \n",
    "        self.categories = categories\n",
    "        self.n_classes = n_classes\n",
    "        \n",
    "        self.num_linear = nn.Linear(d_numerical,n_classes) if d_numerical else None\n",
    "        self.cat_linear = CatLinear(categories,n_classes) if categories else None\n",
    "        \n",
    "        self.num_embedding = NumEmbedding(d_numerical,1,d_embed) if d_numerical else None\n",
    "        self.cat_embedding = CatEmbedding(categories, d_embed) if categories else None\n",
    "        \n",
    "        num_fields = d_numerical+len(categories)\n",
    "        \n",
    "        self.se_attention = SENetAttention(num_fields, reduction_ratio)\n",
    "        self.bilinear = BilinearInteraction(num_fields, d_embed, bilinear_type)\n",
    "        \n",
    "        mlp_in = num_fields * (num_fields - 1) * d_embed\n",
    "        self.mlp = MultiLayerPerceptron(\n",
    "            d_in= mlp_in,\n",
    "            d_layers = mlp_layers,\n",
    "            dropout = mlp_dropout,\n",
    "            d_out = n_classes\n",
    "        )\n",
    "        \n",
    "        \n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        x_num: numerical features\n",
    "        x_cat: category features\n",
    "        \"\"\"\n",
    "        x_num,x_cat = x\n",
    "        \n",
    "        #一，wide部分\n",
    "        x_linear = 0.0\n",
    "        if self.num_linear:\n",
    "            x_linear = x_linear + self.num_linear(x_num) \n",
    "        if self.cat_linear:\n",
    "            x_linear = x_linear + self.cat_linear(x_cat)\n",
    "            \n",
    "        #二，deep部分 \n",
    "        \n",
    "        #1，embedding\n",
    "        x_embedding = []\n",
    "        if self.num_embedding:\n",
    "            x_embedding.append(self.num_embedding(x_num[...,None]))\n",
    "        if self.cat_embedding:\n",
    "            x_embedding.append(self.cat_embedding(x_cat))\n",
    "        x_embedding = torch.cat(x_embedding,dim=1)\n",
    "        \n",
    "        #2，interaction\n",
    "        se_embedding = self.se_attention(x_embedding)\n",
    "        ffm_out = self.bilinear(x_embedding)\n",
    "        se_ffm_out = self.bilinear(se_embedding)\n",
    "        x_interaction = torch.flatten(torch.cat([ffm_out, se_ffm_out], dim=1), start_dim=1)\n",
    "        \n",
    "        #3，mlp\n",
    "        x_deep = self.mlp(x_interaction)\n",
    "        \n",
    "        #三，高低融合\n",
    "        x_out = x_linear+x_deep\n",
    "        if self.n_classes==1:\n",
    "            x_out = x_out.squeeze(-1)\n",
    "        return x_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b39cb4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "##测试 FiBiNET\n",
    "\n",
    "model = FiBiNET(d_numerical = 3, categories = [4,3,2],\n",
    "        d_embed = 4, mlp_layers = [20,20], mlp_dropout=0.25,\n",
    "        reduction_ratio = 3,\n",
    "        bilinear_type = \"field_interaction\",\n",
    "        n_classes = 1)\n",
    "\n",
    "x_num = torch.randn(2,3)\n",
    "x_cat = torch.randint(0,2,(2,3))\n",
    "print(model((x_num,x_cat)))  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d2a960d",
   "metadata": {},
   "source": [
    "```\n",
    "tensor([-0.8621,  0.6743], grad_fn=<SqueezeBackward1>)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec37a4df",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "72a05548",
   "metadata": {},
   "source": [
    "## 三，Criteo数据集完整范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79dd5aef",
   "metadata": {},
   "source": [
    "Criteo数据集是一个经典的广告点击率CTR预测数据集。\n",
    "\n",
    "这个数据集的目标是通过用户特征和广告特征来预测某条广告是否会为用户点击。\n",
    "\n",
    "数据集有13维数值特征(I1-I13)和26维类别特征(C14-C39), 共39维特征, 特征中包含着许多缺失值。\n",
    "\n",
    "训练集4000万个样本，测试集600万个样本。数据集大小超过100G.\n",
    "\n",
    "此处使用的是采样100万个样本后的cretio_small数据集。 \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16474d85",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70d99f21",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd \n",
    "import datetime \n",
    "\n",
    "from sklearn.model_selection import train_test_split \n",
    "\n",
    "import torch \n",
    "from torch import nn \n",
    "from torch.utils.data import Dataset,DataLoader  \n",
    "import torch.nn.functional as F \n",
    "import torchkeras \n",
    "\n",
    "def printlog(info):\n",
    "    nowtime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')\n",
    "    print(\"\\n\"+\"==========\"*8 + \"%s\"%nowtime)\n",
    "    print(info+'...\\n\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e3f720c",
   "metadata": {},
   "source": [
    "### 1，准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edf49cd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder,QuantileTransformer\n",
    "from sklearn.pipeline import Pipeline \n",
    "from sklearn.impute import SimpleImputer \n",
    "\n",
    "dfdata = pd.read_csv(\"./eat_pytorch_datasets/criteo_small.zip\",sep=\"\\t\",header=None)\n",
    "dfdata.columns = [\"label\"] + [\"I\"+str(x) for x in range(1,14)] + [\n",
    "    \"C\"+str(x) for x in range(14,40)]\n",
    "\n",
    "cat_cols = [x for x in dfdata.columns if x.startswith('C')]\n",
    "num_cols = [x for x in dfdata.columns if x.startswith('I')]\n",
    "num_pipe = Pipeline(steps = [('impute',SimpleImputer()),('quantile',QuantileTransformer())])\n",
    "\n",
    "for col in cat_cols:\n",
    "    dfdata[col]  = LabelEncoder().fit_transform(dfdata[col])\n",
    "\n",
    "dfdata[num_cols] = num_pipe.fit_transform(dfdata[num_cols])\n",
    "\n",
    "categories = [dfdata[col].max()+1 for col in cat_cols]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86e22fca",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch.utils.data import Dataset,DataLoader \n",
    "\n",
    "#DataFrame转换成torch数据集Dataset, 特征分割成X_num,X_cat方式\n",
    "class DfDataset(Dataset):\n",
    "    def __init__(self,df,\n",
    "                 label_col,\n",
    "                 num_features,\n",
    "                 cat_features,\n",
    "                 categories,\n",
    "                 is_training=True):\n",
    "        \n",
    "        self.X_num = torch.tensor(df[num_features].values).float() if num_features else None\n",
    "        self.X_cat = torch.tensor(df[cat_features].values).long() if cat_features else None\n",
    "        self.Y = torch.tensor(df[label_col].values).float() \n",
    "        self.categories = categories\n",
    "        self.is_training = is_training\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.Y)\n",
    "    \n",
    "    def __getitem__(self,index):\n",
    "        if self.is_training:\n",
    "            return ((self.X_num[index],self.X_cat[index]),self.Y[index])\n",
    "        else:\n",
    "            return (self.X_num[index],self.X_cat[index])\n",
    "    \n",
    "    def get_categories(self):\n",
    "        return self.categories\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a77fcdf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "dftrain_val,dftest = train_test_split(dfdata,test_size=0.2)\n",
    "dftrain,dfval = train_test_split(dftrain_val,test_size=0.2)\n",
    "\n",
    "ds_train = DfDataset(dftrain,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n",
    "\n",
    "ds_val = DfDataset(dfval,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n",
    "\n",
    "ds_test = DfDataset(dftest,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77e4b9cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "dl_train = DataLoader(ds_train,batch_size = 2048,shuffle=True)\n",
    "dl_val = DataLoader(ds_val,batch_size = 2048,shuffle=False)\n",
    "dl_test = DataLoader(ds_test,batch_size = 2048,shuffle=False)\n",
    "\n",
    "for features,labels in dl_train:\n",
    "    break \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e83da026",
   "metadata": {},
   "source": [
    "### 2，定义模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6107212d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_net():\n",
    "    net = FiBiNET(\n",
    "        d_numerical= ds_train.X_num.shape[1],\n",
    "        categories= ds_train.get_categories(),\n",
    "        d_embed = 8, mlp_layers = [128,64,32], mlp_dropout=0.25,\n",
    "        reduction_ratio = 3,\n",
    "        bilinear_type = \"field_all\",\n",
    "        n_classes = 1\n",
    "        \n",
    "    )\n",
    "    return net \n",
    "\n",
    "from torchkeras import summary\n",
    "\n",
    "net = create_net()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dd54b90",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e51aff2e",
   "metadata": {},
   "source": [
    "### 3，训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3fd5a09",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os,sys,time\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import datetime \n",
    "from tqdm import tqdm \n",
    "\n",
    "import torch\n",
    "from torch import nn \n",
    "from accelerate import Accelerator\n",
    "from copy import deepcopy\n",
    "\n",
    "\n",
    "def printlog(info):\n",
    "    nowtime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')\n",
    "    print(\"\\n\"+\"==========\"*8 + \"%s\"%nowtime)\n",
    "    print(str(info)+\"\\n\")\n",
    "    \n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn,stage = \"train\", metrics_dict = None, \n",
    "                 optimizer = None, lr_scheduler = None,\n",
    "                 accelerator = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler\n",
    "        self.accelerator = accelerator\n",
    "    \n",
    "    def __call__(self, features, labels):\n",
    "        #loss\n",
    "        preds = self.net(features)\n",
    "        loss = self.loss_fn(preds,labels)\n",
    "\n",
    "        #backward()\n",
    "        if self.optimizer is not None and self.stage==\"train\":\n",
    "            if self.accelerator is  None:\n",
    "                loss.backward()\n",
    "            else:\n",
    "                self.accelerator.backward(loss)\n",
    "            self.optimizer.step()\n",
    "            if self.lr_scheduler is not None:\n",
    "                self.lr_scheduler.step()\n",
    "            self.optimizer.zero_grad()\n",
    "            \n",
    "        #metrics\n",
    "        step_metrics = {self.stage+\"_\"+name:metric_fn(preds, labels).item() \n",
    "                        for name,metric_fn in self.metrics_dict.items()}\n",
    "        return loss.item(),step_metrics\n",
    "    \n",
    "    \n",
    "class EpochRunner:\n",
    "    def __init__(self,steprunner):\n",
    "        self.steprunner = steprunner\n",
    "        self.stage = steprunner.stage\n",
    "        self.steprunner.net.train() if self.stage==\"train\" else self.steprunner.net.eval()\n",
    "        \n",
    "    def __call__(self,dataloader):\n",
    "        total_loss,step = 0,0\n",
    "        loop = tqdm(enumerate(dataloader), total =len(dataloader))\n",
    "        for i, batch in loop:\n",
    "            features,labels = batch\n",
    "            if self.stage==\"train\":\n",
    "                loss, step_metrics = self.steprunner(features,labels)\n",
    "            else:\n",
    "                with torch.no_grad():\n",
    "                    loss, step_metrics = self.steprunner(features,labels)\n",
    "\n",
    "            step_log = dict({self.stage+\"_loss\":loss},**step_metrics)\n",
    "\n",
    "            total_loss += loss\n",
    "            step+=1\n",
    "            if i!=len(dataloader)-1:\n",
    "                loop.set_postfix(**step_log)\n",
    "            else:\n",
    "                epoch_loss = total_loss/step\n",
    "                epoch_metrics = {self.stage+\"_\"+name:metric_fn.compute().item() \n",
    "                                 for name,metric_fn in self.steprunner.metrics_dict.items()}\n",
    "                epoch_log = dict({self.stage+\"_loss\":epoch_loss},**epoch_metrics)\n",
    "                loop.set_postfix(**epoch_log)\n",
    "\n",
    "                for name,metric_fn in self.steprunner.metrics_dict.items():\n",
    "                    metric_fn.reset()\n",
    "        return epoch_log\n",
    "\n",
    "class KerasModel(torch.nn.Module):\n",
    "    def __init__(self,net,loss_fn,metrics_dict=None,optimizer=None,lr_scheduler = None):\n",
    "        super().__init__()\n",
    "        self.accelerator = Accelerator()\n",
    "        self.history = {}\n",
    "        \n",
    "        self.net = net\n",
    "        self.loss_fn = loss_fn\n",
    "        self.metrics_dict = nn.ModuleDict(metrics_dict) \n",
    "        \n",
    "        self.optimizer = optimizer if optimizer is not None else torch.optim.Adam(\n",
    "            self.parameters(), lr=1e-2)\n",
    "        self.lr_scheduler = lr_scheduler\n",
    "        \n",
    "        self.net,self.loss_fn,self.metrics_dict,self.optimizer = self.accelerator.prepare(\n",
    "            self.net,self.loss_fn,self.metrics_dict,self.optimizer)\n",
    "\n",
    "    def forward(self, x):\n",
    "        if self.net:\n",
    "            return self.net.forward(x)\n",
    "        else:\n",
    "            raise NotImplementedError\n",
    "\n",
    "\n",
    "    def fit(self, train_data, val_data=None, epochs=10, ckpt_path='checkpoint.pt', \n",
    "            patience=5, monitor=\"val_loss\", mode=\"min\"):\n",
    "        \n",
    "        train_data = self.accelerator.prepare(train_data)\n",
    "        val_data = self.accelerator.prepare(val_data) if val_data else []\n",
    "\n",
    "        for epoch in range(1, epochs+1):\n",
    "            printlog(\"Epoch {0} / {1}\".format(epoch, epochs))\n",
    "            \n",
    "            # 1，train -------------------------------------------------  \n",
    "            train_step_runner = StepRunner(net = self.net,stage=\"train\",\n",
    "                    loss_fn = self.loss_fn,metrics_dict=deepcopy(self.metrics_dict),\n",
    "                    optimizer = self.optimizer, lr_scheduler = self.lr_scheduler,\n",
    "                    accelerator = self.accelerator)\n",
    "            train_epoch_runner = EpochRunner(train_step_runner)\n",
    "            train_metrics = train_epoch_runner(train_data)\n",
    "            \n",
    "            for name, metric in train_metrics.items():\n",
    "                self.history[name] = self.history.get(name, []) + [metric]\n",
    "\n",
    "            # 2，validate -------------------------------------------------\n",
    "            if val_data:\n",
    "                val_step_runner = StepRunner(net = self.net,stage=\"val\",\n",
    "                    loss_fn = self.loss_fn,metrics_dict=deepcopy(self.metrics_dict),\n",
    "                    accelerator = self.accelerator)\n",
    "                val_epoch_runner = EpochRunner(val_step_runner)\n",
    "                with torch.no_grad():\n",
    "                    val_metrics = val_epoch_runner(val_data)\n",
    "                val_metrics[\"epoch\"] = epoch\n",
    "                for name, metric in val_metrics.items():\n",
    "                    self.history[name] = self.history.get(name, []) + [metric]\n",
    "            \n",
    "            # 3，early-stopping -------------------------------------------------\n",
    "            arr_scores = self.history[monitor]\n",
    "            best_score_idx = np.argmax(arr_scores) if mode==\"max\" else np.argmin(arr_scores)\n",
    "            if best_score_idx==len(arr_scores)-1:\n",
    "                torch.save(self.net.state_dict(),ckpt_path)\n",
    "                print(\"<<<<<< reach best {0} : {1} >>>>>>\".format(monitor,\n",
    "                     arr_scores[best_score_idx]),file=sys.stderr)\n",
    "            if len(arr_scores)-best_score_idx>patience:\n",
    "                print(\"<<<<<< {} without improvement in {} epoch, early stopping >>>>>>\".format(\n",
    "                    monitor,patience),file=sys.stderr)\n",
    "                self.net.load_state_dict(torch.load(ckpt_path))\n",
    "                break \n",
    "            \n",
    "        return pd.DataFrame(self.history)\n",
    "\n",
    "    @torch.no_grad()\n",
    "    def evaluate(self, val_data):\n",
    "        val_data = self.accelerator.prepare(val_data)\n",
    "        val_step_runner = StepRunner(net = self.net,stage=\"val\",\n",
    "                    loss_fn = self.loss_fn,metrics_dict=deepcopy(self.metrics_dict),\n",
    "                    accelerator = self.accelerator)\n",
    "        val_epoch_runner = EpochRunner(val_step_runner)\n",
    "        val_metrics = val_epoch_runner(val_data)\n",
    "        return val_metrics\n",
    "        \n",
    "       \n",
    "    @torch.no_grad()\n",
    "    def predict(self, dataloader):\n",
    "        dataloader = self.accelerator.prepare(dataloader)\n",
    "        result = torch.cat([self.forward(t[0]) for t in dataloader])\n",
    "        return result.data\n",
    "              "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81598c6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras.metrics import AUC\n",
    "\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "\n",
    "metrics_dict = {\"auc\":AUC()}\n",
    "\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=0.002, weight_decay=0.001) \n",
    "\n",
    "model = KerasModel(net,\n",
    "                   loss_fn = loss_fn,\n",
    "                   metrics_dict= metrics_dict,\n",
    "                   optimizer = optimizer\n",
    "                  )         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd242b51",
   "metadata": {},
   "outputs": [],
   "source": [
    "dfhistory = model.fit(train_data=dl_train,val_data=dl_val,epochs=100, patience=5,\n",
    "                      monitor = \"val_auc\",mode=\"max\",ckpt_path='checkpoint.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e82e1a5",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2t3twovonj20my0axq4j.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8994bf3e",
   "metadata": {},
   "source": [
    "### 4，评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26b68fdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'svg'\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_metric(dfhistory, metric):\n",
    "    train_metrics = dfhistory[\"train_\"+metric]\n",
    "    val_metrics = dfhistory['val_'+metric]\n",
    "    epochs = range(1, len(train_metrics) + 1)\n",
    "    plt.plot(epochs, train_metrics, 'bo--')\n",
    "    plt.plot(epochs, val_metrics, 'ro-')\n",
    "    plt.title('Training and validation '+ metric)\n",
    "    plt.xlabel(\"Epochs\")\n",
    "    plt.ylabel(metric)\n",
    "    plt.legend([\"train_\"+metric, 'val_'+metric])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27b334a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_metric(dfhistory,\"loss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29c19734",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2t3tup3hxj20gc0af74n.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6efb4d50",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_metric(dfhistory,\"auc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4169cb05",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2t3tuemikj20f70ait90.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b04a53e",
   "metadata": {},
   "source": [
    "### 5，使用模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b356206",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "preds = torch.sigmoid(model.predict(dl_val))\n",
    "labels = torch.cat([x[-1] for x in dl_val])\n",
    "\n",
    "val_auc = roc_auc_score(labels.cpu().numpy(),preds.cpu().numpy())\n",
    "print(val_auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e56ee74b",
   "metadata": {},
   "source": [
    "0.7806176567186112"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48965f11",
   "metadata": {},
   "source": [
    "### 6，保存模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13bda74f",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model.net.state_dict(),\"best_fibinet.pt\")\n",
    "net_clone = create_net()\n",
    "net_clone.load_state_dict(torch.load(\"best_fibinet.pt\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7c343ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "net_clone.eval()\n",
    "preds = torch.cat([torch.sigmoid(net_clone(x[0])).data for x in dl_val]) \n",
    "labels = torch.cat([x[-1] for x in dl_val])\n",
    "\n",
    "val_auc = roc_auc_score(labels.cpu().numpy(),preds.cpu().numpy())\n",
    "print(val_auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f922378d",
   "metadata": {},
   "source": [
    "0.7806176567186112"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22370240",
   "metadata": {},
   "source": [
    "可以看到FiBiNET在验证集的AUC得分为0.7806,相比之下DeepFM的验证集AUC为0.7803。\n",
    "\n",
    "不能说纹丝不动, 只能说了涨了个蚊子腿大小肉的点。\n",
    "\n",
    "并且这是以较大地牺牲模型训练预测效率为代价的。\n",
    "\n",
    "DeepFM训练一个Epoch大约需要20s, 而FiBiNET训练一个Epoch需要大约2min.\n",
    "\n",
    "尽管如此, FiBiNET的结构设计依然是值得我们学习和借鉴的, 集神经网络结构设计三大主流高级技巧于一体, 闪烁着穿越时空的才华与智慧光芒。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b54f14a3",
   "metadata": {},
   "source": [
    "**如果本书对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果对本书内容理解上有需要进一步和作者交流的地方，欢迎在公众号\"算法美食屋\"下留言。作者时间和精力有限，会酌情予以回复。\n",
    "\n",
    "也可以在公众号后台回复关键字：**加群**，加入读者交流群和大家讨论。\n",
    "\n",
    "![算法美食屋logo.png](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,md",
   "main_language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}