强化学习/dqn-nature-notes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0 摘要"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "传统强化学习中，Agent面临着一些困难：\n",
    "- 从高维感知环境的输入中提取有效的环境表达（Representations of the environment）\n",
    "- 使用这些表达去总结概括过去经验，从而应对新的环境\n",
    "\n",
    "过去强化学习的应用受限于某些领域，例如可以人工构造特征的领域或者某些具有完全可观察的、低维的状态空间领域下。\n",
    "\n",
    "提出利用DNN去训练新的智能体Agent的方式，称为**（deep Q-network）**，能够使用端到端的强化学习方式直接从高维输入习得最优策略。\n",
    "\n",
    "DQN在Atari经典的2600个游戏上经过测试，取得了超越过去所有算法性能的效果。通过接收像素与游戏分数作为输入，可以取得非常好的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 不足与改进"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们假设Agent面临的任务是通过观察(observations)、动作(actions)以及奖励(rewards)的序列来与环境交互，即任务可抽象为以下序列：\n",
    "\n",
    "$$(s_0,a_0,r_1,s_1,a_1,\\dots,r_T,s_T,a_T)$$\n",
    "\n",
    "Agent目标是以某一种方式选择actions来最大化累积未来奖励(cumulative future reward)。本文中，作者使用深度卷积神经网络来近似最优的action-value function：\n",
    "\n",
    "$$Q^*(s,a)=\\max_\\pi\\mathbb{E}[r_t+\\gamma r_{t+1} + \\gamma^2 r_{t+2}+\\dots +|s_t=s, a_t=a, \\pi]$$\n",
    "\n",
    "当使用非线性激活函数近似来表达action-value(Q) function时，强化学习的表现是不稳定且容易发散的，原因如下：\n",
    "\n",
    "- 观察序列之间的相关性（序列之间不够独立，只要Q发生一点变化，就会影响到当前的action选择，从而影响后续的数据分布）\n",
    "- action-value(也称Q值)和目标值TD Target($r+\\gamma\\max_{a'}Q(s',a')$)之间具有相关性\n",
    "\n",
    "现存的方法例如neural fitted Q-iteration可以解决这个问题，但不够高效。因此作者提出两种新的关键idea来解决上述问题：\n",
    "\n",
    "- **经验回放(Experience Replay)**\n",
    "- **Q值的周期性迭代更新(Periodically Updated)**(或被称为Fixed Q-Targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 预处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 输入：210 × 160像素点，每个像素点的取值有128个\n",
    "2. 静态化图片。由于游戏中会有闪烁块，其本质上是通过在连续奇偶帧变换像素值实现闪烁，因此对连续的两帧(frame)图片在对应像素点位置取最大，获得静态图片；\n",
    "3. 提取图片亮度Y Channel（忽略颜色），并切割为84×84的大小\n",
    "4. 将当前帧前最近且连续的m个帧图像进行堆叠（实验中使用m=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 模型结构"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 输入：预处理后的84×84×4的image\n",
    "2. conv_layer_1: num_filter=32, size=(8, 8)，stride=4, activation=relu\n",
    "3. conv_layer_2: num_filter=64, size=(4, 4), stride=2, activation=relu\n",
    "4. conv_layer_3: num_filter=64, size=(3, 3), stride=1, activation=relu\n",
    "5. fc_layer: hidden_units=512, activation=relu\n",
    "6. output_layer: hidden_units=action_space.n, activation=None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://raw.githubusercontent.com/NELSONZHAO/paper-learning-and-notes/master/%E5%BC%BA%E5%8C%96%E5%AD%A6%E4%B9%A0/pictures/dqn-nature-1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 算法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Experience Replay\n",
    "\n",
    "将Agent多个episodes的经验存储为一个数据集$D_t$，每次训练网络时候从其中sample一个样本进行参数更新。\n",
    "\n",
    "好处：\n",
    "- 一条样本会被采样多次用来参数更新，数据效率增加\n",
    "- 由于序列样本之间的相关性，直接从原始序列样本中学习是低效率的；随机化采样可以避免相关性，减少更新参数的方差（？）\n",
    "- \n",
    "\n",
    "不足：\n",
    "- 由于采用固定大小的Replay Buffer，只保留最近N个Experience，Buffer中直接进行按顺序覆盖，因此不区分一些重要的experience\n",
    "- 其次采用均匀分布抽样experience也没有考虑一些重要的经验"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://raw.githubusercontent.com/NELSONZHAO/paper-learning-and-notes/master/%E5%BC%BA%E5%8C%96%E5%AD%A6%E4%B9%A0/pictures/dqn-natrue-2.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}