cs221_228_notes_prob_models_brief.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "State-based model\n",
    "  - Search Proplem/ MDP/ Adversarial games\n",
    "  \n",
    "Variable-based model\n",
    "  - Constraint satisfaction problems, e.g. 数独；/ Bayesian networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MDP, 即MP with T, reward, policy, discount, utility; [V,Q](http://web.stanford.edu/class/cs221/lectures/mdp1.pdf#page=42)； [policy evaluation: iter至一定error](http://web.stanford.edu/class/cs221/lectures/mdp1.pdf#page=52)；[Vopt](http://web.stanford.edu/class/cs221/lectures/mdp1.pdf#page=66)\n",
    "\n",
    "\n",
    "加强学习，unknown T, reward\n",
    "\n",
    "  - [Model-free Monte Carlo](http://web.stanford.edu/class/cs221/lectures/mdp2.pdf)：估计$Q_\\pi$\n",
    "    - $Q_\\pi$ 为u平均；等价形式更新，如同梯度下降；advanced $\\eta$ decay setting\n",
    "    - [SARSA](http://web.stanford.edu/class/cs221/lectures/mdp2.pdf#page=40)：对utility补充些许估计\n",
    "    \n",
    "  - [Q-learning](http://web.stanford.edu/class/cs221/lectures/mdp2.pdf#page=46)：估计$Q_{opt}$\n",
    "    - 与SARSA差异在于target中是$V_opt$为max形式\n",
    "    - Exploration/exploitation tradeoff\n",
    "    - epsilon-greedy：本质即以$\\epsilon$探索(random action)，$1-\\epsilon$开发(argmax)，自然地，$\\epsilon_t \\downarrow$；（Problem, real-world millions of states）\n",
    "    \n",
    "\n",
    "Models:\n",
    "  - LDA：如名描述，即由D分布隐变量生成词\n",
    "  \n",
    "  - BayesNet: DAG，想象timeline上许多因->...许多果->...的图结构\n",
    "  \n",
    "  - Markov Random Fields（MRFs）：无向图，P有cliques' factor形式，\n",
    "  \n",
    "  - CRF：上述形式P为条件形式y|x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "web\n",
    "\n",
    "[openai gym](https://gym.openai.com/)\n",
    "\n",
    "[cs228_notes, prob.graph.model](https://ermongroup.github.io/cs228-notes/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A brief intro to CRF/HMMs \n",
    "\n",
    "  - HMM, $p(y|x) \\propto p(x|y)p(y) = \\prod_{t=1}^T p(y_t|y_{t-1}) p(x_t|y_t) $. 想象HMM图\n",
    "  - log-linear model; 即有形式 $p_\\theta(z_c|x_c) \\propto \\exp(\\theta \\phi(x_c, z_c))$\n",
    "  - CRF, in-practice, $p_\\theta(y|x) \\propto \\prod_{t=1}^T  \\exp(\\theta^T f(y_t, y_{t-1}, x_t))$\n",
    "  - Viterbi's，即对于Sequence Tags, $S^T$, 要求最大posterior时，动态规划$V_{t,k}$ 前t个词以tag为k结尾的概率，这样每次\n",
    "    只用记录k个值，和对应的prev即可\n",
    "  - 对于CRF, max log， 有较好的形式[formula](http://www.cs.columbia.edu/~mcollins/crf.pdf#page=9)\n",
    "  - [BiLSTM CRF](https://arxiv.org/abs/1508.01991), [impl-example](https://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html#sphx-glr-beginner-nlp-advanced-tutorial-py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}