korean-ray-tutorial/01-Introduction-to-Reinforcement-Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ray RLlib - Introduction to Reinforcement Learning\n",
    "\n",
    "© 2019-2020, Anyscale. All Rights Reserved\n",
    "\n",
    "![Anyscale Academy](../images/AnyscaleAcademy_Logo_clearbanner_141x100.png)\n",
    "\n",
    "_Reinforcement Learning_ is the category of machine learning that focuses on training one or more _agents_ to achieve maximal _rewards_ while operating in an environment. This lesson discusses the core concepts of RL, while subsequent lessons explore RLlib in depth. We'll use two examples with exercises to give you a taste of RL. If you already understand RL concepts, you can either skim this lesson or skip to the [next lesson](02-Introduction-to-RLlib.ipynb).\n",
    "\n",
    "_강화학습_(이하 RL)은 환경과 상호작용하며 최대 보상을 달성하기 위해 하나 이상의 에이전트를 훈련시키는 데 초점을 맞춘 기계 학습 방법론 중 하나입니다. 이 수업은 RL의 핵심 개념을 살펴보고, RLlib를 깊이 있게 탐구합니다. 우리는 두 가지 예를 사용하여 RL을 체험해볼 것입니다. 만약 이미 RL 개념을 알고있다며, 이 부분은 건너뛸 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What Is Reinforcement Learning?\n",
    "\n",
    "Let's explore the basic concepts of RL, specifically the _Markov Decision Process_ abstraction, and to show its use in Python.\n",
    "\n",
    "RL의 기본 개념, 특히 _Markov Decision Process_ 개념을 탐구하고 Python에서 그 사용법을 봅시다.\n",
    "\n",
    "Consider the following image:\n",
    "\n",
    "![RL Concepts](../images/rllib/RL-concepts.png)\n",
    "\n",
    "In RL, one or more **agents** interact with an **environment** to maximize a **reward**. The agents make **observations** about the **state** of the environment and take **actions** that are believed will maximize the long-term reward. However, at any particular moment, the agents can only observe the immediate reward. So, the training process usually involves lots and lot of replay of the game, the robot simulator traversing a virtual space, etc., so the agents can learn from repeated trials what decisions/actions work best to maximize the long-term, cumulative reward.\n",
    "\n",
    "RL에서 하나 이상의 **agent**는 **environment**과 상호작용하여 **reward**을 최대화 합니다. agent는 environment의 **state**에 대해 **observations**을 하고, 장기 reward을 극대화할 수 있는 **policy**를 취합니다. 그러나 agent들은 어떤 특정한 순간의 즉각적인 reward만을 알 수 있습니다. 따라서 training 과정은 대개 게임의 많은 재생, 가상공간에서의 로봇 시뮬레이터 등을 수반하므로, agent는 장기적인 누적 reward을 극대화하기 위해 어떤 decision/action이 가장 잘 작동하는지 반복적인 실험을 통해 배울 수 있습니다.\n",
    "\n",
    "---\n",
    "\n",
    "The trail and error search and delayed reward are the distinguishing characterists of RL vs. other ML methods ([Sutton 2018](06-RL-References.ipynb#Books)).\n",
    "\n",
    "The way to formalize trial and error is the **exploitation vs. exploration tradeoff**. When an agent finds what appears to be a \"rewarding\" sequence of actions, the agent may naturally want to continue to **exploit** these actions. However, even better actions may exist. An agent won't know whether alternatives are better or not unless some percentage of actions taken **explore** the alternatives. So, all RL algorithms include a strategy for exploitation and exploration.\n",
    "\n",
    "**1)trail and error search** 및 **2)delayed reward**은 RL을 다른 ML 방법론들과 구별할 수 있는 특징입니다.\n",
    "\n",
    "trail and error search 방법은 **exploitation vs. exploration tradeoff**로 나타낼 수 있습니다. agent가 reward를 주는 일련의 action들을 발견하면, agent는 자연스럽게 이러한 action들을 계속 **exploit**하기를 원할 수 있습니다. 하지만, 더 나은 action들이 존재할 수도 있습니다. agent는 지금 찾은 대안이 더 좋은지를 어느정도 다른 대안들을 explore하지 않는 한 알 수 없습니다. 따라서 모든 RL 알고리즘에는 적절한 exploitation과 exploration을 하기 위한 전략이 포함됩니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RL Applications\n",
    "\n",
    "RL has many potential applications. RL became \"famous\" due to these successes, including achieving expert game play, training robots, autonomous vehicles, and other simulated agents:\n",
    "\n",
    "RL은 다양한 분야에 활용될 잠재성을 가지고 있습니다. RL로 전문가처럼 게임 플레이를 하고, 훈련 로봇, 자율 차량 및 기타 시뮬레이션에서 잘 작동하는 agent들을 만들어내는 성공들로 인해 유명해졌습니다.\n",
    "\n",
    "![AlphaGo](../images/rllib/alpha-go.jpg)\n",
    "![Game](../images/rllib/breakout.png)\n",
    "\n",
    "![Stacking Legos with Sawyer](../images/rllib/stacking-legos-with-sawyer.gif)\n",
    "![Walking Man](../images/rllib/walking-man.gif)\n",
    "\n",
    "![Autonomous Vehicle](../images/rllib/daimler-autonomous-car.jpg)\n",
    "![\"Cassie\": Two-legged Robot](../images/rllib/cassie-crouched.png)\n",
    "\n",
    "Credits:\n",
    "* [AlphaGo](https://www.youtube.com/watch?v=l7ngy56GY6k)\n",
    "* [Breakout](https://towardsdatascience.com/tutorial-double-deep-q-learning-with-dueling-network-architectures-4c1b3fb7f756) ([paper](https://arxiv.org/abs/1312.5602))\n",
    "* [Stacking Legos with Sawyer](https://robohub.org/soft-actor-critic-deep-reinforcement-learning-with-real-world-robots/)\n",
    "* [Walking Man](https://openai.com/blog/openai-baselines-ppo/)\n",
    "* [Autonomous Vehicle](https://www.daimler.com/innovation/case/autonomous/intelligent-drive-2.html)\n",
    "* [\"Cassie\": Two-legged Robot](https://mime.oregonstate.edu/research/drl/robots/cassie/) (Uses Ray!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recently other industry applications have emerged, include the following:\n",
    "\n",
    "* **Process optimization:** industrial processes (factories, pipelines) and other business processes, routing problems, cluster optimization.\n",
    "* **Ad serving and recommendations:** Some of the traditional methods, including _collaborative filtering_, are hard to scale for very large data sets. RL systems are being developed to do an effective job more efficiently than traditional methods.\n",
    "* **Finance:** Markets are time-oriented _environments_ where automated trading systems are the _agents_. \n",
    "\n",
    "최근에 다른 산업들에서도 RL을 적용될 수 있는 분야들이 생겨나고 있으며 아래와 같은 분야들을 포함합니다.\n",
    "\n",
    "* **Process optimization:** 산업 프로세스 (공장, 파이프 라인) 및 기타 비즈니스 프로세스, 라우팅 문제, 클러스터 최적화에 대해 연구하고 있습니다.\n",
    "* **Ad serving and recommendations:** _colaborative filtering_ 을 포함한 전통적인 방법 중 일부는 매우 큰 데이터 세트에 대해 스케일링하기가 어렵습니다. RL 시스템은 기존의 방법보다 효과적인 작업을 하기 위해 개발되고 있습니다.\n",
    "* **Finance:** 시장은 시간 의존적인 _environments_ 이며, 자동화된 거래 시스템 _agent_ 를 연구합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hideCode": false,
    "hidePrompt": false,
    "id": "bAptEafKNlhm"
   },
   "source": [
    "## Markov Decision Processes\n",
    "\n",
    "At its core, Reinforcement learning builds on the concepts of [Markov Decision Process (MDP)](https://en.wikipedia.org/wiki/Markov_decision_process), where the current state, the possible actions that can be taken, and overall goal are the building blocks.\n",
    "\n",
    "An MDP models sequential interactions with an external environment. It consists of the following:\n",
    "\n",
    "- a **state space** where the current state of the system is sometimes called the **context**.\n",
    "- a set of **actions** that can be taken at a particular state $s$ (or sometimes the same set for all states).\n",
    "- a **transition function** that describes the probability of being in a state $s'$ at time $t+1$ given that the MDP was in state $s$ at time $t$ and action $a$ was taken. The next state is selected stochastically based on these probabilities.\n",
    "- a **reward function**, which determines the reward received at time $t$ following action $a$, based on the decision of **policy** $\\pi$.\n",
    "\n",
    "---\n",
    "\n",
    "현재 상태(current state), 취할 수 있는 가능한 actions, 전반적인 목표(overall goal)가 구성 요소인 마르코프 의사결정 과정(MDP)의 개념을 바탕으로 RL이 구축된다. MDP는 외부 environment과의 순차적 상호작용을 모델로 한다. 아래는 MDP의 구성 요소들 입니다.\n",
    "- 시스템의 현재 상태를 (**context**라고 부르기도 하는) **state space**\n",
    "- 특정 state $s$ (또는 모든 states에서 동일한 세트)에서 선택할 수 있는 **actions**\n",
    "- 시간 $t$에 agent가 state $s$에 있고 action $a$d을 선택했을 때, 시간 $t+1$에 state $s'$에 있을 확률을 알려주는 함수 **transition function**. 다음 state $s'$는 이러한 확률에 기초하여 확률적으로 선택됩니다.\n",
    "- agent가 **policy** $\\pi$의 결정에 따라 시간 $t$에 action $a$을 선택하고 받을 reward을 결정하는 **reward function**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hideCode": false,
    "hidePrompt": false,
    "id": "bAptEafKNlhm"
   },
   "source": [
    "---\n",
    "The goal of MDP is to develop a **policy** $\\pi$ that specifies what action $a$ should be chosen for a given state $s$ so that the cumulative reward is maximized. When it is possible for the policy \"trainer\" to fully observe all the possible states, actions, and rewards, it can define a deterministic policy, fixing a single action choice for each state. In this scenario, the transition probabilities reduce to the probability of transitioning to state $s'$ given the current state is $s$, independent of actions, because the state now leads to a deterministic action choice. Various algorithms can be used to compute this policy. \n",
    "\n",
    "MDP의 목표는 누적 reward가 극대화 되도록 주어진 state에 대해 어떤 action를 선택해야 하는지를 알려주는 **policy**을 만드는 것입니다. policy \"trainer\"가 가능한 모든 state, action 및 reward을 완전히 관찰할 수 있을 때 각 state에 대한 단일 action 선택을 할 수 있는 결정론적(deterministic) policy을 정의할 수 있습니다. \n",
    "이러한 시나리오에서 state에 따라 확정적인 action을 선택하게 되기 때문에, action과는 독립적으로 transition probabilities은 현재 state $s$에서 다음 state $s'$로 전환되는 확률로 수렴합니다. 이 policy를 구하는 데 다양한 알고리즘을 사용할 수 있습니다.\n",
    "\n",
    "---\n",
    "Put another way, if the policy isn't deterministic, then the transition probability to state $s'$ at a time $t+1$ when action $a$ is taken for state $s$ at time $t$, is given by:\n",
    "\n",
    "\\begin{equation}\n",
    "P_a(s',s) = P(s_{t+1} = s'|s_t=s,a)\n",
    "\\end{equation}\n",
    "\n",
    "When the policy is deterministic, this transition probability reduces to the following, independent of $a$:\n",
    "\n",
    "\\begin{equation}\n",
    "P(s',s) = P(s_{t+1} = s'|s_t=s)\n",
    "\\end{equation}\n",
    "\n",
    "To be clear, a deterministic policy means that one and only one action will always be selected for a given state $s$, but the next state $s'$ will still be selected stochastically.\n",
    "\n",
    "In the general case of RL, it isn't possible to fully know all this information, some of which might be hidden and evolving, so it isn't possible to specify a fully-deterministic policy.\n",
    "\n",
    "다른 방법으로, policy가 결정론적이 아니라면 action $a$를 선택하고 시간 $t+1$에 state $s'$이 될 확률, 즉 transition probability는 다음과 같이 나타낼 수 있습니다.\n",
    "\n",
    "\\begin{equation}\n",
    "P_a(s',s) = P(s_{t+1} = s'|s_t=s,a)\n",
    "\\end{equation}\n",
    "\n",
    "만약 policy가 결정론적이라면, transition probability은 action $a$와 무관하게 아래와 같이 나타내질 수 있습니다.\n",
    "\n",
    "\\begin{equation}\n",
    "P(s',s) = P(s_{t+1} = s'|s_t=s)\n",
    "\\end{equation}\n",
    "\n",
    "분명히, 결정론적 policy는 주어진 state에 대해 오직 하나의 action만 선택되고 다음 state는 여전히 확률 적으로 선택된다는 것을 의미합니다.\n",
    "\n",
    "RL의 일반적인 경우, 이 모든 정보를 완전히 알 수는 없으며 그 중 일부는 숨겨져 있고 변화할 수 있으므로 완전히 결정론적 policy을 정의할 수 없습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hideCode": false,
    "hidePrompt": false,
    "id": "bAptEafKNlhm"
   },
   "source": [
    "---\n",
    "Often this cumulative reward is computed using the **discounted sum** over all rewards observed:\n",
    "\n",
    "\\begin{equation}\n",
    "\\arg\\max_{\\pi} \\sum_{t=1}^T \\gamma^t R_t(\\pi),\n",
    "\\end{equation}\n",
    "\n",
    "where $T$ is the number of steps taken in the MDP (this is a random variable and may depend on $\\pi$), $R_t$ is the reward received at time $t$ (also a random variable which depends on $\\pi$), and $\\gamma$ is the **discount factor**. The value of $\\gamma$ is between 0 and 1, meaning it has the effect of \"discounting\" earlier rewards vs. more recent rewards. \n",
    "\n",
    "The [Wikipedia page on MDP](https://en.wikipedia.org/wiki/Markov_decision_process) provides more details. Note what we said in the third bullet, that the new state only depends on the previous state and the action taken. The assumption is that we can simplify our effort by ignoring all the previous states except the last one and still achieve good results. This is known as the [Markov property](https://en.wikipedia.org/wiki/Markov_property). This assumption often works well and it greatly reduces the resources required.\n",
    "\n",
    "이 누적 reward는 자주 관찰된 모든 reward에 대해 할인된 합계를 사용하여 계산됩니다.\n",
    "\n",
    "\\begin{equation}\n",
    "\\arg\\max_{\\pi} \\sum_{t=1}^T \\gamma^t R_t(\\pi),\n",
    "\\end{equation}\n",
    "\n",
    "여기서 T는 MDP에서 취한 step의 수(이것은 무작위 변수이며 $\\pi$에 의존할 수 있음), $R_t$는 시간 t에서 받은 reward(또한 $\\pi$에 의존하는 무작위 변수), $\\gamma$는 **discount factor**이다. $\\gamma$ 값은 0에서 1 사이이며, 이는 초기 reward 대 최근의 reward을 \"할인\"하는 효과를 가지고 있습니다.\n",
    "\n",
    "[MDP의 위키피디아 페이지](https://en.wikipedia.org/wiki/Markov_decision_process)는 더 많은 세부사항을 제공합니다. 우리가 세 번째 bullet에서 한 말에서 보면, 새로운 state는 이전 state와 선택한 action에만 달려 있다. 마지막 state를 제외한 모든 이전 state를 무시하고 여전히 좋은 결과를 달성함으로써 우리의 노력을 단순화할 수 있다는 [Markov property](https://en.wikipedia.org/wiki/Markov_property) 가정이다. 이 가정은 종종 잘 작동하며 필요한 자원을 크게 감소시킵니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hideCode": false,
    "hidePrompt": false,
    "id": "bAptEafKNlhm"
   },
   "source": [
    "## The Elements of RL\n",
    "\n",
    "Here are the elements of RL that expand on MDP concepts (see [Sutton 2018](https://mitpress.mit.edu/books/reinforcement-learning-second-edition) for more details):\n",
    "\n",
    "여기 MDP 개념에서 확장되는 RL의 요소가 있습니다.\n",
    "\n",
    "#### Policies\n",
    "\n",
    "Unlike MDP, the **transition function** probabilities are often not known in advance, but must be learned. Learning is done through repeated \"play\", where the agent interacts with the environment.\n",
    "\n",
    "This makes the **policy** $\\pi$ harder to determine. Because the fully state space usually can't be fully known, the choice of action $a$ for given state $s$ almostly always remains a stochastic choice, never deterministic, unlike MDP.\n",
    "\n",
    "MDP와 달리 **transition function**에서 나오는 확률은 사전에 알려지지 않은 경우가 많지만 반드시 배워야 합니다. 학습은 agent가 environment와 상호작용하는 반복적인 \"play\"를 통해 이루어집니다.\n",
    "\n",
    "이것은 **policy**를 결정하기 어렵게 만듭니다. 완전한 state 공간은 일반적으로 완전히 알 수 없기 때문에 주어진 state에 대한 action 선택은 거의 항상 (MDP와 달리)결정론적이 아닌 확률적 선택으로 남아있습니다.\n",
    "\n",
    "#### Reward Signal\n",
    "\n",
    "The idea of a **reward signal** encapsulates the desired goal for the system and provides feedback for updating the policy based on how well particular events or actions contribute rewards towards the goal.\n",
    "\n",
    "**reward signal**는 시스템에 대한 원하는 목표를 요약하고 특정 사건이나 action이 목표에 대한 reward에 얼마나 잘 기여하는지에 따라 policy를 업데이트하기 위한 피드백을 제공합니다.\n",
    "\n",
    "#### Value Function\n",
    "\n",
    "The **value function** encapsulates the maximum cumulative reward likely to be achieved starting from a given state for an **episode**. This is harder to determine than the simple reward returned after taking an action. In fact, much of the research in RL over the decades has focused on finding better and more efficient implementations of value functions. To illustrate the challenge, repeatedly taking one sequence of actions may yield low rewards for a while, but eventually provide large rewards. Conversely, always choosing a different sequence of actions may yield a good reward at each step, but be suboptimal for the cumulative reward.\n",
    "\n",
    "**value function**는 **episode**에서 주어진 state에서 시작하여 달성될 가능성이 있는 최대 누적 reward을 요약합니다. 이것은 action를 취한 후에 반환된 단순한 reward보다 결정하기 어렵습니다. 사실, 수십 년 동안 RL의 많은 연구는 value function의 더 좋고 효율적인 구현을 찾는 데 초점을 맞추고 있습니다. 한 가지 도전을 설명해보면, 한 가지 일련의 action을 반복적으로 취하면 잠시 동안은 낮은 reward을 얻을 수 있지만, 결국 큰 reward을 제공할 수 있습니다. 반대로, 항상 다른 일련의 action을 선택하면 각 단계에서 좋은 reward을 얻을 수 있지만 누적된 reward에 대해서는 최적이 아닙니다.\n",
    "\n",
    "#### Episode\n",
    "\n",
    "A sequence of steps by the agent starting in an initial state. At each step, the agent observes the current state, chooses the next action, and receives the new reward. Episodes are used for both training policies and replaying with an existing policy (called _rollout_).\n",
    "\n",
    "초기 state에서 시작하는 agent에 의한 일련의 step들을 말합니다. 각 단계에서 agent는 현재 state를 관찰하고 다음 action을 선택하고 새로운 reward를 받습니다. 에피소드는 training policies과 existing policy으로 다시 play하는 데 모두 사용됩니다(_rollout_라고 함).\n",
    "\n",
    "#### Model\n",
    "\n",
    "An optional feature, some RL algorithms develop or use a **model** of the environment to anticipate the resulting states and rewards for future actions. Hence, they are useful for _planning_ scenarios. Methods for solving RL problems that use models are called _model-based methods_, while methods that learn by trial and error are called _model-free methods_.\n",
    "\n",
    "선택적인 특징으로, 일부 RL 알고리즘은 미래의 action에 대한 결과 state와 reward을 예상하기 위해 env의 **model**을 개발하거나 사용합니다. 따라서, 그것들은 _planning_ 시나리오에 유용합니다. model을 사용하는 RL 문제를 해결하기 위한 방법을 _model 기반 방법_ 이라고 하며 시행착오로 학습하는 메소드를 _model-free 방법_ 이라고 합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hideCode": false,
    "hidePrompt": false,
    "id": "bAptEafKNlhm"
   },
   "source": [
    "## Reinforcement Learning Example\n",
    "\n",
    "Let's finish this introduction let's learn about the popular \"hello world\" (1) example environment for RL, balancing a pole vertically on a moving cart, called `CartPole`. Then we'll see how to use RLlib to train a policy using a popular RL algorithm, _Proximal Policy Optimization_, again using `CartPole`.\n",
    "\n",
    "(1) In books and tutorials on programming languages, it is a tradition that the very first program shown prints the message \"Hello World!\".\n",
    "\n",
    "이 소개를 끝내고 인기 있는 \"hello world\" RL의 환경을 예로 들어보겠습니다. 카트폴이라는 움직이는 카트에 수직으로 폴을 균형을 맞추는 문제입니다. 그러고 나서 대중적인 RL 알고리즘인 _Proximal Policy Optimization_ 를 사용하여 policy을 훈련시키기 위해 RLlib을 사용하는 방법을 다시 'CartPole'을 사용하여 볼 것입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hideCode": false,
    "hidePrompt": false,
    "id": "bAptEafKNlhm"
   },
   "source": [
    "### CartPole and OpenAI\n",
    "\n",
    "The popular [OpenAI \"gym\" environment](https://gym.openai.com/) provides MDP interfaces to a variety of simulated environments. Perhaps the most popular for learning RL is `CartPole`, a simple environment that simulates the physics of balancing a pole on a moving cart. The `CartPole` problem is described at https://gym.openai.com/envs/CartPole-v1. Here is an image from that website, where the pole is currently falling to the right, which means the cart will need to move to the right to restore balance:\n",
    "\n",
    "인기있는 OpenAI \"gym\" 환경은 다양한 시뮬레이션 환경에 MDP 인터페이스를 제공합니다. 아마도 RL을 배우는 데 가장 인기 있는 것은 움직이는 카트에 장대를 매는 물리학을 시뮬레이션하는 간단한 환경인 카트폴일 것입니다. 카트폴 문제는 https://gym.openai.com/envs/CartPole-v1 에서 더 자세히 설명하고 있습니다. 다음은 현재 폴이 오른쪽으로 떨어지고 있는 웹 사이트의 이미지입니다. 즉, 카트가 균형을 회복하기 위해 오른쪽으로 이동해야 합니다.\n",
    "\n",
    "![Cart Pole](../images/rllib/Cart-Pole.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hideCode": false,
    "hidePrompt": false,
    "id": "bAptEafKNlhm"
   },
   "source": [
    "This example fits into the MDP framework as follows:\n",
    "- The **state** consists of the position and velocity of the cart (moving in one dimension from left to right) as well as the angle and angular velocity of the pole that is balancing on the cart.\n",
    "- The **actions** are to decrease or increase the cart's velocity by one unit. A negative velocity means it is moving to the left.\n",
    "- The **transition function** is deterministic and is determined by simulating physical laws. Specifically, for a given **state**, what should we choose as the next velocity value? In the RL context, the correct velocity value to choose has to be learned. Hence, we learn a _policy_ that approximates the optimal transition function that could be calculated from the laws of physics.\n",
    "- The **reward function** is a constant 1 as long as the pole is upright, and 0 once the pole has fallen over. Therefore, maximizing the reward means balancing the pole for as long as possible.\n",
    "- The **discount factor** in this case can be taken to be 1, meaning we treat the rewards at all time steps equally and don't discount any of them.\n",
    "\n",
    "More information about the `gym` Python module is available at https://gym.openai.com/. The list of all the available Gym environments is in [this wiki page](https://github.com/openai/gym/wiki/Table-of-environments). We'll use a few more of them and even create our own in subsequent lessons.\n",
    "\n",
    "이 예는 다음과 같이 MDP 프레임 워크에 적합합니다.\n",
    "\n",
    "- **state**는 카트의 위치와 속도(좌에서 우로 한 차원으로 이동)와 카트에서 균형을 이루고 있는 폴의 각도와 각속도로 구성됩니다.\n",
    "- **actions**은 카트의 속도를 한 단위로 줄이거나 증가시키는 것입니다. 음수는 그것이 왼쪽으로 움직이는 것을 의미합니다.\n",
    "- **transition function**는 결정론적이며 물리적 법칙을 시뮬레이션하여 결정됩니다. 구체적으로, 주어진 **state**의 경우, 다음 속도 값으로 무엇을 선택해야 하는가?에 대한 내용입니다. RL에서 선택해야 할 정확한 속도 값을 배워야 합니다. 따라서 물리학 법칙에서 계산할 수 있는 최적의 transition function를 근사화하는 _policy_ 를 배웁니다.\n",
    "- **reward function**는 pole이 서있을 때 상수 1이고,pole이 넘어지면 0입니다. 따라서 reward을 극대화한다는 것은 가능한 한 오랫동안 pole의 균형을 잡는 것을 의미합니다.\n",
    "- 이 경우 **discount factor**는 1로 간주할 수 있습니다. 즉, 우리는 항상 보상을 동등하게 취급하고 그 중 어느 것도 할인하지 않습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "a9Kwo5ZfNlhn"
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WPpofaxQNlhp"
   },
   "source": [
    "The code below illustrates how to create and manipulate MDPs in Python. An MDP can be created by calling `gym.make`. Gym environments are identified by names like `CartPole-v1`. A **catalog of built-in environments** can be found at https://gym.openai.com/envs.\n",
    "\n",
    "아래 코드는 파이썬에서 MDP를 만들고 조작하는 방법을 보여줍니다. MDP는 `gym.make`로 만들어질 수 있습니다. Gym environments은 `CartPole-v1` 같은 이름으로 구분됩니다. **catalog of built-in environments**는 https://gym.openai.com/envs에서 찾을 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "hideCode": false,
    "hidePrompt": false,
    "id": "6DZ68SG9Nlhp",
    "outputId": "293be60b-8107-42f2-c54a-58f3eaf295f2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created env: <TimeLimit<CartPoleEnv<CartPole-v1>>>\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('CartPole-v1')\n",
    "print('Created env:', env)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Xn5PqgDzNlhr"
   },
   "source": [
    "Reset the state of the MDP by calling `env.reset()`. This call returns the initial state of the MDP.\n",
    "\n",
    "`env.reset()`로 MDP의 상태를 reset합니다. MDP의 초기 state를 불러옵니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "hideCode": false,
    "hidePrompt": false,
    "id": "zRA58dOFNlhs",
    "outputId": "7aba4eac-fb0f-4654-eb49-0fbd6d664f28"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The starting state is: [ 0.03830547  0.00469169  0.01894432 -0.04509305]\n"
     ]
    }
   ],
   "source": [
    "state = env.reset()\n",
    "print('The starting state is:', state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the state is the position of the cart, its velocity, the angle of the pole, and the angular velocity of the pole.\n",
    "\n",
    "state는 카트의 위치, 속도, 극의 각도 및 극의 각속도임을 기억하십시오."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8MuXXesWNlhu"
   },
   "source": [
    "The `env.step` method takes an action. In the case of the `CartPole` environment, the appropriate actions are 0 or 1, for pushing the cart to the left or right, respectively. `env.step()` returns a tuple of four things:\n",
    "1. the new state of the environment\n",
    "2. a reward\n",
    "3. a boolean indicating whether the simulation has finished\n",
    "4. a dictionary of miscellaneous extra information\n",
    "\n",
    "Let's show what happens if we take one step with an action of 0.\n",
    "\n",
    "`env.step`의 방법은 action을 받습니다. 카트폴 환경의 경우 카트를 좌우로 밀어내는 데 필요한 action는 0이나 1입니다. `env.step()`는 4개의 요소를 가지고 있는 튜플을 반환합니다.\n",
    "1. the new state of the environment\n",
    "2. a reward\n",
    "3. a boolean indicating whether the simulation has finished\n",
    "4. a dictionary of miscellaneous extra information\n",
    "\n",
    "0의 동작으로만 한 step가면 어떻게 되는지 봅시다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "hideCode": false,
    "hidePrompt": false,
    "id": "TufVaMz_Nlhu",
    "outputId": "920b9758-7d85-49e8-f8ef-4586b6947dea"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.0383993  -0.19069672  0.01804246  0.25350626] 1.0 False {}\n"
     ]
    }
   ],
   "source": [
    "action = 0\n",
    "state, reward, done, info = env.step(action)\n",
    "print(state, reward, done, info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uBIoIuWYNlhw"
   },
   "source": [
    "A **rollout** is a simulation of a policy in an environment. It is used both during training and when running simulations with a trained policy. \n",
    "\n",
    "The code below performs a rollout in a given environment. It takes **random actions** until the simulation has finished and returns the cumulative reward.\n",
    "\n",
    "**rollout**은 environment에서 policy을 시뮬레이션하는 것입니다. 훈련 중일 때 및 훈련된 policy으로 시뮬레이션을 실행할 때 모두 사용됩니다.\n",
    "\n",
    "아래 코드는 주어진 env에서 rollout을 수행합니다. 시뮬레이션이 완료되고 누적 reward을 반환할 때까지 **random actions**이 필요합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "Zp00mr88Nlhw",
    "outputId": "f0d01977-00c9-4ad2-931a-7f9730b5b005"
   },
   "outputs": [],
   "source": [
    "def random_rollout(env):\n",
    "    state = env.reset()\n",
    "    \n",
    "    done = False\n",
    "    cumulative_reward = 0\n",
    "\n",
    "    # Keep looping as long as the simulation has not finished.\n",
    "    while not done:\n",
    "        # Choose a random action (either 0 or 1).\n",
    "        action = np.random.choice([0, 1])\n",
    "        \n",
    "        # Take the action in the environment.\n",
    "        state, reward, done, _ = env.step(action)\n",
    "        \n",
    "        # Update the cumulative reward.\n",
    "        cumulative_reward += reward\n",
    "    \n",
    "    # Return the cumulative reward.\n",
    "    return cumulative_reward    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try rerunning the following cell a few times. How much do the answers change? Note that the maximum possible reward for `CartPole` is 500. You'll probably get numbers under 200.\n",
    "\n",
    "다음 셀을 몇 번 다시 실행해 보십시오. 답은 얼마나 변하는가? 카트폴의 최대 reward은 500달러다. 아마 200 미만의 숫자를 얻을 수 있을 겁니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "Zp00mr88Nlhw",
    "outputId": "f0d01977-00c9-4ad2-931a-7f9730b5b005"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20.0\n",
      "23.0\n"
     ]
    }
   ],
   "source": [
    "reward = random_rollout(env)\n",
    "print(reward)\n",
    "reward = random_rollout(env)\n",
    "print(reward)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i3FVvEJRNlhy"
   },
   "source": [
    "### Exercise 1\n",
    "\n",
    "Choosing actions at random in `random_rollout` is not a very effective policy, as the previous results showed. Finish implementing the `rollout_policy` function below, which takes an environment *and* a policy. Recall that the *policy* is a function that takes in a *state* and returns an *action*. The main difference is that instead of choosing a **random action**, like we just did (with poor results), the action should be chosen **with the policy** (as a function of the state).\n",
    "\n",
    "> **Note:** Exercise solutions for this tutorial can be found [here](solutions/Ray-RLlib-Solutions.ipynb).\n",
    "\n",
    "앞선 결과에서 알 수 있듯이 무작위로 `random_rollout`에서 action을 선택하는 것은 그다지 효과적인 policy가 아닙니다. 아래의 `rollout_policy` 기능을 실행하여 env과 policy을 취합니다. *policy*은 *state*를 취하여 *action*을 반환하는 함수라는 것을 기억하세요. 차이점은 우리가 방금 했던 것처럼 **random action**을 선택하는 대신 **with the policy**으로 액션을 선택해야 한다는 것입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 375
    },
    "colab_type": "code",
    "id": "9PgmkROqNlhy",
    "outputId": "91445278-aeb7-4e86-e1b7-c92e5ba830a2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The first sample policy got an average reward of 9.5.\n",
      "The second sample policy got an average reward of 28.27.\n"
     ]
    }
   ],
   "source": [
    "def rollout_policy(env, policy):\n",
    "    state = env.reset()\n",
    "    \n",
    "    done = False\n",
    "    cumulative_reward = 0\n",
    "\n",
    "    # EXERCISE: Fill out this function by copying the appropriate part of 'random_rollout'\n",
    "    # and modifying it to choose the action using the policy.\n",
    "    # --------------------------------------------------------------------------\n",
    "    while not done:\n",
    "        # Choose a action using the policy\n",
    "        action = policy(state)\n",
    "        \n",
    "        # Take the action in the environment.\n",
    "        state, reward, done, _ = env.step(action)\n",
    "        \n",
    "        # Update the cumulative reward.\n",
    "        cumulative_reward += reward\n",
    "\n",
    "    # raise NotImplementedError\n",
    "    # --------------------------------------------------------------------------\n",
    "\n",
    "    # Return the cumulative reward.\n",
    "    return cumulative_reward\n",
    "\n",
    "def sample_policy1(state):\n",
    "    return 0 if state[0] < 0 else 1\n",
    "\n",
    "def sample_policy2(state):\n",
    "    return 1 if state[0] < 0 else 0\n",
    "\n",
    "reward1 = np.mean([rollout_policy(env, sample_policy1) for _ in range(100)])\n",
    "reward2 = np.mean([rollout_policy(env, sample_policy2) for _ in range(100)])\n",
    "\n",
    "print('The first sample policy got an average reward of {}.'.format(reward1))\n",
    "print('The second sample policy got an average reward of {}.'.format(reward2))\n",
    "\n",
    "assert 5 < reward1 < 15, ('Make sure that rollout_policy computes the action '\n",
    "                          'by applying the policy to the state.')\n",
    "assert 25 < reward2 < 35, ('Make sure that rollout_policy computes the action '\n",
    "                           'by applying the policy to the state.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll return to `CartPole` in lesson [01: Application Cart Pole](explore-rllib/01-Application-Cart-Pole.ipynb) in the `explore-rllib` section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qXQ8hIB9Nlh0"
   },
   "source": [
    "### RLlib Reinforcement Learning Example: Cart Pole with Proximal Policy Optimization\n",
    "\n",
    "This section demonstrates how to use the _proximal policy optimization_ (PPO) algorithm implemented by [RLlib](http://rllib.io). PPO is a popular way to develop a policy. RLlib also uses [Ray Tune](http://tune.io), the Ray Hyperparameter Tuning framework, which is covered in the [Ray Tune Tutorial](../ray-tune/00-Ray-Tune-Overview.ipynb).\n",
    "\n",
    "We'll provide relatively little explanation of **RLlib** concepts for now, but explore them in greater depth in subsequent lessons. For more on RLlib, see the documentation at http://rllib.io."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qXQ8hIB9Nlh0"
   },
   "source": [
    "PPO is described in detail in [this paper](https://arxiv.org/abs/1707.06347). It is a variant of _Trust Region Policy Optimization_ (TRPO) described in [this earlier paper](https://arxiv.org/abs/1502.05477). [This OpenAI post](https://openai.com/blog/openai-baselines-ppo/) provides a more accessible introduction to PPO.\n",
    "\n",
    "PPO works in two phases. In the first phase, a large number of rollouts are performed in parallel. The rollouts are then aggregated on the driver and a surrogate optimization objective is defined based on those rollouts. In the second phase, we use SGD (_stochastic gradient descent_) to find the policy that maximizes that objective with a penalty term for diverging too much from the current policy.\n",
    "\n",
    "![PPO](../images/rllib/ppo.png)\n",
    "\n",
    "> **NOTE:** The SGD optimization step is best performed in a data-parallel manner over multiple GPUs. This is exposed through the `num_gpus` field of the `config` dictionary. Hence, for normal usage, one or more GPUs is recommended.\n",
    "\n",
    "(The original version of this example can be found [here](https://raw.githubusercontent.com/ucbrise/risecamp/risecamp2018/ray/tutorial/rllib_exercises/)).\n",
    "\n",
    "PPO는 TRPO의 변형 알고리즘이다. 총 2단계로 진행되는데, 첫번째 단계에서 많은 rollout들이 동시에 수행된다. 이 rollout들이 드라이버에서 모아지고 이 rollout들을 기반으로 surrogate(대리) 최적화가 정의된다. 그 다음 단계에서 SGD를 가지고 목적함수를 최대화 시키는 정책을 찾는다. 이때 목적함수에는 현재 정책과 너무 다를(=발산하는) 경우 패널티를 주는 항이 추가되어 있다.\n",
    "\n",
    "> 여러개 GPU로 이루어지는 데이터 병렬과정에서는 SGD가 최고의 퍼포먼스를 나타낸다. 이것은 `config`에 있는 `num_gpus`로 확인할 수 있다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XwPnR2ibNlh2"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "lz4 not available, disabling sample compression. This will significantly impact RLlib performance. To install lz4, run `pip install lz4`.\n"
     ]
    }
   ],
   "source": [
    "# import gym  # imported above already, but listed here for completeness\n",
    "import ray\n",
    "from ray.rllib.agents.ppo import PPOTrainer, DEFAULT_CONFIG\n",
    "from ray.tune.logger import pretty_print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following script checks if the Ray cluster is already running. If not, it tells you what to do to start Ray."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!../tools/start-ray.sh --check --verbose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tQFzEX2BNlh3"
   },
   "source": [
    "Now start Ray in this \"driver\" process. This must be done before we instantiate any RL agents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tQFzEX2BNlh3"
   },
   "outputs": [],
   "source": [
    "# ray.init(address='auto', ignore_reinit_error=True, log_to_driver=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-08-23 14:03:57,645\tINFO resource_spec.py:231 -- Starting Ray with 3.52 GiB memory available for workers and up to 1.78 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).\n",
      "2020-08-23 14:03:58,292\tINFO services.py:1193 -- View the Ray dashboard at \u001b[1m\u001b[32mlocalhost:8265\u001b[39m\u001b[22m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'node_ip_address': '192.168.219.132',\n",
       " 'raylet_ip_address': '192.168.219.132',\n",
       " 'redis_address': '192.168.219.132:6379',\n",
       " 'object_store_address': '/tmp/ray/session_2020-08-23_14-03-57_640839_3701/sockets/plasma_store',\n",
       " 'raylet_socket_name': '/tmp/ray/session_2020-08-23_14-03-57_640839_3701/sockets/raylet',\n",
       " 'webui_url': 'localhost:8265',\n",
       " 'session_dir': '/tmp/ray/session_2020-08-23_14-03-57_640839_3701'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ray.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Tip:** Having trouble starting Ray? See the [Troubleshooting](../reference/Troubleshooting-Tips-Tricks.ipynb) tips."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell prints the URL for the Ray Dashboard. **This is only correct if you are running this tutorial on a laptop.** Click the link to open the dashboard.\n",
    "\n",
    "If you are running on the Anyscale platform, use the URL provided by your instructor to open the Dashboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dashboard URL: http://localhost:8265\n"
     ]
    }
   ],
   "source": [
    "print(f'Dashboard URL: http://{ray.get_webui_url()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f9yhpJZVNlh5"
   },
   "source": [
    "Instantiate a PPOTrainer object. We pass in a config object that specifies how the network and training procedure should be configured. Some of the parameters are the following.\n",
    "\n",
    "- `num_workers` is the number of actors that the agent will create. This determines the degree of parallelism that will be used. In a cluster, these actors will be spread over the available nodes.\n",
    "- `num_sgd_iter` is the number of epochs of SGD (stochastic gradient descent, i.e., passes through the data) that will be used to optimize the PPO surrogate objective at each iteration of PPO, for each _minibatch_ (\"chunk\") of training data. Using minibatches is more efficient than training with one record at a time.\n",
    "- `sgd_minibatch_size` is the SGD minibatch size (batches of data) that will be used to optimize the PPO surrogate objective.\n",
    "- `model` contains a dictionary of parameters describing the neural net used to parameterize the policy. The `fcnet_hiddens` parameter is a list of the sizes of the hidden layers. Here, we have two hidden layers of size 100, each.\n",
    "- `num_cpus_per_worker` when set to 0 prevents Ray from pinning a CPU core to each worker, which means we could run out of workers in a constrained environment like a laptop or a cloud VM.\n",
    "\n",
    "PPOTrainer 인스턴스를 가져온다. config로 네트워크와 학습과정에 대해 명시한다.\n",
    "\n",
    "- `num_workers` 만들어질 agent의 수. 병렬화 정도를 나타낸다. In a cluster, these actors will be spread over the available nodes.\n",
    "- `num_sgd_iter` SGD epoch 수 (stochastic gradient descent, i.e., passes through the data). the PPO surrogate objective를 최적화 시키는 정도는 나타낸다. minibatch를 사용하여 더 효과적인 training이 가능하다.\n",
    "- `sgd_minibatch_size` SGD minibatch 크기 (batches of data). PPO surrogate objective 최적화하는데 사용.\n",
    "- `model` dictionary 자료형으로 policy를 나타내는 the neural net에 대해 나타낸다. `fcnet_hiddens` 는 hidden layers의 크기를 나타낸다. 여기서는 size 100의 hidden layer를 2개 가지고 있다.\n",
    "- `num_cpus_per_worker` when set to 0 prevents Ray from pinning a CPU core to each worker, which means we could run out of workers in a constrained environment like a laptop or a cloud VM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ok210MCfNlh5"
   },
   "outputs": [],
   "source": [
    "config = DEFAULT_CONFIG.copy()\n",
    "config['num_workers'] = 1\n",
    "config['num_sgd_iter'] = 30\n",
    "config['sgd_minibatch_size'] = 128\n",
    "config['model']['fcnet_hiddens'] = [100, 100]\n",
    "config['num_cpus_per_worker'] = 0\n",
    "config['framework']='torch'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ok210MCfNlh5"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-08-23 14:04:21,037\tINFO trainer.py:632 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.\n",
      "2020-08-23 14:04:21,164\tWARNING util.py:37 -- Install gputil for GPU system monitoring.\n",
      "\u001b[2m\u001b[36m(pid=3934)\u001b[0m 2020-08-23 14:04:22,373\tWARNING compression.py:16 -- lz4 not available, disabling sample compression. This will significantly impact RLlib performance. To install lz4, run `pip install lz4`.\n",
      "\u001b[2m\u001b[36m(pid=3934)\u001b[0m /home/jhmbabo/anaconda3/lib/python3.7/site-packages/ray/rllib/utils/torch_ops.py:149: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /opt/conda/conda-bld/pytorch_1595629401553/work/torch/csrc/utils/tensor_numpy.cpp:141.)\n",
      "\u001b[2m\u001b[36m(pid=3934)\u001b[0m   tensor = torch.from_numpy(np.asarray(item))\n"
     ]
    }
   ],
   "source": [
    "agent = PPOTrainer(config, 'CartPole-v1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ty1a6AWVNlh7"
   },
   "source": [
    "Now let's train the policy on the `CartPole-v1` environment for `N` steps. The JSON object returned by each call to `agent.train()` contains a lot of information we'll inspect below. For now, we'll extract information we'll graph, such as `episode_reward_mean`. The _mean_ values are more useful for determining successful training.\n",
    "\n",
    "agent를 training해보고 `episode_reward_mean`를 뽑아보자."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0: Min/Mean/Max reward:   9.0000/ 20.9579/ 67.0000\n",
      "  1: Min/Mean/Max reward:   9.0000/ 40.8700/154.0000\n",
      "  2: Min/Mean/Max reward:   9.0000/ 60.7600/258.0000\n",
      "  3: Min/Mean/Max reward:  12.0000/ 88.9700/461.0000\n",
      "  4: Min/Mean/Max reward:  12.0000/122.2100/500.0000\n",
      "  5: Min/Mean/Max reward:  12.0000/155.0200/500.0000\n",
      "  6: Min/Mean/Max reward:  12.0000/186.6300/500.0000\n",
      "  7: Min/Mean/Max reward:  12.0000/218.0600/500.0000\n",
      "  8: Min/Mean/Max reward:  14.0000/251.9800/500.0000\n",
      "  9: Min/Mean/Max reward:  14.0000/283.2400/500.0000\n"
     ]
    }
   ],
   "source": [
    "N=10\n",
    "results = []\n",
    "episode_data = []\n",
    "episode_json = []\n",
    "for n in range(N):\n",
    "    result = agent.train()\n",
    "    results.append(result)\n",
    "    episode = {'n': n, \n",
    "               'episode_reward_min':  result['episode_reward_min'],  \n",
    "               'episode_reward_mean': result['episode_reward_mean'], \n",
    "               'episode_reward_max':  result['episode_reward_max'],  \n",
    "               'episode_len_mean':    result['episode_len_mean']}    \n",
    "    episode_data.append(episode)\n",
    "    episode_json.append(json.dumps(episode))\n",
    "    print(f'{n:3d}: Min/Mean/Max reward: {result[\"episode_reward_min\"]:8.4f}/{result[\"episode_reward_mean\"]:8.4f}/{result[\"episode_reward_max\"]:8.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's convert the episode data to a Pandas `DataFrame` for easy manipulation. The results indicate how much reward the policy is receiving (`episode_reward_*`) and how many time steps of the environment the policy ran (`episode_len_mean`). The maximum possible reward for this problem is `500`. The reward mean and trajectory length are very close because the agent receives a reward of one for every time step that it survives. However, this is specific to this environment and not true in general.\n",
    "\n",
    "이제 에피소드 데이터를 Pandas `DataFrame`으로 변환해보자. 그 결과 정책이 얼마나 많은 보상을 받고 있는지(`episode_reward_*`), 정책이 실행한 환경의 몇 단계(`episode_len_mean`)를 알 수 있다. 이에 대한 최대 보상은 500달러다. 보상 평균과 궤적 길이는 에이전트가 생존하는 매 단계마다 보상을 받기 때문에 매우 가깝습니다. 그러나 이것은 이러한 환경에만 국한되어 있으며 일반적이지 않다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(data=episode_data)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the data.\n",
    "\n",
    "이제 데이터를 그려보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "from util.line_plots import plot_line, plot_line_with_min_max, plot_line_with_stddev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bokeh.io\n",
    "# The next two lines prevent Bokeh from opening the graph in a new window.\n",
    "bokeh.io.reset_output()\n",
    "bokeh.io.output_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the length and reward means are equal, we'll only plot one line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'plot_line_with_min_max' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-05e1f3b99aa0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m plot_line_with_min_max(df, x_col='n', y_col='episode_reward_mean', min_col='episode_reward_min', max_col='episode_reward_max',\n\u001b[0m\u001b[1;32m      2\u001b[0m                       title='Episode Rewards', x_axis_label='n', y_axis_label='reward')\n",
      "\u001b[0;31mNameError\u001b[0m: name 'plot_line_with_min_max' is not defined"
     ]
    }
   ],
   "source": [
    "plot_line_with_min_max(df, x_col='n', y_col='episode_reward_mean', min_col='episode_reward_min', max_col='episode_reward_max',\n",
    "                      title='Episode Rewards', x_axis_label='n', y_axis_label='reward')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "([image](../images/rllib/Cart-Pole-Episode-Rewards.png))\n",
    "\n",
    "The model is quickly able to hit the maximum value of 500, but the mean is what's most valueable. After 10 steps, we're more than half way there.\n",
    "\n",
    "모델은 500의 최대 값을 빠르게 맞출 수 있지만, 평균은 가장 가치 있는 것이다. 10단계를 밟고 나면, 우리는 절반 이상 거기에 있다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FYI, here are two views of the whole value for one result. First, a \"pretty print\" output.\n",
    "\n",
    "> **Tip:** The output will be long. When this happens for a cell, right click and select _Enable scrolling for outputs_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3o0wjdZ3Nlh7",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "custom_metrics: {}\n",
      "date: 2020-08-22_16-06-58\n",
      "done: false\n",
      "episode_len_mean: 292.49\n",
      "episode_reward_max: 500.0\n",
      "episode_reward_mean: 292.49\n",
      "episode_reward_min: 21.0\n",
      "episodes_this_iter: 9\n",
      "episodes_total: 408\n",
      "experiment_id: 06033fc525044632a35ea7c94f0a833b\n",
      "hostname: k-14z970-gr30k\n",
      "info:\n",
      "  learner:\n",
      "    default_policy:\n",
      "      cur_kl_coeff: 0.07500000298023224\n",
      "      cur_lr: 4.999999873689376e-05\n",
      "      entropy: 0.47810235619544983\n",
      "      entropy_coeff: 0.0\n",
      "      kl: 0.006814178079366684\n",
      "      model: {}\n",
      "      policy_loss: -0.007620224263519049\n",
      "      total_loss: 676.917724609375\n",
      "      vf_explained_var: 0.00036377483047544956\n",
      "      vf_loss: 676.9246826171875\n",
      "  num_steps_sampled: 40000\n",
      "  num_steps_trained: 40000\n",
      "iterations_since_restore: 10\n",
      "node_ip: 192.168.219.132\n",
      "num_healthy_workers: 1\n",
      "off_policy_estimator: {}\n",
      "perf:\n",
      "  cpu_util_percent: 81.88571428571429\n",
      "  ram_util_percent: 70.23809523809524\n",
      "pid: 8209\n",
      "policy_reward_max: {}\n",
      "policy_reward_mean: {}\n",
      "policy_reward_min: {}\n",
      "sampler_perf:\n",
      "  mean_env_wait_ms: 0.12451672703784579\n",
      "  mean_inference_ms: 1.332949180291734\n",
      "  mean_processing_ms: 0.21946828076482064\n",
      "time_since_restore: 148.0465931892395\n",
      "time_this_iter_s: 14.594220876693726\n",
      "time_total_s: 148.0465931892395\n",
      "timers:\n",
      "  learn_throughput: 490.273\n",
      "  learn_time_ms: 8158.721\n",
      "  load_throughput: 260062.283\n",
      "  load_time_ms: 15.381\n",
      "  sample_throughput: 607.29\n",
      "  sample_time_ms: 6586.634\n",
      "  update_time_ms: 2.483\n",
      "timestamp: 1598080018\n",
      "timesteps_since_restore: 0\n",
      "timesteps_total: 40000\n",
      "training_iteration: 10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(pretty_print(results[-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll learn about more of these values as continue the tutorial.\n",
    "\n",
    "The whole, long JSON blob, which includes the historical stats about episode rewards and lengths:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'episode_reward_max': 500.0,\n",
       " 'episode_reward_min': 21.0,\n",
       " 'episode_reward_mean': 292.49,\n",
       " 'episode_len_mean': 292.49,\n",
       " 'episodes_this_iter': 9,\n",
       " 'policy_reward_min': {},\n",
       " 'policy_reward_max': {},\n",
       " 'policy_reward_mean': {},\n",
       " 'custom_metrics': {},\n",
       " 'hist_stats': {'episode_reward': [500.0,\n",
       "   500.0,\n",
       "   370.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   493.0,\n",
       "   336.0,\n",
       "   26.0,\n",
       "   196.0,\n",
       "   202.0,\n",
       "   51.0,\n",
       "   106.0,\n",
       "   33.0,\n",
       "   188.0,\n",
       "   93.0,\n",
       "   88.0,\n",
       "   104.0,\n",
       "   108.0,\n",
       "   21.0,\n",
       "   250.0,\n",
       "   190.0,\n",
       "   274.0,\n",
       "   332.0,\n",
       "   166.0,\n",
       "   85.0,\n",
       "   348.0,\n",
       "   142.0,\n",
       "   149.0,\n",
       "   63.0,\n",
       "   158.0,\n",
       "   245.0,\n",
       "   200.0,\n",
       "   209.0,\n",
       "   129.0,\n",
       "   46.0,\n",
       "   303.0,\n",
       "   70.0,\n",
       "   332.0,\n",
       "   106.0,\n",
       "   84.0,\n",
       "   28.0,\n",
       "   190.0,\n",
       "   204.0,\n",
       "   168.0,\n",
       "   312.0,\n",
       "   158.0,\n",
       "   222.0,\n",
       "   290.0,\n",
       "   343.0,\n",
       "   267.0,\n",
       "   189.0,\n",
       "   196.0,\n",
       "   229.0,\n",
       "   271.0,\n",
       "   237.0,\n",
       "   387.0,\n",
       "   175.0,\n",
       "   190.0,\n",
       "   155.0,\n",
       "   363.0,\n",
       "   198.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   253.0,\n",
       "   500.0,\n",
       "   341.0,\n",
       "   466.0,\n",
       "   500.0,\n",
       "   105.0,\n",
       "   388.0,\n",
       "   369.0,\n",
       "   499.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   414.0,\n",
       "   395.0,\n",
       "   235.0,\n",
       "   169.0,\n",
       "   500.0,\n",
       "   296.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   421.0,\n",
       "   361.0,\n",
       "   334.0,\n",
       "   500.0,\n",
       "   225.0,\n",
       "   222.0,\n",
       "   188.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   500.0,\n",
       "   500.0],\n",
       "  'episode_lengths': [500,\n",
       "   500,\n",
       "   370,\n",
       "   500,\n",
       "   500,\n",
       "   500,\n",
       "   500,\n",
       "   493,\n",
       "   336,\n",
       "   26,\n",
       "   196,\n",
       "   202,\n",
       "   51,\n",
       "   106,\n",
       "   33,\n",
       "   188,\n",
       "   93,\n",
       "   88,\n",
       "   104,\n",
       "   108,\n",
       "   21,\n",
       "   250,\n",
       "   190,\n",
       "   274,\n",
       "   332,\n",
       "   166,\n",
       "   85,\n",
       "   348,\n",
       "   142,\n",
       "   149,\n",
       "   63,\n",
       "   158,\n",
       "   245,\n",
       "   200,\n",
       "   209,\n",
       "   129,\n",
       "   46,\n",
       "   303,\n",
       "   70,\n",
       "   332,\n",
       "   106,\n",
       "   84,\n",
       "   28,\n",
       "   190,\n",
       "   204,\n",
       "   168,\n",
       "   312,\n",
       "   158,\n",
       "   222,\n",
       "   290,\n",
       "   343,\n",
       "   267,\n",
       "   189,\n",
       "   196,\n",
       "   229,\n",
       "   271,\n",
       "   237,\n",
       "   387,\n",
       "   175,\n",
       "   190,\n",
       "   155,\n",
       "   363,\n",
       "   198,\n",
       "   500,\n",
       "   500,\n",
       "   253,\n",
       "   500,\n",
       "   341,\n",
       "   466,\n",
       "   500,\n",
       "   105,\n",
       "   388,\n",
       "   369,\n",
       "   499,\n",
       "   500,\n",
       "   500,\n",
       "   414,\n",
       "   395,\n",
       "   235,\n",
       "   169,\n",
       "   500,\n",
       "   296,\n",
       "   500,\n",
       "   500,\n",
       "   500,\n",
       "   421,\n",
       "   361,\n",
       "   334,\n",
       "   500,\n",
       "   225,\n",
       "   222,\n",
       "   188,\n",
       "   500,\n",
       "   500,\n",
       "   500,\n",
       "   500,\n",
       "   500,\n",
       "   500,\n",
       "   500,\n",
       "   500]},\n",
       " 'sampler_perf': {'mean_env_wait_ms': 0.12451672703784579,\n",
       "  'mean_processing_ms': 0.21946828076482064,\n",
       "  'mean_inference_ms': 1.332949180291734},\n",
       " 'off_policy_estimator': {},\n",
       " 'num_healthy_workers': 1,\n",
       " 'timesteps_total': 40000,\n",
       " 'timers': {'sample_time_ms': 6586.634,\n",
       "  'sample_throughput': 607.29,\n",
       "  'load_time_ms': 15.381,\n",
       "  'load_throughput': 260062.283,\n",
       "  'learn_time_ms': 8158.721,\n",
       "  'learn_throughput': 490.273,\n",
       "  'update_time_ms': 2.483},\n",
       " 'info': {'learner': {'default_policy': {'cur_kl_coeff': 0.07500000298023224,\n",
       "    'cur_lr': 4.999999873689376e-05,\n",
       "    'total_loss': 676.9177,\n",
       "    'policy_loss': -0.0076202243,\n",
       "    'vf_loss': 676.9247,\n",
       "    'vf_explained_var': 0.00036377483,\n",
       "    'kl': 0.006814178,\n",
       "    'entropy': 0.47810236,\n",
       "    'entropy_coeff': 0.0,\n",
       "    'model': {}}},\n",
       "  'num_steps_sampled': 40000,\n",
       "  'num_steps_trained': 40000},\n",
       " 'done': False,\n",
       " 'episodes_total': 408,\n",
       " 'training_iteration': 10,\n",
       " 'experiment_id': '06033fc525044632a35ea7c94f0a833b',\n",
       " 'date': '2020-08-22_16-06-58',\n",
       " 'timestamp': 1598080018,\n",
       " 'time_this_iter_s': 14.594220876693726,\n",
       " 'time_total_s': 148.0465931892395,\n",
       " 'pid': 8209,\n",
       " 'hostname': 'k-14z970-gr30k',\n",
       " 'node_ip': '192.168.219.132',\n",
       " 'config': {'num_workers': 1,\n",
       "  'num_envs_per_worker': 1,\n",
       "  'rollout_fragment_length': 200,\n",
       "  'sample_batch_size': -1,\n",
       "  'batch_mode': 'truncate_episodes',\n",
       "  'num_gpus': 0,\n",
       "  'train_batch_size': 4000,\n",
       "  'model': {'conv_filters': None,\n",
       "   'conv_activation': 'relu',\n",
       "   'fcnet_activation': 'tanh',\n",
       "   'fcnet_hiddens': [100, 100],\n",
       "   'free_log_std': False,\n",
       "   'no_final_linear': False,\n",
       "   'vf_share_layers': True,\n",
       "   'use_lstm': False,\n",
       "   'max_seq_len': 20,\n",
       "   'lstm_cell_size': 256,\n",
       "   'lstm_use_prev_action_reward': False,\n",
       "   'state_shape': None,\n",
       "   'framestack': True,\n",
       "   'dim': 84,\n",
       "   'grayscale': False,\n",
       "   'zero_mean': True,\n",
       "   'custom_model': None,\n",
       "   'custom_model_config': {},\n",
       "   'custom_action_dist': None,\n",
       "   'custom_preprocessor': None,\n",
       "   'custom_options': -1},\n",
       "  'optimizer': {},\n",
       "  'gamma': 0.99,\n",
       "  'horizon': None,\n",
       "  'soft_horizon': False,\n",
       "  'no_done_at_end': False,\n",
       "  'env_config': {},\n",
       "  'env': 'CartPole-v1',\n",
       "  'normalize_actions': False,\n",
       "  'clip_rewards': None,\n",
       "  'clip_actions': True,\n",
       "  'preprocessor_pref': 'deepmind',\n",
       "  'lr': 5e-05,\n",
       "  'monitor': False,\n",
       "  'log_level': 'WARN',\n",
       "  'callbacks': ray.rllib.agents.callbacks.DefaultCallbacks,\n",
       "  'ignore_worker_failures': False,\n",
       "  'log_sys_usage': True,\n",
       "  'fake_sampler': False,\n",
       "  'framework': 'tf',\n",
       "  'eager_tracing': False,\n",
       "  'no_eager_on_workers': False,\n",
       "  'explore': True,\n",
       "  'exploration_config': {'type': 'StochasticSampling'},\n",
       "  'evaluation_interval': None,\n",
       "  'evaluation_num_episodes': 10,\n",
       "  'in_evaluation': False,\n",
       "  'evaluation_config': {},\n",
       "  'evaluation_num_workers': 0,\n",
       "  'custom_eval_function': None,\n",
       "  'sample_async': False,\n",
       "  '_use_trajectory_view_api': False,\n",
       "  'observation_filter': 'NoFilter',\n",
       "  'synchronize_filters': True,\n",
       "  'tf_session_args': {'intra_op_parallelism_threads': 2,\n",
       "   'inter_op_parallelism_threads': 2,\n",
       "   'gpu_options': {'allow_growth': True},\n",
       "   'log_device_placement': False,\n",
       "   'device_count': {'CPU': 1},\n",
       "   'allow_soft_placement': True},\n",
       "  'local_tf_session_args': {'intra_op_parallelism_threads': 8,\n",
       "   'inter_op_parallelism_threads': 8},\n",
       "  'compress_observations': False,\n",
       "  'collect_metrics_timeout': 180,\n",
       "  'metrics_smoothing_episodes': 100,\n",
       "  'remote_worker_envs': False,\n",
       "  'remote_env_batch_wait_ms': 0,\n",
       "  'min_iter_time_s': 0,\n",
       "  'timesteps_per_iteration': 0,\n",
       "  'seed': None,\n",
       "  'extra_python_environs_for_driver': {},\n",
       "  'extra_python_environs_for_worker': {},\n",
       "  'num_cpus_per_worker': 0,\n",
       "  'num_gpus_per_worker': 0,\n",
       "  'custom_resources_per_worker': {},\n",
       "  'num_cpus_for_driver': 1,\n",
       "  'memory': 0,\n",
       "  'object_store_memory': 0,\n",
       "  'memory_per_worker': 0,\n",
       "  'object_store_memory_per_worker': 0,\n",
       "  'input': 'sampler',\n",
       "  'input_evaluation': ['is', 'wis'],\n",
       "  'postprocess_inputs': False,\n",
       "  'shuffle_buffer_size': 0,\n",
       "  'output': None,\n",
       "  'output_compress_columns': ['obs', 'new_obs'],\n",
       "  'output_max_file_size': 67108864,\n",
       "  'multiagent': {'policies': {},\n",
       "   'policy_mapping_fn': None,\n",
       "   'policies_to_train': None,\n",
       "   'observation_fn': None,\n",
       "   'replay_mode': 'independent'},\n",
       "  'replay_sequence_length': 1,\n",
       "  'use_pytorch': -1,\n",
       "  'eager': -1,\n",
       "  'use_critic': True,\n",
       "  'use_gae': True,\n",
       "  'lambda': 1.0,\n",
       "  'kl_coeff': 0.2,\n",
       "  'sgd_minibatch_size': 128,\n",
       "  'shuffle_sequences': True,\n",
       "  'num_sgd_iter': 30,\n",
       "  'lr_schedule': None,\n",
       "  'vf_share_layers': False,\n",
       "  'vf_loss_coeff': 1.0,\n",
       "  'entropy_coeff': 0.0,\n",
       "  'entropy_coeff_schedule': None,\n",
       "  'clip_param': 0.3,\n",
       "  'vf_clip_param': 10.0,\n",
       "  'grad_clip': None,\n",
       "  'kl_target': 0.01,\n",
       "  'simple_optimizer': False,\n",
       "  '_fake_gpus': False},\n",
       " 'time_since_restore': 148.0465931892395,\n",
       " 'timesteps_since_restore': 0,\n",
       " 'iterations_since_restore': 10,\n",
       " 'perf': {'cpu_util_percent': 81.88571428571429,\n",
       "  'ram_util_percent': 70.23809523809524}}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the `episode_reward` values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'plot_line' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-23-ae9090d885fe>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mepisode_rewards\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresults\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'hist_stats'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'episode_reward'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mdf_episode_rewards\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m'episode'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepisode_rewards\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'reward'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mepisode_rewards\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mplot_line\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdf_episode_rewards\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx_col\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'episode'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_col\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'reward'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtitle\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'Episode Rewards'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx_axis_label\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'episode'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_axis_label\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'reward'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'plot_line' is not defined"
     ]
    }
   ],
   "source": [
    "episode_rewards = results[-1]['hist_stats']['episode_reward']\n",
    "df_episode_rewards = pd.DataFrame(data={'episode':range(len(episode_rewards)), 'reward':episode_rewards})\n",
    "plot_line(df_episode_rewards, x_col='episode', y_col='reward', title='Episode Rewards', x_axis_label='episode', y_axis_label='reward')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "([image](../images/rllib/Cart-Pole-Episode-Rewards2.png))\n",
    "\n",
    "For a well-trained model, most runs do very well while occasional runs do poorly. Try plotting other results episodes by changing the array index in `results[-1]` to another number between `0` and `9`. (The length of `results` is `10`.)\n",
    "\n",
    "잘 훈련된 모델의 경우, 대부분의 런은 매우 잘 작동하는 반면, 때때로 런은 잘 작동하지 않는다. 결과[-1]의 배열 지수를 0과 9 사이의 다른 숫자로 변경하여 다른 결과 에피소드를 플롯해 보십시오.(결과의 길이는 10이다.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FPdkWLrENlh9"
   },
   "source": [
    "### Exercise 2\n",
    "\n",
    "The current network and training configuration are too large and heavy-duty for a simple problem like `CartPole`. Modify the configuration to use a smaller network (the `config['model']['fcnet_hiddens']` setting) and to speed up the optimization of the surrogate objective. (Fewer SGD iterations and a larger batch size should help.)\n",
    "\n",
    "현재의 네트워크와 훈련 구성은 `카트폴` 과 같은 단순한 문제로서는 너무 크고 무겁다. 더 작은 네트워크(`config[`model`]['fcnet_hiddens']` 설정을 사용하고 대리 목적의 최적화를 가속화하도록 구성을 수정해보자. (SGD 반복 횟수를 줄이고 배치 크기를 늘리면 도움이 된다.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3lp6tqkNNlh9"
   },
   "outputs": [],
   "source": [
    "# Make edits here:\n",
    "config = DEFAULT_CONFIG.copy()\n",
    "config['num_workers'] = 3\n",
    "config['num_sgd_iter'] = 30\n",
    "config['sgd_minibatch_size'] = 128\n",
    "config['model']['fcnet_hiddens'] = [100, 100]\n",
    "config['num_cpus_per_worker'] = 0\n",
    "\n",
    "agent = PPOTrainer(config, 'CartPole-v1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "64FmVP7kNlh_"
   },
   "source": [
    "Train the agent and try to get a reward of 200. If it's training too slowly you may need to modify the config above to use fewer hidden units, a larger `sgd_minibatch_size`, a smaller `num_sgd_iter`, or a larger `num_workers`.\n",
    "\n",
    "This should take around `N` = 20 or 30 training iterations.\n",
    "\n",
    "\n",
    "에이전트가 200의 보상을 받도록 학습해보자. 만약 학습이 너무 느리다면, 위의 구성중에 `fcnet_hiddens`의 수를 줄이거나, `sgd_minibatch_size`의 수를 늘리거나, `num_sgd_iter`의 수를 줄이거나, `num_workers`의 수를 늘리는 방향으로 수정하면 된다. \n",
    "\n",
    "이 작업에는 N = 20-30회 정도의 훈련을 반복해야 한다.\n",
    "\n",
    "----------------------\n",
    "이전 세팅\n",
    "\n",
    "config['num_workers'] = 1\n",
    "config['num_sgd_iter'] = 30\n",
    "config['sgd_minibatch_size'] = 128\n",
    "config['model']['fcnet_hiddens'] = [100, 100]\n",
    "config['num_cpus_per_worker'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XB7sdKUzNliA",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "N=5\n",
    "results = []\n",
    "episode_data = []\n",
    "episode_json = []\n",
    "for n in range(N):\n",
    "    result = agent.train()\n",
    "    results.append(result)\n",
    "    episode = {'n': n, \n",
    "               'episode_reward_mean': result['episode_reward_mean'], \n",
    "               'episode_reward_max': result['episode_reward_max'],  \n",
    "               'episode_len_mean': result['episode_len_mean']}    \n",
    "    episode_data.append(episode)\n",
    "    episode_json.append(json.dumps(episode))\n",
    "    print(f'Max reward: {episode[\"episode_reward_max\"]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PW6bN9CYNliB"
   },
   "source": [
    "# Using Checkpoints\n",
    "\n",
    "You checkpoint the current state of a trainer to save what it has learned. Checkpoints are used for subsequent _rollouts_ and also to continue training later from a known-good state.  Calling `agent.save()` creates the checkpoint and returns the path to the checkpoint file, which can be used later to restore the current state to a new trainer. Here we'll load the trained policy into the same process, but often it would be loaded in a new process, for example on a production cluster for serving that is separate from the training cluster.\n",
    "\n",
    "당신은 트레이너가 배운 것을 저장하기 위해 트레이너의 체크포인트를 생성한다. 체크포인트는 후속 _롤아웃_을 위해 사용되며, 또한 알려진 양호한 상태에서 나중에 훈련을 계속하기 위해 사용된다. `agent.save()`를 호출하면 체크포인트가 생성되고 체크포인트 파일의 경로가 반환되는데, 나중에 이 경로를 사용하여 현재 상태를 새 트레이너에게 복원할 수 있다. 여기서 교육된 정책을 동일한 프로세스에 로드하지만, 예를 들어 교육 클러스터와 별도로 서비스를 제공하기 위한 프로덕션 클러스터에 로드하는 경우가 종종 있다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6uf808LMNliC"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jhmbabo/ray_results/PPO_CartPole-v1_2020-08-22_16-04-129asrrz4k/checkpoint_10/checkpoint-10\n"
     ]
    }
   ],
   "source": [
    "checkpoint_path = agent.save()\n",
    "print(checkpoint_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "05icI8bfNliD"
   },
   "source": [
    "Now load the checkpoint in a new trainer:\n",
    "\n",
    "새로운 trainer에 체트포인트를 로드해보자:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8Qq2_AYVNliE"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m   np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m WARNING:tensorflow:From /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorflow/python/compat/v2_compat.py:61: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m Instructions for updating:\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m non-resource variables are not supported in the long term\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m WARNING:tensorflow:From /home/jhmbabo/anaconda3/envs/tf1/lib/python3.7/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m Instructions for updating:\n",
      "\u001b[2m\u001b[36m(pid=8316)\u001b[0m Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "2020-08-22 18:19:27,320\tWARNING util.py:37 -- Install gputil for GPU system monitoring.\n",
      "2020-08-22 18:19:27,623\tINFO trainable.py:473 -- Restored on 192.168.219.132 from checkpoint: /home/jhmbabo/ray_results/PPO_CartPole-v1_2020-08-22_16-04-129asrrz4k/checkpoint_10/checkpoint-10\n",
      "2020-08-22 18:19:27,624\tINFO trainable.py:480 -- Current state after restoring: {'_iteration': 10, '_timesteps_total': None, '_time_total': 148.0465931892395, '_episodes_total': 408}\n"
     ]
    }
   ],
   "source": [
    "trained_config = config.copy()\n",
    "test_agent = PPOTrainer(trained_config, 'CartPole-v1')\n",
    "test_agent.restore(checkpoint_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c2gUUlqkNliG"
   },
   "source": [
    "Use the previously-trained policy to act in an environment. The key line is the call to `test_agent.compute_action(state)` which uses the trained policy to choose an action. This is an example of _rollout_, which we'll study in a subsequent lesson.\n",
    "\n",
    "Verify that the cumulative reward received roughly matches up with the reward printed above. It will be at or near 200.\n",
    "\n",
    "이전에 학습한 정책을 사용하여 환경에서 작업하십시오. 키 라인은 훈련된 정책을 사용하여 액션을 선택하는 `test_agent.compute_action(state)`에 대한 호출이다. 이것은 롤아웃의 한 예로서, 다음 수업에서 연구할 것이다.\n",
    "\n",
    "받은 누적 보상액이 위에 인쇄된 보상과 대략 일치하는지 확인하라. 그것은 거의 200이 될 것이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9asL5Z5lNliH"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "500.0\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('CartPole-v1')\n",
    "state = env.reset()\n",
    "done = False\n",
    "cumulative_reward = 0\n",
    "\n",
    "while not done:\n",
    "    action = test_agent.compute_action(state)  # key line; get the next action\n",
    "    state, reward, done, _ = env.step(action)\n",
    "    cumulative_reward += reward\n",
    "\n",
    "print(cumulative_reward)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "ray.shutdown()  # \"Undo ray.init()\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next lesson, [02: Introduction to RLlib](02-Introduction-to-RLlib.ipynb) steps back to introduce to RLlib, its goals and the capabilities it provides."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "Copy of RLlib Tutorial",
   "provenance": []
  },
  "hide_code_all_hidden": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}