examples/multi_discrete_setup_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discrete actions multi-data, multi-asset setup intro."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "from logbook import INFO, WARNING, DEBUG\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\") # suppress h5py deprecation warning\n",
    "\n",
    "import numpy as np\n",
    "import os\n",
    "import backtrader as bt\n",
    "\n",
    "from btgym.research.casual_conv.strategy import CasualConvStrategyMulti\n",
    "from btgym.research.casual_conv.networks import conv_1d_casual_attention_encoder\n",
    "\n",
    "from btgym.algorithms.launcher.base import Launcher\n",
    "from btgym.algorithms.aac import A3C\n",
    "from btgym.algorithms.policy import StackedLstmPolicy\n",
    "\n",
    "from btgym import MultiDiscreteEnv\n",
    "from btgym.datafeed.casual import BTgymCasualDataDomain\n",
    "from btgym.datafeed.multi import BTgymMultiData\n",
    "\n",
    "from collections import OrderedDict\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Problem formulation\n",
    "Consider setup with one riskless asset acting as broker account cash and K risky assets.\n",
    "For every risky asset there exists track of historic price records referred as `data-line`.\n",
    "Apart from assets data lines there [optionally] exists number of exogenous data lines holding some\n",
    "information and statistics, e.g. economic indexes, encoded news, macroeconomic indicators, weather forecasts\n",
    "etc. which are considered relevant to decision-making.\n",
    "It is supposed for this setup that:\n",
    "    1. there is no interest rates for any asset;\n",
    "    2. broker actions are fixed-size market orders (`buy`, `sell`, `close`); short selling is permitted;\n",
    "    3. transaction costs are modelled via broker commission;\n",
    "    4. 'market liquidity' and 'capital impact' assumptions are met;\n",
    "    6. time indexes match for all data lines provided;\n",
    "\n",
    "#### Model\n",
    "The problem is modelled as discrete-time finite-horizon partially observable Markov decision process for equity/currency trading:\n",
    "    - *for every asset* traded agent action space is discrete `(0: `hold` [do nothing], 1:`buy`, 2: `sell`, 3:`close` [position])`;\n",
    "    - environment is episodic: maximum  episode duration and episode termination conditions\n",
    "      are set;\n",
    "    - for every timestep of the episode agent is given environment state observation as tensor of last\n",
    "      `m` time-embedded preprocessed values for every data-line included and emits actions according some stochastic policy.\n",
    "    - agent's goal is to maximize expected cumulative capital by learning optimal policy;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Environment setup explanation:\n",
    "\n",
    "        1. This environment expects Dataset to be instance of `btgym.datafeed.multi.BTgymMultiData`, which sets\n",
    "        number,  specifications and sampling synchronisation for historic data for all assets and datalines\n",
    "        one want to consider jointly.\n",
    "\n",
    "        2. Internally every episodic asset data is converted to single bt.feed and added to environment strategy\n",
    "        as separate named data-line (see backtrader docs for extensive explanation of data-lines concept).\n",
    "        Strategy is expected to properly handle all received data-lines.\n",
    "\n",
    "        3. btgym.spaces.ActionDictSpace and order execution. Strategy expects to receive separate action\n",
    "        for every asset in form of dictionary: `{data_line_name_1: action, ..., data_line_name_K: action}`\n",
    "        for K assets added, and issues orders for all assets within a single strategy step.\n",
    "        It is supposed that actions are discrete [for this environment] and same for every asset.\n",
    "        Base actions are set by strategy.params.portfolio_actions, default is: ('hold', 'buy', 'sell', 'close')\n",
    "        which equals to `gym.spaces.Discrete` with depth `N=4 (~number of actions: 0, 1, 2, 3)`. \n",
    "        That is, for `K` assets environment action space will be a shallow dictionary \n",
    "        `(DictSpace)` of discrete spaces:\n",
    "        `{data_line_name_1: gym.spaces.Discrete(N), ..., data_line_name_K: gym.spaces.Discrete(N)}`\n",
    "\n",
    "            Example::\n",
    "\n",
    "                if datalines added via BTgymMultiData are: ['eurchf', 'eurgbp', 'eurgpy', 'eurusd'],\n",
    "                and base asset actions are ['hold', 'buy', 'sell', 'close'], than:\n",
    "\n",
    "                env.action.space will be:\n",
    "                    DictSpace(\n",
    "                        {\n",
    "                            'eurchf': gym.spaces.Discrete(4),\n",
    "                            'eurgbp': gym.spaces.Discrete(4),\n",
    "                            'eurgpy': gym.spaces.Discrete(4),\n",
    "                            'eurusd': gym.spaces.Discrete(4),\n",
    "                        }\n",
    "                    )\n",
    "                single environment action instance (as seen inside strategy):\n",
    "                    {\n",
    "                        'eurchf': 'hold',\n",
    "                        'eurgbp': 'buy',\n",
    "                        'eurgpy': 'hold',\n",
    "                        'eurusd': 'close',\n",
    "                    }\n",
    "                corresponding action integer encoding as passed to environment via .step():\n",
    "                    {\n",
    "                        'eurchf': 0,\n",
    "                        'eurgbp': 1,\n",
    "                        'eurgpy': 0,\n",
    "                        'eurusd': 3,\n",
    "                    }\n",
    "                vector of integers (categorical):\n",
    "                    (0, 1, 0, 3)\n",
    "\n",
    "        4. Environment actions cardinality and encoding. Note that total set of environment actions for `K` assets\n",
    "        and `N` base actions is a `cartesian product of K sets of N elements each`. \n",
    "        It can be encoded as `vector of integers,\n",
    "        single scalar, binary or one_hot`. As cardinality skyrockets with `K`, \n",
    "        `multi-discrete` action setup is only suited\n",
    "        for small number of assets.\n",
    "\n",
    "            Example::\n",
    "\n",
    "                Setup with 4 assets and 4 base actions [hold, buy, sell, close] spawns total of 256 possible\n",
    "                environment actions expressed by single integer in [0, 255] or binary encoding:\n",
    "                    vector str :                            vector int:     int:   binary:\n",
    "                    ('hold', 'hold', 'hold', 'hold')     -> (0, 0, 0, 0) -> 0   -> 00000000\n",
    "                    ('hold', 'hold', 'hold', 'buy')      -> (0, 0, 0, 1) -> 1   -> 00000001\n",
    "                    ...         ...         ...\n",
    "                    ('close', 'close', 'close', 'sell')  -> (3, 3, 3, 2) -> 254 -> 11111110\n",
    "                    ('close', 'close', 'close', 'close') -> (3, 3, 3, 3) -> 255 -> 11111111\n",
    "\n",
    "        Internally there is some weirdness with encodings as we jump forth and back between\n",
    "        dictionary of names or categorical encodings and binary encoding or one-hot encoding.\n",
    "        As a rule: strategy operates with dictionary of string names of actions, environment \n",
    "        sees action as dictionary of integer numbers while policy estimator operates with \n",
    "        either binary or one-hot encoding.\n",
    "\n",
    "        5. Observation space: is nested DictSpace, where 'external' part part of space \n",
    "        should hold specifications for every data line added.\n",
    "\n",
    "            Example::\n",
    "\n",
    "                if datalines added via BTgymMultiData are:\n",
    "                    'eurchf', 'eurgbp', 'eurgpy', 'eurusd';\n",
    "\n",
    "                environment observation space should be DictSpace:\n",
    "                 {\n",
    "                    'raw': spaces.Box(low=-1000, high=1000, shape=(128, 4), dtype=np.float32),\n",
    "                    'external': DictSpace(\n",
    "                        {\n",
    "                            'eurusd': spaces.Box(low=-1000, high=1000, shape=(128, 1, num_features), dtype=np.float32),\n",
    "                            'eurgbp': spaces.Box(low=-1000, high=1000, shape=(128, 1, num_features), dtype=np.float32),\n",
    "                            'eurchf': spaces.Box(low=-1000, high=1000, shape=(128, 1, num_features), dtype=np.float32),\n",
    "                            'eurgpy': spaces.Box(low=-1000, high=1000, shape=(128, 1, num_features), dtype=np.float32),\n",
    "                        }\n",
    "                    ),\n",
    "                    'internal': spaces.Box(...),\n",
    "                    'datetime': spaces.Box(...),\n",
    "                    'metadata': DictSpace(...)\n",
    "                }\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reference:\n",
    "- **MultiDataFeed:** https://kismuz.github.io/btgym/btgym.datafeed.html#btgym.datafeed.multi.BTgymMultiData\n",
    "- **ActionSpace:** https://kismuz.github.io/btgym/btgym.html#btgym.spaces.ActionDictSpace\n",
    "- **MultiDiscreteEnv:** https://kismuz.github.io/btgym/btgym.envs.html#btgym.envs.multidiscrete.MultiDiscreteEnv\n",
    "- **PortfolioEnv:** https://kismuz.github.io/btgym/btgym.envs.html#btgym.envs.portfolio.PortfolioEnv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "engine = bt.Cerebro()\n",
    "\n",
    "num_features = 16\n",
    "\n",
    "engine.addstrategy(\n",
    "    CasualConvStrategyMulti,\n",
    "    cash_name='EUR',  # just a naming for cash asset\n",
    "    start_cash=2000,\n",
    "    commission=0.0001, \n",
    "    leverage=10.0,\n",
    "    asset_names={'USD', 'CHF'},  # that means we use JPY and GBP as information data lines only\n",
    "    drawdown_call=10,\n",
    "    target_call=10,\n",
    "    skip_frame=10,\n",
    "    gamma=0.99,\n",
    "    state_ext_scale = {  # strategy specific CWT preprocessing params:\n",
    "        'USD': np.linspace(1, 2, num=num_features),\n",
    "        'GBP': np.linspace(1, 2, num=num_features),\n",
    "        'CHF': np.linspace(1, 2, num=num_features),\n",
    "        'JPY': np.linspace(5e-3, 1e-2, num=num_features),\n",
    "    },\n",
    "    cwt_signal_scale=4e3,\n",
    "    cwt_lower_bound=4.0, \n",
    "    cwt_upper_bound=90.0,\n",
    "    reward_scale=7, \n",
    "    order_size={\n",
    "        'CHF': 1000,\n",
    "        #'GBP': 1000,\n",
    "        #'JPY': 1000,\n",
    "        'USD': 1000,\n",
    "    },\n",
    ")\n",
    "\n",
    "data_config = [\n",
    "    ('USD', {'filename': './data/DAT_ASCII_EURUSD_M1_2017.csv'}, ),\n",
    "    ('GBP', {'filename': './data/DAT_ASCII_EURGBP_M1_2017.csv'}, ),\n",
    "    ('JPY', {'filename': './data/DAT_ASCII_EURJPY_M1_2017.csv'}, ),\n",
    "    ('CHF', {'filename': './data/DAT_ASCII_EURCHF_M1_2017.csv'}, ),\n",
    "]\n",
    "\n",
    "data_config = OrderedDict(data_config)\n",
    "\n",
    "dataset = BTgymMultiData(\n",
    "    data_class_ref=BTgymCasualDataDomain,\n",
    "    data_config=data_config,\n",
    "    trial_params=dict(\n",
    "        start_weekdays={0, 1, 2, 3, 4, 5, 6},\n",
    "        sample_duration={'days': 30, 'hours': 0, 'minutes': 0},\n",
    "        start_00=False,\n",
    "        time_gap={'days': 15, 'hours': 0},\n",
    "        test_period={'days': 7, 'hours': 0, 'minutes': 0},\n",
    "        expanding=True,\n",
    "    ),\n",
    "    episode_params=dict(\n",
    "        start_weekdays={0, 1, 2, 3, 4, 5, 6},\n",
    "        sample_duration={'days': 2, 'hours': 23, 'minutes': 55},\n",
    "        start_00=False,\n",
    "        time_gap={'days': 2, 'hours': 15},\n",
    "    ),\n",
    "    frozen_time_split={'year': 2017, 'month': 3, 'day': 1},\n",
    ")\n",
    "#########################\n",
    "\n",
    "env_config = dict(\n",
    "    class_ref=MultiDiscreteEnv, \n",
    "    kwargs=dict(\n",
    "        dataset=dataset,\n",
    "        engine=engine,\n",
    "        render_modes=['episode'],\n",
    "        render_state_as_image=True,\n",
    "        render_size_episode=(12,16),\n",
    "        render_size_human=(9, 4),\n",
    "        render_size_state=(11, 3),\n",
    "        render_dpi=75,\n",
    "        port=5000,\n",
    "        data_port=4999,\n",
    "        connect_timeout=90,\n",
    "        verbose=0,\n",
    "    )\n",
    ")\n",
    "\n",
    "cluster_config = dict(\n",
    "    host='127.0.0.1',\n",
    "    port=12230,\n",
    "    num_workers=4,  # Set according CPU's available or so\n",
    "    num_ps=1,\n",
    "    num_envs=1,\n",
    "    log_dir=os.path.expanduser('~/tmp/multi_discrete'),\n",
    ")\n",
    "policy_config = dict(\n",
    "    class_ref=StackedLstmPolicy,\n",
    "    kwargs={\n",
    "        'lstm_layers': (256, 256),\n",
    "        'dropout_keep_prob': 1.0,  # 0.25 - 0.75 opt\n",
    "        'encode_internal_state': False,\n",
    "        'conv_1d_num_filters': 64,\n",
    "        'state_encoder_class_ref': conv_1d_casual_attention_encoder,\n",
    "    }\n",
    ")\n",
    "\n",
    "trainer_config = dict(\n",
    "    class_ref=A3C,\n",
    "    kwargs=dict(\n",
    "        opt_learn_rate=1e-4,\n",
    "        opt_end_learn_rate=1e-5,\n",
    "        opt_decay_steps=50*10**6,\n",
    "        model_gamma=0.99,\n",
    "        model_gae_lambda=1.0,\n",
    "        model_beta=0.001, # entropy reg: 0.001 for non_shared encoder_params; 0.05 if 'share_encoder_params'=True\n",
    "        rollout_length=20,\n",
    "        time_flat=True, \n",
    "        model_summary_freq=10,\n",
    "        episode_summary_freq=1,\n",
    "        env_render_freq=5,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "launcher = Launcher(\n",
    "    cluster_config=cluster_config,\n",
    "    env_config=env_config,\n",
    "    trainer_config=trainer_config,\n",
    "    policy_config=policy_config,\n",
    "    test_mode=False,\n",
    "    max_env_steps=100*10**6,\n",
    "    root_random_seed=0,\n",
    "    purge_previous=1,  # ask to override previously saved model and logs\n",
    "    verbose=0\n",
    ")\n",
    "\n",
    "# Train it:\n",
    "launcher.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}