Exploiting Distributional Temporal Difference Learning
To Deal With Tail Risk

This repository contains both the environment and the agents of the leptokurtosis project.

For a detailed description of the experiment, please refer to our paper.

Environment (envs):

• Leptokurtosis

Implemented algorithms (agents):

• Tabular SARSA
• Categorical Temporal Difference Learning (a tabular version of Categorical distributional reinforcement learning)
• (Efficient) Distributional Temporal Difference Learning:
  • Integration over reward distribution (sample average) method.
  • Maximum Likelihood Estimator (EM-MLE) method.

Prerequisites

see requirements.txt

• Python 3.6
• Numpy
• Pandas
• OpenAI-gym
• Matplotlib
• Seaborn

Results replication

• To replicate the simulation results, run stats_and_plots/run_game_100_times.py (warning: takes time)
• Archived simulation results (in json format) are available in a zip file.
• The performance statistics can be calculated by stats_and_plot/analysis.py
• To generate the data for plots in the paper, run stats_and_plots/figures.py
• To see all the essential meta-parameters used in the paper, please refer to utils/config.py

To reference this repository

@misc{distributionalRLTailRisk,
  author = {Peter Bossaerts, Shijie Huang and Nitin Yadav},
  title = {Exploiting Distributional Temporal Difference Learning To Deal With Tail Risk},
  year = {2020}
}

Key references

Outlier and Leptokurtosis:

• Neural Mechanisms Behind Identification of Leptokurtic Noise and Adaptive Behavioral Response
• Supplementary Material: behavioural results and reinforcement learning

Statistics:

• Casella, G., & Berger, R. L. (2002). Statistical inference (Vol. 2). Pacific Grove, CA: Duxbury.
• Schervish, M. J. (2012). Theory of statistics. Springer Science & Business Media.

(Distributional) Reinforcement Learning:

Primary:

• Reinforcement Learning
• A Distributional Perspective on Reinforcement Learning
• Statistics and Samples in Distributional Reinforcement Learning

Others:

• Distributional Reinforcement Learning with Quantile Regression
• An Analysis of Categorical Distributional Reinforcement Learning
• Implicit Quantile Networks for Distributional Reinforcement Learning
• A Comparative Analysis of Expected and Distributional Reinforcement Learning