Description

FRONTIER is a deep reinforcement learning model for portfolio management that takes investor preferences into account. The original version allows long-only trades (see this paper for more details). This repo will extend the original version to allow short trades as well.

Usage

There are two ways to set up this development environment. One for development with VS Code, and the other for development through a browser interface using Jupyter Lab. Please refer to the relevant section below:

Running parallelised experiments

Training

Specify your training configuration settings in src/config/train_config.json and run:

python src/utilities/train_parallel.py

Backtesting

Specify your backtesting configuration settings in src/config/backtest_config.json and run:

python src/utilities/backtest_parallel.py

Developing in VS Code

Prerequisites: You must have Docker and VS Code installed.

1. Ensure your VS Code has the Remote-Containers extension installed
2. Clone this repo
3. Open the root folder using the Remote-Containers extension:
   1. Open your command pallette in VS Code (fn+F1)
   2. Type in and select: Remote-Containers: Open Folder in Container
   3. Select the frontier-rl folder
   4. Wait for the container to build (first time will take a couple of minutes)
4. Once the container is built, you can start developing.
5. Insert your API keys and desired plot theme into .env as per .env_example
6. Browse through src/models/frontier.py to see the details of the environment and model architecture, training, backtesting, etc.
7. Run notebooks/train_template.ipynb to see an example of how the models are trained
8. Run notebooks/backtest_template.ipynb to see an example of how trained models are backtested

Developing in Jupyter Lab

Prerequisites: You must have Docker and Docker Compose installed.

1. Clone this repo
2. Navigate to the .devcontainer folder by running cd frontier-rl/.devcontainer
3. Build the development environment container by running docker-compose up (this will build and configure the environment and start a jupyter lab server inside the notebook which can be accessed remotely)
4. Once done, open a web browser and navigate to the jupyter server using your token. It should be something like http://127.0.0.1:10000/lab?token=f1731cd54965375ea245efc131ef6c1172f415139e38e8e9
5. Now you can start developing.
6. Insert your API keys and desired plot theme into .env as per .env_example
7. Browse through src/models/frontier.py to see the details of the environment and model architecture, training, backtesting, etc.
8. Run notebooks/train_template.ipynb to see an example of how the models are trained
9. Run notebooks/backtest_template.ipynb to see an example of how trained models are backtested

Roadmap (TODO)

Setting up contributors and development environment

• Add code from original study
• Add data folders containing market data
• Set up containerised development environment to easily reproduce results
  • Fore development with VS Code through the Remote-Containers extension
  • For remote development with Jupyter Lab through jupyter server in container
• Set up Discord server for project
• Add other paper authors as repo contributors (in progress)
  • Add Prof. van Zyl
  • Add Andrew
• Run simple test on small portfolio (5 assets) to see if containerised environment setup works properly
  • Training of RL models with CNN policy network
  • Backtesting trained RL models with CNN policy network
• Reorganise folders and files to improve repo readability
  • Restructure folders, scripts, notebooks, etc.
  • Update contents of notebooks and scripts to respect new folder structure
• Rename REINFORCE to FRONTIER (folder names, script imports, documentation, etc.)
• Put Quandl key in dotenv, make sure it's in .gitignore, and update loading key in src/utilities/data_manager.py
• Test all noteboks and scripts to see if everything works as expected after folder restructure and renaming
  • notebooks/train_template.ipynb
  • notebooks/backtest_template.ipynb
  • notebooks/backtest_actions.ipynb
  • notebooks/get_all_frontiers.ipynb
  • notebooks/inspect_backtest_actions_template.ipynb
  • notebooks/data_preprocessor.ipynb
  • notebooks/get_index_data.ipynb
  • notebooks/hot-fix.ipynb
• Add plot preferences config (LaTeX-like font and light/dark theme selector)

Add changes to extend study

• Branch off main to add new features for NCAA paper
• Disable transaction costs and investor preferences and run a simple long-only test to maximise portfolio returns to see how it compares to MA-FDRNN, DDPG, PPO, etc.
• Update transaction cost function to accomodate short trades - see Boyd et al. (2017)
• Update transaction cost faunction in reward signal
• Update activation function of policy network output layer to allow short positions (and make sure weights sum to 1)
• Run a couple of tests to confirm transaction costs and portfolio balances are modelled correctly
• Rerun study experiments on different markets to see how performance is impacted after allowing short trades

Publish paper and code

• Write paper
• Make docs from docstrings for all code
• Include detailed README or documentation for steps to use repo and reproduce results
• Publish paper and open source this repo