MATLAB example on how to use Reinforcement Learning for developing a financial trading model
MATLAB
Latest commit ae31921 Feb 5, 2020
Type Name Latest commit message Commit time
Failed to load latest commit information.
agents Feb 4, 2020
data Feb 5, 2020
docs
images Feb 4, 2020
resources/project Feb 5, 2020
myResetFunction.m Feb 5, 2020
myStepFunction.m Feb 5, 2020
workflow.m Feb 5, 2020
workflow.mlx Feb 5, 2020

# Reinforcement Learning For Financial Trading📈

How to use Reinforcement learning for financial trading using Simulated Stock Data using MATLAB.

## Setup

To run:

2. Open workflow.mlx (MATLAB Live Script preferred) or workflow.m (MATLAB script viewable in GitHub)
3. Run workflow.mlx

Environment and Reward can be found in: myStepFunction.m

Requires

## Overview

The goal of the Reinforcement Learning agent is simple. Learn how to trade the financial markets without ever losing money.

Note, this is different from learn how to trade the market and make the most money possible.

## Reinforcement Learning for Financial Trading

Lets apply some of the terminology and concepts of teaching a reinforcement learning agent to trade.

• The agent in this case study is the computer.
• It will observe financial market indicators (states).
• The financial market is the environment.
• The actions it can take are buy, hold sell.
• It needs to learn which action to take and when (policy).

## Case Study

Our environment consists of 3 stocks, \$20000 cash & 15 years of historical data:

Stocks are:

Actions (buy, sell ,hold) for 3 stocks = 27 total actions

The States being observed are:

• Stocks Owned
• Price Different when Bought
• Cash In Hand
• Price change from yesterday
• % Price change from 2 days ago
• % Price change from 7 days ago
• % Price change from average price of 7 days ago

## Strategy

• Given 3 stocks
• Try to find the best time to buy, sell, or hold each stock
• If selling a stock, sell all of it.
• If buying a stock, buy the maximum allowed given cash in hand.

## Reward

The reward system was what took the most time to develop and required several iterations.

The details are listed, but to simplify

A good reward is given when a trade results in a profit and a stock is bought/held/sold at the right time. For example buying a stock on the way up.

The reverse goes for giving a penalty except for one thing.

A large penalty is given if ANY trade of the 3 stocks is determined as a bad trade. WHY? In the first iteration of the reward system, this was not there. What was observed is that the agent quickly learnt the best of the 3 stocks to trade and ignored the other 2.

• A good reward is given when
• A selling trade results in profit
• A selling trade occurs with momentum
• I.e. selling when prices start to fall -Holding a stock results in the portfolio value increasing
• A bad reward is given when
• Selling results in a negative profit
• A selling trade occurs against momentum
• I.e. selling when prices are increasing, buying when prices are falling
• Holding a stock results in the portfolio value decreasing
• A good reward is overwritten to be bad if any illogical trade occurs
• I.e. sell a stock you don’t have or buy when you don’t had cash

## Training

• Based 12years of data
• 3000 episodes
• ~80hrs to train

Here is an overview of how long it took to learn. You might think 80hrs is a long time. But remember, how long do you think it takes a human to learn how to trade successfully over 12 years?

## Results

Highlights

• 100 Simulations of 3 years:
• Average profit - ~\$30k
• 0 simulations returned negative
• But most did not outperform individual stocks

Histogram of 100 Simulations

Best Simulation

Worst Simulation

The histogram shows that for 100 simulations, not once did the agent ever lose money. So the goal was achieved!

However, you can see that the range varies quite a bit. If you inspect the plots on the right, just buying and holding 1 stock would returned a profit just as good if not better than the agent.

BUT – It’s easy to judge retrospectively. The agent was trading each day as it occurred. None of the simulations resulted in a loss.

## Further Improvements

The case study did ignore some common things to consider when trading the market. Here a few areas that could improve the performance, and make the trained agent more robust:

• Include Transaction costs
• Refined reward system
• Compare different agents

## Conclusion

The aim of this example was to show:

• What reinforcement learning is
• How it can be applied to trading the financial markets
• Leave a starting point for financial professionals to use and enhance using their own domain expertise.

## Resources

Getting Started with Reinforcement Learning (YouTube series)