Skip to content
CAIS++ Spring 2019 Project: Building an Agent to Trade with Reinforcement Learning
Jupyter Notebook Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.idea Fixed some issues with the gym Feb 24, 2019
Backtester removing useless file Mar 20, 2019
LSTMs actually added lstm folder lol Feb 3, 2019
RL demos remove .DS_Store Feb 18, 2019
Supervised supervised learning Apr 18, 2019
data sin data Mar 3, 2019
model fix bugs in training.py Apr 10, 2019
pre-processing updated pre-processing Mar 3, 2019
tradinggym fix bugs Mar 3, 2019
.gitignore merging Feb 24, 2019
README.md Merge branch 'master' of https://github.com/gstenger98/rl-finance Apr 2, 2019
training.py fix bugs in training.py Apr 10, 2019

README.md

Reinforcement Learning in Finance

Description

CAIS++ Spring 2019 Project: Building an Agent to Trade with Reinforcement Learning

Timeline

  • February 3rd:

    • In meeting:
      • First Meeting, Environment Set-Up, DQN Explanation, Project Planning
    • Homework:
      • Read first three chapter of Spinning Up
      • Watch the first half of Stanford RL Lecture
      • Code up your own LSTM on the AAPL data. Check each other's out for inspiration, find online resources, ask questions in the slack. Should be a useful exercise for everyone!
      • (Optional: Watch LSTM Intro Video)
  • February 10th: Working LSTM Model

    • State:
      • Current Stock Price
      • percent change from (n-1) to n open
    • Action Space: Buy, Sell, Hold (3-dimensional continuous) as percentage of bankroll
    • Reward:
      • Life Span (define maximum length agent can interact with environment)
        • Receives reward based on profit/loss at the end
        • Sparse reward, harder to train
    • Punishment
      • Test extra severe punishment for losses
      • set thresh-hold time before it can trade again based on punishment
    • Architecture of model
      • One day of encoding LSTM
        • Observation, no actions taken
      • Second day of Policy Net
        • Based on methodology learned from encoding
        • Actions are taken
      • Two-day batches
    • Model Dimensions
      • Encoding LSTM
        • #layers of LSTM, #layers of FCs
        • input size, hidden units size, encoding vector size
      • Policy LSTM
        • input size (state space size)
        • output size (action space size: 3d continuous)
    • Homework:
      • Jincheng and Yang: Begin building Encoding / Policy Net Models
      • Chris: Look through Andrew's current LSTM model
      • Grant: Do the preprocessing data
      • Tomas: - Continue working on RL architecture - Make graph of prices + volume over batch - Visualize price gaps
    • Pre-Process Data
    • Visualization
    • Gym Trading Environment
    • Integrate LSTM into DDDQN
  • February 18th: Working DQN

    • Done for homework
      • Built first policy gradient model (Jincheng)
      • Worked on data pre-processing (Tomas)
    • Today's plan
      • Data pre-processing
      • Use data as input into the gym
      • Finalize the model
  • February 24th: Work day

    • Finish pre-processing
    • Finish trading gym
      • simulate.py
        • Change action 'quit' to quit when timeseries ends
        • change time series to remove seconds
      • series_env.py
        • in class seriesenv
          • do not need daily_start_time, daily_end_time
          • remove randomization of start index (in def 'seed')
    • Finish pipelining <<<<<<< HEAD
  • February 28th: Hackathon

    • TODO
      • Review the current reward function in series_env
      • Finish building the dataset
      • Merge dataset with environment and test
      • Begin building the model
      • Create sine wave csv for testing
  • March 3rd: Working Actor-Critic Model

  • March 10th: Add details like trading costs, slippage, and ask-bid spread; compute performance statistics; data visualization

Long-Term To Do's

=======

  • March 3rd:
    • Work on implementing LSTM (Chris & Caroline)
    • Create test datasets (Grant)
    • Integrate dataset with gym (Yang & Jincheng)
  • March 31th:
    • Finish LSTM

Outstanding To Do's:

  • Working Actor-Critic Model
  • Add details like trading costs, slippage, and ask-bid spread; compute performance statistics; data visualization

13bd296a9295db93c71581f958b043ea734bdeda

  • Build back testing environment
  • Integrate NLP sentiment analysis as feature
  • Add more indicators to model
  • Clean up README
  • Do we hold positions overnight? I think initially no. There are also weird jumps over holidays and weekends.
  • Take into account high, low, close, volume data

Interesting Resources

Reinforcement Learning Education

Papers

Primary
Secondary

Medium Articles

You can’t perform that action at this time.