Evaluating Deep Reinforcement Learning for Asset Management in the Crypto Market

Evaluating Deep Reinforcement Learning for Asset Management in the Crypto Market

This project evaluates two different deep learning models for Asset Management using Reinforcement Learning, namely Convolutional Neural Networks (CNNs) and Vision Transformers (ViT).

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. About the Author
5. Contact
6. Acknowledgments

About The Project

This project is built in partial fulfilment of the Bachelor of Science in Data Science at Duke Kunshan University.

Inspired by the two AlphaFold papers by DeepMind. I decided to tackle the problem of Asset Management using Deep Reinforcement Learning. I test using two different models to handle the task and two different inputs or observations x.

Therefore, I test 4 different approaches to handle the task of Deep Reinforcement Learning for Asset Management.

1. Approach A. Vision Transformers taking a snapshot of the last 24 hourly close prices.
2. Approach B. Convolutional Neural Networks taking a covariance matrix using hourly data from the last 24 hours.
3. Approach C. Convolutional Neural Networks taking a snapshot of the last 24 hourly close prices.
4. Approach D. Vision Transformers taking a covariance matrix using hourly data from the last 24 hours.

The raw data consists of 1000 datapoints, from which the first 24 are used for calculations of the input and hence lost. Then, of the 976 remaining hours, 75% are used (732 datapoints) are used for training and the rest (244 datapoints) are used for evaluation.

(back to top)

Built With

(back to top)

Getting Started

Tp get started in this project, you will need to install the required libraries and dependencies.

Installation

1. Clone the repo

   git clone https://github.com/cs582/SignatureWorkReinforcementLearning.git

2. Install Requirements

   pip install -f requirements.txt

(back to top)

Usage

There are two main files, the training loop and the evaluation loop. The training loop takes several arguments to play around with. However, most of the values are set as default except for -r (reward metric), -lb (lookback window i.e. number of hours to consider when making an observation), -algorithm (whether to use DQN or Double DQN), -model (whether to use CNN or ViT), -use (2 to use a snapshot and 3 to use the covariance matrix), -episodes (number of training episodes).

1. Training

   # Approach A
   python main.py -r "sharpe" -lb 24 -algorithm "Single_DQN" -model "ViT" -use 3 -episodes 50 2>log.txt
   
   # Approach B
   python main.py -r "sharpe" -lb 24 -algorithm "Single_DQN" -model "CNN" -use 2 -episodes 50 2>log.txt
   
   # Approach C
   python main.py -r "sharpe" -lb 24 -algorithm "Single_DQN" -model "ViT" -use 2 -episodes 50 2>log.txt
   
   # Approach D
   python main.py -r "sharpe" -lb 24 -algorithm "Single_DQN" -model "CNN" -use 3 -episodes 50 2>log.txt

2. Evaluation

   # Approach A
   python eval.py -r "sharpe" -lb 24 -algorithm "Single_DQN" -model "ViT" -use 3 -episodes 50 -path "models/saved_models/Single_DQN_ViT_29_2023-03-03_19:19:36.pt" 2>log.txt
   
   # Approach B
   python eval.py -r "sharpe" -lb 24 -algorithm "Single_DQN" -model "CNN" -use 2 -episodes 50 -path "models/saved_models/Single_DQN_CNN_29_2023-03-03_21:15:35.pt" 2>log.txt
   
   # Approach C
   python eval.py -r "sharpe" -lb 24 -algorithm "Single_DQN" -model "ViT" -use 2 -episodes 50 -path "models/saved_models/Single_DQN_ViT_29_2023-03-03_18:55:00.pt" 2>log.txt
   
   # Approach D
   python eval.py -r "sharpe" -lb 24 -algorithm "Single_DQN" -model "CNN" -use 3 -episodes 50 -path "models/saved_models/Single_DQN_CNN_29_2023-03-03_20:14:42.pt" 2>log.txt

(back to top)

About the Author

Carlos Gustavo Salas Flores

Carlos Gustavo Salas Flores is a senior undergraduate student at Duke Kunshan University majoring in Data Science. He has been an active member of the university by engaging in competitive activities such as the Alibaba GET Challenge where his team and him finished in the top 10 of 250+ teams across China.

He is also a trained Software Engineer and Data Scientist. His team and him participated at Hack Duke 2022 where they got 2nd place for their project "Resume Blinder" which is a Website that uses GPT-3 to hide gender/race related information from job applicants and also show other relevant information about the candidate that may have not been highlighted by the applicant.

Last summer, he interned at Amazon.com remotely from Durham, NC at the AWS Lambda team as a Software Engineer/Data Science Intern and is also going back as a Full-time Software Developer Engineer in Seattle.

Contact Information

Carlos Gustavo Salas Flores - yuseicarlos2560@gmail.com - cs582@duke.edu

Project Link: https://github.com/cs582/SignatureWorkReinforcementLearning

(back to top)

Acknowledgments

• Advisor: Luyao Zhang Ph.D.

(back to top)