When is the Actual REAL-TIME trading tests going to happen ?

[developeralgo8888]

Use this template when reporting bugs, errors or unexpected behaviour.
Override for general questions, feature requests, proposals etc.

Running environment:

Files or part of package has been run:

Expected behaviour:

Actual behaviour:

Steps to reproduce:

[Kismuz]

@developeralgo8888, when good backtest generalisation results will be obtained. Nevertheless, any real-time trading interface development ideas are welcome.

[mysl]

Hi, @Kismuz ,
I am planning to take a deeper look into the GPS/imitation learning approach. Could you pls kindly share some experience with that? Does this direction show any good potential for the trading task? Thank you so much!

[Kismuz]

@mysl,
My implementation of GPS is quite simple, code can be found at btgym.research.gps; notebook: examples/guided_a3c.ipynb;
First, there is very simple Oracle class: for a backtesting training episode we know data in advance so we can estimate optimal trading strategy. It is bit complex because we need solve optimisation task taking in account all broker and account conditions. Instead I implemented 'advisor' indicating is it either time to buy, hold or sell. It uses quite primitive algorithm: just estimates local price peaks and emits signal with some repetition. Those signals can be seen at episode rendering chart if ExpertObserver is added to strategy. So Oracle scans entire episode data just before episode starts, estimates advises and appends it to observation step-by-step.
Next we need incorporate it to our loss. Oracle signals are actually encoded to action probabilities so we can compare it against those emitted by policy; I have found that it is sufficient to estimate loss only on buy and sell actions and omit the rest. And we just sum this loss with base A3C loss with some lambda weight to control the strength Oracle have over algorithm.
The trick here is to find balance between guidance and actual learning.
This approach works and it works well.
Guided loss especially beneficial at early stages of learning when there is danger to stuck at local 'do nothing' solution. Guided loss effectively prevents that and almost doubles convergence speed.
It could be annealed to zero at later stages as imperfect advices can prevent from finding optimal policy, especially with such primitive advisor as mine.
I advice you to look at the code as it is very simple and play with notebook to get the feeling of GPS impact on training:

• try to train at bigger (one year) dataset with guided_lambda=0 and see gradients dying and policy doing nothing; set lambda to 1.0 - 5.0 to see gradients remain consistent and policy improving;

• on synthetic sine wave dataset set lower and higher lambdas and see how higher values can slow down convergence speed as policy approaches optimal (it can bee seen by how fast episode length is contracting: as policy became close to optimal, it can reach the target and terminate earlier, so episode length should drop);

[mysl]

@Kismuz Thanks for the detail explanation. I will take a look based on your advice. Does this approach help on the generalization issue you mentioned?
BTW, it looks like the recent ICLR 2018 best paper is dealing with nonstationary environments. Maybe that could be helpful in trading context too?

[mysl]

https://arxiv.org/abs/1710.03641

[Kismuz]

One of my pillow-books now :)