Market Replay for testing?

[lavalampmj]

I typically use Market Replay as part of my strategy testing, prior to live simulation, to make sure things behave as expected WRT order flow and error handling.

For the life of me I could not figure out why stop order rejections were not being handled (cancel orders, close position) when using Replay. Rather, if an error occurred in one or all stop orders, I'd end up with a bunch of targets, or neither stops or targets and an open position.

Then I spotted in OnOrderUpdate,

if (Playback) return;

prior to any of the order.OrderState case checking and order addition / removal from the Orders List

Why is there "if (Playback) return;" in OnOrderUpdate? Is there something about replay that negates it from being a valid way to check this frameworks behavior? I note elsewhere in the code less invocation of the workflow engine with with checking around IsHistoricalTradeOrPlayBack. I suspect that there some limitation of Replay that I don't yet understand?

BTW: When I removed the if (Playback) return; everything seemed to be handled as expected, much more gracefully! :-)

[MicroTrendsTom]

Replay Differences
is handled a bit differently essentially the same as backtests it assumes your entry and exits are perfect

1. Replay skips events - is not the same order and event flow as NT uses in realtime sim/live
2. ATSQuadroStrategyBase - Does not use the full gamut of the workflow as it would slow it down etc.

So for replay:
Derived Strategy Solution
A) is it a speed issue at what speed does it occur - to prevent the scenario
B) can code be more selective with orders and do the checking pre submission to prevent rejections

Engine Code Solution
or if its really afterwards only more realistic to handle as you say no problem"

We can add a bool to allow handling for that context in replay etc
would need more details of the section, logs flow as to what is occurring
than can add in a flag to allow that flow etc

Ok have made a simple change to main- added
public bool IsPlayBackHandleOnOrderUpdate { get;set; }
and edited
protected override void OnOrderUpdate etc

See commit: 29df98a

[MicroTrendsTom]

we will need to leave this open and wait for test feedback - in case this has broken something else when execution goes into the new section instead of returning :-)

[lavalampmj]

I'm pretty new to NinjaTrader, understanding it's nuances is proving to be quite a steep learning curve. Nonetheless, its by far the best product I've used when it comes to documentation and design, if only it were opensource ;-)

The approach I'm taking is definitely

"B) can code be more selective with orders and do the checking pre submission to prevent rejections"

RTFM: from the manual:

https://ninjatrader.com/support/helpGuides/nt8/?developing_for__tick_replay.htm

In the notes section:

Tick Replay was NOT designed to provide accuracy in backtesting concerning order fills and execution and should NOT be used to expect the exact sequence of executions as running a strategy on live data. For greater order-fill resolution and accuracy in strategy backtesting, you can use the High Fill Resolution in the Strategy Analyzer. Furthermore you cannot combine both Tick Replay and High Order Fill resolution.

I guess if you're spooling replay ticks at 1000x times actual speed and as the broker sim behaves asynchronously WRT fills/rejections, which means that order states could change at significantly different times or even in different actual order of orders.

The switch you added therefore probably makes sense to use if playback is at slow speeds only IMO.

Thanks for your patience.

[BurnOutTrader]

Hi, hopefully you can understand what I am trying to say, as I am not a developer.

I have found when testing on Maximum replay speed that even Ninjatrader managed mode strategies will get the occasional ghost fill, or ghost order that will stay in market when there is no logical reason and this will then corrupt the rest of the test by stopping the system operating correctly, it has resulted in hours of hair pulling frustration whilst also learning to code, sometimes it will happen in the exact same part of a replay no matter what system I am using and in this case it is obviously the replay data its self.

My solution has been to have an internal 'safety workflow' inside each system that checks between Position.MarketPosition and Position.Quantity and my internal strategy enums (StrategyIntention.IsFlat / .IsLong / . IsShort) , StrategyIntention would be reset to IsFlat before checking for any remaining orders or positions and closing them, this can be done from OnExecutionUpdate(), OnPositionUpdate() when an exit order is partially or fully filled etc in testing it doesn't really matter which methods you use to call a safety workflow most methods work for calling a custom check; Order.State can be useful for handling rejections, the only thing that matters is it is done quickly and regularly in a way that is consistent with the intentions of the individual strategy, then when going live I simply move the calling of the 'safety workflow' to a TimedWorkFlow() for operating on Rithmic so that it is not reliant on the correct sequence of events, sometimes this will be a simple timer, sometimes it will be called every x amount of ticks.

I have also been using internal enums for individual signals to check StrategyIntention vs TradeSignal vs Position.MarketPosition and many problems were overcome doing this, even if the strategy believes is not in sync or has positions but believes it is flat it is possible to keep the strategy correctly aligned with the Strategy-Intentions. Order rejections are definitely better prevented than igonored but by having a _StrategyIntention and _TradeSignal, Order rejections will simply be ignored and the strategy will replace the orders whenever necessary as fast as requested based on its own logic.

the key point is that the individual system will need to constantly check for reasons to cancel pending orders and flatten open positions if they no longer represent strategy intentions, I don't think there is any way around this based upon the 1000's of hours testing I have done in the last few months.

I have come back to using this solution as I have accepted that Toms way is the best way forward I will start building my templates from scratch and see how I solve those problems in the unmanaged mode this time around, I have been lingering on the verge of a breakthrough with this problem for weeks as I have tested every possible entry style combination, hopefully soon I can contribute a strategy template encompassing many entry types that self corrects its own bugs in testing and in Realtime.

[lavalampmj]

Like the manual says: "Tick Replay was NOT designed to provide accuracy in backtesting concerning order fills and execution and should NOT be used to expect the exact sequence of executions as running a strategy on live data."

My mistake, and why I posted the original question was that I had assumed as all Last ticks get spooled through OnMarketUpdate etc; I'd thought that Replay must be an accurate way to ensure order logic is sound. This is not the case due to the async nature of NT and the broker sim (there is no guarantee of things happening at the same repeatable actual time as real live trading especially when spooling at 1000x).

I can only assume the point of Market Replay is to enable discretionary traders to practice. A useful feature but not always helpful in algo debugging.

Having said that, enabling OnOrderUpdate with Replay has proven that the system logic behaves closer to what would have occured in real live mode. More testing needed though.

The bottom line is, you need to program defensively such as you (BurnOut) outline above and Tom suggested too. Other areas of defence can include:

a) checking that actual real time ticks arrive close to local computer time and drop stale ticks (delays can be caused by network glitches or bursts overwhelming tick processing at the feed provider or local hangs)
b) not running your strat in a chart (or even having an open chart of the market being traded) but having it run within the control center - this reduces pc resource consumption by orders of magnitude thus helping avoid local hangs/ delays
c) with NT there is no substitute for live sim testing prior to production

my 2 cents :-)

[MicroTrendsTom]

Replay
useful for testing algo logic without having to wait for the realtime signal etc
The replay is a subset of the real thing. order workflows are skipped and position update is all cut down to avoid bottlenecks

Pre-ProductionTesting
the only way to test correctly is sim realtime and then live market real money
but maybe some demo account connection will also allow that

FAST MARKET Soak Testing
For fast market soak testing use small bars to thrash out the errors - eg small tick, range or second etc

a) checking that actual real time ticks arrive close to local computer time and drop stale ticks (delays can be caused by network glitches or bursts overwhelming tick processing at the feed provider or local hangs)

VERY VERY Good point. NT8 REnder delay as they call it - runs a lot deeper in fact
I use lag detection and purge/ignore in my production algo engines and indicators - i can transfer some snippets over

b) not running your strat in a chart (or even having an open chart of the market being traded) but having it run within the control center - this reduces pc resource consumption by orders of magnitude thus helping avoid local hangs/ delays

Run in a chart or not
case by case -
1)Hybrid trading shows an interactive realtime UI (NinjaBuddy) for user control/discretion with the algo
2)It can run in the strategy tab only - bewarer this loads data differently to a chart so you cannot always reconcile the same performance - no control over the system.

Performance
100% but the less graphics and charts render you have the better
Bottlenecks in NT8
1)obfuscated agile.net virtual machine dlls mode can be super slow compared to normal.
2)WPF and the graphics layer
3) Intrinsic bars are super slow - doing lookups for session's and all kinds can be cut out where its superfluous..etc.
2 and 3 can be avoided

c) with NT there is no substitute for live sim testing prior to production
10000000% agreed - To meet compliance requirements when trading algos in a hedge fund - to take a system live it can take months of waiting - added to the sometimes weeks/months or years it took to innovate it etc.

1. Backtesting ( even as useless this generally is)
2. Pre-Production testing/ Walkfowards paper trading - say 3 months to 6 months. depending on version and evolution
3. approval and go live Production

[MicroTrendsTom]

Hi, hopefully you can understand what I am trying to say, as I am not a developer.

I have found when testing on Maximum replay speed that even Ninjatrader managed mode strategies will get the occasional ghost fill, or ghost order that will stay in market when there is no logical reason and this will then corrupt the rest of the test by stopping the system operating correctly, it has resulted in hours of hair pulling frustration whilst also learning to code, sometimes it will happen in the exact same part of a replay no matter what system I am using and in this case it is obviously the replay data its self.

My solution has been to have an internal 'safety workflow' inside each system that checks between Position.MarketPosition and Position.Quantity and my internal strategy enums (StrategyIntention.IsFlat / .IsLong / . IsShort) , StrategyIntention would be reset to IsFlat before checking for any remaining orders or positions and closing them, this can be done from OnExecutionUpdate(), OnPositionUpdate() when an exit order is partially or fully filled etc in testing it doesn't really matter which methods you use to call a safety workflow most methods work for calling a custom check; Order.State can be useful for handling rejections, the only thing that matters is it is done quickly and regularly in a way that is consistent with the intentions of the individual strategy, then when going live I simply move the calling of the 'safety workflow' to a TimedWorkFlow() for operating on Rithmic so that it is not reliant on the correct sequence of events, sometimes this will be a simple timer, sometimes it will be called every x amount of ticks.

I have also been using internal enums for individual signals to check StrategyIntention vs TradeSignal vs Position.MarketPosition and many problems were overcome doing this, even if the strategy believes is not in sync or has positions but believes it is flat it is possible to keep the strategy correctly aligned with the Strategy-Intentions. Order rejections are definitely better prevented than igonored but by having a _StrategyIntention and _TradeSignal, Order rejections will simply be ignored and the strategy will replace the orders whenever necessary as fast as requested based on its own logic.

the key point is that the individual system will need to constantly check for reasons to cancel pending orders and flatten open positions if they no longer represent strategy intentions, I don't think there is any way around this based upon the 1000's of hours testing I have done in the last few months.

I have come back to using this solution as I have accepted that Toms way is the best way forward I will start building my templates from scratch and see how I solve those problems in the unmanaged mode this time around, I have been lingering on the verge of a breakthrough with this problem for weeks as I have tested every possible entry style combination, hopefully soon I can contribute a strategy template encompassing many entry types that self corrects its own bugs in testing and in Realtime.

Cool awesome - not sure its the best this was a migration from a Nt7 engine from a decade ago :-)
So hopefully it will do what you need. i used to have other engines that did keep tabs on state to micro details and in fact this adds overhead and a kind of 1 fix for all occasions approach seems to be ok - :-)
i would be happy to use this engine for
a) backtesting and prototyping
b)in attendance hybrid man and machine mode trading
c) for some unattended pre production -Sunday to Friday - hopefully no bad disconnects to spoil it.
d)production trading with monitoring - subject to a suitable/viable algo etc