Missing Tick Data when using continuous Futures in Research Environment

[ArthurAsenheimer]

Expected Behavior

Calling qb.History[Tick](continuous_future.Symbol, start, end) returns all the Tick data for the interval [start, end].

Actual Behavior

There are a lot of trade ticks missing.

Reproducing the Problem

Run all cells of the research notebook of this project and check the output.

qb = QuantBook()
future = qb.AddFuture(ticker=Futures.Indices.SP500EMini, 
                    resolution=Resolution.Tick, 
                    dataNormalizationMode=DataNormalizationMode.BackwardsRatio, 
                    dataMappingMode=DataMappingMode.OpenInterest, 
                    contractDepthOffset=0)

start = datetime(2022, 9, 1, 8, 0)
end = start + timedelta(minutes=1000)
qb.SetStartDate(end + timedelta(1))

ticks = list(qb.History[Tick](future.Symbol, start, end))
trade_ticks = [tick for tick in ticks if tick.TickType is TickType.Trade] 
total_quantity = sum(abs(tick.Quantity) for tick in trade_ticks) 

print(f"Start: {ticks[0].Time}, End: {ticks[-1].Time}.")
print(f"Total number of Ticks: {len(ticks)}.")
print(f"Number of Trade Ticks: {len(trade_ticks)}.")
print(f"Total traded quantity: {total_quantity}.")

Output

Start: 2022-09-01 09:30:00.001000, End: 2022-09-01 16:59:59.939000.
Total number of Ticks: 1963951.
Number of Trade Ticks: 8489.
Total traded quantity: 48928.0.

The number of trade ticks and the total traded quantity is much lower than the official numbers from CME of ~2M contracts traded.

Note: Could be related to #6649.

System Information

QC Cloud

Checklist

• I have completely filled out this template
• I have confirmed that this issue exists on the current master branch
• I have confirmed that this is not a duplicate issue by searching issues

• I have provided detailed steps to reproduce the issue

[ArthurAsenheimer]

Turned out that this is not a mapping issue with continuous Futures and also not a data issue.
It seems to be related to the specific method signature History[Tick](symbol, start, end).

When using History(Tick, symbol start, end) I receive correct results as you can see below:

total_quantity = ticks = trade_ticks = 0 
stepsize = 100 
_start = start 
_end = start + timedelta(minutes=stepsize) 

while True:
    df = qb.History(Tick, future.Symbol, _start, _end)
    if not df.empty and 'quantity' in df.columns:
        ticks += df.dropna(how='all').shape[0]
        trade_ticks += df.loc[df.quantity.ne(0)].shape[0] 
        total_quantity += df.quantity.sum() 
    _start = _end 
    _end += timedelta(minutes=stepsize)  
    _end = min(_end, end) 
    if _end >= end:
        break 

print(f"Start: {start}, End: {_end}.")
print(f"Total number of Ticks: {ticks}.")
print(f"Number of Trade Ticks: {trade_ticks}.")
print(f"Total traded quantity: {total_quantity}.") 

Output

Start: 2022-09-01 09:00:00, End: 2022-09-02 01:40:00.
Total number of Ticks: 8021995.
Number of Trade Ticks: 446230.
Total traded quantity: 1682733.0.

The values are correct now. It's not a nice workaround though. As you can see in the code snippet above, I had to split it into many smaller History requests since calling History(Tick, ...) consumes a huge amount of RAM (more than 8 GB RAM is needed for a single trading day). Not surprising considering that it returns a pandas.DataFrame instead of an Enumerable. Hence, a fix of History[Tick](...) is highly appreciated. Thanks!

[gmwhipple]

@Martin-Molinero @jhonabreul will this be fixed anytime soon?

[jhonabreul]

Closing with #6932