index out of range

[ghost]

Hi Ran, Reza again ;)

I have this code which is essentially the piece of code you saw before. The only difference is that I have 100 symbols added instead of 4 symbols. The file sp500.csv is downloaded from [https://blog.quandl.com/useful-lists] , just in case you would like to run the code.

from qtpylib.algo import Algo
from qtpylib.indicators import macd#, atr
import pandas as pd

#import qtpylib.indicators
class TestStrategy(Algo):
    def on_start(self):
        print("Started...")
   
    def on_bar(self, instrument):
        lookback = 28
        bars = instrument.get_bars(lookback = lookback)
        print("{:<5}".format(instrument.get_symbol()), len(bars))
                    
            
if __name__ == "__main__":
    sp500 = pd.DataFrame.from_csv('../datas/sp500.csv', index_col = None)

    strategy = TestStrategy(
        bar_window = 28,
        instruments = sp500.iloc[:,0].tolist()[0:99],
        resolution  = "1T" # 1Min bar resolution (Pandas "resample" resolutions)
    )
    strategy.run()

After I run the code, the blotter adds all the 100 symbols (no problem), however, the algorithm code start giving "out of range exception" and some of threads exit. Here is an example of the output

File "C:\Users\Reza\AppData\Local\Programs\Python\Python35\lib\threading.py",
line 914, in _bootstrap_inner
    self.run()
  File "C:\Users\Reza\trading\glt\env\lib\site-packages\qtpylib\asynctools.py",
line 116, in run
    self._func()
  File "C:\Users\Reza\trading\glt\env\lib\site-packages\qtpylib\algo.py", line 1
93, in add_stale_tick
    tick = self.ticks[self.ticks['symbol']==sym][-1:].to_dict(orient='records')[
0]
IndexError: list index out of range

CERN  1
AFL   1
ALXN  1
CAT   1
AIZ   1
BK    1
AMT   1

and after longer run:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\Reza\AppData\Local\Programs\Python\Python35\lib\threading.py",
line 914, in _bootstrap_inner
    self.run()
  File "C:\Users\Reza\trading\glt\env\lib\site-packages\qtpylib\asynctools.py",
line 116, in run
    self._func()
  File "C:\Users\Reza\trading\glt\env\lib\site-packages\qtpylib\algo.py", line 1
93, in add_stale_tick
    tick = self.ticks[self.ticks['symbol']==sym][-1:].to_dict(orient='records')[
0]
IndexError: list index out of range

[ranaroussi]

I'm running this code now with the entire s&p stocks without problems, tho I did a room for a code improvement, which is now released in 1.5.54

[ghost]

1. after 10 min running, some very liquid/popular stocks such as GOOGL only receive three (minute-) bar. Is this normal to you?

"****************************** 537
15:38:00 +1 BXP {'ACN': 1, 'APA': 1, 'AVY': 1, 'CTL': 9, 'HRB': 13, 'CNP': 11,
'AAP': 1, 'ADSK': 1, 'ADI': 1, 'COF': 10, 'BIIB': 1, 'ARG': 0, 'BBBY': 12, 'APH
': 1, 'ALLE': 9, 'AAL': 1, 'AKAM': 1, 'BSX': 1, 'ABT': 1, 'BXLT': 0, 'BA': 13, '
CAH': 1, 'ACE': 0, 'GOOGL': 3, 'AGN': 1, 'AES': 1, 'CHRW': 1, 'APD': 1, 'CBS': 9
, 'CERN': 1, 'CAT': 9, 'BHI': 11, 'BRK-B': 0, 'AMP': 1, 'AIV': 1, 'CCL': 1, 'HSI
C': 9, 'BF-B': 0, 'CBG': 1, 'BCR': 1, 'ADBE': 1, 'ALTR': 0, 'AMZN': 1, 'ALL': 1,
'BXP': 1, 'CELG': 12, 'MO': 1, 'CAM': 0, 'AVB': 1, 'AEP': 1, 'A': 1, 'AET': 1,
'CF': 1, 'BK': 1, 'AA': 1, 'AMAT': 1, 'AAPL': 1, 'APC': 1, 'SCHW': 1, 'ADS': 12,
'BLL': 1, 'AFL': 1, 'ALXN': 1, 'BDX': 8, 'KMX': 1, 'BBY': 1, 'T': 1, 'AON': 1,
'ATVI': 1, 'BBT': 1, 'ADM': 1, 'ABC': 5, 'AN': 1, 'CA': 1, 'AXP': 1, 'MMM': 1, '
GOOG': 1, 'BAX': 1, 'AMG': 1, 'BMY': 1, 'BRCM': 0, 'AIG': 1, 'AIZ': 1, 'GAS': 0,
'COG': 1, 'BLK': 1, 'AMT': 1, 'AVGO': 1, 'BAC': 1, 'AMGN': 1, 'ADP': 1, 'ABBV':
1, 'CVC': 0, 'ADT': 0, 'BWA': 3, 'AZO': 1, 'AEE': 1, 'CPB': 1, 'AME': 1}"

2. I stopped the blotter and then I Ctl+Ced the strategy. Even though the blotter was down, I was receiving bars for 2 minutes.

[ghost]

I implemented another code with more information printed:

from qtpylib.algo import Algo
import pandas as pd
import sys
import threading
import datetime


class TestStrategy(Algo):
    def on_start(self):
        self.bar_count = dict.fromkeys(self.symbols, 0)
        self.start_time = datetime.datetime.now()
        self.counter = 0
        self.lock = threading.Lock()
        print("Started...")

    def on_bar(self, instrument):
        if not instrument:
            print('Recieved Null instrument.')
            return

        self.lock.acquire()
        bars = instrument.get_bars(28)
        print('-'*10, 'This is for instrument:', instrument.get_symbol(), '-'*10)
        elapsedTime =  datetime.datetime.now() - self.start_time
        print("Time passed since start:", divmod(elapsedTime.total_seconds(), 60))

        if len(bars.index):
            self.bar_count[instrument] = self.bar_count[instrument] + 1
            print(bars.tail(1))
        else:
            print('Recieved empty bars.')
        self.counter = self.counter + 1
        allbars = 0
        for symbl in self.symbols:
             allbars = self.bar_count[symbl] + allbars
        print(self.bar_count)
        print('Total number of on_bar called:', self.counter)    
        print('Total number of non empty bars recieved:', allbars)
        print('-'*50)
        self.lock.release()
        sys.stdout.flush()


if __name__ == "__main__":
    sp500 = pd.DataFrame.from_csv('../datas/sp500.csv', index_col = None)

    strategy = TestStrategy(
        bar_window = 28,
        #instruments = sp500.iloc[:,0].tolist()[0:98],
        instruments = ['MMM', 'T', 'ABBV', 'ABT', 'ACN', 'AGN', 'ALL', 'GOOG', 'GOOGL', 'MP', 'AMZN', 'AXP', 'AIG', 'AMGN', 'AAPL',
                        'BAC', 'BIIB', 'BLK', 'BA', 'BMY', 'CVS', 'COF'],
        resolution  = "1T" # 1Min bar resolution (Pandas "resample" resolutions)
    )
    strategy.run()

This is an example of output. Basically after 50 minutes most of the symbols have not reached the 25-window bars.

---------- This is for instrument: BAC ----------
Time passed since start: (50.0, 15.73700000000008)
                            open   high  close  volume    low symbol  \
datetime
2016-12-23 21:22:00+00:00  22.59  22.59  22.59       4  22.59    BAC

                          symbol_group asset_class  signal
datetime
2016-12-23 21:22:00+00:00          BAC         STK     NaN
{'AGN': 18, 'GOOGL': 15, 'ACN': 14, 'BLK': 18, 'ABBV': 19, 'COF': 11, 'MMM': 12,
 'AXP': 12, 'AIG': 13, 'AAPL': 28, 'GOOG': 14, 'AMZN': 19, 'T': 16, 'CVS': 16, '
ALL': 13, 'BMY': 13, 'BAC': 25, 'ABT': 19, 'AMGN': 17, 'BA': 16, 'MP': 0, 'BIIB'
: 14}
Total number of on_bar called: 343
Total number of non empty bars recieved: 342
--------------------------------------------------

[ranaroussi]

The source of this issue as the same one as issue #49. Closing this for now and continuing on #49