Error encountered while running on Live Mode

[elijahho]

Describe the bug
Could not run on Live Mode on Oanda.

Traceback (most recent call last):
  File "runfile.py", line 23, in <module>
    at.run()
  File "/usr/local/lib/python3.8/site-packages/autotrader/autotrader.py", line 218, in run
    self._main()
  File "/usr/local/lib/python3.8/site-packages/autotrader/autotrader.py", line 295, in _main
    bot = AutoTraderBot(instrument, self.strategies[strategy],
  File "/usr/local/lib/python3.8/site-packages/autotrader/autobot.py", line 146, in __init__
    data, quote_data, MTF_data = self._retrieve_data(instrument, self.feed)
  File "/usr/local/lib/python3.8/site-packages/autotrader/autobot.py", line 546, in _retrieve_data
    data = getattr(self.get_data, feed.lower())(instrument,
  File "/usr/local/lib/python3.8/site-packages/autotrader/lib/autodata.py", line 99, in oanda
    data        = self.response_to_df(response)
  File "/usr/local/lib/python3.8/site-packages/autotrader/lib/autodata.py", line 222, in response_to_df
    candles = response.body["candles"]
KeyError: 'candles'

To Reproduce
Here is a copy of my strategy yaml file

NAME: 'MACD Momentum Strategy'
MODULE: 'macd'
CLASS: 'SimpleMACD'
INTERVAL: '5m'
PERIOD: 200
RISK_PC: 1
SIZING: 'risk'
PARAMETERS:
  ema_period_fast: 20
  ema_period_slow: 200
  MACD_fast: 12
  MACD_slow: 26
  MACD_smoothing: 9

WATCHLIST: ['EURUSD=X']

Expected behavior
To be able to run as expected on Live Mode.

Version of AutoTrader being used
Latest.

Any idea what the issue could be?

[kieran-mackle]

That particular error is coming from your watchlist specification - EURUSD=X is the format for the Yahoo Finance API. As per the Strategy Configuation and AutoData documentation, you must specify instruments in WATCHLIST and candlestick granularities in INTERVAL in accordance with the Broker API/data feed being used. For Oanda, EUR/USD should be specified as EUR_USD in the WATCHLIST, with M5 for INTERVAL.

This format is in place to ensure that backtesting is completed using the same data feed as will be used for live trading, since data may vary from different providers.

[elijahho]

Ah yes, I somehow assumed that the backtest feed was pointing to Oanda because I had set my Oanda account details in the GLOBAL.yaml file. I had updated the config to the correct Oanda values, and tried running in backtest mode, and still hitting the same error.

NAME: 'MACD Momentum Strategy'
MODULE: 'macd'
CLASS: 'SimpleMACD'
INTERVAL: 'M5'
PERIOD: 10000000
RISK_PC: 1
SIZING: 'risk'
PARAMETERS:
  ema_period_fast: 20
  ema_period_slow: 200
  MACD_fast: 12
  MACD_slow: 26
  MACD_smoothing: 9

WATCHLIST: ['EUR_USD']

Any idea what the cause might be? Thanks!

[kieran-mackle]

Could you please share your runfile? Specifically, have you set the feed parameter to Oanda in the AutoTrader configuration?

[elijahho]

Sure, here is my runfile

from autotrader.autotrader import AutoTrader

at = AutoTrader()

at.configure(show_plot=True, verbosity=1, feed="Oanda")
at.add_strategy('macd')
at.backtest(start = '7/10/2021',
            end = '4/12/2021',
            initial_balance=10000,
            spread=0.9,
            leverage = 30)
at.run()

[kieran-mackle]

My suspicion is that the API token is being rejected for some reason, and the data is not being returned as expected.
As a test, could you please save the following code as a file in your project directory (the same as where your runfile is), run it, and then share the message printed to the console?

import v20
from autotrader.lib import environment_manager, read_yaml

global_config = read_yaml.read_yaml('./config' + '/GLOBAL.yaml')
broker_config = environment_manager.get_config('demo', global_config, 'Oanda')

API = broker_config["API"]
ACCESS_TOKEN = broker_config["ACCESS_TOKEN"]
port = broker_config["PORT"]
api = v20.Context(hostname=API, token=ACCESS_TOKEN, port=port)

response = api.instrument.candles('EUR_USD', granularity = 'M5', count = 500)
print(response)

[elijahho]

Sure. I'm hitting the live API instead of the demo API but works. Got a 200 response.

Method = GET
Path = https://api-fxtrade.oanda.com:443/v3/instruments/EUR_USD/candles?granularity=M5&count=500
Status = 200
Reason = OK
Content-Type = application/json

[kieran-mackle]

In that case, could you replace 'demo' with 'real', as per the following line:
broker_config = environment_manager.get_config('real', global_config, 'Oanda')

[elijahho]

Yup that's what I did, and I pasted the response just before your reply.

[kieran-mackle]

That's very strange, not sure exactly what is going on. Could you try replacing the last two lines with the following, and comment the response:

response = api.instrument.candles('EUR_USD', granularity = 'M5', count = 10)
print(response.body)

[elijahho]

Very strange indeed. Res body looks good too.

{'instrument': 'EUR_USD', 'granularity': 'M5', 'candles': [<v20.instrument.Candlestick object at 0x1073ad9d0>, <v20.instrument.Candlestick object at 0x1073ad7c0>, <v20.instrument.Candlestick object at 0x1073c6790>, <v20.instrument.Candlestick object at 0x1073c6a90>, <v20.instrument.Candlestick object at 0x1073c6c70>, <v20.instrument.Candlestick object at 0x1073c6760>, <v20.instrument.Candlestick object at 0x1073c6a00>, <v20.instrument.Candlestick object at 0x1073c6ca0>, <v20.instrument.Candlestick object at 0x1073c6d30>, <v20.instrument.Candlestick object at 0x1073c6dc0>]}

[kieran-mackle]

That does look as expected... I'm grasping for ideas here, so could you please try the following to see if there are any errors returned.

print(response.body["candles"])
print(response.body["candles"][0])

[elijahho]

Looks good.

[<v20.instrument.Candlestick object at 0x1033a09d0>, <v20.instrument.Candlestick object at 0x1033a07c0>, <v20.instrument.Candlestick object at 0x1033b9790>, <v20.instrument.Candlestick object at 0x1033b9a90>, <v20.instrument.Candlestick object at 0x1033b9c70>, <v20.instrument.Candlestick object at 0x1033b9760>, <v20.instrument.Candlestick object at 0x1033b9a00>, <v20.instrument.Candlestick object at 0x1033b9ca0>, <v20.instrument.Candlestick object at 0x1033b9d30>, <v20.instrument.Candlestick object at 0x1033b9dc0>]
time: '2021-12-06T02:45:00.000000000Z'
mid:
  o: 1.12944
  h: 1.12944
  l: 1.12921
  c: 1.12928
volume: 99
complete: true

I had no issues backtesting with Yahoo Finance feeds though.

[kieran-mackle]

Yep, that's very strange. The initial error you posted is suggesting that the response object from the API does not contain any candles, but it looks like everything is working from the above. I very occasionally see the same error, but hitting run again usually has no problems. Is the error happening every time you run?

Also, have you tried running a backtest with the Oanda feed, but with other start/end dates?

[elijahho]

Yup it doesn't make much sense. Yes, it is happening each time I run for Oanda feed.

Yes, I also just tried running it different start/end dates.

at = AutoTrader()
at.configure(show_plot=True, verbosity=1, feed="Oanda", account_id='xxx')
at.add_strategy('macd')
at.backtest(start = '1/1/2021',
            end = '30/3/2021',
            initial_balance=10000,
            spread=0.9,
            leverage = 30)
at.run()

Getting the same error too.

The only other reason is the keys in GLOBAL.yaml file is not being read in correctly?

[kieran-mackle]

Since the test script above returned a 200 response, I would imagine everything is being loaded correctly - the functions used in that script are the same used throughout the code. Regardless, have a look at the docs for the global config file to make sure the structure and spacing matches.

The last thing I can think of to help debug this error is to add a print statement on line 221 of autodata.py, located at /usr/local/lib/python3.8/site-packages/autotrader/lib/autodata.py (first line in the response_to_df method). It's a bit hacky, but will hopefully give us some more information.

print(response.body.keys())

Run AutoTrader again, then see what the output is.

[elijahho]

Ok I added this print statement and another one print(response.body)

Getting a more informative error message now -

{'errorMessage': 'Insufficient authorization to perform request.'}
dict_keys(['errorMessage'])

Strange I'm hitting authorization error here?

[kieran-mackle]

Okay, sorry I missed this before, but set the environment to 'real' when configuring AutoTrader in your runfile:

at.configure(show_plot=True, verbosity=1, feed="Oanda", environment='real')

Let me know if that works!

[elijahho]

Yes, that is it! Backtest works now :) Thank you!

Btw, since I have you now, what is the spread datatype? Is it in pips or absolute value of the currency?

[elijahho]

Btw, I'm keen to help work on this project with you, if you're planning to extend the features

[kieran-mackle]

Great! Very happy to hear.

Good question, it is in pips. I've just updated the docs to reflect that.

That would be great, contributions are very much welcomed! I am very busy with other commitments at the moment though, so wasn't planning any major additions. That said, there are always things which can be added/improved. Feel free to send me an email (kemackle98@gmail.com) if you would like to continue this discussion. Otherwise, open a pull request with any additions you make and would like to share. Thanks!