Some BUY orders fail during Backtesting

[Spinlogic]

I've tried to so some backtesting using basana to find that some BUY orders fail despite being "market orders". This should never happen in reality.
Checking the code in the examples (e.g. in samples/backtesting_bbands.py) , I can see that the order_size is calculated as

_, ask = await self._exchange.get_bid_ask(trading_signal.pair)

before placing the BUY order in

await self._exchange.create_market_order(trading_signal.operation, trading_signal.pair, order_size)

Then as part of the create_market_order, the actual price for the pair is recalculated (seems to use the "open" price for the next candle, but not sure). This re-calculation happens inside order.py -> MarketOrder in lines:

if self.operation == OrderOperation.BUY:
            price = slipped_price(bar.open, self.operation, amount, liquidity_strategy, cap_high=bar.high)

In some cases, the value of ask above is lower that the recalculated price and this results in the order not being filled. Which screws up the backtest because it skips a buy and the sequent sell orders.

In this case, I do not understand why order_size in terms of base pair needs to be passed to the create_market_order at all. It should be the order_size in terms of quote pair, isn't it?

[gbeced]

Hola Juan,
Could you please set logging to DEBUG and include the output ? You can do so like this:

logging.basicConfig(level=logging.DEBUG, format="[%(asctime)s %(levelname)s] %(message)s")

Note that the backtesting exchange uses the following liquidity strategy by default: https://basana.readthedocs.io/en/latest/backtesting_liquidity.html#basana.backtesting.liquidity.VolumeShareImpact
and that may affect you in two different ways:

• The slipped price * the order size is higher than the available funds.
• The order size is greater than the 25% of the next bar's volume.

In any case, the DEBUG log should give you a clue of what is going on.

Another option is to use this liquidity strategy: https://basana.readthedocs.io/en/latest/backtesting_liquidity.html#basana.backtesting.liquidity.InfiniteLiquidity
like this:

diff --git a/samples/backtesting_bbands.py b/samples/backtesting_bbands.py
index 55fd391..2dc778a 100644
--- a/samples/backtesting_bbands.py
+++ b/samples/backtesting_bbands.py
@@ -25,6 +25,7 @@ import logging
 from basana.external.binance import csv
 import basana as bs
 import basana.backtesting.exchange as backtesting_exchange
+from basana.backtesting import liquidity
 import bbands
 
 
@@ -66,7 +67,8 @@ async def main():
     pair = bs.Pair("BTC", "USDT")
     exchange = backtesting_exchange.Exchange(
         event_dispatcher,
-        initial_balances={"BTC": Decimal(0), "USDT": Decimal(10000)}
+        initial_balances={"BTC": Decimal(0), "USDT": Decimal(10000)},
+        liquidity_strategy_factory=liquidity.InfiniteLiquidity
     )
     exchange.set_pair_info(pair, bs.PairInfo(8, 2))

Please let me know if that helps.

[gbeced]

That is fine.. any library that you want to use for technical indicators should be fine.. but that has nothing to do with the problem that you described related to orders not getting filled.

[Spinlogic]

Sorry for the delay. This is what I see in the log:

2022-11-29 21:59:59,999 INFO     [sample.py:61 - on_trading_signal()] 2022-11-29 20:59:59.999999+00:00 Creating buy market order for ADA/BNB: amount=890.80459770
2022-11-29 22:59:59,999 DEBUG    [exchange.py:429 - _process_order()] Processing order {"order_id": "bbae4cc408404312817ec0486908a116", "balance_updates": {"ADA": "890.80459770", "BNB": "-0.94"}}
2022-11-29 22:59:59,999 DEBUG    [exchange.py:437 - _process_order()] Processing order {"order_id": "bbae4cc408404312817ec0486908a116", "fees": {}}
2022-11-29 22:59:59,999 DEBUG    [exchange.py:449 - _process_order()] Balance short processing order {"order_id": "bbae4cc408404312817ec0486908a116", "symbol": "BNB", "short": "-0.01"}
2022-11-29 22:59:59,999 DEBUG    [exchange.py:411 - order_not_filled()] Order not filled {"order_id": "bbae4cc408404312817ec0486908a116", "order_state": "OrderState.CANCELED"}

I configure my own logger and had to do some changes in the source code of basana to get the logs. And I do not know if I am getting everything that you asked for.

It looks like the order gets cancelled because of a spread issue. But I think that this would not happen in real maket orders as they would get filled on different prices (ADA prices in this case).

InfiniteLiquidity solves the issue.

[gbeced]

From the logs it seems that you're short 1 cent (-0.01) on BNB. You need 0.94 BNB and your account's balance should be 0.93. Could you add the BNB balance to the log when creating the market order ?

[Spinlogic]

Hello,
Not sure, but I think that this contains what you asked.

2022-11-29 21:59:59,999 DEBUG    [sma.py:66 - on_bar_event()] BUY signal: 2022-11-29 20:00:00+00:00
2022-11-29 21:59:59,999 INFO     [sample.py:35 - on_trading_signal()] Trading signal: operation=buy pair=ADA/BNB
2022-11-29 21:59:59,999 DEBUG    [sample.py:48 - on_trading_signal()] Balance available: 0.93 BNB
2022-11-29 21:59:59,999 DEBUG    [sample.py:62 - on_trading_signal()] Order size: 890.80459770 ADA
2022-11-29 21:59:59,999 INFO     [sample.py:63 - on_trading_signal()] 2022-11-29 20:59:59.999999+00:00 Creating buy market order for ADA/BNB: amount=890.80459770 ADA   balance: Balance(available=Decimal('0.93'), total=Decimal('0.93')) BNB
2022-11-29 22:59:59,999 DEBUG    [exchange.py:429 - _process_order()] Processing order {"order_id": "c61b0ba21af74f2b98d85fff0a548b71", "balance_updates": {"ADA": "890.80459770", "BNB": "-0.94"}}
2022-11-29 22:59:59,999 DEBUG    [exchange.py:437 - _process_order()] Processing order {"order_id": "c61b0ba21af74f2b98d85fff0a548b71", "fees": {}}
2022-11-29 22:59:59,999 DEBUG    [exchange.py:449 - _process_order()] Balance short processing order {"order_id": "c61b0ba21af74f2b98d85fff0a548b71", "symbol": "BNB", "short": "-0.01"}
2022-11-29 22:59:59,999 DEBUG    [exchange.py:411 - order_not_filled()] Order not filled {"order_id": "c61b0ba21af74f2b98d85fff0a548b71", "order_state": "OrderState.CANCELED"}

I start the backtest with a balance of 1 BNB and let it run with no fees. If I add a fee, all order fail. Seems to be some problem with the rounding of decimals.

I can clone your repo on a temporary github repo if you would like to check further, or send the files to you via email or some other means.

Best regards,
Juan

[gbeced]

Hi Juan,
I'm taking a look at the ADA/BNB prices and it seems to me that at the moment you place the order (2022-11-29 21:59:59,999) ADA/BNB was around 0.001044 and you calculated the order size using all your available BNB balance: 0.93.
The problem is that the opening price for next bar the price was higher (0.00104700), and that is why you don't have enough cash to execute that order.
Instead of using your full BNB balance to calculate the order size you could try using 99% of it.