fix: exceeding max open trades #462

uhliksk · 2022-08-08T14:38:32Z

Description

Allow buy orders to be placed for already open trades but prevent opening new trades.
If Grid Trade #1 buy order is executed and max. open trades limit is reached, all other Grid Trade #1 buy orders are cancelled. This will prevent overshooting the limit while it will allow to wait in multiple symbol for the first buy order to be triggered.

Related Issue

Motivation and Context

This will solve the problem with more opened trades than allowed while also will fix the issue when buy orders in already open trades weren't allowed if the limit was exceeded.

How Has This Been Tested?

I tested changing the limit on the fly to open more trades and then limiting them to lower value. Then waiting if excess buy orders are cancelled and buy orders in already opened trades are allowed.

Screenshots (if appropriate):

uhliksk · 2022-08-08T14:43:43Z

@chrisleekr Please, don't merge yet as there is still an issue with less traded symbols. As the executeTrailingTrade needs to be executed to cancel the excess buy orders, sometimes those orders are not cancelled on time and with just next price change they are executed on Binance right before the bot had a chance to remove them. I'm working on commit to fire executeTrailingTrade when needed on those buy orders which have to be cancelled before there are any websocket data received from Binance.

chrisleekr · 2022-08-10T12:48:30Z

@uhliksk
Is this ready to review?

uhliksk · 2022-08-10T14:05:38Z

@chrisleekr Not yet. I was busy and I didn't finished the commit which will help to cancel the buy order even when there is no price change since the max. open trade limit is reached.

uhliksk · 2022-08-14T08:00:06Z

@habibalkhabbaz I'm modifying app/cronjob/trailingTrade/step/ensure-grid-trade-order-executed.js. I simply added const queue = require('../../trailingTradeHelper/queue'); and then later in code I used queue.executeFor(logger, symbol);, but it's throwing TypeError: queue.executeFor is not a function. Am I overlooking something?

chrisleekr · 2022-08-14T08:01:36Z

@uhliksk Ah, just to confirm, do you want to include this fix in the release? Then we can wait for the release.

uhliksk · 2022-08-14T08:06:15Z

@chrisleekr Yes, I was just waiting until #464 is merged to replace my simple queue to the new queueing mechanism. I thought it'll be simple function replace but I have an issue with queue.executeFor and I don't know what I'm doing wrong right now.

habibalkhabbaz · 2022-08-14T08:28:35Z

@habibalkhabbaz I'm modifying app/cronjob/trailingTrade/step/ensure-grid-trade-order-executed.js. I simply added const queue = require('../../trailingTradeHelper/queue'); and then later in code I used queue.executeFor(logger, symbol);, but it's throwing TypeError: queue.executeFor is not a function. Am I overlooking something?

Hello @uhliksk
I couldn't see an issue with your code. Maybe you forgot to rebuild or so after migrating?

uhliksk · 2022-08-14T08:33:33Z

@habibalkhabbaz Sorry, I forgot to push last commit.

uhliksk · 2022-08-14T08:36:49Z

@habibalkhabbaz This is the line throwing the error.

binance-trading-bot/app/cronjob/trailingTrade/step/ensure-grid-trade-order-executed.js

Line 262 in 1fe9e72

queue.executeFor(logger, symbolToQueue);

uhliksk · 2022-08-14T08:48:46Z

@chrisleekr If it's not throwing the error for you, then you can merge, but for me it's logging this when buy order is filled:

{"name":"binance-api","version":"0.0.88","hostname":"2fb1d3018816","pid":46,"gitHash":"unspecified","level":50,"err":{"message":"p.executeFor is not a function","name":"TypeError","stack":"TypeError: p.executeFor is not a function\n at /srv/dist/server.js:1:32112\n at Array.map (<anonymous>)\n at execute (/srv/dist/server.js:1:32088)\n at async /srv/dist/server.js:1:20849\n at async errorHandlerWrapper (/srv/dist/server.js:1:100025)\n at async execute (/srv/dist/server.js:1:19521)"},"msg":"p.executeFor is not a function","time":"2022-08-14T08:41:25.672Z","v":0}

habibalkhabbaz · 2022-08-14T09:09:35Z

app/cronjob/trailingTrade/step/ensure-grid-trade-order-executed.js

@@ -254,6 +255,13 @@ const execute = async (logger, rawData) => {
        },
        temporaryDisableActionAfterConfirmingOrder
      );
+
+      // Queue other symbols to check if max. open trade is reached
+      symbols.map(async symbolToQueue => {


map can be replaced by forEach and I think no need for async here because we are not expecting a result.

habibalkhabbaz · 2022-08-14T09:19:04Z

@chrisleekr If it's not throwing the error for you, then you can merge, but for me it's logging this when buy order is filled:

{"name":"binance-api","version":"0.0.88","hostname":"2fb1d3018816","pid":46,"gitHash":"unspecified","level":50,"err":{"message":"p.executeFor is not a function","name":"TypeError","stack":"TypeError: p.executeFor is not a function\n at /srv/dist/server.js:1:32112\n at Array.map (<anonymous>)\n at execute (/srv/dist/server.js:1:32088)\n at async /srv/dist/server.js:1:20849\n at async errorHandlerWrapper (/srv/dist/server.js:1:100025)\n at async execute (/srv/dist/server.js:1:19521)"},"msg":"p.executeFor is not a function","time":"2022-08-14T08:41:25.672Z","v":0}

@uhliksk
It seems that the reported error not exist with the latest commit because I see the reported error for p.executeFor but your code is fine.
I just added one comment.

uhliksk · 2022-08-14T10:44:14Z

@chrisleekr
You're right, thank you. I replaced map with forEach and removed async. I'm waiting for buy order to be executed to see if the error will be logged again.

habibalkhabbaz · 2022-08-14T10:52:04Z

That's great @uhliksk
If no more issues I think this PR will be ready to be reviewed by @chrisleekr
Do you agree @uhliksk?

uhliksk · 2022-08-14T11:21:35Z

@habibalkhabbaz Maybe. It's still throwing p.executeFor is not a function to me, but I'm also getting another error in another part of code Error: Illegal characters found in parameter 'symbol'. Is it possible I have corrupted database? I'll check. If there is something corrupted then the error is probably problem in my environment only and the code itself is fine for review.

uhliksk · 2022-08-14T11:24:47Z

@habibalkhabbaz Another observation, the error is logged single time when buy order is executed, but forEach will run it multiple times for all symbols so it looks like just a single symbol is throwing the error. That's should proof it's just my local problem and the code is really good to go.

Edit: Actually I removed async so it looks like the error will be thrown single time because of this. I didn't found any corruption in database. Strange issue.

habibalkhabbaz · 2022-08-14T20:21:44Z

@habibalkhabbaz Bad news.

{"name":"binance-api","version":"0.0.88","hostname":"9a8f2c00b037","pid":46,"gitHash":"unspecified","server":"binance","helper":"queue","jobName":"trailingTrade","correlationId":"1c9e262b-47b1-4eb0-9dce-1e5d392bde17","symbol":"ETHBUSD","level":50,"err":{"message":"u.executeFor is not a function","name":"TypeError","stack":"TypeError: u.executeFor is not a function\n at /srv/dist/server.js:1:75727\n at Array.forEach (<anonymous>)\n at checkIfOpenOrdersExceedingMaxOpenTrades (/srv/dist/server.js:1:75713)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async execute (/srv/dist/server.js:1:31914)\n at async /srv/dist/server.js:1:20849\n at async errorHandlerWrapper (/srv/dist/server.js:1:99150)\n at async execute (/srv/dist/server.js:1:19521)"},"debug":true,"saveLog":true,"msg":"⚠ Execution failed.","time":"2022-08-14T20:12:42.038Z","v":0}

Ops.
I am running my bot on your branch right now. Let me see if I will face the same error.

habibalkhabbaz · 2022-08-14T20:23:18Z

@habibalkhabbaz

This is what I was trying in ensure-grid-trade-order-executed but also didn't worked as expected.

      symbols.forEach(async symbolToCheck => {
        if (symbolToCheck !== symbol) {
          const orderData = {
            symbol,
            symbolConfiguration: {},
            openOrders: [],
            sell: {}
          };
          orderData.symbol = symbolToCheck;
          orderData.symbolConfiguration = await getConfiguration(
            logger,
            orderData.symbol
          );
          const lastBuyPriceDoc = getLastBuyPrice(logger, orderData.symbol);
          orderData.sell = {
            lastBuyPrice: _.get(lastBuyPriceDoc, 'lastBuyPrice', null)
          };

          // Only the initial buy orders have to be cancelled
          if (!orderData.sell.lastBuyPrice) {
            orderData.openOrders =
              JSON.parse(
                await cache.hget('trailing-trade-open-orders', orderData.symbol)
              ) || [];

            orderData.openOrders.forEach(order => {
              if (
                order.type === 'STOP_LOSS_LIMIT' &&
                order.side.toLowerCase() === 'buy' &&
                isExceedingMaxOpenTrades(logger, orderData)
              ) {
                // Cancel the buy order if max. open trades exceeded
                orderData.action = 'buy-order-cancelled';
                logger.info(
                  { orderData, saveLog: true },
                  `The current number of open trades has reached the maximum number of open trades. ` +
                    `The buy order will be cancelled.`
                );

                // Cancel current order
                cancelOrder(logger, orderData.symbol, order);
              }
            });
          }
        }
      });

@uhliksk That's great but I think it will be complicated and it will be hard to debug.
I will try to find a solution for that issue.

uhliksk · 2022-08-14T20:23:21Z

@habibalkhabbaz
Will it help to split queue into queue-init and queue-executeFor to not have const { executeTrailingTrade } = require('../index'); included in executeFor part and thus workaround the issue?

habibalkhabbaz · 2022-08-14T20:30:52Z

@uhliksk
Yes, the same problem as I expected. The issue comes from circular dependency.
We cannot reference the queue inside the steps because the queue has a reference to the trailing trade.

Warning: Accessing non-existent property 'executeFor' of module exports inside circular dependency

{
  "name": "binance-api",
  "version": "0.0.88",
  "hostname": "6ccb1958e63c",
  "pid": 65,
  "gitHash": "unspecified",
  "server": "binance",
  "helper": "queue",
  "jobName": "trailingTrade",
  "correlationId": "127742bd-01e5-45cc-a5d6-e6a1da135422",
  "symbol": "MANAUSDT",
  "level": 50,
  "err": {
    "message": "queue.executeFor is not a function",
    "name": "TypeError",
    "stack": "TypeError: queue.executeFor is not a function\n    at /srv/app/cronjob/trailingTradeHelper/common.js:1151:35\n    at Array.forEach (<anonymous>)\n    at checkIfOpenOrdersExceedingMaxOpenTrades (/srv/app/cronjob/trailingTradeHelper/common.js:1151:11)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at async execute (/srv/app/cronjob/trailingTrade/step/ensure-grid-trade-order-executed.js:216:5)\n    at async /srv/app/cronjob/trailingTrade.js:156:14\n    at async errorHandlerWrapper (/srv/app/error-handler.js:71:5)\n    at async execute (/srv/app/cronjob/trailingTrade.js:37:3)"
  },
  "debug": true,
  "saveLog": true,
  "msg": "⚠ Execution failed.",
  "time": "2022-08-14T20:27:21.404Z",
  "v": 0
}

Maybe we have to refactor the queue as you said.

uhliksk · 2022-08-14T20:38:13Z

Maybe we have to refactor the queue as you said.

@habibalkhabbaz
But I'm actually not sure how to do it properly. I'll keep that queue thing on you. ;)

uhliksk · 2022-08-14T20:47:53Z

@habibalkhabbaz
Another idea I have is to set queued-symbol into cache instead of calling executeFor directly and everytime executeTrailingTrade is finishing it'll look into cache if there are some queued-symbol keys and will call executeFor for those symbols and then will remove those keys from cache. What do you think?

Edit: Or will it also cause the circular dependency to call executeFor from executeTrailingTrade?

habibalkhabbaz · 2022-08-14T21:20:04Z

@uhliksk
I use PubSub to do the trick. It should work properly now. Try and let me know :)
Maybe @chrisleekr has other thoughts but at least for now, this implementation does the trick without changing the queue implementation.

uhliksk · 2022-08-14T21:30:45Z

@habibalkhabbaz
It looks like it doesn't work. Orders has been cancelled on another tick in handle-open-orders. One of them even 6 minutes after UNIBUSD buy order has been executed so it was clearly not fired by executeFor.

Edit: I was wrong. There is a time of creation for orders.

habibalkhabbaz · 2022-08-14T21:44:08Z

Hello @uhliksk
Is this with the latest commit ?

uhliksk · 2022-08-14T21:52:48Z

@habibalkhabbaz
With last commit it looks working as expected.

uhliksk · 2022-08-14T21:58:34Z

Just a note: Orders are cancelled on Binance instantly, notifications in bot are triggered instantly, but from Buy Open Orders some of them are still not removed and are wating for next Action. It looks like order cancellation received by websocket is missing the call of executeFor.

uhliksk · 2022-08-14T22:02:53Z

It's the same with manual orders. I'll create order, it will show on Binance and bot almost instantly. I'll cancel order, it is cancelled on Binance instantly, but in bot on another price change. Do you know @habibalkhabbaz where it should be fixed?

habibalkhabbaz · 2022-08-14T22:11:06Z

Just a note: Orders are cancelled on Binance instantly, notifications in bot are triggered instantly, but from Buy Open Orders some of them are still not removed and are wating for next Action. It looks like order cancellation received by websocket is missing the call of executeFor.

I pushed a fix for this.

It's the same with manual orders. I'll create order, it will show on Binance and bot almost instantly. I'll cancel order, it is cancelled on Binance instantly, but in bot on another price change. Do you know @habibalkhabbaz where it should be fixed?

However, for manual order cancellation, it should be reflected on the frontend directly.

https://github.com/uhliksk/binance-trading-bot/blob/d964d22970acec275b382caf21f96f2190dea7af/app/frontend/websocket/handlers/cancel-order.js#L26

uhliksk · 2022-08-14T22:59:41Z

I pushed a fix for this.

I fixed a fix for this. :)

habibalkhabbaz · 2022-08-15T09:18:54Z

@uhliksk @chrisleekr
My bot running on this branch till now and it seems stable enough.

chrisleekr · 2022-08-15T11:15:58Z

@habibalkhabbaz @uhliksk

Sorry for joining the late party.

Let me review and merge in!

app/cronjob/trailingTrade/step/ensure-grid-trade-order-executed.js

chrisleekr · 2022-08-15T11:27:41Z

app/cronjob/trailingTrade/step/handle-open-orders.js

+
+        // Reset buy open orders
+        if (cancelResult === true) {
+          data.buy.openOrders = [];


Cancelling the order often fails even if it was successful.

And when it succeeds to cancel, it should process the same as stop > limit price cancel process.

https://github.com/chrisleekr/binance-trading-bot/pull/462/files#diff-48b71d5ac0a2a4fd94708519e33e808e09b1a98ab8df1b8996f499d8b71a244bR124

Agree? @uhliksk @habibalkhabbaz

Let me quickly refactor it.

I agree. It'll be cleaner way even as this is actually cancelling the first buy order which have no impact on anything. It will save some headache in future if it will be reused for another purpose.

I was gonna just update this file, but @uhliksk is right. I will make a generic function in the common.js and update other steps to reduce duplicated codes.

Allow me some time. There will be additional commit.

@uhliksk @habibalkhabbaz

I've added a generic function refreshOpenOrdersAndAccountInfo to common.js - https://github.com/chrisleekr/binance-trading-bot/pull/462/files#diff-478c565f3dc02bbdb099f674677ebcd7d1f16fa4ddd06220dd3d6119b8467b84R1130

And I updated handle-open-orders.js to use refreshOpenOrdersAndAccountInfo - https://github.com/chrisleekr/binance-trading-bot/pull/462/files#diff-48b71d5ac0a2a4fd94708519e33e808e09b1a98ab8df1b8996f499d8b71a244bR127

I checked and found some other steps have exactly the same code.
However, refactoring those steps seems out of scope. Hence, I've added FIXME comments instead of refactoring them - https://github.com/chrisleekr/binance-trading-bot/pull/462/files#diff-ad373fc5bc2ac395a075a4ef44bdaebd17d66fc02707c5c07c83e1483199bab6R205

Let's test it out for another day and if there is no issue, I will merge in.

This is a very good fix @uhliksk @habibalkhabbaz

chrisleekr · 2022-08-16T07:39:24Z

Let’s merge it!

uhliksk force-pushed the fix/exceeding-max-open-trades branch from 572d4ec to 9b7b396 Compare August 8, 2022 17:20

fix: exceeding max open trades

fea6d39

uhliksk force-pushed the fix/exceeding-max-open-trades branch from 9b7b396 to fea6d39 Compare August 8, 2022 22:19

This was linked to issues Aug 10, 2022

Bot buys coins, even though orderLimitMaxOpenTrades is exceeded #398

Closed

When max. open trades is exceeded not even existing trades will trigger buy orders #457

Closed

chrisleekr added the bug Something isn't working label Aug 10, 2022

chrisleekr assigned uhliksk Aug 10, 2022

chrisleekr self-requested a review August 10, 2022 12:49

Merge branch 'chrisleekr:master' into fix/exceeding-max-open-trades

6dfc0d0

uhliksk mentioned this pull request Aug 11, 2022

feat: introduce queue ✨ #464

Merged

uhliksk force-pushed the fix/exceeding-max-open-trades branch from 635128d to 6dfc0d0 Compare August 14, 2022 07:18

Merge branch 'chrisleekr:master' into fix/exceeding-max-open-trades

2f39816

fix: queue symbols to check max. open trades

1fe9e72

habibalkhabbaz reviewed Aug 14, 2022

View reviewed changes

fix: replace map with forEach

0f92374

habibalkhabbaz added 2 commits August 15, 2022 00:15

fix: check open orders

4c1cc08

chore: lint

7672ba4

fix: minor change

5c4fbc2

fix: clear open orders after cancellation

d964d22

habibalkhabbaz and others added 2 commits August 15, 2022 01:15

test: handle-open-orders

6c8467f

fix: clean open orders when order cancellation is successfull

d7bb220

chrisleekr reviewed Aug 15, 2022

View reviewed changes

app/cronjob/trailingTrade/step/ensure-grid-trade-order-executed.js Show resolved Hide resolved

chrisleekr reviewed Aug 15, 2022

View reviewed changes

chrisleekr added 4 commits August 15, 2022 22:17

refactor: updated handle open orders

b7b47ab

refactor: updated handle open orders

12bbd3a

chore: added comments

0acf1e6

docs: updated CHANGELOG.md

b492360

chrisleekr merged commit 9a25f13 into chrisleekr:master Aug 16, 2022

fix: exceeding max open trades #462

fix: exceeding max open trades #462

Conversation

uhliksk commented Aug 8, 2022

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

uhliksk commented Aug 8, 2022

chrisleekr commented Aug 10, 2022

uhliksk commented Aug 10, 2022

uhliksk commented Aug 14, 2022

chrisleekr commented Aug 14, 2022

uhliksk commented Aug 14, 2022

habibalkhabbaz commented Aug 14, 2022 • edited

uhliksk commented Aug 14, 2022

uhliksk commented Aug 14, 2022

uhliksk commented Aug 14, 2022

habibalkhabbaz Aug 14, 2022 • edited

Choose a reason for hiding this comment

habibalkhabbaz commented Aug 14, 2022 • edited

uhliksk commented Aug 14, 2022

habibalkhabbaz commented Aug 14, 2022

uhliksk commented Aug 14, 2022

uhliksk commented Aug 14, 2022 • edited

habibalkhabbaz commented Aug 14, 2022

habibalkhabbaz commented Aug 14, 2022

uhliksk commented Aug 14, 2022

habibalkhabbaz commented Aug 14, 2022

uhliksk commented Aug 14, 2022

uhliksk commented Aug 14, 2022 • edited

habibalkhabbaz commented Aug 14, 2022 • edited

uhliksk commented Aug 14, 2022 • edited

habibalkhabbaz commented Aug 14, 2022

uhliksk commented Aug 14, 2022

uhliksk commented Aug 14, 2022

uhliksk commented Aug 14, 2022

habibalkhabbaz commented Aug 14, 2022

uhliksk commented Aug 14, 2022

habibalkhabbaz commented Aug 15, 2022

chrisleekr commented Aug 15, 2022

chrisleekr Aug 15, 2022

Choose a reason for hiding this comment

habibalkhabbaz Aug 15, 2022

Choose a reason for hiding this comment

chrisleekr Aug 15, 2022

Choose a reason for hiding this comment

uhliksk Aug 15, 2022

Choose a reason for hiding this comment

chrisleekr Aug 15, 2022 • edited

Choose a reason for hiding this comment

chrisleekr Aug 15, 2022

Choose a reason for hiding this comment

chrisleekr commented Aug 16, 2022

habibalkhabbaz commented Aug 14, 2022 •

edited

habibalkhabbaz Aug 14, 2022 •

edited

habibalkhabbaz commented Aug 14, 2022 •

edited

uhliksk commented Aug 14, 2022 •

edited

uhliksk commented Aug 14, 2022 •

edited

habibalkhabbaz commented Aug 14, 2022 •

edited

uhliksk commented Aug 14, 2022 •

edited

chrisleekr Aug 15, 2022 •

edited