Jesse websocket problem

[ynzheng]

[INFO | 2022-09-14T11:58:40] Debug mode is ENABLED
[INFO | 2022-09-14T11:58:40] Local candle generation mode is DISABLED
[INFO | 2022-09-14T11:58:40] Trading 2 routes with 0 extra routes:
Binance Perpetual Futures-DEFI-USDT-1h
Binance Perpetual Futures-ATOM-USDT-1h

[INFO | 2022-09-14T11:58:47] Persistency setting is disabled, will close all existing positions and orders on the exchange before starting the new session
[INFO | 2022-09-14T11:58:47] Canceling all existing orders (if any) for DEFI-USDT on Binance Perpetual Futures
[INFO | 2022-09-14T11:58:47] Canceling all existing orders (if any) for ATOM-USDT on Binance Perpetual Futures
[INFO | 2022-09-14T11:58:47] Updated leverage for DEFI-USDT on Binance Perpetual Futures to 2x
[INFO | 2022-09-14T11:58:47] Updated leverage for ATOM-USDT  on Binance Perpetual Futures to 2x

[INFO | 2022-09-14T11:58:47] Authenticated to Binance Perpetual Futures successfully.
[INFO | 2022-09-14T11:58:47] Successfully opened WS connection to "wss://fstream.binance.com/stream?streams=xxxxxxxxxxxxxxxxxxxx"

[INFO | 2022-09-14T14:00:00] Executing  SlowTrendFollowing-Binance Perpetual Futures-ATOM-USDT-1h
[INFO | 2022-09-14T14:00:00] SUBMITTED order: ATOM-USDT, MARKET, sell, -0.67, $14.18
[INFO | 2022-09-14T14:00:00] EXECUTED order: ATOM-USDT, MARKET, sell, 0.67, $14.18
[INFO | 2022-09-14T14:00:00] OPENED short position for ATOM-USDT: qty: -0.67, entry_price: 14.178
[INFO | 2022-09-14T14:00:00] QUEUED order: ATOM-USDT, LIMIT, buy, 0.67, $11.28
[INFO | 2022-09-14T14:00:00] SUBMITTED order: ATOM-USDT, STOP, buy, 0.67, $14.7

[INFO | 2022-09-15T07:47:49] Authenticated to Binance Perpetual Futures successfully.
[INFO | 2022-09-15T08:16:49] Authenticated to Binance Perpetual Futures successfully.
[INFO | 2022-09-15T08:45:49] Authenticated to Binance Perpetual Futures successfully.

Problem 1:
2022-09-15 07:48:14 listenKeyExpired (websocket pushed Event: User Data Stream Expired)

2022-09-15 08:25:54 ATOM-USDT, STOP order has been EXECUTED in exchange,But Jesse get nothing from websocket...

so the position is still there in Jesse (and forever) ,but in fact, no position in exchange anymore

[INFO | 2022-09-15T09:00:00] Executing  SlowTrendFollowing-Binance Perpetual Futures-DEFI-USDT-1h
[INFO | 2022-09-15T09:00:00] SUBMITTED order: DEFI-USDT, MARKET, sell, -0.012, $723.2

[INFO | 2022-09-15T10:00:00] Executing  SlowTrendFollowing-Binance Perpetual Futures-DEFI-USDT-1h
[INFO | 2022-09-15T10:00:00] cancel all remaining orders to prepare for a fresh start...
[INFO | 2022-09-15T10:00:00] CANCELED order: DEFI-USDT, MARKET, sell, 0.012, $723.2
[INFO | 2022-09-15T10:00:00] SUBMITTED order: DEFI-USDT, MARKET, sell, -0.013, $722.1

Problem 2:
2022-09-15 09:00:00 DEFI-USDT, MARKET, sell has been EXECUTED in exchange,But Jesse get nothing from websocket...
2022-09-15 09:00:00 DEFI-USDT, MARKET, sell has been EXECUTED in exchange,But Jesse get nothing from websocket...

at last, in Jesse,there is no any DEFI-USDT position, but in exchange,there is 0.025 DEFI position...

[ynzheng]

~$ pip list |grep websockets
websockets 9.1

[saleh-mir]

Hi there. So basically the WebSocket stopped working without disconnecting. Where did you get 2022-09-15 07:48:14 listenKeyExpired (websocket pushed Event: User Data Stream Expired) from? Is in your logs?

[saleh-mir]

P.S: I edited the post to add the correct formating to make it easier to read the logs.

[ynzheng]

Hi there. So basically the WebSocket stopped working without disconnecting. Where did you get 2022-09-15 07:48:14 listenKeyExpired (websocket pushed Event: User Data Stream Expired) from? Is in your logs?

not in my logs, I am watching websocket msg by websocat (https://github.com/vi/websocat) on linux shell ..

[saleh-mir]

At 07:47:49 Jesse's logs say:

[INFO | 2022-09-15T07:47:49] Authenticated to Binance Perpetual Futures successfully.

But your log at 07:48:14 says:

2022-09-15 07:48:14 listenKeyExpired (websocket pushed Event: User Data Stream Expired)

Which means the private WS connection couldn't renew authentication as the logged file said which is the cause of the other problems you mentioned.

I looked at my code to see how I'm logging it, the error handling of it is only handling a certain case that I could reproduce. There are two cases:

1. There's something in the response that I didn't expect
2. The exchange said it authenticated but it didn't really.

I added a new line to log the raw response of the exchange to look into it if it happened again. But there is a good change that it won't because I wrote this code a long time ago for Binance and this is the first time I get a report about this.

The new line I added will be in the next version which I'll release in the coming days. If it happened again, please report it via the custom report button in the GUI because that way I get to see the raw exchange logs as well.

[ynzheng]

I think it will be great and better if jesse can handle the websocket pushed listenKeyExpired event to delete the old key and got a new one...

[ynzheng]

I added a new line to log the raw response of the exchange to look into it if it happened again. But there is a good change that it won't because I wrote this code a long time ago for Binance and this is the first time I get a report about this.

great, its so weird...and in my hongkong and japan servers,it happend once in a while..

[saleh-mir]

I think it will be great and better if jesse can handle the websocket pushed listenKeyExpired event to delete the old key and got a new one...

We normally don't allow the listenKey to be expired in the first place. But I guess it's a good idea to implement handling for when it does happen. I'll add it. Thank you for the suggestion!

[saleh-mir]

This should be fixed with the upcoming release in the next few days. Thanks for reporting it.

[ynzheng]

I have doing a further test on the old v0.34.0 version, it seem maybe the problem is "no WS connection reopened"

Just FYI

[INFO | 2022-09-18T12:40:01] Executing  SlowTrendFollowing-Binance Futures-BTC-USDT-5m

[INFO | 2022-09-18T12:38:43] Successfully opened WS connection to "wss://fstream.binance.com/stream?streams=umCjy2181cDzcKa7zB5rLErBdLidBtU3abdsq0YZ4ruMmkJrgF7qNLt6OoWd5ZlV"

we got the first listenKey here, but as the binance-docs,it usually will be listenKeyExpired after 24 hours cos:
"A single connection is only valid for 24 hours; expect to be disconnected at the 24 hour mark"
@https://binance-docs.github.io/apidocs/futures/en/#user-data-streams

[INFO | 2022-09-19T16:27:45] Authenticated to Binance Futures successfully.

at this moment, the umCjy2181cDzcKa7zB5rLErBdLidBtU3abdsq0YZ4ruMmkJrgF7qNLt6OoWd5ZlV listenKey is still alive

[INFO | 2022-09-19T16:56:45] Authenticated to Binance Futures successfully.

at this time, we got a new listenKey ...and the old listenKey listenKeyExpired...

[INFO | 2022-09-19T17:05:00] BTC-USDT go_short  blabla...

[INFO | 2022-09-19T17:25:45] Authenticated to Binance Futures successfully.

after 17:05:00, Jesse submit go_short and go_long orders every 5m... all orders have been filled in exchange, but no anything on the Jesse side...like Jesse got nothing from the new listenKey websocket..

So I think maybe ,this is reason, no WS connection reopened with the new listenKey after old listenKey Expired...Jesse just used the Expired WS connection even we had got a new one...

cos in the log, I cannot find anything like "Successfully reopened/opened WS connection to" the new listenKey msg..

[saleh-mir]

The Authenticated to Binance Futures successfully message is for renewing listenKey. The expired happens if it's not renewed before the 30 minutes. I changed it in the coming update to update it every 15 minutes so we'll have another chance in case it's not successful (which is very rare)

[ynzheng]

The Authenticated to Binance Futures successfully message is for renewing listenKey. The expired happens if it's not renewed before the 30 minutes. I changed it in the coming update to update it every 15 minutes so we'll have another chance in case it's not successful (which is very rare)

on the latest Jesse and live 0.34 version , nothing changed, the bug is still there

I have sent report via the report button in the GUI , pls check

thanks

[ynzheng]

I think this can be fix, if jesse live plugin got websocket pushed listenKeyExpired event and handle it to delete the old key and got a new one...then close the old websocket ,reconnect websocket with the new key ..

[saleh-mir]

Pushing the 0.40.1 for this. Thank you for the detailed report.