Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Binance "invalid WebSocket Extension handshake" #3907

Closed
feelus opened this issue Feb 8, 2021 · 42 comments
Closed

Binance "invalid WebSocket Extension handshake" #3907

feelus opened this issue Feb 8, 2021 · 42 comments

Comments

@feelus
Copy link

feelus commented Feb 8, 2021 •
edited
Loading

Hello,

I'm suddenly experiencing issues connecting to Binance streaming API, since 13:20 UTC today. I checked their changelog and there seem to have been some changes today to the API, see https://binance-docs.github.io/apidocs/spot/en/

It can be reproduced with this example

    StreamingExchange exchange = StreamingExchangeFactory.INSTANCE.createExchange(BinanceStreamingExchange.class);

    ProductSubscription streamSubscription = ProductSubscription.create()
    .addOrderbook(CurrencyPair.BTC_EUR)
    .build();

    exchange.connect(streamSubscription).blockingAwait();

    Disposable subscription1 = exchange.getStreamingMarketDataService()
        .getOrderBook(CurrencyPair.BTC_EUR)
        .subscribe(
            book -> System.out.println(book),
            throwable -> System.out.println(throwable));

The error I'm getting is

Exception in thread "main" io.netty.handler.codec.CodecException: invalid WebSocket Extension handshake for "permessage-deflate; server_no_context_takeover; client_max_window_bits=15"
        at io.netty.handler.codec.http.websocketx.extensions.WebSocketClientExtensionHandler.channelRead(WebSocketClientExtensionHandler.java:110)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1533)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:832)

Could anyone help? Was there a Binance API change that introduced some incompatible changes?
@ajutzeler
Copy link

Hi,

I am experiencing the exact same thing.

It is relevant to mention that Binance has been suffering from major availability issues since that massive BTC surge (around 13:00 UTC today).

Could this error be the result of a maneuver from Binance as an attempt to reduce the traffic?

@makarid
Copy link
Collaborator

makarid commented Feb 8, 2021 via email

@VadimsGrusas
Copy link
Contributor

Same here.

@ajutzeler
Copy link

ajutzeler commented Feb 8, 2021 •
edited
Loading

All the people I know of who are not currently being impacted by this gather the two following conditions:

  • They are using ccxt integration
  • They reached a certain VIP level

Could anyone here confirm that it is either

  • working fine with XChange on an account at VIP0 level
  • broken with XChange on an account at superior VIP level

to invalidate my hypothesis?

@paolo-rendano
Copy link
Contributor

paolo-rendano commented Feb 8, 2021 •
edited
Loading

same for me. What is happening?
VIP0 - not working
Authenticated - not working

@VadimsGrusas
Copy link
Contributor

VIP0 - not working.
Without authorization not working.
This js example works - https://github.com/binance-exchange/binance-websocket-examples

@VadimsGrusas
Copy link
Contributor

That issue looks similar to:
netty/netty#10191
#3803

@feelus
Copy link
Author

feelus commented Feb 8, 2021

That issue looks similar to:
netty/netty#10191
#3803

I thought so too, I tried to fix it as it was used here #3803 but did not seem to help. I'm not a Java guy so I'm not sure if maybe I accidentally messed something up, so maybe someone else can confirm

@earce
Copy link
Collaborator

earce commented Feb 8, 2021

@feelus it is related to #3803, what probably happened is that Binance upgraded the websocket and changed this flag. We need to update the library to be able to properly hand-shake with the websocket, I can approve a PR if someone submits it.

@earce
Copy link
Collaborator

earce commented Feb 8, 2021

Also in the future Discord is better for faster engagement about ongoing issues.

@feelus
Copy link
Author

feelus commented Feb 8, 2021

Also in the future Discord is better for faster engagement about ongoing issues.

I agree, but couldn't a mention of it anywhere

@earce
Copy link
Collaborator

earce commented Feb 8, 2021

@feelus search for discord on this projects readme

@paolo-rendano
Copy link
Contributor

my 10c$.
just made some experiments and now it works again. What I have done:
1 - get the code from coinbase
private WebSocketClientCompressionAllowClientNoContextHandler() {
super(
new PerMessageDeflateClientExtensionHandshaker(
6, ZlibCodecFactory.isSupportingWindowSizeAndMemLevel(), MAX_WINDOW_SIZE, true, true),
new DeflateFrameClientExtensionHandshaker(false),
new DeflateFrameClientExtensionHandshaker(true));
}
2 - override BinanceStreamingService:
@OverRide
protected WebSocketClientExtensionHandler getWebSocketClientExtensionHandler() {
return WebSocketClientCompressionAllowClientNoContextHandler.INSTANCE;
}

Look at the code at 1. I had to set both parameters to true to get it work. The second is "requestedServerNoContext" that I had to set to true.

Hope it helps, maybe this is only a workaround. I don't know what Binance changed.

@feelus
Copy link
Author

feelus commented Feb 8, 2021

@feelus search for discord on this projects readme

Oh damn, it's an image, can't find it with search. Ok!

@Greekcc
Copy link

Greekcc commented Feb 8, 2021

my 10c$.
just made some experiments and now it works again. What I have done:
1 - get the code from coinbase
private WebSocketClientCompressionAllowClientNoContextHandler() {
super(
new PerMessageDeflateClientExtensionHandshaker(
6, ZlibCodecFactory.isSupportingWindowSizeAndMemLevel(), MAX_WINDOW_SIZE, true, true),
new DeflateFrameClientExtensionHandshaker(false),
new DeflateFrameClientExtensionHandshaker(true));
}
2 - override BinanceStreamingService:
@OverRide
protected WebSocketClientExtensionHandler getWebSocketClientExtensionHandler() {
return WebSocketClientCompressionAllowClientNoContextHandler.INSTANCE;
}

Look at the code at 1. I had to set both parameters to true to get it work. The second is "requestedServerNoContext" that I had to set to true.

Hope it helps, maybe this is only a workaround. I don't know what Binance changed.

Where did you found that code? On CoinbaseProSteamingService?

@paolo-rendano
Copy link
Contributor

my 10c$.
just made some experiments and now it works again. What I have done:
1 - get the code from coinbase
private WebSocketClientCompressionAllowClientNoContextHandler() {
super(
new PerMessageDeflateClientExtensionHandshaker(
6, ZlibCodecFactory.isSupportingWindowSizeAndMemLevel(), MAX_WINDOW_SIZE, true, true),
new DeflateFrameClientExtensionHandshaker(false),
new DeflateFrameClientExtensionHandshaker(true));
}
2 - override BinanceStreamingService:
@OverRide
protected WebSocketClientExtensionHandler getWebSocketClientExtensionHandler() {
return WebSocketClientCompressionAllowClientNoContextHandler.INSTANCE;
}
Look at the code at 1. I had to set both parameters to true to get it work. The second is "requestedServerNoContext" that I had to set to true.
Hope it helps, maybe this is only a workaround. I don't know what Binance changed.

Where did you found that code? On CoinbaseProSteamingService?

exactly.
info.bitrich.xchangestream.coinbasepro.netty.WebSocketClientCompressionAllowClientNoContextHandler

@earce
Copy link
Collaborator

earce commented Feb 8, 2021

You guys can probably create a new class for this

@paolo-rendano
Copy link
Contributor

You guys can probably create a new class for this

yes, I created a new class. You can copy the Coinbase one and change the parameter to true. and then reference it in the BinanceServiceStreaming to use that.

@Greekcc
Copy link

Greekcc commented Feb 8, 2021

Can someone make a PR, i am still trying to make it work.

@paolo-rendano
Copy link
Contributor

Sure I can, but as I said, not clear why the "true" solves the problem. It is "magic". I don't see anything on it in Binance API changelog. Maybe also the problem will be solved by them without changing anything.

@Greekcc
Copy link

Greekcc commented Feb 8, 2021

I have done the same thing as you did, but i still getting the same error. Maybe they have done something with Geographic location or i don't know

@ajutzeler
Copy link

I did exactly what @paolo-rendano suggested as well, but it did not help.

@paolo-rendano Can you try again without your fix? Maybe it now works for you all the same?

@paolo-rendano
Copy link
Contributor

@ajutzeler sure the old code does not work, since I have it in production in amazon AJP datacenter. Just restarted and still broken there. The same error here from Italy. Now I'm building the code with my change and I'll test it also in production. I'll update you.

@VadimsGrusas
Copy link
Contributor

my 10c$.
just made some experiments and now it works again. What I have done:
1 - get the code from coinbase
private WebSocketClientCompressionAllowClientNoContextHandler() {
super(
new PerMessageDeflateClientExtensionHandshaker(
6, ZlibCodecFactory.isSupportingWindowSizeAndMemLevel(), MAX_WINDOW_SIZE, true, true),
new DeflateFrameClientExtensionHandshaker(false),
new DeflateFrameClientExtensionHandshaker(true));
}
2 - override BinanceStreamingService:
@OverRide
protected WebSocketClientExtensionHandler getWebSocketClientExtensionHandler() {
return WebSocketClientCompressionAllowClientNoContextHandler.INSTANCE;
}

Look at the code at 1. I had to set both parameters to true to get it work. The second is "requestedServerNoContext" that I had to set to true.

Hope it helps, maybe this is only a workaround. I don't know what Binance changed.

Tried it locally, thank you. that helped

@paolo-rendano
Copy link
Contributor

Ok, I confirm. Back to working in production as well. I'll open the PR for that, but you should wait some hours for that. I'm on a different branch, so I have to take some time to do that.

paolo-rendano added a commit to paolo-rendano/XChange that referenced this issue Feb 8, 2021
@paolo-rendano
knowm#3907 Quick fix for binance stream due to invalid websocket exce…
ef38c98 
…ption
@paolo-rendano
Copy link
Contributor

I pushed the fix. who will merge that in develop?

paolo-rendano added a commit to paolo-rendano/XChange that referenced this issue Feb 8, 2021
@paolo-rendano
knowm#3907 Quick fix for binance stream due to invalid websocket exce…
d3ba079 
…ption
paolo-rendano added a commit to paolo-rendano/XChange that referenced this issue Feb 8, 2021
@paolo-rendano
knowm#3907 Quick fix for binance stream due to invalid websocket exce…
057c560 
…ption
@earce earce closed this as completed in 906e030 Feb 8, 2021
@ajutzeler
Copy link

I do not want to be a killjoy, but I built everything clean from @paolo-rendano branch and I am still getting the same error.

If I am the only one in this situation, I gladly accept my fate of being such a big noob.

@earce
Copy link
Collaborator

earce commented Feb 8, 2021

@ajutzeler noobs are welcome :) but did you try checking out from trunk?

@paolo-rendano
Copy link
Contributor

paolo-rendano commented Feb 8, 2021 •
edited
Loading

I do not want to be a killjoy, but I built everything clean from @paolo-rendano branch and I am still getting the same error.

If I am the only one in this situation, I gladly accept my fate of being such a big noob.

I'm sure the point is solved since just after the change everything is running fine again both in local and on aws. Just to give you some hints, the xchange project is quite large and maybe the ide sometimes makes some jokes on maven artifacts. would be better if you reload the maven project (in IntelliJ) or if you are on CLI to run mvn clean install -U to be sure you are really running the updated project.
Another hint: put a debug on the line I've added in the Binance service and run in debug to check this is passing from there.

@marcohc
Copy link

marcohc commented Feb 8, 2021

Hi guys, this might be a bit offtopic but, is there a way cloning the project to make use of the fix in a different project avoiding to wait for the next release? thanks!

@earce
Copy link
Collaborator

earce commented Feb 8, 2021

The best way is using the snapshot jars as dependencies

@ajutzeler
Copy link

@paolo-rendano Yes, I completely removed my local maven repo before any attempt and used mvn clean install to build the project.

I checked and I am loading the correct binaries that have just been built.

Added a couple of debug lines just to be sure, and of course it does show.

Can we reasonably assume that Binance is using an heterogeneous pool of websocket servers? Which could explain why only certain user on certain market were impacted by this issue. And why I still am...

Can anyone try to reproduce it with MarketData trades on EGLD_USDT?

@paolo-rendano
Copy link
Contributor

paolo-rendano commented Feb 8, 2021 via email

@ajutzeler
Copy link

Sorry, it is actually the UserTrade stream that crashes!

I can successfully receive market data (orderbook and trades) as well.

Please check it and tell me that you can reproduce it so I can clear my name ;)

@paolo-rendano
Copy link
Contributor

paolo-rendano commented Feb 8, 2021 via email

@earce
Copy link
Collaborator

earce commented Feb 8, 2021

If this fixes it submit another PR

paolo-rendano added a commit to paolo-rendano/XChange that referenced this issue Feb 8, 2021
@paolo-rendano
knowm#3907 Extended fix for binance stream due to invalid websocket e…
b244962 
…xception on user data subscription
@paolo-rendano
Copy link
Contributor

paolo-rendano commented Feb 8, 2021 via email

@makarid
Copy link
Collaborator

makarid commented Feb 8, 2021 via email

@paolo-rendano
Copy link
Contributor

paolo-rendano commented Feb 8, 2021 via email

@ajutzeler
Copy link

@paolo-rendano Yes, I confirm it works again thanks to your fix.

Thank you, Sir!

@2pd
Copy link

2pd commented Feb 8, 2021

From the API document, https://binance-docs.github.io/apidocs/spot/en/#change-log
there is nothing change related to Websocket service. Only a few new RESTful endpoints added.

@earce
Copy link
Collaborator

earce commented Feb 8, 2021

@paolo-rendano there isn't a team, it's a bunch of individual maintainers/contributors for the library @timmolter is the creator and owner of the library, discord usually has some fairly active developers on it

earce added a commit that referenced this issue Feb 8, 2021
@earce
Merge pull request #3911 from paolo-rendano/binance-invalid-websocket…
cac4904 
…-fix-ud

#3907 Extended fix for binance stream due to invalid websocket exception
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@2pd @feelus @paolo-rendano @earce @marcohc @ajutzeler @makarid @Greekcc @VadimsGrusas