xhr_streaming infinite loop after connection closed #308
Comments
|
We have the same problem. You can reproduce disconnecting internet and then, after a few second reconnect. I could see when my server close de session for lack of communication by the client( c[3000, go away!]), then the loop occurs on the client.
|
|
I'm having trouble reproducing this on sockjs-client |
|
@brycekahle i use Java Spring as backend technology. I can see when disconnect and reconnect how sockjs-client try to use the old and new xhr_streaming sessión. The backend close the connection but again and again receive xhr_streaming with the old session. the backend respond with c[3000, "go away!"]. Note: My backend log:
We can see in this screen how sockjs-client have 2 sessions active: @brycekahle Can you explain me where and how sockjs-client manage xhr_streaming invokes? i think that sockjs-client after reconnect (i delete SockJS object in reconnections) maintains the old xhr_streaming petitions (memory leak?) Edit:
|
|
I could play in local turning off and starting the server several times (Not always happen). I cant understand how SockJS once reconnected ( We destroy and create the Sockjs instance from 0 ) send xhr-streaming requests with the id of the old session ( Maintains the pre-reconnection id session? ). I have spoken with other colleagues and I all agree in saying that SockJS not managed well the connection drops. I activated the traces of SockJS and this brings me back : New Sessión Id -> h2pystok `<<< PONG browser.js:120 sockjs-client:receiver:xhr finish +0ms 200 hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh browser.js:120 sockjs-client:receiver:xhr _chunkHandler +0ms 200
Im using: |
|
I found where is the possible error. In polling.js i have see a special treatment when the close reason is 'network'. When is network again call the function _scheduleReceiver(). When we reconnect the infinite loop occurs. I dont know what is the reason for this treatment but I could try this deleting the special treatment of 'network' and everything works correctly. @skozin Can you try?
the workaround is:
|
|
@gravitycode @brycekahle thanks, I'll try your suggestions in a few days. For us, the quickest solution was to temporarily disable our disconnection logic on the backends, but it is not a solution really, so I'll need to fix this pretty soon. |
|
After a few days testing , I have found that in iOS (Iphone 6 with the latest version of iOS ) can not connect . I have solved the infinite loop in desktop but now I can´t connect in mobile |
|
Reproduced on v1.1.0 and with NettoSphere+AtmosphereSockJs on a server side |
|
We are seeing this problem in production..any one has workaround to fix it? |
|
We have encountered this problem as well. I have successfully replicated it using Fiddler (HTTP(S) transport watcher for Windows):
This "buffering proxy" should be somehow detected by SockJS and the protocol should be downgraded to the polling one as it is the only one that will work within such environment. |
|
@petr-ujezdsky i detected the same issue with my firewall. The problem was buffering mode in this layer. We had to downgrade the transport protocol to xhr_polling until the firewall provider made the changes to support streaming |
|
Hi! We are heavily hit by this issue. Basically we have a few customers, that is ddos'ing our production servers because the sockjs client is calling in an infinite loop. Any info on a possible fix? We will try and implement the workaround mentioned by @gravitycode. But it would be nice to get an official fix. |
|
@sudpaw Remember that my workaround didnt work in my phone (iphone). What kind of transport are you using? websockets, xhr-streaming? |
|
@gravitycode We are mainly using websocket. But we have some customers that are behind corporate proxies and they are getting xhr_streaming - this is where it is hurting us. |
|
i could try with only this transports: ['websockets', 'xhr-polling']. xhr-streaming can have many errors with some layer that implement buffering like firewalls, traffic check, etc. I had that problem with my firewall. |
|
BTW one of my collegues have found another major issue, this time in |
|
Using websocket protocol after a connection loss sockJS was slamming our server indefinitely. @gravitycode solution worked for us. We are using spring 4.3 with spring security. Not too concerned about iPhone so this is great, thanks @gravitycode |
|
Hey folks, can you try 1.1.2 and let me know if you can still reproduce the problem? |
|
Looks like 1.1.2 has resolved the issue. Thanks! |
|
The problem has been resolved! Excellent! Thank you so much |
|
Still having this issue even on 1.1.2. |
|
We also still have this issue with a backend using |
|
Given that it resolved the issue for other folks, it is possible you are having a different issue. Can you please provide more details and repro steps on your issue? |
|
I still have this issue in v1.1.2. We are using "sockjs-client" and "stompjs" on client side and "spring-websocket" v4.2.7 on the server side. Our app times out after x minutes. after timeout, we disconnect our stomp client and close the websocket. This happens as expected and the app doesn't makes any calls to server for sometime. After some minutes, they is flood of POST request of 'xhr_streaming' and it stops sometimes after 100 requests, sometimes 25, but the app keeps doing in regular intervals. @gravitycode's workaround fixed this issue. Thank you. Connect Code: Disconnect code: I checked the code. In polling.js, in this part of code. when websocket is closed, we get reason = 'user' and pollIsClosing = 'true' and control doesn't go inside the first if, as it check whether 'pollIsClosing' is undefined. So, i changed the logic to: This did not fix the issue, again after sometime, this method is called with reason='network' and floods our server with network requests. |
Please refer to this issue for more information:sockjs#308
|
I had the same issue. My app was running on secure "https://" where as I am trying to fetch the sockjs-0.3.min.js from an "unsecured CDN location" which fails to download the .js file. Which causes it to look for Xhr_Streaming instead of my sockjs methods while disconnecting the socket. I fixed this issue by changing the "unsecured cdn path to secured cdn path" i.e., |
|
I have the same problem with sockjs 1.1.4, spring boot websocket starter 1.5.10 |
|
Same problem with sockjs 1.1.4, spring boot websocket |
|
Here is our workaround for this issue: var sockJs = new SockJS(...);
var _transportClose = sockJs._transportClose;
sockJs._transportClose = function(code, reason) {
if (this._transport && this._transport.close) {
this._transport.close();
}
_transportClose.call(this, code, reason);
}In our case, loop was triggered by (occasional) initialization timeout in xhr-polling transport. After the timeout, function _transportTimeout calls _transportClose, and _transportClose removes all listeners from transport, however, transport remains active. When transport receives message from spring with close request (like c[3000,Go away!]), no one will process this message (because there are no listeners!), and transport will loop (because response code is 200). Update: I see this is already fixed in master, but not in 1.1.4 |
|
@brycekahle any chance of a release? I encountered this problem using sockjs via Primus (primus/primus#635 (comment)), hand-patching with eef1957 fixed it but it would be nice to be able to rebuild Primus and depend on an official sockjs version. |
|
@brycekahle, We also have problems with spawning multiple POST requests. |
|
mark！Have the same problem |
|
I have xhr_streaming hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh it does it two times when i refresh |
|
Still can reproduces. Webpack dev server spams with XHR requests. Why it couldnt be fixied somehow? It is a critical ussue, which doesn't allow to use web pack dev server at all. |
|
@asartem Is webpack using newer versions of sockjs-client? Do you have code you can share with a reproduction? |
|
I got this because of my shadowsocks, I turned it off and no more errors. |
|
I spent a lot of time debugging SockJS and Spring code to figure out with that issue and what I have found. xhr.js file has next method to parse inbound text message: XhrReceiver.prototype._chunkHandler = function(status, text) {
debug('_chunkHandler', status);
if (status !== 200 || !text) {
return;
}
for (var idx = -1; ; this.bufferPosition += idx + 1) {
var buf = text.slice(this.bufferPosition);
idx = buf.indexOf('\n');
if (idx === -1) {
break;
}
var msg = buf.slice(0, idx);
if (msg) {
debug('message', msg);
this.emit('message', msg);
}
}
};That method expects \n at the end of the SockJS text frame. But in case of "Connection already closed (but not removed yet) forsessionId" Spring Framework sends unformatted SockJsFrame.closeFrameGoAway() without ending \n. AbstractHttpSendingTransportHandler.class protected void handleRequestInternal(ServerHttpRequest request, ServerHttpResponse response,
AbstractHttpSockJsSession sockJsSession) throws SockJsException {
if (sockJsSession.isNew()) {
if (logger.isDebugEnabled()) {
logger.debug(request.getMethod() + " " + request.getURI());
}
sockJsSession.handleInitialRequest(request, response, getFrameFormat(request));
}
else if (sockJsSession.isClosed()) {
if (logger.isDebugEnabled()) {
logger.debug("Connection already closed (but not removed yet) for " + sockJsSession);
}
SockJsFrame frame = SockJsFrame.closeFrameGoAway();
try {
response.getBody().write(frame.getContentBytes());
}
catch (IOException ex) {
throw new SockJsException("Failed to send " + frame, sockJsSession.getId(), ex);
}
}SockJsFrame public static SockJsFrame closeFrame(int code, @Nullable String reason) {
return new SockJsFrame("c[" + code + ",\"" + (reason != null ? reason : "") + "\"]");
}As a quick and dirty fix I just created copy of org.springframework.web.socket.sockjs.frame.SockJsFrame and put in my classpath to substitute the original file with modified one. public static SockJsFrame closeFrame(int code, @Nullable String reason) {
return new SockJsFrame("c[" + code + ",\"" + (reason != null ? reason : "") + "\"]\n");
}But in reality AbstractHttpSendingTransportHandler class should be fixed to apply format to the SockJS frame. SockJsFrameFormat frameFormat = this.getFrameFormat(request);
SockJsFrame frame = SockJsFrame.closeFrameGoAway();
String formattedFrame = frameFormat.format(frame);
try {
response.getBody().write(formattedFrame.getBytes(SockJsFrame.CHARSET));
}After that fix all works as expected and c[1000, "Go Away!"] successfully routes to the 'c' section of next switch SockJS.prototype._transportMessage = function(msg) {
...
switch (type) {
case 'a':
if (Array.isArray(payload)) {
payload.forEach(function(p) {
debug('message', self.transport, p);
self.dispatchEvent(new TransportMessageEvent(p));
});
}
break;
case 'm':
debug('message', this.transport, payload);
this.dispatchEvent(new TransportMessageEvent(payload));
break;
case 'c':
if (Array.isArray(payload) && payload.length === 2) {
this._close(payload[0], payload[1], true);
}
break;
}
};Without that fix message is not parsed as close message. |
and others
Steps to reproduce:
websocket: falsein options);conn.close(1000, 'init_message_timeout')(the exact code and message don't matter);1.0.1(latest) or0.3.4from the latest Chrome.During the first 500 ms, the connection stays alive. After that, it gets closed as expected. However, the
xhr_streamingtransport starts an infinite loop: it tries to connect to the server, then SockJS server (not the app logic) immediately closes that connection with the code/reason provided in the step 2, and, after that,xhr_streamingattempts to connect again, and so on:Here is HTTP log from the server:
This infinite loop takes significant amount of CPU on the client (up to 70%) and on the server and load balancers, effectively causing DDOS when there are multiple such forcefully-closed clients.
It seems to me that
xhr_streamingshould not attempt to reconnect after it gets user-provided close reason from the server. Or am I misunderstanding something?This issue hits us in production, and we need to resolve it as fast as possible.
The text was updated successfully, but these errors were encountered: