CORS pre-flight breaks socket.io behind load balancer #279
Comments
|
This is one of the best issue descriptions I've read in a while. Thanks for taking the time. Can you send me the complete headers of the request that yields Requests that do not need pre-flight according to MDN are: Before proceeding with the OPTIONS fix, I want to make sure it's happening for a good reason. |
|
Plus the performance hit and additional server load.... do what you can to avoid them as part of your connection. |
|
+1 :datfeel: |
|
There are two issues here. We should be supporting OPTIONS unconditionally so that even if the browser makes a pre-flight it doesn't break the existing session stickiness. And we should also see if we can make engine.io not do the pre-flight request at all. |
|
I think the default behavior should stay as today (break) since in 90% of cases it's unintended behavior and we don't want this happening silently. We can add a flag that if explicitly set the server sends OK on every OPTIONS request. |
|
Agreed with @mokesmokes — On Tue, Sep 2, 2014 at 9:14 PM, mokesmokes notifications@github.com wrote:
|
|
@guille Thanks for looking into this so quickly! The |
|
Why don't we want this happening silently? I would think we want this to just work without messing about with settings. |
|
@defunctzombie because in many cases preflighted requests occur due to careless coding, and can be avoided. I think if they occur the dev should be fully aware that they are happening, not make it a silent default decision. Just my $0.02 :) |
|
@mokesmokes Here the preflight requests are occurring because of the request's content-type. Is choosing this content-type a decision made by socket.io? I am not configuring it anywhere explicitly. If that's the case then the default treatment of the OPTIONS req makes sense. |
|
We should fix this. OPTIONS preflights can happen when doing CORS stuff and this behavior is really annoying to an end user to deal with. It basically renders this module unusable behind the amazon ELB even when sticky sessions are on. I do not think this has anything to do with careless coding and is happening under the simplest uses of this module. |
|
For anyone that runs into this before we fix it. You can work around it by using the manual request handling (on request from the http server) and responding to OPTIONS yourself. |
|
Yup, it's not the user's fault, it's engine.io: https://github.com/Automattic/engine.io-client/blob/master/lib/transports/polling-xhr.js#L166 |
|
So basically what I think we need is:
So basically, by default all works smoothly and no OPTIONS as most people would probably prefer. If people insist on using octet-stream then they should manually configure this, but it should be supported. I really don't think we should just send binary data blindly in CORS scenarios. |
|
+1. Big time priority. On Mon Dec 01 2014 at 3:17:05 AM Mark Mokryn notifications@github.com
|
|
Just to get this straight, you would rather have a increased file size for the data that you transfer (for every single message) instead of a one time OPTIONS request? The upgrading of transports is already happening through probing so it shouldn't delay the current connection and I highly doubt that a single extra request to your sever is so heavy that it would affect performance or dramatically increases server load. The suggestions solutions only add more odd edge cases on the client instead of a simple fix on the server. This doesn't make a lot of sense to me. |
|
Yes. Bandwidth is hardly ever the bottleneck, latency is. |
|
In this case, an additional OPTIONS request guarantees an extra roundtrip for us. The extra bandwidth cost for frames would hardly ever result in extra roundtrips. |
|
@guille But latency isn't an issue here as you are already connected using |
|
@3rd-Eden it's only a one-time request if Access-Control-Max-Age is honored by the browser, and I'm not sure (bears checking) if we satisfy the requirements. Even then, browser performance is inconsistent, see http://stackoverflow.com/questions/23543719/cors-access-control-max-age-is-ignored EDIT: I think we will see OPTIONS on every request due to our cache busting :-/ |
|
+1 |
|
Need this for the fix: socketio/engine.io-parser#36 , so we can encode binary data when sending in a CORS situation |
|
Mildly related: #300 |
|
+1 for this issue |
|
For anyone else with this issue, the following code will fix it: module.exports = function(srv) {
var listeners = srv.listeners('request').slice(0);
srv.removeAllListeners('request');
srv.on('request', function(req, res) {
if(req.method === 'OPTIONS' && req.url.indexOf('/socket.io') === 0) {
var headers = {};
if (req.headers.origin) {
headers['Access-Control-Allow-Credentials'] = 'true';
headers['Access-Control-Allow-Origin'] = req.headers.origin;
} else {
headers['Access-Control-Allow-Origin'] = '*';
}
headers['Access-Control-Allow-Headers'] = 'origin, content-type, accept';
res.writeHead(200, headers);
res.end();
} else {
listeners.forEach(function(fn) {
fn.call(srv, req, res);
});
}
});
};Applied after socket.io: e.g: io.listen(server);
handleOptions(server); |
|
There is a small addition to the above code that is needed to fix it. module.exports = function(srv) {
var listeners = srv.listeners('request').slice(0);
srv.removeAllListeners('request');
srv.on('request', function(req, res) {
if(req.method === 'OPTIONS' && req.url.indexOf('/socket.io') === 0) {
var headers = {};
if (req.headers.origin) {
headers['Access-Control-Allow-Credentials'] = 'true';
headers['Access-Control-Allow-Origin'] = req.headers.origin;
} else {
headers['Access-Control-Allow-Origin'] = '*';
}
headers['Access-Control-Allow-Methods'] = 'GET,HEAD,PUT,PATCH,POST,DELETE';
headers['Access-Control-Allow-Headers'] = 'origin, content-type, accept';
res.writeHead(200, headers);
res.end();
} else {
listeners.forEach(function(fn) {
fn.call(srv, req, res);
});
}
});
};This solved the problem for me. |
|
Has this been fixed yet?
|
|
Hello, I'm trying to set up socket.io servers behind AWS ALB.
Because the web application and the ALB are not on the same domain, OPTIONS requests are sent during the handshake phase for long-polling. While I agree that the aforementioned points should be addressed to avoid doing OPTIONS requests, OPTIONS requests should not trigger all the described logic. The rfc2616 says:
I've come up with a way to by-pass the verification steps and delegate earlier the processing of the OPTIONS requests to the transport with MiLk@c8012eb I'm willing to spend more time on this issue. I'll explore the other solution which is trying to disable CORS by changing the content-type and see where I go with that. Edit: |
|
Hi @MiLk, I am also trying to use socket.io behind an AWS ALB. I see a lot of peoples having the same issue on stackoverflow, it would be really helpful! Thanks! |
|
Hi @JohnCoding94, We ended-up disabling the polling transport and forcing all the clients to use websockets, because the What I did in #484, would allow the handshake to work, but the session would still not be right. However, it might work if you are not using CORS, because the only issue I can think of is the OPTIONS request done during the handshake. https://github.com/socketio/socket.io-client/blob/master/docs/API.md#with-websocket-transport-only We have now a few thousands WS connections established staying stable over 24h periods. We force connections to be closed in the middle of the night). |
|
The flow I currently see is:
In 1., the session is created. socket.io assigns a sid and ALB create a new cookie. All of that is happening because:
The only way to make the polling transport in the current state of ALB would be to avoir the preflight requests. However, it should work fine with any loadbalancer using a consistent hashing algorithm, or without setting a new cookie on preflight requests. Basically, the issue didn't change much since the original post. #484 is just making it fail at the POST request instead of the OPTIONS request. |
|
@MiLk You may already be aware of that but just for information, i ended up setting a connection from a subdomain to my racine domain using SSL, and any OPTIONS request is made, i didn't set any specific options on my socket.io server. So it performs the classic long pooling handshake, then switch to websocket protocol, and everything seems to work fine. |
|
@MiLk maybe a |
|
No, it's because the OPTIONS request get a new ALB session cookie, which makes the POST request to go to a server where the socket.io session doesn't exist. |
|
What I meant is adding
|
|
I'm trying to reproduce the issue, but so far the connection seems actually stable. I'm using https://github.com/socketio/engine.io/tree/a63c7b787c54b3a47da7f355826bf2770139c62b. var app = require('express')();
var cors = require('cors');
app.options('*', cors({
origin: true,
methods: 'POST',
allowedHeaders: ['Content-Type'],
credentials: true,
}));
var server = require('http').Server(app);
var io = require('socket.io')(server, {
handlePreflightRequest: false
});
... |
|
The issue is still happening on Safari |
|
What solved the same issue for me: @MiLk How do you receive a |
|
Chrome and FF seem to be both OK. Only Safari 10.1.1 has an issue. However, my socket.io client and server libraries are not up-to-date (1.1.0 and 1.8 respectively). |
|
Any updates on this issue? Is it really enough to use |
|
Using We didn't do any change on the server since last August (except Node version upgrade). The setup is what I described earlier in #279 (comment) |
|
That should be fixed in engine.io v4 (see 61b9492). The new syntax: new Server({
cors: {
origin: "https://example.com",
methods: ["GET"],
allowedHeaders: ["Authorization"],
credentials: true
}
})Previously: new Server({
handlePreflightRequest: (req, res) => {
res.writeHead(200, {
"Access-Control-Allow-Origin": 'https://example.com',
"Access-Control-Allow-Methods": 'GET',
"Access-Control-Allow-Headers": 'Authorization',
"Access-Control-Allow-Credentials": true
});
res.end();
}
}) |
I ran into an issue on our servers. We are running socket.io v1.0.6 on multiple server instances behind a load balancer. For polling, the requests go through the ELB with sticky sessions turned on. Our real-time service is on a subdomain, and thanks to CORS pre-flight requests, socket.io fucks up. Here is what happens on the client when the polling transport is used:
sid, and the headers include the AWS ELB cookie.OPTIONSrequest is made by the browser. The ELB cookie is not included by the browser here. As a result, theOPTIONSrequest is routed to a potentially different server which will not recognize thesidin the query string.Session ID unknownerror.OPTIONSpre-flight request fairly regularly as opposed to doing it only once, so this cycle repeats over and over.The fix on our end currently is to respond to all
OPTIONSrequests with a 200 and all the usualAccess-Control-Allow-…headers the browser knows and loves. We do this before they even get to socket.io in our nginx config.Now, engine.io appears to already handle this case here: https://github.com/Automattic/engine.io/blob/master/lib/transports/polling-xhr.js#L40
However, that check is only reached if the
sidis valid here: https://github.com/Automattic/engine.io/blob/master/lib/server.js#L180which it isn't, of course. I can submit a PR but I'd like to know how you guys think it'd be best to handle this. AFAIK, if a request method is
OPTIONS, we can make the assumption that we are polling. But, since we don't have a validsidto look up a client by, this might mean moving fairly transport-specific logic intoserver.jswhich sounds less than ideal.Thoughts?
The text was updated successfully, but these errors were encountered: