-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SockJS issue through many layers of proxying (debugging) #1358
Comments
So, if you are using tomcat7, you have to use the absolute most recent version, and the default Ubuntu 12 / 14 tomcat7 is typically not recent enough to support websockets. Sorry, should have thought of that sooner. You can drop a jar in, or use the tomcat8 container, which should work. That being said, if you don't have it installed it will fall back to regular long-polling. The web-services will be fine with or web sockets. You can tell because in the browser you'll see something like this: Also, have you looked at https://dockstore.org/? .. not sure what the scope of your project is. |
|
If you’re getting 500 errors, it must be failing on the server-side. The first 400 sometimes happens as the interface is refreshing itself.
Look in the catalina.out logs and see if anything is showing up.
Nathan
… On Dec 1, 2016, at 5:09 PM, Eric Rasche ***@***.***> wrote:
We're using tomcat 8
The problem doesn't seem to be fallback, it seems to be that websockets fail, and then all of the fallback methods trigger a 500 when SockJS thinks the hostname is invalid.
<https://cloud.githubusercontent.com/assets/458683/20819308/6b280744-b82b-11e6-9897-1a1b76bebb32.png>
Dockerstore is something quite different, they're just for packaging up bio-software in containers (of which there are dozens of those types of projects, "bioboxes", "biodckr", mulled <https://github.com/mulled/auto-mulled/>)
Rancher is for orchestration. Think kubernetes / docker swarm + web ui.
<https://cloud.githubusercontent.com/assets/458683/20819399/eff4f1a8-b82b-11e6-95de-50d5a8fa7286.png>
<https://cloud.githubusercontent.com/assets/458683/20819405/002753d6-b82c-11e6-9478-a3fefd01efc6.png>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <#1358 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAt2qnt3Yy3Jf5s5M2vhIyati9mubsOpks5rD2_bgaJpZM4LCECn>.
|
Catalina logs. I'm not seeing the 400 bad request that's specific to the websockets, but I do see the 500s.
|
I don't think its an Apollo issue per se. |
Yeah, both of those came up in my searches, but neither yielded useful solutions. As mentioned in first comment, I can make the request from localhost (i.e. apollo container), but as soon as I get one container away, I cannot and I'm not sure why. |
This likely isn't an apollo issue, it's likely a configuration problem somewhere in the stack. E.g. a specific hostname needs to be provided somewhere, some magic flag needs to be set on one of the proxies, etc. Opened issue to track debugging + in case I had documentation if/when I fixed this that other people could make use of. |
Hmm, from localhost, it works if it goes to 127.0.0.1. But if the request is made to a different IP address associated with the same machine, it fails. (apollo.apollo resolves to same server as localhost)
|
Adding
in my apache conf helps. I'm really struggling to understand why it's but it's lies and the XHR fallback still fails. |
Also http://stackoverflow.com/questions/28385798/host-name-cant-be-null-using-grails-spring-websocket-plugin seems exactly right but that answer is beyond useless. I've tried setting the hostname to all manner of things and nothing good comes of it. |
Update: Spent a long time wiring up JMX and mucking with that. No joy there either. |
I just saw this issue, do you still have proxy/websocket problems? internet ---> apache 2.4 proxy ---> nginx proxy ---> another nginx proxy --> apollo I can share my config if it can help |
@abretaud haven't tested with the latest image which is now based on tomcat:8, so that might be an improvement. I'll do a test deployment here soon. For us the chain is: internet → apache → haproxy → (docker networking) → apollo And intuition says it might be haproxy but I don't have any way to test websockets easily. (I really wish people would post demo client / servers for testing these stupid new protocols). Thanks for the offer, I'll ping you / this issue if I'm still experiencing it. I really, really want apollo deployed on rancher so it isn't separaetly managed since that's painful for me. |
Ok, no problem (though I've never used haproxy) |
@abretaud / @erasche I'm sure you've seen this, but wanted to repost. The default Ubuntu 14 version of tomcat 7 would not have worked (the more recent stable versions do). The long-polling fallback will work fine, though its not ideal (though I don't you're users would notice unless you have weird firewall rules). Configuring with haproxy for websockets looks like it may have been tricky. Have you tried removing haproxy from the equation? I guess you saw my comment above about how to confirm that they are working. You just have to watch the "frames" tab. |
@nathandunn. We don't use ubuntu14. We (used to) run tomcat:7 (which defaults to jre7), we don't use the ubuntu images as they make for gigantic docker images.
Not possible. It's heavily tied to rancher. However haproxy can trivially proxy even mysql connections, so I'm somewhat dubious that it's really to blame and not SockJS + java.
As you can see from the screenshot in #1358 (comment), that's not quite the case. We were having internal server errors during the fallback. Hence me thinking it was sockjs/java. Maybe the proxy was stripping a header / adding a header that caused issues. Hopefully the update to tomcat:8-jre8 fixes this, I should know later today. |
After the upgrade (still testing my image, has a bunch of other local changes), this is solved! 😂 Websockets still aren't proxying right, but hey, who cares, fallback works, I can move apollo off my frontend machine and on to compute infrastructure, and I can have my remote user stuff working now. Thanks for debugging input everyone. |
Weird. Did you explicitly setup apache or nginx proxy (we have some doc on this if not) or are you going straight through tomcat? Or you think its still the haproxy? Anyway, glad it worked one way or another. |
It wasn't any explicit changes, just the change to 1) more recent version of apollo (we were on 2.0.3? 4?) + 2) tomcat8/jre8 We hit apache and haproxy: We actually have nginx in the route as well, but that's just for a special case of API access. There the route goes The apache proxy hasn't changed at all, still using pretty standard proxying rules (proxypass, and wstunnel, quite similar to what your docs describe.) That works perfectly with the direct connection, so I'll make some efforts to figure out why that doesn't work over my proxy setup some other. Just happy to have made progress. |
I'm hitting this when trying to deploy to rancher. Opening this issue mostly to track my debugging.
My setup looks like:
Currently not sure if websockets are functional at all, not sure how I'd test that. It fails before that. Wonder if I can "get away" with
x-forwarded-for
type headers and that will fix this?curl 'http://localhost:8080/apollo-dev/stomp/084/yf3adjwl/xhr' -X POST -H 'Accept: */*' -H 'Cookie: JSESSIONID=......;' -i | head
curl 'http://10.42.182.231:8080/apollo-dev/stomp/084/yf3adjwl/xhr' -X POST -H 'Accept: */*' -H 'Cookie: JSESSIONID=....;' -i
The text was updated successfully, but these errors were encountered: