Document use of AWS ELB

[oberstet]

Scenario: run Crossbar.io on AWS EC2 instances behind an AWS ELB. When ELB runs in raw TCP-mode, this should work.

But from the mailing list:

this is what I've tried so far
(elb = elastic load balancer, ec2 = elastic compute 2 which is basically the Amazon server)

elb tcp 443 to ec2 tcp 443
elb tcp 443 to ec2 tcp 80

elb ssl 443 to ec2 tcp 443
elb ssl 443 to ec2 tcp 80

elb ssl 443 to ec2 ssl 443
elb ssl 443 to ec2 ssl 80

elb https 443 to ec2 https 443
elb https 443 to ec2 http 80

I've tried the above with proxy protocol turned on and off

security groups for the elb and ec2 instances are set to allow all traffic

I've tried with nginx terminating the ssl and proxying to the ec2 instance

the tried using external_port set to 443 in the crossbar config.json

none of these worked to establish the websocket connection. errors I've received were 400, 500, upgrade header missing, connection lost and connection unreachable.

what does work is connecting directly to the ec2 instance and connecting directly to nginx proxying to an ec2 instance.

I was able to get the crossbar static web service working with the elb but the websocket would not connect.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[gregkeys]

@oberstet any update to this, I gave it another round of testing with 0.12.1 and was not able to get a connection, Im able to get the secure webserver working but no luck on getting a websocket connection

[oberstet]

FWIW, Mozilla is running its autopush service on top of AutobahnPython - currently (2016/04), up to 6 million long-lived WebSocket connections, 30 EC2 instances, all via ELB in one AWS region. They terminate the TLS on the ELB, and forward plain TCP (WebSocket) to AutobahnPython. They plan to ramp to to 10's of millions of connections.

As Crossbar.io is based on AutobahnPython, that means, it does indeed work (ELB).

We nevertheless should investigate ..

[gregkeys]

it's been a while since i did the tests, i don't think i spent much time terminating at the load balancer because we want to control the TLS; amazon uses insecure cyphers as a default with most if not all of their services.

we're moving away from Amazon dependencies as a result and relying more and more on docker solutions, we're rolling our own load balancers now.

[oberstet]

ELB using insecure ciphers .. the latest policy (2015-05) doesn't look thaat bad http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-security-policy-table.html - well, AES128-SHA and DES-CBC3-SHA .. bad.

However, I'd never give them a way to impersonate me (uploading my key/cert to them) anyway. That doesn't feel right.

For our public demo instances, we let Crossbar.io terminate the TLS. No load-balancer at all, but Route 53 latency-based routing to the nearest AWS region.

This works great - only issue left: need to figure out how to let the certs update automatically, as we're using Let's Encrypt, and that doesn't know about the multiple instances (with different keys) behind a single hostname.

[gregkeys]

yeah it works great having crossbar terminate tls.

we've also done the latency based routing it works great for cross region balance , however solving for local load balancing isn't so easy, currently exploring ha proxy on docker.

clustering should help once it's ready

[blues-man]

Hello,
I can finally confirm, at least by my direct experience, that everything is fine with AWS ELB and Crossbar, either terminating TLS at ELB, Reverse Proxy and Crossbar level.

The most important part is to have a TCP listener for the Balancer, for which is also possible to use Proxy Protocol.
In my case I tested this pipeline:

Cross-zone ELB TCP raw Proxy Protocol-> NGINX SSL Proxy Protocol-> EC2 with Crossbar 0.13.x on default port 8080

Another important thing to understand in this scenario, is tunnel timeout. By default, ELB sets Idle Timeout to 60s, this timeout take precedence over client and server timeout, thus any timeout set at NGINX or Crossbar level, could be uneffective having connection simply dropped by the ELB after that amount of time (I've seen a FIN,ACK explicitly from the ELB IP)

I also put another TCP Internal ELB between NGINX and Crossbar and everything is fine, with Idle Timeout set to a greater value, 3600s is the maximum.

Hope it helps, but I would say "yes ELB works but use TCP listener, Proxy Protocol and increase Idle Timeout"