LBs stuck in Reinitialising and services in initialising after upgrading from ELB to ALB

[jurajseffer]

Rancher Versions:
Server: 1.1.4

Docker Version:

Containers: 67
 Running: 55
 Paused: 0
 Stopped: 12
Images: 111
Server Version: 1.11.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 805
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.0-53-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.795 GiB
Name: dev-stack-1
ID: 2UDS:TWSJ:CXZ4:NF4J:7ECY:A2TH:6ISU:MLCH:D3JN:H3XU:KIGO:EZFA
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

OS and where are the hosts located? (cloud, bare metal, etc):
AWS EC2

Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)
Single + external DB

Environment Type: (Cattle/Kubernetes/Swarm/Mesos)
Cattle

Steps to Reproduce:
Upgraded infra to use AWS ALB instead of ELB with proxy policies.

Security has not changed, I could manually curl and ping the container's IP address from the stack host as well as rancher-agent container.

Results:
Agents can connect as before, hosts show as online, containers can be deployed, UI works. Upon deploying, services with health checks are stuck in "initialising" or "reinitialising" (restart) state. Same of load balancers. Restarting an existing "running" container that was deployed before the switch results in "initialising" state.

Expected:
Containers pass as healthy.

[aemneina]

ALB only works for rancher v1.2.0+. There were no changes made to rancher 1.1.x to have ALB supported.

[jurajseffer]

I was wondering about that. But sockets work without a problem and so does SSL stuff. I'm surprised that load balancer affects health checks which from my understanding are ran from rancher agents using Rancher networking - in my case a single host.

[aemneina]

ALB did some weird url rewriting, which prompted us to shift our recommendation back to ELB. Have you had issues since @jurajseffer ?

[jurajseffer]

@aemneina We've switched to Rancher 1.3.0 with ALB and I've not noticed any problems since.