nginx proxy does not distribute client connections to all synapse workers

[JohannesKleine]

The ip_hash does not work as intended, only one synapse worker gets the requests. I assume it is through the cascading nginx proxy of client calls to synapse. In element the client ip was the one from the nginx proxy. Removiing this line:

matrix-docker-ansible-deploy/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2

Line 14 in 7cf2f35

ip_hash;

results in correct distribution to all synapse workers. Without ip_hash nginx falls back to round_robin. This could lead to more resource usage from ram and cpu, since one client might be handled by more than one worker. If one user uses more than one client simultaneously the differences between ip_hasch and round_robin gravitate to have less impact.

Work around: remove this line.

Possible fix: add some headers to roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 and roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2 so that the last nginx gets the original client ip for ip_hash. Help needed from an experienced nginx config writer.

Possible fix: integrate the content off roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 into roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2 so that instead of using the

matrix-docker-ansible-deploy/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2

Line 239 in 7cf2f35

set $backend "{{ matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container }}";

backend use

matrix-docker-ansible-deploy/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2

Line 79 in 7cf2f35

location ~ {{ location }} {

instead. Preferred method.

[spantaleev]

Good catch! This is indeed a problem.

A similar problem is being discussed here: https://serverfault.com/questions/660323/nginx-ip-hash-behind-proxy-cloudflare

I've tried one of the solutions, which I believe should work for us. I haven't tested it though. If you have multiple generic workers and can test it out, that'd be great!

[JohannesKleine]

Thank you, it looks like it is working, the cpu load is not equal on each worker, but i guess it depends on some power users. I will monitor it a while.