http/2 traffic stopping with hitch/varnish 5.2.0 #2431
Description
I'm not certain if i encountered a bug or design flaw. The Varnish 5.2.0 final version didn't solved my problems.
The sess_fail counter is raising to 33804924 in seconds if im enabling alpn-protos with h2 support in hitch. if im watching the logs, i see that only http2 requests getting logged, so i'm asume that h1 request can't pass anymore.
The effects reminds me about same issues (haproxy SSL offloading -> nginx, solution see: https://ispire.me/http2-ssl-offloading-with-haproxy-and-nginx/) i encountered when i was pointing/mixing traffic (h1 traffic to h2 or vice versa) to wrong endpoint protocols.
i ran hitch with this settings:
Hitch:
frontend = "[PUBLIC IP]:443+/etc/hitch/certs/certfile.pem"
backend = "[127.0.0.1]:6086"
alpn-protos = "h2, http/1.1"
write-proxy-v2 = on # Write PROXY header
Varnish 5.2.0:
-a PUBLIC IP:80 -a 127.0.0.1:6086,PROXY
-T 127.0.0.1:6082
-f /etc/varnish/myvcl.vcl
-p feature=+http2
-p thread_pool_add_delay=2
-p thread_pools=2
-p thread_pool_min=200
-p thread_pool_max=4000
-p syslog_cli_traffic=off
-t 120
-S /etc/varnish/secret
-s malloc,3G"
What i finally dont understand is how Varnish will distinguish between PUBLICIP:80 HTTP Traffic HTTP1 only coming directly via HTTP to Varnish and feature=+http2 (INTERNALIP:6068,PROXY HTTPS Traffic Protocol: HTTP1+HTTP2) coming via TCP from Hitch which will only listen for SSL Connections.
My current idea is to replace hitch with haproxy as SSL Offloader and doing same with varnish what i do on haproxy > nginx only setups already (see Howto) but for that i need a separate definition parameter h1+PROXY,h2+PROXY to point each protocol to the right listener.
So a definition like setting protocol per Listener would be necessary to make such setup possible:
PublicIP:80 Varnish (non SSL) -> Nginx 10.0.10.2:80
PublicIP:443 Haproxy (SSL Offloading) via Proxy Protocolv1 http/1.1 > Varnish 127.0.0.1:6443 -> Nginx 10.0.10.2:6443 (proxy_protocol)
PublicIP:443 Haproxy (SSL Offloading) via Proxy Protocolv1 h2 -> Varnish 127.0.0.1:6086 -> Nginx 10.0.10.2:6086 (proxy_protocol)
haproxy config:
frontend varnish-ssl
bind PublicIP:443 ssl crt /etc/haproxy/certs/combined.yourcert.pem alpn h2,http/1.1
mode tcp
use_backend varnish-ssl-http2 if { ssl_fc_alpn -i h2 }
default_backend varnish-ssl-http1-fallback
backend varnish-ssl-http2
mode tcp
option tcpka
server varnish-ssl-h2 127.0.0.1:6086 check-send-proxy inter 2500 maxconn 10000
backend varnish-ssl-http1-fallback
mode tcp
option tcpka
server varnish-ssl-h1 127.0.0.1:6443 check-send-proxy inter 2500 maxconn 10000
This is how the listener definition would look like in nginx:
listen 10.0.10.2:6086 http2 proxy_protocol; # haproxy TCP SSL termination + HTTP/2
listen 10.0.10.2:6443 proxy_protocol; # haproxy TCP SSL termination for HTTP/1.1 and lower
listen 10.0.10.2:80; # HTTP only Traffic (non SSL)
So in varnish i need something like this:
-a PUBLICIP:80 -a 127.0.0.1:6443,h1+PROXY -a 127.0.0.1:6086,h2+PROXY
And to finalize point each connection to the right end extending the varnish vcl by this:
if (std.port(local.ip) == 6086) {
set req.backend_hint = nginx_ssl_h1;
} elseif (std.port(local.ip) == 6443) {
set req.backend_hint = nginx_ssl_h2;
} else {
set req.backend_hint = nginx_h1;
}
backend nginx_ssl_h1 {
.host = "10.0.10.2";
.port = "6443";
.proxy_header = 1;
}
backend nginx_ssl_h2 {
.host = "10.0.10.2";
.port = "6086";
.proxy_header = 1;
}
backend nginx_h1 {
.host = "10.0.10.2";
.port = "80";
}
varnish listen config definition without h1/h2 or PROXY (non SSL) should enable HTTP layer while h1/h2 or PROXY (SSL only) varnish config definition should enable raw TCP socket layer
might this be the proper way or does have varnish another approach doing it?