[bug/question]: Is multiplexing HTTP/1 and gRPC supported?

[dilyevsky]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug(?)

NGINX Ingress controller version: 0.22.0

Kubernetes version (use kubectl version): 1.12 (gke)

Environment: GKE

What happened:

It appears that when running an upstream server with gRPC and HTTP/1 multiplexed on the same port Nginx ingress gRPC connection gets knocked down by drive-by HTTP/1 requests.

At first we were getting this error:

no connection data found for keepalive http2 connection while sending request to upstream, client: 10.x.x.x, server: fortune.example.com, request: "POST /fortuneteller/TellFortune HTTP/2.0", upstream: "grpc://172.18.10.28:3000", host: "fortune.example.com:443"

After I patched the nginx binary to ignore non-grpc keepalive connections - it still breaks further down:

upstream sent too large http2 frame: 4740180 while reading response header from upstream, client: 10.x.x.x, server: fortune.example.com, request: "POST /fortuneteller/TellFortune HTTP/2.0", upstream: "grpc://172.18.10.28:3000", host: "fortune.example.com:443"

4740180 is ascii-encoded HTT so it appears it's getting stumbled on HTTP/1 reply when reading from upstream connection socket.

What you expected to happen: Is having same ip:port upstream for both plain HTTP and gRPC supposed to work?

How to reproduce it (as minimally and precisely as possible):

Our generated config:

## start server fortune.example.com
	server {
		server_name fortune.example.com ;
		
		listen 80;
		
		listen [::]:80;
		
		set $proxy_upstream_name "-";
		
		listen 443  ssl http2;
		
		listen [::]:443  ssl http2;
		
		ssl_certificate                         /etc/ingress-controller/ssl/ingress-star.example.com.pem;
		ssl_certificate_key                     /etc/ingress-controller/ssl/ingress-star.example.com.pem;
		
		location ~* "^/fortuneteller.TellFortune/(.+)" {
			
			set $namespace      "default";
			set $ingress_name   "fortune-teller-grpc";
			set $service_name   "fortune-teller";
			set $service_port   "80";
			set $location_path  "/fortuneteller.TellFortune/(.+)";
			
			rewrite_by_lua_block {
				balancer.rewrite()
			}
			
			header_filter_by_lua_block {
				
			}
			body_filter_by_lua_block {
				
			}
			
			log_by_lua_block {
				
				balancer.log()
				
				monitor.call()
				
			}
			
			if ($scheme = https) {
				more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
			}
			
			port_in_redirect off;
			
			set $proxy_upstream_name    "default-fortune-teller-80";
			set $proxy_host             $proxy_upstream_name;
			
			# enforce ssl on server side
			if ($redirect_to_https) {
				
				return 308 https://$best_http_host$request_uri;
				
			}
			
			client_max_body_size                    1m;
			
			grpc_set_header Host                   $best_http_host;
			
			# Pass the extracted client certificate to the backend
			
			# Allow websocket connections
			grpc_set_header                        Upgrade           $http_upgrade;
			
			grpc_set_header                        Connection        $connection_upgrade;
			
			grpc_set_header X-Request-ID           $req_id;
			grpc_set_header X-Real-IP              $the_real_ip;
			
			grpc_set_header X-Forwarded-For        $the_real_ip;
			
			grpc_set_header X-Forwarded-Host       $best_http_host;
			grpc_set_header X-Forwarded-Port       $pass_port;
			grpc_set_header X-Forwarded-Proto      $pass_access_scheme;
			
			grpc_set_header X-Original-URI         $request_uri;
			
			grpc_set_header X-Scheme               $pass_access_scheme;
			
			# Pass the original X-Forwarded-For
			grpc_set_header X-Original-Forwarded-For $http_x_forwarded_for;
			
			# mitigate HTTPoxy Vulnerability
			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
			grpc_set_header Proxy                  "";
			
			# Custom headers to proxied server
			
			proxy_connect_timeout                   5s;
			proxy_send_timeout                      60s;
			proxy_read_timeout                      60s;
			
			proxy_buffering                         off;
			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;
			proxy_request_buffering                 on;
			
			proxy_http_version                      1.1;
			
			proxy_cookie_domain                     off;
			proxy_cookie_path                       off;
			
			# In case of errors try the next upstream server before returning an error
			proxy_next_upstream                     error timeout;
			proxy_next_upstream_tries               3;
			
			grpc_pass grpc://upstream_balancer;
			
			proxy_redirect                          off;
			
		}
		
		location ~* "^/" {
			
			set $namespace      "default";
			set $ingress_name   "fortune-teller";
			set $service_name   "fortune-teller";
			set $service_port   "80";
			set $location_path  "/";
			
			rewrite_by_lua_block {
				balancer.rewrite()
			}
			
			header_filter_by_lua_block {
				
			}
			body_filter_by_lua_block {
				
			}
			
			log_by_lua_block {
				
				balancer.log()
				
				monitor.call()
				
			}
			
			if ($scheme = https) {
				more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
			}
			
			port_in_redirect off;
			
			set $proxy_upstream_name    "default-fortune-teller-80";
			set $proxy_host             $proxy_upstream_name;
			
			# enforce ssl on server side
			if ($redirect_to_https) {
				
				return 308 https://$best_http_host$request_uri;
				
			}
			
			client_max_body_size                    9m;
			
			proxy_set_header Host                   $best_http_host;
			
			# Pass the extracted client certificate to the backend
			
			# Allow websocket connections
			proxy_set_header                        Upgrade           $http_upgrade;
			
			proxy_set_header                        Connection        $connection_upgrade;
			
			proxy_set_header X-Request-ID           $req_id;
			proxy_set_header X-Real-IP              $the_real_ip;
			
			proxy_set_header X-Forwarded-For        $the_real_ip;
			
			proxy_set_header X-Forwarded-Host       $best_http_host;
			proxy_set_header X-Forwarded-Port       $pass_port;
			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
			
			proxy_set_header X-Original-URI         $request_uri;
			
			proxy_set_header X-Scheme               $pass_access_scheme;
			
			# Pass the original X-Forwarded-For
			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
			
			# mitigate HTTPoxy Vulnerability
			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
			proxy_set_header Proxy                  "";
			
			# Custom headers to proxied server
			
			proxy_connect_timeout                   5s;
			proxy_send_timeout                      60s;
			proxy_read_timeout                      60s;
			
			proxy_buffering                         off;
			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;
			proxy_request_buffering                 on;
			
			proxy_http_version                      1.1;
			
			proxy_cookie_domain                     off;
			proxy_cookie_path                       off;
			
			# In case of errors try the next upstream server before returning an error
			proxy_next_upstream                     error timeout;
			proxy_next_upstream_tries               3;
			
			proxy_pass http://upstream_balancer;
			
			proxy_redirect                          off;
			
		}
		
	}
	## end server fortune.example.com

Our service definition:

apiVersion: v1
kind: Service
metadata:
  name: fortune-teller
  namespace: default
spec:
  clusterIP: 192.168.17.79
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http-port
  selector:
    app: fortune-teller-app
  sessionAffinity: None
  type: ClusterIP

Then we define two separate ingresses to match HTTP/1 and gRPC respectively and point them to the above service/port.

Is there a way to tell nginx to initiate new upstream connection instead of reusing existing socket that already exists from HTTP/1 one or there is something else I'm missing here?

[bruse-peng]

i meet the same issue, have you have solved this issue?

[aledbf]

s having same ip:port upstream for both plain HTTP and gRPC supposed to work?

No. This is not supported by NGINX. You need to enable SSL.

[ensonic]

And it is even flaky if using TLS, see #4836