Can't change nginx injected directives causes upstream sent too large http2 frame error

[RevengeComing]

Hi,

Recently we are trying to migrate from kong v1 to kong v2 because of the GRPC feature, But unfortunately, we faced a problem which I don't exactly know if it's a bug or its lack of knowledge about Nginx. So I came up asking here.

Summary

Requesting kong with grpcurl, half the times cause this error although the request is empty and response is not big:

2020/12/14 15:43:07 [error] 27#0: *839 upstream sent too large http2 frame: 3940676 while reading response header from upstream, client: 172.19.2.57, server: kong, request: "POST /some_package.SomeService/SomeRPC HTTP/2.0", upstream: "grpc://10.233.96.27:9000", host: "upstream_balancer"

I also tried to add Nginx injected directives for nginx_http and nginx_proxy and nginx_sproxy but it didn't work:

The injected directives

export KONG_NGINX_HTTP_CLIENT_BODY_BUFFER_SIZE: "64M"
export KONG_NGINX_HTTP_LARGE_CLIENT_HEADER_BUFFERS: "4 128m"
export KONG_NGINX_HTTP_CLIENT_MAX_BODY_SIZE: "128m"
export KONG_NGINX_PROXY_PROXY_BUFFER_SIZE: "64m"
export KONG_NGINX_PROXY_PROXY_BUSY_BUFFERS_SIZE: "128m"
export KONG_NGINX_PROXY_PROXY_BUFFERS: "4 128m"
export KONG_NGINX_PROXY_LARGE_CLIENT_HEADER_BUFFERS: "4 128m"
export KONG_NGINX_PROXY_CLIENT_BODY_BUFFER_SIZE: "128m"
export KONG_NGINX_PROXY_CLIENT_MAX_BODY_SIZE: "128m"
export KONG_NGINX_SPROXY_PROXY_BUFFER_SIZE: "64m"
export KONG_NGINX_SPROXY_PROXY_BUSY_BUFFERS_SIZE: "128m"
export KONG_NGINX_SPROXY_PROXY_BUFFERS: "4 128m"
export KONG_NGINX_SPROXY_LARGE_CLIENT_HEADER_BUFFERS: "4 128m"
export KONG_NGINX_SPROXY_CLIENT_BODY_BUFFER_SIZE: "128m"
export KONG_NGINX_SPROXY_CLIENT_MAX_BODY_SIZE: "128m"

Additional Details & Logs

• Using docker kong:2.2

- Kong debug-level startup logs (`$ kong start --vv`):

2020/12/14 15:59:44 [verbose] Kong: 2.2.0
2020/12/14 15:59:44 [debug] ngx_lua: 10017
2020/12/14 15:59:44 [debug] nginx: 1017008
2020/12/14 15:59:44 [debug] Lua: LuaJIT 2.1.0-beta3
2020/12/14 15:59:44 [verbose] no config file found at /etc/kong/kong.conf
2020/12/14 15:59:44 [verbose] no config file found at /etc/kong.conf
2020/12/14 15:59:44 [verbose] no config file, skip loading
2020/12/14 15:59:44 [debug] reading environment variables
2020/12/14 15:59:44 [debug] KONG_PLUGINS ENV found with "bundled,jwt,oauth-plugin,device-plugin,trace-info"
2020/12/14 15:59:44 [debug] KONG_ADMIN_LISTEN ENV found with "0.0.0.0:8001"
2020/12/14 15:59:44 [debug] KONG_NGINX_PROXY_LARGE_CLIENT_HEADER_BUFFERS ENV found with "4 128m"
2020/12/14 15:59:44 [debug] KONG_PROXY_LISTEN ENV found with "0.0.0.0:9080 http2, 0.0.0.0:9081 http2 ssl"
2020/12/14 15:59:44 [debug] KONG_NGINX_PROXY_PROXY_BUFFERS ENV found with "4 128m"
2020/12/14 15:59:44 [debug] KONG_HEADERS ENV found with "off"
2020/12/14 15:59:44 [debug] KONG_NGINX_HTTP_CLIENT_MAX_BODY_SIZE ENV found with "128m"
2020/12/14 15:59:44 [debug] KONG_NGINX_PROXY_CLIENT_MAX_BODY_SIZE ENV found with "128m"
2020/12/14 15:59:44 [debug] KONG_PG_DATABASE ENV found with "kong"
2020/12/14 15:59:44 [debug] KONG_NGINX_SPROXY_PROXY_BUFFER_SIZE ENV found with "64m"
2020/12/14 15:59:44 [debug] KONG_PG_PORT ENV found with "5432"
2020/12/14 15:59:44 [debug] KONG_NGINX_PROXY_PROXY_BUFFER_SIZE ENV found with "64m"
2020/12/14 15:59:44 [debug] KONG_PG_USER ENV found with "****"
2020/12/14 15:59:44 [debug] KONG_PG_PASSWORD ENV found with "******"
2020/12/14 15:59:44 [debug] KONG_PG_HOST ENV found with "kong-v2-pg"
2020/12/14 15:59:44 [debug] KONG_NGINX_SPROXY_CLIENT_MAX_BODY_SIZE ENV found with "128m"
2020/12/14 15:59:44 [debug] KONG_DATABASE ENV found with "postgres"
2020/12/14 15:59:44 [debug] KONG_MEM_CACHE_SIZE ENV found with "256m"
2020/12/14 15:59:44 [debug] KONG_NGINX_WORKER_PROCESSES ENV found with "8"
2020/12/14 15:59:44 [debug] KONG_NGINX_PROXY_CLIENT_BODY_BUFFER_SIZE ENV found with "128m"
2020/12/14 15:59:44 [debug] KONG_UPSTREAM_KEEPALIVE ENV found with "512"
2020/12/14 15:59:44 [debug] KONG_NGINX_PROXY_PROXY_BUSY_BUFFERS_SIZE ENV found with "128m"
2020/12/14 15:59:44 [debug] KONG_NGINX_DAEMON ENV found with "off"
2020/12/14 15:59:44 [debug] KONG_LUA_SOCKET_POOL_SIZE ENV found with "200"
2020/12/14 15:59:44 [debug] KONG_NGINX_SPROXY_CLIENT_BODY_BUFFER_SIZE ENV found with "128m"
2020/12/14 15:59:44 [debug] KONG_ADMIN_ERROR_LOG ENV found with "/dev/stderr"
2020/12/14 15:59:44 [debug] KONG_PROXY_ERROR_LOG ENV found with "/dev/stderr"
2020/12/14 15:59:44 [debug] KONG_ADMIN_ACCESS_LOG ENV found with "/dev/stdout"
2020/12/14 15:59:44 [debug] KONG_NGINX_HTTP_LARGE_CLIENT_HEADER_BUFFERS ENV found with "4 128m"
2020/12/14 15:59:44 [debug] KONG_PROXY_ACCESS_LOG ENV found with "/dev/null"
2020/12/14 15:59:44 [debug] KONG_NGINX_SPROXY_PROXY_BUSY_BUFFERS_SIZE ENV found with "128m"
2020/12/14 15:59:44 [debug] KONG_NGINX_SPROXY_PROXY_BUFFERS ENV found with "4 128m"
2020/12/14 15:59:44 [warn] the 'upstream_keepalive' configuration property is deprecated, use 'upstream_keepalive_pool_size' instead
2020/12/14 15:59:44 [warn] the 'nginx_http_upstream_keepalive' configuration property is deprecated, use 'upstream_keepalive_pool_size' instead
2020/12/14 15:59:44 [debug] KONG_NGINX_SPROXY_LARGE_CLIENT_HEADER_BUFFERS ENV found with "4 128m"
2020/12/14 15:59:44 [debug] KONG_NGINX_HTTP_CLIENT_BODY_BUFFER_SIZE ENV found with "64M"
2020/12/14 15:59:44 [debug] admin_access_log = "/dev/stdout"
2020/12/14 15:59:44 [debug] admin_error_log = "/dev/stderr"
2020/12/14 15:59:44 [debug] admin_listen = {"0.0.0.0:8001"}
2020/12/14 15:59:44 [debug] anonymous_reports = true
2020/12/14 15:59:44 [debug] cassandra_contact_points = {"127.0.0.1"}
2020/12/14 15:59:44 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
2020/12/14 15:59:44 [debug] cassandra_keyspace = "kong"
2020/12/14 15:59:44 [debug] cassandra_lb_policy = "RequestRoundRobin"
2020/12/14 15:59:44 [debug] cassandra_port = 9042
2020/12/14 15:59:44 [debug] cassandra_read_consistency = "ONE"
2020/12/14 15:59:44 [debug] cassandra_refresh_frequency = 60
2020/12/14 15:59:44 [debug] cassandra_repl_factor = 1
2020/12/14 15:59:44 [debug] cassandra_repl_strategy = "SimpleStrategy"
2020/12/14 15:59:44 [debug] cassandra_schema_consensus_timeout = 10000
2020/12/14 15:59:44 [debug] cassandra_ssl = false
2020/12/14 15:59:44 [debug] cassandra_ssl_verify = false
2020/12/14 15:59:44 [debug] cassandra_timeout = 5000
2020/12/14 15:59:44 [debug] cassandra_username = "****"
2020/12/14 15:59:44 [debug] cassandra_write_consistency = "ONE"
2020/12/14 15:59:44 [debug] client_body_buffer_size = "8k"
2020/12/14 15:59:44 [debug] client_max_body_size = "0"
2020/12/14 15:59:44 [debug] client_ssl = false
2020/12/14 15:59:44 [debug] cluster_control_plane = "127.0.0.1:8005"
2020/12/14 15:59:44 [debug] cluster_data_plane_purge_delay = 1209600
2020/12/14 15:59:44 [debug] cluster_listen = {"0.0.0.0:8005"}
2020/12/14 15:59:44 [debug] cluster_mtls = "shared"
2020/12/14 15:59:44 [debug] database = "postgres"
2020/12/14 15:59:44 [debug] db_cache_ttl = 0
2020/12/14 15:59:44 [debug] db_cache_warmup_entities = {"services","plugins"}
2020/12/14 15:59:44 [debug] db_resurrect_ttl = 30
2020/12/14 15:59:44 [debug] db_update_frequency = 5
2020/12/14 15:59:44 [debug] db_update_propagation = 0
2020/12/14 15:59:44 [debug] dns_error_ttl = 1
2020/12/14 15:59:44 [debug] dns_hostsfile = "/etc/hosts"
2020/12/14 15:59:44 [debug] dns_no_sync = false
2020/12/14 15:59:44 [debug] dns_not_found_ttl = 30
2020/12/14 15:59:44 [debug] dns_order = {"LAST","SRV","A","CNAME"}
2020/12/14 15:59:44 [debug] dns_resolver = {}
2020/12/14 15:59:44 [debug] dns_stale_ttl = 4
2020/12/14 15:59:44 [debug] error_default_type = "text/plain"
2020/12/14 15:59:44 [debug] go_plugins_dir = "off"
2020/12/14 15:59:44 [debug] go_pluginserver_exe = "/usr/local/bin/go-pluginserver"
2020/12/14 15:59:44 [debug] headers = {"off"}
2020/12/14 15:59:44 [debug] host_ports = {}
2020/12/14 15:59:44 [debug] kic = false
2020/12/14 15:59:44 [debug] log_level = "notice"
2020/12/14 15:59:44 [debug] lua_package_cpath = ""
2020/12/14 15:59:44 [debug] lua_package_path = "./?.lua;./?/init.lua;"
2020/12/14 15:59:44 [debug] lua_socket_pool_size = 200
2020/12/14 15:59:44 [debug] lua_ssl_trusted_certificate = {}
2020/12/14 15:59:44 [debug] lua_ssl_verify_depth = 1
2020/12/14 15:59:44 [debug] mem_cache_size = "256m"
2020/12/14 15:59:44 [debug] nginx_admin_directives = {}
2020/12/14 15:59:44 [debug] nginx_daemon = "off"
2020/12/14 15:59:44 [debug] nginx_events_directives = {{name="worker_connections",value="auto"},{name="multi_accept",value="on"}}
2020/12/14 15:59:44 [debug] nginx_events_multi_accept = "on"
2020/12/14 15:59:44 [debug] nginx_events_worker_connections = "auto"
2020/12/14 15:59:44 [debug] nginx_http_client_body_buffer_size = "64M"
2020/12/14 15:59:44 [debug] nginx_http_client_max_body_size = "128m"
2020/12/14 15:59:44 [debug] nginx_http_directives = {{name="client_max_body_size",value="128m"},{name="ssl_prefer_server_ciphers",value="off"},{name="client_body_buffer_size",value="64M"},{name="ssl_protocols",value="TLSv1.2 TLSv1.3"},{name="ssl_session_tickets",value="on"},{name="large_client_header_buffers",value="4 128m"},{name="ssl_session_timeout",value="1d"}}
2020/12/14 15:59:44 [debug] nginx_http_large_client_header_buffers = "4 128m"
2020/12/14 15:59:44 [debug] nginx_http_ssl_prefer_server_ciphers = "off"
2020/12/14 15:59:44 [debug] nginx_http_ssl_protocols = "TLSv1.2 TLSv1.3"
2020/12/14 15:59:44 [debug] nginx_http_ssl_session_tickets = "on"
2020/12/14 15:59:44 [debug] nginx_http_ssl_session_timeout = "1d"
2020/12/14 15:59:44 [debug] nginx_http_status_directives = {}
2020/12/14 15:59:44 [debug] nginx_http_upstream_directives = {{name="keepalive",value="512"}}
2020/12/14 15:59:44 [debug] nginx_http_upstream_keepalive = "512"
2020/12/14 15:59:44 [debug] nginx_main_daemon = "off"
2020/12/14 15:59:44 [debug] nginx_main_directives = {{name="daemon",value="off"},{name="worker_rlimit_nofile",value="auto"},{name="worker_processes",value="8"}}
2020/12/14 15:59:44 [debug] nginx_main_worker_processes = "8"
2020/12/14 15:59:44 [debug] nginx_main_worker_rlimit_nofile = "auto"
2020/12/14 15:59:44 [debug] nginx_optimizations = true
2020/12/14 15:59:44 [debug] nginx_proxy_client_body_buffer_size = "128m"
2020/12/14 15:59:44 [debug] nginx_proxy_client_max_body_size = "128m"
2020/12/14 15:59:44 [debug] nginx_proxy_directives = {{name="large_client_header_buffers",value="4 128m"},{name="proxy_buffers",value="4 128m"},{name="client_max_body_size",value="128m"},{name="proxy_buffer_size",value="64m"},{name="proxy_busy_buffers_size",value="128m"},{name="client_body_buffer_size",value="128m"},{name="real_ip_recursive",value="off"},{name="real_ip_header",value="X-Real-IP"}}
2020/12/14 15:59:44 [debug] nginx_proxy_large_client_header_buffers = "4 128m"
2020/12/14 15:59:44 [debug] nginx_proxy_proxy_buffer_size = "64m"
2020/12/14 15:59:44 [debug] nginx_proxy_proxy_buffers = "4 128m"
2020/12/14 15:59:44 [debug] nginx_proxy_proxy_busy_buffers_size = "128m"
2020/12/14 15:59:44 [debug] nginx_proxy_real_ip_header = "X-Real-IP"
2020/12/14 15:59:44 [debug] nginx_proxy_real_ip_recursive = "off"
2020/12/14 15:59:44 [debug] nginx_sproxy_client_body_buffer_size = "128m"
2020/12/14 15:59:44 [debug] nginx_sproxy_client_max_body_size = "128m"
2020/12/14 15:59:44 [debug] nginx_sproxy_directives = {{name="proxy_buffer_size",value="64m"},{name="client_max_body_size",value="128m"},{name="client_body_buffer_size",value="128m"},{name="proxy_buffers",value="4 128m"},{name="proxy_busy_buffers_size",value="128m"},{name="large_client_header_buffers",value="4 128m"}}
2020/12/14 15:59:44 [debug] nginx_sproxy_large_client_header_buffers = "4 128m"
2020/12/14 15:59:44 [debug] nginx_sproxy_proxy_buffer_size = "64m"
2020/12/14 15:59:44 [debug] nginx_sproxy_proxy_buffers = "4 128m"
2020/12/14 15:59:44 [debug] nginx_sproxy_proxy_busy_buffers_size = "128m"
2020/12/14 15:59:44 [debug] nginx_status_directives = {}
2020/12/14 15:59:44 [debug] nginx_stream_directives = {{name="ssl_session_tickets",value="on"},{name="ssl_protocols",value="TLSv1.2 TLSv1.3"},{name="ssl_session_timeout",value="1d"},{name="ssl_prefer_server_ciphers",value="off"}}
2020/12/14 15:59:44 [debug] nginx_stream_ssl_prefer_server_ciphers = "off"
2020/12/14 15:59:44 [debug] nginx_stream_ssl_protocols = "TLSv1.2 TLSv1.3"
2020/12/14 15:59:44 [debug] nginx_stream_ssl_session_tickets = "on"
2020/12/14 15:59:44 [debug] nginx_stream_ssl_session_timeout = "1d"
2020/12/14 15:59:44 [debug] nginx_supstream_directives = {}
2020/12/14 15:59:44 [debug] nginx_upstream_directives = {{name="keepalive",value="512"}}
2020/12/14 15:59:44 [debug] nginx_upstream_keepalive = "512"
2020/12/14 15:59:44 [debug] nginx_worker_processes = "8"
2020/12/14 15:59:44 [debug] pg_database = "kong"
2020/12/14 15:59:44 [debug] pg_host = "kong-v2-pg"
2020/12/14 15:59:44 [debug] pg_max_concurrent_queries = 0
2020/12/14 15:59:44 [debug] pg_password = "******"
2020/12/14 15:59:44 [debug] pg_port = 5432
2020/12/14 15:59:44 [debug] pg_ro_ssl = false
2020/12/14 15:59:44 [debug] pg_ro_ssl_verify = false
2020/12/14 15:59:44 [debug] pg_semaphore_timeout = 60000
2020/12/14 15:59:44 [debug] pg_ssl = false
2020/12/14 15:59:44 [debug] pg_ssl_verify = false
2020/12/14 15:59:44 [debug] pg_timeout = 5000
2020/12/14 15:59:44 [debug] pg_user = "****"
2020/12/14 15:59:44 [debug] plugins = {"bundled","jwt","oauth-plugin","device-plugin","trace-info"}
2020/12/14 15:59:44 [debug] port_maps = {}
2020/12/14 15:59:44 [debug] prefix = "/usr/local/kong/"
2020/12/14 15:59:44 [debug] proxy_access_log = "/dev/null"
2020/12/14 15:59:44 [debug] proxy_error_log = "/dev/stderr"
2020/12/14 15:59:44 [debug] proxy_listen = {"0.0.0.0:9080 http2","0.0.0.0:9081 http2 ssl"}
2020/12/14 15:59:44 [debug] real_ip_header = "X-Real-IP"
2020/12/14 15:59:44 [debug] real_ip_recursive = "off"
2020/12/14 15:59:44 [debug] role = "traditional"
2020/12/14 15:59:44 [debug] ssl_cipher_suite = "intermediate"
2020/12/14 15:59:44 [debug] ssl_ciphers = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"
2020/12/14 15:59:44 [debug] ssl_prefer_server_ciphers = "on"
2020/12/14 15:59:44 [debug] ssl_protocols = "TLSv1.1 TLSv1.2 TLSv1.3"
2020/12/14 15:59:44 [debug] ssl_session_tickets = "on"
2020/12/14 15:59:44 [debug] ssl_session_timeout = "1d"
2020/12/14 15:59:44 [debug] status_access_log = "off"
2020/12/14 15:59:44 [debug] status_error_log = "logs/status_error.log"
2020/12/14 15:59:44 [debug] status_listen = {"off"}
2020/12/14 15:59:44 [debug] stream_listen = {"off"}
2020/12/14 15:59:44 [debug] trusted_ips = {}
2020/12/14 15:59:44 [debug] upstream_keepalive = 512
2020/12/14 15:59:44 [debug] upstream_keepalive_idle_timeout = 60
2020/12/14 15:59:44 [debug] upstream_keepalive_max_requests = 100
2020/12/14 15:59:44 [debug] upstream_keepalive_pool_size = 512
2020/12/14 15:59:44 [debug] worker_consistency = "strict"
2020/12/14 15:59:44 [debug] worker_state_update_frequency = 5
2020/12/14 15:59:44 [verbose] prefix in use: /usr/local/kong
2020/12/14 15:59:45 [debug] loading subsystems migrations...
2020/12/14 15:59:45 [verbose] retrieving database schema state...
2020/12/14 15:59:45 [verbose] schema state retrieved
2020/12/14 15:59:45 [verbose] preparing nginx prefix directory at /usr/local/kong
2020/12/14 15:59:45 [verbose] SSL enabled, no custom certificate set: using default certificate
2020/12/14 15:59:45 [verbose] generating default SSL certificate and key
2020/12/14 15:59:45 [debug] searching for OpenResty 'nginx' executable
2020/12/14 15:59:45 [debug] /usr/local/openresty/nginx/sbin/nginx -v: 'nginx version: openresty/1.17.8.2'
nginx: [warn] load balancing method redefined in /usr/local/kong/nginx-kong.conf:55
2020/12/14 15:59:47 [notice] 24#0: using the "epoll" event method
2020/12/14 15:59:47 [notice] 24#0: openresty/1.17.8.2
2020/12/14 15:59:47 [notice] 24#0: built by gcc 9.3.0 (Alpine 9.3.0) 
2020/12/14 15:59:47 [notice] 24#0: OS: Linux 4.15.0-46-generic
2020/12/14 15:59:47 [notice] 24#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2020/12/14 15:59:47 [notice] 24#0: start worker processes
2020/12/14 15:59:47 [notice] 24#0: start worker process 31
2020/12/14 15:59:47 [notice] 24#0: start worker process 32
2020/12/14 15:59:47 [notice] 24#0: start worker process 33
2020/12/14 15:59:47 [notice] 24#0: start worker process 34
2020/12/14 15:59:47 [notice] 24#0: start worker process 35
2020/12/14 15:59:47 [notice] 24#0: start worker process 36
2020/12/14 15:59:47 [notice] 24#0: start worker process 37
2020/12/14 15:59:47 [notice] 24#0: start worker process 38
2020/12/14 15:59:47 [notice] 24#0: signal 17 (SIGCHLD) received from 25
2020/12/14 15:59:47 [notice] 31#0: *1 [lua] cache_warmup.lua:50: cache_warmup_single_entity(): Preloading 'services' into the core_cache..., context: init_worker_by_lua*
2020/12/14 15:59:47 [notice] 31#0: *1 [lua] cache_warmup.lua:89: cache_warmup_single_entity(): finished preloading 'services' into the core_cache (in 588ms), context: init_worker_by_lua*
2020/12/14 15:59:47 [notice] 31#0: *1 [lua] cache_warmup.lua:50: cache_warmup_single_entity(): Preloading 'plugins' into the core_cache..., context: init_worker_by_lua*
2020/12/14 15:59:47 [notice] 31#0: *1 [lua] cache_warmup.lua:89: cache_warmup_single_entity(): finished preloading 'plugins' into the core_cache (in 96ms), context: init_worker_by_lua*
2020/12/14 15:59:47 [notice] 31#0: *16 [lua] cache_warmup.lua:29: warming up DNS entries ..., context: ngx.timer
2020/12/14 15:59:48 [notice] 31#0: *16 [lua] cache_warmup.lua:39: finished warming up DNS entries' into the cache (in 233ms), context: ngx.timer
2020/12/14 15:59:45 [debug] found OpenResty 'nginx' executable at /usr/local/openresty/nginx/sbin/nginx10.233.101.175 - - [14/Dec/2020:16:00:01 +0000] "GET /status HTTP/1.1" 200 1410 "-" "-"
2020/12/14 16:00:31 [warn] 31#0: *975 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000000001, client: 172.19.2.57, server: kong, request: "POST /some_package.SomeService/SomeRPC HTTP/2.0", host: "upstream_balancer"
2020/12/14 16:00:34 [error] 31#0: *1060 upstream sent too large http2 frame: 3940676 while reading response header from upstream, client: 172.19.2.60, server: kong, request: "POST /some_package.SomeService/SomeRPC HTTP/2.0", upstream: "grpc://10.233.96.27:9000", host: "upstream_balancer"
2020/12/14 16:18:21 [error] 30#0: *975 upstream sent too large http2 frame: 3940676 while reading response header from upstream, client: 172.19.2.58, server: kong, request: "POST /some_package.SomeService/SomeRPC HTTP/2.0", upstream: "grpc://10.233.96.27:9000", host: "upstream_balancer"
2020/12/14 16:18:27 [error] 30#0: *1105 upstream sent too large http2 frame: 3940676 while reading response header from upstream, client: 172.19.2.62, server: kong, request: "POST /some_package.SomeService/SomeRPC HTTP/2.0", upstream: "grpc://10.233.109.130:9000", host: "upstream_balancer"
2020/12/14 16:18:32 [error] 30#0: *1202 upstream sent too large http2 frame: 3940676 while reading response header from upstream, client: 172.19.2.69, server: kong, request: "POST /some_package.SomeService/SomeRPC HTTP/2.0", upstream: "grpc://10.233.96.27:9000", host: "upstream_balancer"
2020/12/14 16:18:34 [error] 30#0: *1276 upstream sent too large http2 frame: 3940676 while reading response header from upstream, client: 172.19.2.51, server: kong, request: "POST /some_package.SomeService/SomeRPC HTTP/2.0", upstream: "grpc://10.233.111.43:9000", host: "upstream_balancer"
2020/12/14 16:18:36 [error] 30#0: *1294 upstream sent too large http2 frame: 3940676 while reading response header from upstream, client: 172.19.2.69, server: kong, request: "POST /some_package.SomeService/SomeRPC HTTP/2.0", upstream: "grpc://10.233.111.43:9000", host: "upstream_balancer"

- Kong configuration:

{
  "plugins": {
    "enabled_in_cluster": [
      "trace-info",
      "jwt",
      "correlation-id",
      "request-transformer",
      "cors",
      "device-plugin",
      "prometheus"
    ],
    "available_on_server": {
      "grpc-web": true,
      "correlation-id": true,
      "pre-function": true,
      "cors": true,
      "rate-limiting": true,
      "loggly": true,
      "hmac-auth": true,
      "zipkin": true,
      "device-plugin": true,
      "request-size-limiting": true,
      "azure-functions": true,
      "request-transformer": true,
      "trace-info": true,
      "oauth2": true,
      "response-transformer": true,
      "ip-restriction": true,
      "statsd": true,
      "jwt": true,
      "proxy-cache": true,
      "basic-auth": true,
      "key-auth": true,
      "http-log": true,
      "syslog": true,
      "jwt": true,
      "session": true,
      "datadog": true,
      "tcp-log": true,
      "prometheus": true,
      "post-function": true,
      "ldap-auth": true,
      "acl": true,
      "grpc-gateway": true,
      "file-log": true,
      "oauth-plugin": true,
      "udp-log": true,
      "response-ratelimiting": true,
      "aws-lambda": true,
      "bot-detection": true,
      "acme": true,
      "request-termination": true
    }
  },
  "tagline": "Welcome to kong",
  "configuration": {
    "plugins": [
      "bundled",
      "jwt",
      "oauth-plugin",
      "device-plugin",
      "trace-info"
    ],
    "cluster_mtls": "shared",
    "admin_listen": [
      "0.0.0.0:8001"
    ],
    "nginx_proxy_large_client_header_buffers": "4 128m",
    "prefix": "/usr/local/kong",
    "nginx_conf": "/usr/local/kong/nginx.conf",
    "cassandra_username": "****",
    "nginx_events_directives": [
      {
        "value": "on",
        "name": "multi_accept"
      },
      {
        "value": "auto",
        "name": "worker_connections"
      }
    ],
    "dns_resolver": {},
    "nginx_http_upstream_directives": [
      {
        "value": "512",
        "name": "keepalive"
      }
    ],
    "nginx_main_daemon": "off",
    "stream_proxy_ssl_enabled": false,
    "nginx_acc_logs": "/usr/local/kong/logs/access.log",
    "pg_semaphore_timeout": 60000,
    "proxy_listen": [
      "0.0.0.0:9080 http2",
      "0.0.0.0:9081 http2 ssl"
    ],
    "client_ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
    "go_pluginserver_exe": "/usr/local/bin/go-pluginserver",
    "dns_no_sync": false,
    "db_update_propagation": 0,
    "nginx_proxy_proxy_buffers": "4 128m",
    "stream_listen": [
      "off"
    ],
    "nginx_err_logs": "/usr/local/kong/logs/error.log",
    "pg_ro_ssl_verify": false,
    "headers": [
      "off"
    ],
    "nginx_http_client_max_body_size": "128m",
    "status_ssl_enabled": false,
    "status_listen": [
      "off"
    ],
    "dns_stale_ttl": 4,
    "cluster_control_plane": "127.0.0.1:8005",
    "nginx_http_ssl_prefer_server_ciphers": "off",
    "nginx_proxy_client_max_body_size": "128m",
    "pg_database": "kong",
    "lua_package_cpath": "",
    "admin_acc_logs": "/usr/local/kong/logs/admin_access.log",
    "cassandra_refresh_frequency": 60,
    "nginx_pid": "/usr/local/kong/pids/nginx.pid",
    "nginx_sproxy_proxy_buffer_size": "64m",
    "nginx_main_worker_rlimit_nofile": "auto",
    "cassandra_contact_points": [
      "127.0.0.1"
    ],
    "proxy_listeners": [
      {
        "listener": "0.0.0.0:9080 http2",
        "proxy_protocol": false,
        "reuseport": false,
        "backlog=%d+": false,
        "deferred": false,
        "ssl": false,
        "ip": "0.0.0.0",
        "port": 9080,
        "http2": true,
        "bind": false
      },
      {
        "listener": "0.0.0.0:9081 ssl http2",
        "proxy_protocol": false,
        "reuseport": false,
        "backlog=%d+": false,
        "deferred": false,
        "ssl": true,
        "ip": "0.0.0.0",
        "port": 9081,
        "http2": true,
        "bind": false
      }
    ],
    "db_cache_warmup_entities": [
      "services",
      "plugins"
    ],
    "enabled_headers": {
      "latency_tokens": false,
      "X-Kong-Response-Latency": false,
      "Server": false,
      "X-Kong-Admin-Latency": false,
      "X-Kong-Upstream-Status": false,
      "Via": false,
      "X-Kong-Proxy-Latency": false,
      "server_tokens": false,
      "X-Kong-Upstream-Latency": false
    },
    "nginx_http_ssl_protocols": "TLSv1.2 TLSv1.3",
    "upstream_keepalive_idle_timeout": 60,
    "db_cache_ttl": 0,
    "nginx_events_multi_accept": "on",
    "status_listeners": {},
    "pg_ssl": false,
    "status_access_log": "off",
    "cluster_listeners": [
      {
        "listener": "0.0.0.0:8005",
        "proxy_protocol": false,
        "reuseport": false,
        "backlog=%d+": false,
        "deferred": false,
        "ssl": false,
        "ip": "0.0.0.0",
        "port": 8005,
        "http2": false,
        "bind": false
      }
    ],
    "ssl_protocols": "TLSv1.1 TLSv1.2 TLSv1.3",
    "kong_env": "/usr/local/kong/.kong_env",
    "cassandra_schema_consensus_timeout": 10000,
    "log_level": "notice",
    "admin_ssl_cert_key_default": "/usr/local/kong/ssl/admin-kong-default.key",
    "ssl_session_timeout": "1d",
    "real_ip_recursive": "off",
    "cassandra_repl_factor": 1,
    "ssl_cipher_suite": "intermediate",
    "port_maps": {},
    "pg_port": 5432,
    "cassandra_keyspace": "kong",
    "ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
    "nginx_http_ssl_session_timeout": "1d",
    "nginx_upstream_directives": [
      {
        "value": "512",
        "name": "keepalive"
      }
    ],
    "upstream_keepalive_pool_size": 512,
    "role": "traditional",
    "nginx_stream_ssl_session_timeout": "1d",
    "nginx_proxy_proxy_buffer_size": "64m",
    "admin_ssl_enabled": false,
    "trusted_ips": {},
    "loaded_plugins": {
      "grpc-web": true,
      "session": true,
      "pre-function": true,
      "cors": true,
      "ldap-auth": true,
      "loggly": true,
      "hmac-auth": true,
      "zipkin": true,
      "device-plugin": true,
      "request-size-limiting": true,
      "azure-functions": true,
      "request-transformer": true,
      "trace-info": true,
      "oauth2": true,
      "prometheus": true,
      "ip-restriction": true,
      "statsd": true,
      "jwt": true,
      "proxy-cache": true,
      "basic-auth": true,
      "key-auth": true,
      "http-log": true,
      "rate-limiting": true,
      "acme": true,
      "bot-detection": true,
      "datadog": true,
      "tcp-log": true,
      "response-transformer": true,
      "post-function": true,
      "response-ratelimiting": true,
      "acl": true,
      "grpc-gateway": true,
      "correlation-id": true,
      "oauth-plugin": true,
      "udp-log": true,
      "file-log": true,
      "aws-lambda": true,
      "syslog": true,
      "jwt": true,
      "request-termination": true
    },
    "nginx_supstream_directives": {},
    "ssl_cert_key": "/usr/local/kong/ssl/kong-default.key",
    "host_ports": {},
    "pg_user": "****",
    "pg_password": "******",
    "ssl_ciphers": "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384",
    "nginx_admin_directives": {},
    "nginx_http_directives": [
      {
        "value": "64M",
        "name": "client_body_buffer_size"
      },
      {
        "value": "128m",
        "name": "client_max_body_size"
      },
      {
        "value": "4 128m",
        "name": "large_client_header_buffers"
      },
      {
        "value": "prometheus_metrics 5m",
        "name": "lua_shared_dict"
      },
      {
        "value": "off",
        "name": "ssl_prefer_server_ciphers"
      },
      {
        "value": "TLSv1.2 TLSv1.3",
        "name": "ssl_protocols"
      },
      {
        "value": "on",
        "name": "ssl_session_tickets"
      },
      {
        "value": "1d",
        "name": "ssl_session_timeout"
      }
    ],
    "pg_host": "kong-v2-pg",
    "nginx_kong_stream_conf": "/usr/local/kong/nginx-kong-stream.conf",
    "ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
    "go_plugins_dir": "off",
    "cluster_listen": [
      "0.0.0.0:8005"
    ],
    "cassandra_write_consistency": "ONE",
    "dns_order": [
      "LAST",
      "SRV",
      "A",
      "CNAME"
    ],
    "dns_error_ttl": 1,
    "nginx_sproxy_client_max_body_size": "128m",
    "nginx_sproxy_proxy_buffers": "4 128m",
    "worker_consistency": "strict",
    "lua_ssl_verify_depth": 1,
    "pg_timeout": 5000,
    "cassandra_data_centers": [
      "dc1:2",
      "dc2:3"
    ],
    "nginx_stream_directives": [
      {
        "value": "stream_prometheus_metrics 5m",
        "name": "lua_shared_dict"
      },
      {
        "value": "off",
        "name": "ssl_prefer_server_ciphers"
      },
      {
        "value": "TLSv1.2 TLSv1.3",
        "name": "ssl_protocols"
      },
      {
        "value": "on",
        "name": "ssl_session_tickets"
      },
      {
        "value": "1d",
        "name": "ssl_session_timeout"
      }
    ],
    "database": "postgres",
    "mem_cache_size": "256m",
    "cassandra_read_consistency": "ONE",
    "nginx_upstream_keepalive": "512",
    "cassandra_lb_policy": "RequestRoundRobin",
    "nginx_worker_processes": "8",
    "ssl_cert_csr_default": "/usr/local/kong/ssl/kong-default.csr",
    "nginx_http_status_directives": {},
    "upstream_keepalive_max_requests": 100,
    "admin_ssl_cert_default": "/usr/local/kong/ssl/admin-kong-default.crt",
    "nginx_sproxy_client_body_buffer_size": "128m",
    "admin_access_log": "/dev/stdout",
    "lua_package_path": "./?.lua;./?/init.lua;",
    "cassandra_timeout": 5000,
    "upstream_keepalive": 512,
    "nginx_daemon": "off",
    "worker_state_update_frequency": 5,
    "nginx_sproxy_proxy_busy_buffers_size": "128m",
    "nginx_proxy_proxy_busy_buffers_size": "128m",
    "pg_max_concurrent_queries": 0,
    "nginx_proxy_real_ip_header": "X-Real-IP",
    "nginx_stream_ssl_session_tickets": "on",
    "nginx_proxy_client_body_buffer_size": "128m",
    "proxy_ssl_enabled": true,
    "nginx_http_upstream_keepalive": "512",
    "nginx_http_client_body_buffer_size": "64M",
    "lua_socket_pool_size": 200,
    "cassandra_ssl_verify": false,
    "error_default_type": "text/plain",
    "db_resurrect_ttl": 30,
    "pg_ssl_verify": false,
    "ssl_cert": "/usr/local/kong/ssl/kong-default.crt",
    "client_max_body_size": "0",
    "admin_error_log": "/dev/stderr",
    "nginx_main_directives": [
      {
        "value": "off",
        "name": "daemon"
      },
      {
        "value": "8",
        "name": "worker_processes"
      },
      {
        "value": "auto",
        "name": "worker_rlimit_nofile"
      }
    ],
    "dns_not_found_ttl": 30,
    "nginx_http_ssl_session_tickets": "on",
    "proxy_access_log": "/dev/null",
    "nginx_main_worker_processes": "8",
    "cassandra_repl_strategy": "SimpleStrategy",
    "status_error_log": "logs/status_error.log",
    "dns_hostsfile": "/etc/hosts",
    "kic": false,
    "proxy_error_log": "/dev/stderr",
    "nginx_kong_conf": "/usr/local/kong/nginx-kong.conf",
    "real_ip_header": "X-Real-IP",
    "status_ssl_cert_key_default": "/usr/local/kong/ssl/status-kong-default.key",
    "admin_listeners": [
      {
        "listener": "0.0.0.0:8001",
        "proxy_protocol": false,
        "reuseport": false,
        "backlog=%d+": false,
        "deferred": false,
        "ssl": false,
        "ip": "0.0.0.0",
        "port": 8001,
        "http2": false,
        "bind": false
      }
    ],
    "cassandra_ssl": false,
    "nginx_http_large_client_header_buffers": "4 128m",
    "nginx_proxy_real_ip_recursive": "off",
    "pg_ro_ssl": false,
    "nginx_stream_ssl_prefer_server_ciphers": "off",
    "cluster_data_plane_purge_delay": 1209600,
    "client_ssl": false,
    "nginx_events_worker_connections": "auto",
    "nginx_stream_ssl_protocols": "TLSv1.2 TLSv1.3",
    "nginx_proxy_directives": [
      {
        "value": "128m",
        "name": "client_body_buffer_size"
      },
      {
        "value": "128m",
        "name": "client_max_body_size"
      },
      {
        "value": "4 128m",
        "name": "large_client_header_buffers"
      },
      {
        "value": "64m",
        "name": "proxy_buffer_size"
      },
      {
        "value": "4 128m",
        "name": "proxy_buffers"
      },
      {
        "value": "128m",
        "name": "proxy_busy_buffers_size"
      },
      {
        "value": "X-Real-IP",
        "name": "real_ip_header"
      },
      {
        "value": "off",
        "name": "real_ip_recursive"
      }
    ],
    "nginx_sproxy_large_client_header_buffers": "4 128m",
    "anonymous_reports": true,
    "nginx_optimizations": true,
    "lua_ssl_trusted_certificate": {},
    "status_ssl_cert_default": "/usr/local/kong/ssl/status-kong-default.crt",
    "client_ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
    "client_body_buffer_size": "8k",
    "db_update_frequency": 5,
    "stream_listeners": {},
    "nginx_sproxy_directives": [
      {
        "value": "128m",
        "name": "client_body_buffer_size"
      },
      {
        "value": "128m",
        "name": "client_max_body_size"
      },
      {
        "value": "4 128m",
        "name": "large_client_header_buffers"
      },
      {
        "value": "64m",
        "name": "proxy_buffer_size"
      },
      {
        "value": "4 128m",
        "name": "proxy_buffers"
      },
      {
        "value": "128m",
        "name": "proxy_busy_buffers_size"
      }
    ],
    "ssl_session_tickets": "on",
    "nginx_status_directives": {},
    "cassandra_port": 9042,
    "ssl_prefer_server_ciphers": "on"
  },
  "version": "2.2.0",
  "node_id": "5e8d2647-6ea3-4787-b2b9-bd9373ad8a7f",
  "lua_version": "LuaJIT 2.1.0-beta3",
  "prng_seeds": {
    "pid: 28": 169712640124,
    "pid: 31": 143234221214,
    "pid: 27": 371023114816,
    "pid: 1": 371411301497,
    "pid: 34": 673512920045,
    "pid: 33": 431551862102,
    "pid: 30": 131832042021,
    "pid: 32": 191113765616,
    "pid: 29": 441421061721
  },
  "timers": {
    "pending": 9,
    "running": 0
  },
  "hostname": "apigateway-kong-v2-7b7998fb68-hjw76"
}

[bungle]

Just wondering would this have any effect:
http://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_buffer_size

[RevengeComing]

I have added these env variables but nothing changed.

KONG_NGINX_PROXY_GRPC_BUFFER_SIZE: "128m"
KONG_NGINX_PROXY_HTTP2_MAX_HEADER_SIZE: "128m"
KONG_NGINX_HTTP_GRPC_BUFFER_SIZE: "128m"
KONG_NGINX_HTTP_HTTP2_MAX_HEADER_SIZE: "128m"
KONG_NGINX_HTTP_HTTP2_RECV_BUFFER_SIZE: "128m"
KONG_NGINX_SPROXY_GRPC_BUFFER_SIZE: "128m"
KONG_NGINX_SPROXY_HTTP2_MAX_HEADER_SIZE: "128m"

[bungle]

@RevengeComing, thanks for feedback. There must be something more to it. Could you give us some kind of easily reproducable example of a ws service that would cause this (it would speed up fixing this)?

[RevengeComing]

@bungle Ok, I'll write a service that responds with the exact response to reproduce it. It must take a day.

[RevengeComing]

I found the problem ... the problem lies within kong's DNS resolver ... It resolves with SRV record and ignores the given port I have set, which in my case DNS server returns both port 80 and 9000 and kong balances the traffic between these two ports and one of them is GRPC server and the other is Prometheus metric server.

Those requests sent to Prometheus metric server is returning the error I mentioned and it will log the mentioned error. And If I set IP for Host instead of domain everything works perfectly.

[RevengeComing]

Setting dns_order to LAST,A,CNAME,SRV fixed the problem.
Thank you.

[RevengeComing]

@bungle Btw I think it was kind of miss-information to send such an error on receiving not GRPC response from upstream.