Does envoy have limit on the single header size?

[korjwl1]

Currently, I am testing Langfuse V3 with Istio on the AWS EKS.
Everything except the Export Traces of the Langfuse V3 worked nicely, but Export Traces with many traces keep failing.

This function sends a request to clickhouse to extract trace data using the node.js library @clickhouse/client, and the relevant tcp stream packet is like below.

POST /?query_id=a3b4ae4e-6c5f-4f14-bd10-be2ca6672b83&param_limit=&5CN&param_offset=0&param_traceTimestamp=1744608584911&param_projectId=cm9b59nrn000u1107q2nunwuy&param_stringFiltereiDNI=cm9b59nrn000u1i07q2nunwu
y&param_dateTimeFiltereennQ=1744608584911&param_string0ptionsFilterAMyHi=%5B%27default%27%5D&param_string0ptionsFilterliXLL=%5B%273e8ac5bf-91e9-47b6-9e08-385dc2082b6a%27%2C%270716634f-bdbb-4fd9-8d89-1e6abf4ef32 d%27%2%2783346665-388c-49ca-89bb-aaa3199fefca%27%2C%27ed645bae-433c-4565-948f-5263433c378%27%2C%270fdb436b-060c-458b-9a55-e0c58a945120%27%2C%27ced96b12-0972-401e-97d5-e5 feeb7780b8%27%2C&270b462a42-9bfb-4d4d-a
e11-ab5fd99dd6c0%27%2C%270a71722f-64b5-4416-9cdc-41ee52137180%27%2C%275c4214c9-3d11-48f4-b750-3a9f71d8c06f%27%2C%270cf82ccb-c6cd-448c-a3c3-fd57f9ef91a3%27%2C%2742a17f49-1439-4bcc-b8ea-f18430c62075%27%2C%27f5d70 
.... < lots of query ids in a single line > ...
6dd5d9eb-db20-4171-bb3d-0a331b605d65%27%2C%27c2beef83-02f2-4f5b-8662-5ad0a822c1dd%27%2C%27b98e6f42-e778-4cb7-a9cb-f52f8f1111b9%27%2C%277c5c2489-c04b-48f3-ba4f-fa73402f698a%27%2C%273f54c31b-012c-427d-98ea-909cf3
Cc08a0%27%2C%27786f5720-3483-40c4-9978-29be73595f7d%27%2C%27234c76f8-01e4-48e7-bdb4-f974f0429e6c%27%2C%27a94e70a6-06f4-434c-8020-d819d239b9d4%27%2C%2785dd91a6-12c6-42e2-aabb-34717e3be09e%27%2C%279adc6798-9b93-4 0f1-be6e-908e31a9ff88%27%2C%270cd098fd-6b85-42bd-b3df-ee200ed5b0a4%27%2C%27d34c905a-b7dc-4f2a-9069-ee725328f634%27%2C%272b6ce54c-343d-4660-b2d4-b02d090eb56e%27%2C%2794d2f2de-8b88-4227-9fe0-10480da4075e%27%2C%27
dcf58482-46d0-4d10-9ec5-f53c314c98e1%27%2C%27b6659a98-6bbb-4597-98dc-cf cea542b10c%27%2C%2788ed7102-4a58-4558-93a7-c83bc2fb8a2c%27%2C%2774da4b33-d860-4e6f-b639-ed8161ca5767%27%2C%278bb104b7-1d9c-493b-8f34-e32cf7
5dc1e5%27%5D&log_comment=%7B%22kind%22%3A%22analytic%22%2C%22feature%22%3A%22tracing%22%2C%22type%22%3A%22traces-table%22%2C%22projectId%22%3A&22cm9b59nrn000u1i07q2nunwuy%22%7D&async_insert=1&wait_for_async_ins
ert=1 HTTP/1.1
traceparent: 00-187ae54adf6a0565ac3824d6e499cd2f-a73e4edd96ae3856-01
Connection: keep-alive
User-Agent: clickhouse-js/1.4.0 (lv:nodejs/v20.19.0; os: linux)
Authorization: Basic 
Host: langfuse-clickhouse:8123
Transfer-Encoding: chunked

c31
WITH observations_stats AS (
    SELECT
        COUNT(*) AS observation_count,
        sumMap (usage_details) as usage_details, SUM(total_cost) AS total_cost,
        .... < ClickHouse SQL Sentences> ....
FORMAT JSONEachRow
0

In our case, we found out that the packet size was about 28 KB ~ 90 KB
Response packet was JSON Body packet with size 170KB ~ 400KB

When we applied istio sidecar to the pod, we found out that the Export Traces failed since the above POST request was dropped. To be concise, the packets with too large single header were all dropped while the packets with large body were not affected at all.

I adjusted max_request_headers_kb to 1024 KB but the problem was not solved.
My assume right now is that maybe Envoy has a limit on the single header size regardless of the entire header size.
Does anybody know about this problem? Or is there any setting value that I missed to solve this problem?

[phlax]

cc @yanavlasov @mattklein123 as HCM codeowners

(fwiw i ran up against header limit size recently when working LLM prompts)

[yanavlasov]

If you have increased overall header map size to 1Mb and still had problems, my guess is that there is some other limit, most likely on the HTTP/1.1 start line. @paul-r-gall if he knows that Balsa has the limit on the start-line.

[paul-r-gall]

No limit in Balsa; we actually have integration tests validating large paths:

envoy/test/integration/protocol_integration_test.cc

Lines 2460 to 2468 in 62fbf5e

	TEST_P(DownstreamProtocolIntegrationTest, LargeRequestUrlRejected) {
	// Send one 95 kB URL with limit 60 kB headers.
	testLargeRequestUrl(95, 60);
	}
	TEST_P(DownstreamProtocolIntegrationTest, LargeRequestUrlAccepted) {
	// Send one 95 kB URL with limit 96 kB headers.
	testLargeRequestUrl(95, 96);
	}

. I modified this to validate large query parameters, and it also passed.

[paul-r-gall]

What response code is Envoy generating? Are there any access logs that you can see?

[korjwl1]

@paul-r-gall Sorry for my misunderstanding. Today, I and my partner recollected packets and logs using dumpcap and then tracked the issue again to give you the logs.

At first, I thought the problem was due to the header size since the huge header-size packets were all seem to be dropped in one of the pcap files.

However, today I found out that sometimes the huge header-size packets passed without any problem. The real problem seems to be the lack of connection pool in our setting, and since the langfuse sends many simultaneous requests to the clickhouse, some of the packets were dropped due to the zero-window issue but it looked as if the huge header-size packets dropped.