Transfer encoding issues between 1.6 and 1.7 sidecar breaking http 204 responses

[Stono]

Bug description
Given two services, on different sidecar versions:

• portal: 1.7
• leads-service: 1.6

When we make a HEAD request from a to be, which should return a 204, we see:

[atcloud@portal-6d6c4f6474-242bl app]$ curl 'http://app.leads-service/api/leads/count?retailerId=4400&status=NEW_LEAD&status=NEW_MESSAGE&fromDate=2020-07-30&toDate&archived=false' -H 'authorization: Basic xxxx' -H 'retailer-id: 4400' -I
HTTP/1.1 503 Service Unavailable
content-length: 95
content-type: text/plain
date: Fri, 30 Oct 2020 15:47:30 GMT
server: envoy

Looking at the envoy logs, we see:

2020-10-30T14:39:36.458392Z debug envoy client  [C3662] protocol error: http/1.1 protocol error: transfer encoding not allowed in 1xx or 204

The issue appears to be that the destination service (leads-service) is returning a Transfer-Encoding: chunked header. We validated this by going onto the destination and making the same request:

[atcloud@leads-service-699b889948-qshsd app]$ curl 'http://app.leads-service/api/leads/count?retailerId=4400&status=NEW_LEAD&status=NEW_MESSAGE&fromDate=2020-07-30&toDate&archived=false' -H 'authorization: Basic xxxx' -H 'retailer-id: 4400' -I
HTTP/1.1 204 No Content
x-total-count: 50
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: 0
x-frame-options: DENY
set-cookie: JSESSIONID=1ED459A111CC65E666A3CA790204FDFA; Path=/; HttpOnly
date: Fri, 30 Oct 2020 15:47:10 GMT
x-envoy-upstream-service-time: 6
server: envoy
transfer-encoding: chunked

Note: Transfer encoding.

This is not being returned from the application, because if we curl localhost instead, bypassing the proxy, we get:

[atcloud@leads-service-699b889948-qshsd app]$ curl 'http://127.0.0.1:8080/api/leads/count?retailerId=4400&status=NEW_LEAD&status=NEW_MESSAGE&fromDate=2020-07-30&toDate&archived=false' -H 'authorization: Basic xxxx' -H 'retailer-id: 4400' -I
HTTP/1.1 204
X-Total-Count: 50
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Set-Cookie: JSESSIONID=60F3A3FA8FB0E4A52A3215B5DEBCD59A; Path=/; HttpOnly
Date: Fri, 30 Oct 2020 15:51:24 GMT

Note: No transfer encoding header.

Therefore it appears that a 1.6 proxy is adding transfer-encoding: chunked to this request, which in turn is being rejected by a 1.7 proxy.

[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[x] User Experience
[ ] Developer Infrastructure

Expected behavior
1.6 proxy should not be adding Transfer-Encoding: chunked, on a <200 or 204 status code.

Steps to reproduce the bug
As above

Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)
1.6.13 & 1.7.4

How was Istio installed?
Helm

Environment where bug was observed (cloud vendor, OS, etc)
GKE

[Stono]

Associated I believe: #24753 (comment)

[Monkeyanator]

I'm able to reproduce by curling from a sleep container running with proxyv2:1.6.13 to {httpbin-addr}/status/204 on a httpbin instance with proxyv2:1.7.3. Setting envoy.reloadable_features.strict_1xx_and_204_response_headers=false on the sleep container's sidecar allows the request to succeed with 204.

Will look into whether just setting that in our envoy_bootstrap.json is the right way forward.

[Monkeyanator]

This should be fixed with #28450 (with a follow-up fix coming later, as explained here). Thanks so much for finding this @Stono

[Stono]

@Monkeyanator no problemo, thanks for the fix!
Just rolled it out in our testing environment and it seems to have done done the job.

[Stono]

For anyone coming to this the fix will be released in 1.7.5

[hxn36]

@Monkeyanator seeing this issue to all POST requests where its returning header with Transfer Encoding: Chunked. We see envoy adding one more chunked to it, making it Transfer Encoding: chunked chunked and because of this istio is retrying the request twice and eventually throwing 503s. This got fixed by running the command curl -X POST http://localhost:15000/runtime_modify?envoy.reloadable_features.reject_unsupported_transfer_encodings=false. We are using istio 1.7.4 on GKE.

[Monkeyanator]

@hxn36 this issue shows up when sending traffic between different versions of envoy, shouldn't have anything to do with reject_unsupported_transfer_encodings runtime flag IIUC. The issue you're describing sounds more like this one. (two Transfer Encoding envoy-related issues but actually completely different)