Bug description
Another set of apps that doesn't work. We upgrade the control plane to 1.6, but don't touch the apps (so still running 1.5 sidecar) and we start getting the old upstream connect error or disconnect/reset before headers. reset reason: connection failure when making requests from within the mesh (from another app to this one).
Looking at the istio-proxy debug logs for the destination service we see:
[Envoy (Epoch 0)] [2020-10-16 07:46:36.184][31][debug][filter] [external/envoy/source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
[Envoy (Epoch 0)] [2020-10-16 07:46:36.184][31][debug][conn_handler] [external/envoy/source/server/connection_handler_impl.cc:361] [C83] new connection
[Envoy (Epoch 0)] [2020-10-16 07:46:36.184][31][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:279] [C83] dispatch error: http/1.1 protocol error: HPE_INVALID_METHOD
[Envoy (Epoch 0)] [2020-10-16 07:46:36.184][31][debug][connection] [external/envoy/source/common/network/connection_impl.cc:101] [C83] closing data_to_write=66 type=2
[Envoy (Epoch 0)] [2020-10-16 07:46:36.184][31][debug][connection] [external/envoy/source/common/network/connection_impl_base.cc:30] [C83] setting delayed close timer with timeout 1000 ms
[Envoy (Epoch 0)] [2020-10-16 07:46:36.184][31][debug][connection] [external/envoy/source/common/network/connection_impl.cc:600] [C83] write flush complete
Here are config dumps from istio-proxy on the destination service from 1.5 and 1.6 control planes:
config-dumps.tar.gz
The pod exposes three ports:
ports:
- containerPort: 9080
name: http-web-admin
protocol: TCP
- containerPort: 8317
name: http-web
protocol: TCP
- containerPort: 8318
name: tcp-tcp-sftp
protocol: TCP
❯ k -n vds-acceptor get ep
NAME ENDPOINTS AGE
admin 10.198.10.11:9080 338d
app 10.198.10.11:8317 435d
sftp 10.198.10.11:8318 356d
The cluster is configured with strict mtls:
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system
spec:
mtls:
mode: STRICT
The only difference I could draw was that this app (and the other broken services) also has a non-mesh service (stfp):
However that is excluded from istio with a Sidecar:
ingress:
- defaultEndpoint: 127.0.0.1:8317
port:
name: http-app
number: 8317
protocol: HTTP
- defaultEndpoint: 127.0.0.1:9080
port:
name: http-admin
number: 9080
protocol: HTTP
and
traffic.sidecar.istio.io/includeInboundPorts: 8317,9080
We tried adding an explicit PeerAuthentication policy (even though it shouldn't actually make any difference):
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
spec:
mtls:
mode: STRICT
portLevelMtls:
8318:
mode: DISABLE
selector:
matchLabels:
app: vds-acceptor
group: delivery-platform
And as expected it didn't change anything.
Restarting the app to pull in a 1.6 sidecar fixed the issue, therefore it seems to only be an issue with a 1.5 sidecar & 1.6 control plane.
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[x] Test and Release
[x] User Experience
[ ] Developer Infrastructure
Expected behavior
Upgrades to not break services
Steps to reproduce the bug
🤷 happy to live debug my cluster with you
Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)
1.6.11
How was Istio installed?
helm
Environment where bug was observed (cloud vendor, OS, etc)
gke
Bug description
Another set of apps that doesn't work. We upgrade the control plane to
1.6, but don't touch the apps (so still running1.5sidecar) and we start getting the oldupstream connect error or disconnect/reset before headers. reset reason: connection failurewhen making requests from within the mesh (from another app to this one).Looking at the istio-proxy debug logs for the destination service we see:
Here are config dumps from
istio-proxyon the destination service from 1.5 and 1.6 control planes:config-dumps.tar.gz
The pod exposes three ports:
The cluster is configured with strict mtls:
The only difference I could draw was that this app (and the other broken services) also has a non-mesh service (
stfp):However that is excluded from istio with a
Sidecar:and
We tried adding an explicit
PeerAuthenticationpolicy (even though it shouldn't actually make any difference):And as expected it didn't change anything.
Restarting the app to pull in a 1.6 sidecar fixed the issue, therefore it seems to only be an issue with a 1.5 sidecar & 1.6 control plane.
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[x] Test and Release
[x] User Experience
[ ] Developer Infrastructure
Expected behavior
Upgrades to not break services
Steps to reproduce the bug
🤷 happy to live debug my cluster with you
Version (include the output of
istioctl version --remoteandkubectl version --shortandhelm versionif you used Helm)1.6.11
How was Istio installed?
helm
Environment where bug was observed (cloud vendor, OS, etc)
gke