Upsteam error issue [503] since side-car-istio-proxy cannot connect to pilot #6085
Comments
|
Same issue: Deploy using quick start: Deploy sample app: Test: Returns: |
|
I have the same issue, the dotwiz looks quite interesting: I am trying to upgrade pgadmin on a fresh cluster. |
|
@johnzheng1975 is the log you attached from the istio-proxy by the pilot? Did you use ingress or gateway? Could you please send logs from the gateway/ingress pods, as well as from pilot, also kubectl get all? |
istio-proxy
ingress
|
|
@andraxylia For logs you requested, I will send all when I find this issue next time. |
|
Any update on this issue? same issue here. |
|
Similar issue here: |
|
I am hitting this issue as well. Is there a workaround? Or an older version that does not exhibit this issue? |
|
Oh no, sorry to say that I am joining the party here. Seems to only occur when a deployment is in flight. Do I have a messed up policy config? I have the same setup as @johnzheng1975 |
|
The logs seem to show that pilot closes the connection every 5 minutes - but it reconnects after immediately. It's actually a feature ( an accidental one ) - so pilot connections gets re-balanced. We're working on a better way to rebalance - and after we'll fix this 5-min reconnect. AFAIK it should not cause any problems. |
|
@costinm re: "AFAIK it should not cause any problems." I am getting 503 while accessing the bookinfo sample application. If this message is not indicative of the problem, is there something else I can check? Relevant logs are below: Gateway: productpage sidecar: |
|
Just hit the same issue, a workaround would be awsome, if somebody knows one? |
|
I just ran into this on Istio 1.0.3. Not sure if I saw this on previous versions. Deleting the istio-pilot pods seems to help, but is probably only a temporary fix. The pods were only a day old (upgraded Istio 1.0.2 -> 1.0.3 yesterday) and I didn't notice anything obviously bad in the pilot dashboard. Perhaps the recent activity in this issue is people running 1.0.3? |
|
I'm also hitting UNAVAILABLE:upstream connect error or disconnect/reset before headers on Istio 1.0.3 |
Yeah, deleting pilot seems to fix issue, but it's not really ideal 😄 Ps. running 1.0.3 as well. |
|
I am seeing this with 1.0.3 as well. My scenario: A nginx proxy sits in front of apache httpd. I have two apaches deployed with different versions of a web app. All pods are within the service mesh and have automatically injected sidecars. I have a virtual service sitting in front of the apache. Any time I restart the apache pods I see the disconnect/reset errors. Restarting pilot or applying new virutalservices routes to the apache virtual service fixes the issues. Any logs I should look for? |
|
Same problem on Istio 1.0.3. |
|
same problem on Istio 1.0.3, qps is 40~50 |
|
Possibly related to #10360 |
|
Similar behaviour with 1.0.5. Restarting the pilot POD helps. |
|
I am facing a similar issue with 1.0.5. Even restarting the pilot pod didn't solve my problem. |
|
I am facing the same issue, restarting pilot doesn't seem to help. |
|
Is the problem reproducible with Minikube? Is it hosted? If it only happens on a hosted platform it might be related to an issue I was having. |
|
My issue is in GKE, running Istio 1.0.5(mtls not enabled), noticed this when I was trying to access one of my service(https) through Gateway. |
|
My issue on AKS, running Istio 1.0.5(mtls not enabled). Sample Book info app is running fine, but when deployed own application( simple web app) VS not routing to the route path of the pod. |
|
I probably found the cause of the problem. The reasonI found that the cluster in istio-proxy contains the Kubernetes pod IP that no longer exists. How to solve thisIn my case,i solve it by apply the destinationRule again,and istio sync the right cluster. More questionWhy isio still retains the pod ip that has been stopped? The logIn my case,the cloudspidergateway have 2 pod in Kubernetes which ip is 10.244.25.4，10.244.14.34 ,but the istio-proxy think i have three pod (10.244.25.4，10.244.7.51，10.244.14.34) the error log of istio-proxyThe Kubernetes service describekubectl describe service cloudspidergateway -n cloudspider
Name: cloudspidergateway
Namespace: cloudspider
Labels: app=cloudspidergateway
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"creationTimestamp":null,"labels":{"app":"cloudspidergateway"},"name":"cloudspidergate...
Selector: app=cloudspidergateway
Type: ClusterIP
IP: 10.96.123.71
Port: http-9000-9000-ztdzg 9000/TCP
TargetPort: 9000/TCP
Endpoints: 10.244.14.34:9000,10.244.25.4:9000
Session Affinity: None
Events: <none>the cluster info find in istio-proxy |
|
I'll try this one, but it won't make it into 1.1, sorry |
|
@liutanrong thanks for the tip |
|
@howardjohn please do what you can here. Not sure if we can get a fix into 1.1, but please give it a shot |
|
This bug has become a collector for many problems that all end up looking similar to the downstream client because of the 503s. I'm going to close this since I think we have a large batch of 503 mitigations going into 1.1. Most are already in 1.1rc2, but a few more will go into 1.1rc3 which we will create later this week. If you still have issues with 1.1rc3+ please file a fresh bug. |
|
I'm seeing this on 1.1rc2. After a rolling release of a service, the Ingress returns 503s until a restart of Pilot. |
|
Is there any follow up issue, @seanclerkin , @duderino ? I would like to track the work on it 😉 |
|
Seeing upstream 503 issues still on 1.1.2. |
|
If you are seeing issues please open a new issue with details @jaygorrell |
Similar behaviour with 1.0.5. But, How to restarting the pilot POD.I delete the pilot pod,but it Cannot start automatically。how to restarting the pilot。 |
|
Similar behaviour with 1.1.4. |
|
Version: 1.1.15 I am encountering following issue that stops my application and my pod is not able to launch. So application never launch. As specified in the thread I have delete istio-pilot but the error persists. |
|
This solved my issue,
This does carry security implications and I assume a new meshpolicy will have to be defined - but it does stop the 503's |
|
I'm also facing the same problem too sometime after upgrade service version my workload are calling to another Kubernetes Service that none exist (namespace not found) and in Kiali it depict the picture service call out to PassthroughCluster ! (even though it internal service in the same cluster) |
|
@wdrdres3qew5ts21 please file a separate issue. It's very likely to be completely unrelated to this issue. |
and others
istio 0.8.0 (mTLS disabled, also no control plane security)
k8s 1.9.5
cilium 1.1.0
Steps
Redeploy a service with new version
Expect result:
service can be accessed through istio-ingress
Actual result,
It show 503 upstream error issue.
From attached side-car istio-proxy log, it show cannot connect to istio-pilot.
UPSTREQM-istio-proxyerror.log
The text was updated successfully, but these errors were encountered: