istio is sending request of application to node on which application is no scheduled in its lifetime and getting 504

[navneet-we]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

I am using the AWS EKS kubernetes version 1.27 and istio version 1.20.6, and i am using the karpenter as node autoscaler
i had upgraded istio from 1.15.2 to 1.20.6, in both istio version issue is same
issue i am getting is, when karpenter is draining the node called as consolidation, istio is sending the request to draining node and request to pod for which istio is sending is and was not available on that node, when we are making consolidation=false, then we are not getting the issue, we are using the AMI family as bottlerocket, and the request which istio had send to application is running fine with 3 replicas in different nodes, and all pod are running fine at that time and i am getting 504 error.
can someone help me to apply which config i have to change.
installation command

istioctl install --revision=1-20-6 --set profile=default --set hub=361428600572.dkr.ecr.ap-south-1.amazonaws.com --skip-confirmation

Version

client version: 1.20.6
istiod version: 1.15.2
istiod version: 1.15.2
istiod version: 1.20.6
istiod version: 1.21.0
data plane version: 1.15.2 (2 proxies), 1.20.6 (213 proxies)

kubectl version---
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.12-eks-adc7111

Additional Information

No response

[ankur07garg]

Facing same issue, anyone got breakthrough?

[amanmallsops]

Facing same issue

[navneet-we]

@howardjohn please have view on this
also connected with aws support is there any infra issue we are getting, aws support pointed the issue of istio.