After upgrading to istio 1.20.1 , communication fails to rds mysql

[himanshu-tyagi-3pg]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

I am new to istio , so pardon for stupid questions

1. After upgrading istio from 1.18.2 to 1.20.1 I have observed some erratic behavior . I have one app namespace where java springboot app is running and also one external service deployed as below to connect to rds mysql

apiVersion: v1
kind: Service
metadata:
labels:
app: mysql-service
name: mysql-service
namespace: app-prd
spec:
ports:
- protocol: TCP
port: 3306
targetPort: 3306
externalName: "mysqlrds.abcdef.us-west-2.rds.amazonaws.com"
selector:
app: mysql-service
type: ExternalName

after upgrading istio to 1.20.1 app pods fails to connect to rds giving communication link fail error.

2024-01-10 09:00:02.249 ERROR 7 --- [ main] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Exception during pool initialization.
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

but when I make istiod/ingress/egress pods replica to 0 and then again 1 (means bouncing istio processes) then communication link setup again and app pods connects to rds.

This was not the case until 1.18.2 . What is wrong with this version ? can you guide me ?

2. Secondly for production , is it OK that we keep istiod/ingress/egress replicas more than 1 ? e.g. 2 or 3 . will it be a good practice ?

Version

$ istioctl version
client version: 1.20.1
control plane version: 1.20.1
data plane version: 1.20.1 (6 proxies)

Additional Information

No response

[keithmattix]

You'll need to explicitly set your protocol: https://istio.io/latest/docs/ops/deployment/requirements/#server-first-protocols

[himanshu-tyagi-3pg]

Infact I did check , even after pod was able to start , but sometimes in application log I can see communication failure in between.

We are not using mysql ssl . I didnt get how to set this protocol ? couldnt find steps
Also it was not issue in 1.18.2 istio. is it something new now ?

[himanshu-tyagi-3pg]

any help ? Also second question was that istiod/ingress/egress can have replica more than 1 ?

[keithmattix]

If you change the appProtocol to "tcp-server-first" in your Service, does that change anything?

And yes when you use the helm chart, there is are HPAs deployed for istiod and the gateways

[himanshu-tyagi-3pg]

issue is it is now intermittent :-( no firm way to test. but still I try this option. Thanks Keith.

[whmmm]

same question

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2024-01-12. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.