Unable to connect only to external MongoDB v4.0 (AWS DocumentDB) & read replica DNS (Mongo v3.6 and primary DNS work) with istio-proxy

[hasakura12]

Bug Description

• two AWS EKS clusters (us east: K8s v1.19, istio v1.8, us west: k8s 1.22, istio 1.13.3)
• namespace enables istio injection
• pod in the namespace CAN already connect to AWS DocDB v3.6 primary DNS
• istio ServiceEntry is configured already (Unable to connect to external MongoDB instance with istio-proxy #8410 (comment)) and has been working for DocDB v3.6's primary DNS

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry # ref: https://preliminary.istio.io/blog/2018/egress-mongo/#control-tcp-egress-traffic-without-a-gateway
metadata:
  name: aws-docdb
  namespace: staging-sandbox
spec:
  hosts: # L7 host will be ignroed for L4 TCP/MONGO protocol 
  - mongo.tcp.svc
  ports:
  - number: 27017
    name: aws-docdb
    protocol: TCP # specify TCP instead of MONGO if traffic is encrypted, because if the MongoDB protocol runs on top of TLS, then the encrypted MongoDB protocol cannot be parsed by the Istio proxy. Ref: https://preliminary.istio.io/blog/2018/egress-mongo/#control-tcp-egress-traffic-without-a-gateway
  location: MESH_EXTERNAL
  resolution: DNS #NONE
  endpoints:
  - address: docdb-cluster-uxxx.docdb.amazonaws.com   # primary (load balanced)
  - address: docdb-instance-xxx.docdb.amazonaws.com  # read replicas (load balanced)xx
  - address: test-docdb40.xxx.docdb.amazonaws.com   # test DocDB v4.0

What does work:

• pod in istio-enabled namespaces CAN connects to AWS DocDB v3.6's primary DNS

# mongo --host docdb-cluster-xxx.cluster-xxx.docdb.amazonaws.com:27017 \
  --username xxx --password>  xxx
MongoDB shell version v4.0.28
MongoDB server version: 3.6.0
WARNING: shell and server versions do not match

rs0:PRIMARY> exit

• pod in istio-DISABLED namespaces CAN connects to v3.6's primary & read replica DNSs, and AWS DocDB v4.0

# TEST PASS: v3.6's read replica DNSs
$ k exec -it -n test  test-mongo sh

# mongo --host docdb-instance-xxx.xxx.docdb.amazonaws.com:27017 \
  --username xxx --pass> word xxx
MongoDB shell version v4.0.28
MongoDB server version: 3.6.0
WARNING: shell and server versions do not match
rs0:SECONDARY> 


# TEST PASS: v4.0's primay DNS
# mongo --host test-docdb40.cluster-xxx.docdb.amazonaws.com:27017 \
  --username xxx--password xxx
MongoDB shell version v4.0.28
MongoDB server version: 4.0.0

rs0:PRIMARY> 

What does NOT work:

• pod in istio-enabled namespaces can't connects to v3.6's read replicas and AWS DocDB v4.0's primary and read replica DNSs

# TEST FAILED: connect to DocDB v3.6's reader DNS 
# mongo --host docdb-instance-xxx.docdb.amazonaws.com:27017 \
  --username xxx --pass> word xxx
MongoDB shell version v4.0.28
connecting to: mongodb://docdb-instance-xxx.docdb.amazonaws.com:27017/?gssapiServiceName=mongodb
MongoDB server version: 3.6.0
WARNING: shell and server versions do not match

rs0:PRIMARY> exit.           # <------- should've said SECONDARY if connected to read replicas


# TEST FAILED: connect to DocDB v4.0's primary DNS 
$ k exec -it -n staging-sandbox test-mongo sh
Defaulted container "test-mongo" out of: test-mongo, istio-proxy, istio-init (init)
# 
# mongo --host test-docdb40.cluster-xxx.docdb.amazonaws.com:27017 \
  --username xxx --password xxx
MongoDB shell version v4.0.28
connecting to: mongodb://test-docdb40.cluster-xxx.docdb.amazonaws.com:27017/?gssapiServiceName=mongodb
2022-08-25T06:45:19.057+0000 E QUERY    [js] Error: Authentication failed. :
connect@src/mongo/shell/mongo.js:356:17
@(connect):2:6
exception: connect failed
# exit

Version

# us-east EKS cluster
istioctl version
client version: 1.8.2
control plane version: 1.8.2
data plane version: 1.8.2 (75 proxies)

kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:41:42Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.16-eks-a05fdea", GitCommit:"a05fdea9a04815782f9c4b08eed73bb2d78b07e3", GitTreeState:"clean", BuildDate:"2022-06-09T21:50:47Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}



# us-west EKS cluster
$ istioctl version
client version: 1.8.2
control plane version: 1.13.3
data plane version: 1.13.3 (40 proxies)

$ k version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:41:42Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.10-eks-84b4fe6", GitCommit:"cc6a1b4915a99f49f5510ef0667f94b9ca832a8a", GitTreeState:"clean", BuildDate:"2022-06-09T18:24:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Additional Information

No response

[zirain]

1.8.2 is too old

please follow this doc https://istio.io/latest/docs/ops/deployment/requirements/#server-first-protocols

[hasakura12]

1.8.2 is too old

please follow this doc https://istio.io/latest/docs/ops/deployment/requirements/#server-first-protocols

@zirain

We have two EKS clusters and one of them uses istio 1.13.3. Still the same issue.

# us-west EKS cluster
$ istioctl version
client version: 1.8.2
control plane version: 1.13.3
data plane version: 1.13.3 (40 proxies)

$ k version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:41:42Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.10-eks-84b4fe6", GitCommit:"cc6a1b4915a99f49f5510ef0667f94b9ca832a8a", GitTreeState:"clean", BuildDate:"2022-06-09T18:24:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Also, the ServerEntry config is using 27017 already, so not sure what you mean. AWS DocDB is TLS DISABLED btw.

spec:
  hosts: # L7 host will be ignroed for L4 TCP/MONGO protocol 
  - mongo.tcp.svc
  ports:
  - number: 27017
    name: aws-docdb
    protocol: TCP # # specify TCP instead of MONGO if traffic is encrypted, because if the MongoDB protocol runs on top of TLS, then the encrypted MongoDB protocol cannot be parsed by the Istio proxy. Ref: https://preliminary.istio.io/blog/2018/egress-mongo/#control-tcp-egress-traffic-without-a-gateway

Tried spec.ports[].protocol=MONGO to no avail:

spec:
  hosts: # L7 host will be ignroed for L4 TCP/MONGO protocol 
  - mongo.tcp.svc
  ports:
  - number: 27017
    name: aws-docdb
    protocol: MONGO 
  location: MESH_EXTERNAL
  resolution: DNS

[zirain]

Also, the ServerEntry config is using 27017 already, so not sure what you mean. AWS DocDB is TLS DISABLED btw.

what I say is the mTLS mode of istio, which should be PERMISSIVE by default, you may need disabled it.

[hasakura12]

@zirain

It's already set to PERMISSIVE and not working.

$ k get peerauthentication --all-namespaces
NAMESPACE         NAME      MODE         AGE
istio-system      default   PERMISSIVE   94d
staging-sandbox   default   PERMISSIVE   65s

[zirain]

you need disable mTLS for server first protocol

[hasakura12]

@zirain

$ k get peerauthentication  --all-namespaces
NAMESPACE         NAME      MODE         AGE
istio-system      default   PERMISSIVE   104d
staging-sandbox   default   DISABLE      10d

The result is the same:

$ k exec -it test-mongo -n staging-sandbox sh
# mongo --host test-xxxx.cluster-xxxx.us-east-1.docdb.amazonaws.com:27017 --username xxxx --password  xxxx
MongoDB shell version v4.0.28
connecting to: mongodb://test-xxxx.cluster-xxxx.us-east-1.docdb.amazonaws.com:27017/?gssapiServiceName=mongodb
2022-09-05T14:04:56.823+0000 E QUERY    [js] Error: Authentication failed. :
connect@src/mongo/shell/mongo.js:356:17
@(connect):2:6
exception: connect failed
# ^[[A^C
# exit

Is disabling mTLS in namespace enough for disabling mTLS for server first protocol?

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2022-08-29. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.