Egress to cross-region extenal Mongo DB doesn't seem to work with istio-injection enabled namespace #50597
Open
3 of 17 tasks
Labels
Comments
hasakura12
changed the title
hasakura12
changed the title
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is this the right place to submit this?
Bug Description
egress connection from ue1 EKS cluster to uw2 AWS DocDB in istio-injection enabled namespace does NOT work:
$ k get ns prod -oyaml apiVersion: v1 kind: Namespace metadata: labels: istio-injection: enabled istio.io/rev: prod-stable kubernetes.io/metadata.name: prod name: prod $ k get peerauthentication --all-namespaces NAMESPACE NAME MODE AGE istio-system default PERMISSIVE 364d $ k get destinationrule,serviceentry -n prod No resources found in prod namespace. $ istioctl proxy-status NAME CLUSTER CDS LDS EDS RDS ECDS ISTIOD VERSION istio-egressgateway-cbc78779f-mvm69.istio-system Kubernetes SYNCED SYNCED SYNCED NOT SENT NOT SENT istiod-6b65588b8b-ckmjv 1.21.0 istio-ingressgateway-768dcfdbcb-stc5n.istio-system Kubernetes SYNCED SYNCED SYNCED NOT SENT NOT SENT istiod-6b65588b8b-ckmjv 1.21.0 istio-private-ingressgateway-5854c7b95c-xlmb4.istio-system Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-6b65588b8b-ckmjv 1.21.0 test-mongo.prod Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-6b65588b8b-ckmjv 1.21.0 # ref: https://istio.io/latest/docs/tasks/traffic-management/egress/egress-control/#envoy-passthrough-to-external-services $ kubectl get configmap istio -n istio-system -o yaml apiVersion: v1 data: mesh: |- accessLogFile: /dev/stdout defaultConfig: discoveryAddress: istiod.istio-system.svc:15012 tracing: datadog: address: $(HOST_IP):8126 defaultProviders: metrics: - prometheus enablePrometheusMerge: true rootNamespace: istio-system trustDomain: cluster.local meshNetworks: 'networks: {}' kind: ConfigMap metadata: labels: install.operator.istio.io/owning-resource: installed-state install.operator.istio.io/owning-resource-namespace: istio-system istio.io/rev: default operator.istio.io/component: Pilot operator.istio.io/managed: Reconcile operator.istio.io/version: 1.21.0 release: istio name: istio namespace: istio-system # accessing external HTTPS works $ k exec -it test-mongo -n prod -- curl -sSI https://www.google.com | grep "HTTP/" HTTP/1.1 200 OK # telnet connection works initially, then gets closed after $ k exec -it test-mongo -n prod -- telnet docdb.cluster-XXX.us-west-2.docdb.amazonaws.com 27017 Trying 10.2.105.XXX... Connected to docdb.cluster-XXX.us-west-2.docdb.amazonaws.com. Escape character is '^]'. Connection closed by foreign host. # <-------- this doesn't happen on istio-injection disabled namespace command terminated with exit code 1 # <-------- # accessing external AWS DocDB in ue1 from ue1 cluster does work $ k exec -it test-mongo -n test -- mongo --verbose --host ${host_2}:27017 --username ${master_username} --password ${master_password} MongoDB shell version v3.6.23 connecting to: mongodb://docdb-XXX.us-east-1.docdb.amazonaws.com:27017/?gssapiServiceName=mongodb 2024-04-22T14:52:24.631+0000 D NETWORK [thread1] creating new connection to:docdb-XXXus-east-1.docdb.amazonaws.com:27017 2024-04-22T14:52:24.636+0000 D NETWORK [thread1] connected to server docdb-XXX.us-east-1.docdb.amazonaws.com:27017 (10.1.105.239) 2024-04-22T14:52:24.656+0000 D NETWORK [thread1] connected connection! Implicit session: session { "id" : UUID("901a06db-faf2-4e69-b2e1-6b9ca7f8c7ae") } MongoDB server version: 4.0.0 WARNING: shell and server versions do not match Warning: Non-Genuine MongoDB Detected This server or service appears to be an emulation of MongoDB rather than an official MongoDB product. Some documented MongoDB features may work differently, be entirely missing or incomplete, or have unexpected performance characteristics. To learn more please visit: https://dochub.mongodb.org/core/non-genuine-mongodb-server-warning. rs0:SECONDARY> exit bye # accessing external AWS DocDB in uw2 from ue1 cluster does NOT work $ k exec -it test-mongo -n prod -- mongo --verbose --host ${host}:27017 --username ${master_username} --password ${master_password} MongoDB shell version v3.6.23 connecting to: mongodb://docdb.XXX.db:27017/?gssapiServiceName=mongodb 2024-04-15T18:35:21.620+0000 D NETWORK [thread1] creating new connection to:docdb.XXX.db:27017 2024-04-15T18:35:21.629+0000 D NETWORK [thread1] connected to server docdb.XXX.db:27017 (10.2.105.XXX) 2024-04-15T18:35:31.629+0000 I NETWORK [thread1] Socket recv() Connection reset by peer 10.2.105.XXX:27017 2024-04-15T18:35:31.629+0000 I NETWORK [thread1] SocketException: remote: (NONE):0 error: SocketException socket exception [RECV_ERROR] server [10.2.105.XXX:27017] 2024-04-15T18:35:31.629+0000 D - [thread1] User Assertion: 6:network error while attempting to run command 'isMaster' on host 'docdb.XXX.db:27017' src/mongo/client/dbclient.cpp 241 2024-04-15T18:35:31.629+0000 D - [thread1] User Assertion: 1:network error while attempting to run command 'isMaster' on host 'docdb.XXX.db:27017' src/mongo/scripting/mozjs/mongo.cpp 761 2024-04-15T18:35:31.630+0000 E QUERY [thread1] Error: network error while attempting to run command 'isMaster' on host 'docdb.XXX.db:27017' : connect@src/mongo/shell/mongo.js:275:13 @(connect):1:6 2024-04-15T18:35:31.630+0000 D - [thread1] User Assertion: 12513:connect failed src/mongo/shell/shell_utils.cpp 279 2024-04-15T18:35:31.630+0000 I QUERY [thread1] MozJS GC prologue heap stats - total: 3841623 limit: 0 2024-04-15T18:35:31.631+0000 I QUERY [thread1] MozJS GC epilogue heap stats - total: 284815 limit: 0 2024-04-15T18:35:31.631+0000 I QUERY [thread1] MozJS GC prologue heap stats - total: 175975 limit: 0 2024-04-15T18:35:31.631+0000 I QUERY [thread1] MozJS GC epilogue heap stats - total: 15 limit: 0 2024-04-15T18:35:31.631+0000 D - [main] User Assertion: 12513:connect failed src/mongo/scripting/mozjs/proxyscope.cpp 302 exception: connect failed command terminated with exit code 1 # I can see log entries $ k logs -n prod test-mongo -c istio-proxy 2024-04-22T13:41:10.228635Z info Envoy proxy is ready [2024-04-22T13:42:22.946Z] "- - -" 0 UF,URX - - "-" 0 0 10001 - "-" "-" "-" "-" "10.2.105.XXX:27017" PassthroughCluster - 10.2.105.XXX:27017 10.1.104.247:45362 - -But istio-injection disabled namespace works:
Accessing DocDB in ue1 and uw2 from uw2 cluster works
Version
Additional Information
ue1 debug outputs
Debug outputs in uw2 cluster
Affected product area
The text was updated successfully, but these errors were encountered: