Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cookie not applied in inter-namespace communication #50643

Open
Raven6681 opened this issue Apr 23, 2024 Discussed in #50415 · 8 comments
Open

Cookie not applied in inter-namespace communication #50643

Raven6681 opened this issue Apr 23, 2024 Discussed in #50415 · 8 comments

Comments

@Raven6681
Copy link

Discussed in #50415

Originally posted by Raven6681 April 12, 2024
Hello,

I am trying to apply cookies on a service and my setup is the following:

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myservice-sticky-vs
  namespace: myservice
spec:
  hosts:
    - my.gateway.fqdn
    - myservice.namespace.svc.cluster.local
  gateways:
    - istio-ingress-internal/gateway-internal
    - mesh
  http:
  - match:
    - uri:
        prefix: /path
    route:
    - destination:
        host: myservice.namespace.svc.cluster.local
        port:
          number: 8080
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: myservice-sticky-dr
  namespace: myservice
spec:
  host: myservice.namespace.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie:
          name: istio
          ttl: 0s
    tls:
      mode: ISTIO_MUTUAL

EKS cluster version: 1.27
Istio version: 1.21.1

The Gateway sits behind an AWS ALB (gateway deployment installed with a NodePort service and the ALB as ingress).
If I hit the Gateway FQDN with curl, I can see the cookie (real path and fqdn have been substituted), < set-cookie: istio="a9bbe686634a6ce8"; HttpOnly:

curl -v https://my.gateway.fqdn/path
*   Trying 10.102.5.74:443...
* Connected to my.gateway.fqdn (10.102.5.74) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=my.gateway.fqdn
*  start date: Jan 26 00:00:00 2024 GMT
*  expire date: Feb 23 23:59:59 2025 GMT
*  subjectAltName: host "my.gateway.fqdn" matched cert's "my.gateway.fqdn"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: my.gateway.fqdn]
* h2 [:path: /path]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7fb1ca80c800)
> GET /path HTTP/2
> Host: my.gateway.fqdn
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/2 200
< date: Fri, 12 Apr 2024 09:58:35 GMT
< content-type: application/json
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< x-frame-options: DENY
< content-security-policy: script-src 'self'
< referrer-policy: no-referrer
< x-envoy-upstream-service-time: 5
< server: istio-envoy
< set-cookie: istio="a9bbe686634a6ce8"; HttpOnly
<
{
    ...
    some_json_output
}

but when I am using curl from a pod within the cluster, hitting the service, I see no cookie:

curl -v http://myservice.namespace.svc.cluster.local:8080/path
*   Trying 10.102.3.57:8080...
* Connected to myservice.namespace.svc.cluster.local (10.102.3.57) port 8080 (#0)
> GET /path HTTP/1.1
> Host: myservice.namespace.svc.cluster.local:8080
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< x-frame-options: DENY
< content-security-policy: script-src 'self'
< referrer-policy: no-referrer
< content-type: application/json
< date: Fri, 12 Apr 2024 10:03:12 GMT
< x-envoy-upstream-service-time: 2
< server: envoy
< transfer-encoding: chunked
<
{
    ...
    some_json_output
}

I have enabled debug level in logs and I can see the cookie in the ingressgateway pod logs:

2024-04-12T09:18:46.800592Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1869	[Tags: "ConnectionId":"5628","StreamId":"1461258412630245050"] encoding headers via codec (end_stream=false):
':status', '200'
'x-content-type-options', 'nosniff'
'x-xss-protection', '1; mode=block'
'cache-control', 'no-cache, no-store, max-age=0, must-revalidate'
'pragma', 'no-cache'
'expires', '0'
'x-frame-options', 'DENY'
'content-security-policy', 'script-src 'self''
'referrer-policy', 'no-referrer'
'content-type', 'application/json'
'date', 'Fri, 12 Apr 2024 09:18:46 GMT'
'x-envoy-upstream-service-time', '3'
'server', 'istio-envoy'
'set-cookie', 'istio="a9bbe686634a6ce8"; HttpOnly'

but not in the istio-proxy logs of the pod that originates the curl request:

2024-04-12T09:19:42.290888Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1869	[Tags: "ConnectionId":"998","StreamId":"8952761916989281991"] encoding headers via codec (end_stream=false):
':status', '200'
'x-content-type-options', 'nosniff'
'x-xss-protection', '1; mode=block'
'cache-control', 'no-cache, no-store, max-age=0, must-revalidate'
'pragma', 'no-cache'
'expires', '0'
'x-frame-options', 'DENY'
'content-security-policy', 'script-src 'self''
'referrer-policy', 'no-referrer'
'content-type', 'application/json'
'date', 'Fri, 12 Apr 2024 09:19:42 GMT'
'x-envoy-upstream-service-time', '2'
'server', 'envoy'
@j2gg0s
Copy link
Contributor

j2gg0s commented Apr 23, 2024

Your curl pod can see DestinationRule/VirtualService?

You can check by istioctl proxy-config route Your-Curl-Pod.
Or check your default exportTo

@Raven6681
Copy link
Author

yes, both:

istioctl -n agi-connector proxy-config route agi-connector-689d57fd64-8h4f9 | grep -E "sticky|VIRTUAL"
NAME                                                                             VHOST NAME                                                                       DOMAINS                                                                                              MATCH                  VIRTUAL SERVICE
80                                                                               my_gateway_fqdn:80                                                               my_gateway_fqdn                                                                                      /DiaManT*              diamant-workers-sticky-vs.diamant
80                                                                               diamant-workers.diamant.svc.cluster.local:80                                     diamant-workers.diamant.svc.cluster.local                                                            /DiaManT*              diamant-workers-sticky-vs.diamant
8080                                                                             my_gateway_fqdn:8080                                                             my_gateway_fqdn                                                                                      /DiaManT*              diamant-workers-sticky-vs.diamant
8080                                                                             diamant-workers.diamant.svc.cluster.local:8080                                   diamant-workers.diamant, *.diamant-workers.diamant                                                   /DiaManT*              diamant-workers-sticky-vs.diamant

istioctl proxy-config cluster  agi-connector-689d57fd64-8h4f9 -n agi-connector | grep -E "sticky|DESTINATION"
SERVICE FQDN                                                               PORT      SUBSET     DIRECTION     TYPE             DESTINATION RULE
diamant-workers.diamant.svc.cluster.local                                  8080      -          outbound      ORIGINAL_DST     diamant-workers-sticky-dr.diamant

agi-connector is the source NS, diamant is the destination NS.

I have tried adding both NSs in exportTo, adding exportTo: ["*"], with no change.

curl -v http://diamant-workers.diamant.svc.cluster.local:8080/DiaManT/admin/state
*   Trying 10.102.4.14:8080...
* Connected to diamant-workers.diamant.svc.cluster.local (10.102.4.14) port 8080 (#0)
> GET /DiaManT/admin/state HTTP/1.1
> Host: diamant-workers.diamant.svc.cluster.local:8080
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< x-frame-options: DENY
< content-security-policy: script-src 'self'
< referrer-policy: no-referrer
< content-type: application/json
< date: Wed, 24 Apr 2024 07:48:03 GMT
< x-envoy-upstream-service-time: 3
< server: envoy
< transfer-encoding: chunked
<
{ ... some json output }

@j2gg0s
Copy link
Contributor

j2gg0s commented Apr 24, 2024

Maybe we should modify the exportTo of destination service, https://istio.io/latest/docs/reference/config/annotations/#NetworkingExportTo

I test with source ping.test-ohai, destination pong.ohai

istioctl pc route ping-vz-6578fdfd76-vm6n2 -n test-ohai --name 80 -o json | jq '.[].virtualHosts[] | select(.name == "pong.ohai.svc.cluster.local:80") | .routes[].route.hashPolicy'
[
  {
    "cookie": {
      "name": "istio",
      "ttl": "0s"
    }
  }
]

@Raven6681
Copy link
Author

added the annotation networking.istio.io/exportTo: '*' to the destination Service, indeed before that your query above returned nothing.
Now it returns for both port 80 and 8080:

istioctl pc route agi-connector-689d57fd64-vvq5f -n agi-connector --name 8080 -o json | jq '.[].virtualHosts[] | select(.name == "diamant-workers.diamant.svc.cluster.local:8080") | .routes[].route.hashPolicy'
[
  {
    "cookie": {
      "name": "istio",
      "ttl": "0s"
    }
  }
]
istioctl pc route agi-connector-689d57fd64-vvq5f -n agi-connector --name 80 -o json | jq '.[].virtualHosts[] | select(.name == "diamant-workers.diamant.svc.cluster.local:80") | .routes[].route.hashPolicy'
[
  {
    "cookie": {
      "name": "istio",
      "ttl": "0s"
    }
  }
]

but again, no set-cookie seen in curl or the istio-proxy logs

2024-04-24T11:11:40.119094Z	debug	envoy filter external/envoy/source/extensions/filters/listener/original_dst/original_dst.cc:69	original_dst: set destination to 10.102.4.14:8080	thread=23
2024-04-24T11:11:40.119237Z	debug	envoy filter external/envoy/source/extensions/filters/listener/http_inspector/http_inspector.cc:139	http inspector: set application protocol to http/1.1	thread=23
2024-04-24T11:11:40.119308Z	debug	envoy conn_handler external/envoy/source/common/listener_manager/active_tcp_listener.cc:160	[Tags: "ConnectionId":"45"] new connection from 10.102.4.254:47248	thread=23
2024-04-24T11:11:40.119350Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:393	[Tags: "ConnectionId":"45"] new stream	thread=23
2024-04-24T11:11:40.119440Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1192	[Tags: "ConnectionId":"45","StreamId":"4363738411584164444"] request headers complete (end_stream=true):
':authority', 'diamant-workers.diamant.svc.cluster.local:8080'
':path', '/DiaManT/admin/state'
':method', 'GET'
'user-agent', 'curl/7.88.1'
'accept', '*/*'
	thread=23
2024-04-24T11:11:40.119456Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1175	[Tags: "ConnectionId":"45","StreamId":"4363738411584164444"] request end stream	thread=23
2024-04-24T11:11:40.119482Z	debug	envoy connection external/envoy/source/common/network/connection_impl.h:98	[Tags: "ConnectionId":"45"] current connecting state: false	thread=23
2024-04-24T11:11:40.119551Z	debug	envoy filter source/extensions/filters/http/alpn/alpn_filter.cc:92	override with 3 ALPNs	thread=23
2024-04-24T11:11:40.119572Z	debug	envoy router external/envoy/source/common/router/router.cc:514	[Tags: "ConnectionId":"45","StreamId":"4363738411584164444"] cluster 'outbound|8080||diamant-workers.diamant.svc.cluster.local' match for URL '/DiaManT/admin/state'	thread=23
2024-04-24T11:11:40.119630Z	debug	envoy router external/envoy/source/common/router/router.cc:731	[Tags: "ConnectionId":"45","StreamId":"4363738411584164444"] router decoding headers:
':authority', 'diamant-workers.diamant.svc.cluster.local:8080'
':path', '/DiaManT/admin/state'
':method', 'GET'
':scheme', 'http'
'user-agent', 'curl/7.88.1'
'accept', '*/*'
'x-forwarded-proto', 'http'
'x-request-id', 'f679d60e-4d87-4cd9-a711-a7b4fb1dab43'
'x-envoy-decorator-operation', 'diamant-workers.diamant.svc.cluster.local:8080/DiaManT*'
'x-envoy-peer-metadata-id', 'sidecar~10.102.4.254~agi-connector-689d57fd64-rghl7.agi-connector~agi-connector.svc.cluster.local'
'x-envoy-peer-metadata', 'CiEKDkFQUF9DT05UQUlORVJTEg8aDWFnaS1jb25uZWN0b3IKGgoKQ0xVU1RFUl9JRBIMGgpLdWJlcm5ldGVzCh4KDElOU1RBTkNFX0lQUxIOGgwxMC4xMDIuNC4yNTQKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjIxLjEK8wEKBkxBQkVMUxLoASrlAQotChphcHAua3ViZXJuZXRlcy5pby9pbnN0YW5jZRIPGg1hZ2ktY29ubmVjdG9yCikKFmFwcC5rdWJlcm5ldGVzLmlvL25hbWUSDxoNYWdpLWNvbm5lY3RvcgokChlzZWN1cml0eS5pc3Rpby5pby90bHNNb2RlEgcaBWlzdGlvCjIKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSDxoNYWdpLWNvbm5lY3RvcgovCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIIGgZsYXRlc3QKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsCigKBE5BTUUSIBoeYWdpLWNvbm5lY3Rvci02ODlkNTdmZDY0LXJnaGw3ChwKCU5BTUVTUEFDRRIPGg1hZ2ktY29ubmVjdG9yClcKBU9XTkVSEk4aTGt1YmVybmV0ZXM6Ly9hcGlzL2FwcHMvdjEvbmFtZXNwYWNlcy9hZ2ktY29ubmVjdG9yL2RlcGxveW1lbnRzL2FnaS1jb25uZWN0b3IKgwEKEVBMQVRGT1JNX01FVEFEQVRBEm4qbAolChVhd3NfYXZhaWxhYmlsaXR5X3pvbmUSDBoKZXUtd2VzdC0xYgooCg9hd3NfaW5zdGFuY2VfaWQSFRoTaS0wYzRhMjAzODViMzlhZTEwYQoZCgphd3NfcmVnaW9uEgsaCWV1LXdlc3QtMQogCg1XT1JLTE9BRF9OQU1FEg8aDWFnaS1jb25uZWN0b3I='
'x-envoy-attempt-count', '1'
	thread=23
2024-04-24T11:11:40.119662Z	debug	envoy pool external/envoy/source/common/http/conn_pool_base.cc:78	queueing stream due to no available connections (ready=0 busy=0 connecting=0)	thread=23
2024-04-24T11:11:40.119667Z	debug	envoy pool external/envoy/source/common/conn_pool/conn_pool_base.cc:291	trying to create new connection	thread=23
2024-04-24T11:11:40.119671Z	debug	envoy pool external/envoy/source/common/conn_pool/conn_pool_base.cc:145	creating a new connection (connecting=0)	thread=23
2024-04-24T11:11:40.119742Z	debug	envoy connection external/envoy/source/common/network/connection_impl.h:98	[Tags: "ConnectionId":"46"] current connecting state: true	thread=23
2024-04-24T11:11:40.119754Z	debug	envoy client external/envoy/source/common/http/codec_client.cc:57	[Tags: "ConnectionId":"46"] connecting	thread=23
2024-04-24T11:11:40.119758Z	debug	envoy connection external/envoy/source/common/network/connection_impl.cc:1009	[Tags: "ConnectionId":"46"] connecting to 10.102.4.14:8080	thread=23
2024-04-24T11:11:40.119849Z	debug	envoy connection external/envoy/source/common/network/connection_impl.cc:1028	[Tags: "ConnectionId":"46"] connection in progress	thread=23
2024-04-24T11:11:40.119885Z	debug	envoy connection external/envoy/source/common/network/connection_impl.cc:746	[Tags: "ConnectionId":"46"] connectedthread=23
2024-04-24T11:11:40.120856Z	debug	envoy client external/envoy/source/common/http/codec_client.cc:88	[Tags: "ConnectionId":"46"] connected	thread=23
2024-04-24T11:11:40.120879Z	debug	envoy pool external/envoy/source/common/conn_pool/conn_pool_base.cc:328	[Tags: "ConnectionId":"46"] attaching to next stream	thread=23
2024-04-24T11:11:40.120886Z	debug	envoy pool external/envoy/source/common/conn_pool/conn_pool_base.cc:182	[Tags: "ConnectionId":"46"] creating stream	thread=23
2024-04-24T11:11:40.120899Z	debug	envoy router external/envoy/source/common/router/upstream_request.cc:578	[Tags: "ConnectionId":"45","StreamId":"4363738411584164444"] pool ready	thread=23
2024-04-24T11:11:40.120938Z	debug	envoy client external/envoy/source/common/http/codec_client.cc:141	[Tags: "ConnectionId":"46"] encode complete	thread=23
2024-04-24T11:11:40.124091Z	debug	envoy router external/envoy/source/common/router/router.cc:1506	[Tags: "ConnectionId":"45","StreamId":"4363738411584164444"] upstream headers complete: end_stream=false	thread=23
2024-04-24T11:11:40.124200Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1869	[Tags: "ConnectionId":"45","StreamId":"4363738411584164444"] encoding headers via codec (end_stream=false):
':status', '200'
'x-content-type-options', 'nosniff'
'x-xss-protection', '1; mode=block'
'cache-control', 'no-cache, no-store, max-age=0, must-revalidate'
'pragma', 'no-cache'
'expires', '0'
'x-frame-options', 'DENY'
'content-security-policy', 'script-src 'self''
'referrer-policy', 'no-referrer'
'content-type', 'application/json'
'date', 'Wed, 24 Apr 2024 11:11:40 GMT'
'x-envoy-upstream-service-time', '4'
'server', 'envoy'
	thread=23
2024-04-24T11:11:40.124392Z	debug	envoy client external/envoy/source/common/http/codec_client.cc:128	[Tags: "ConnectionId":"46"] response completethread=23
2024-04-24T11:11:40.124436Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1974	[Tags: "ConnectionId":"45","StreamId":"4363738411584164444"] Codec completed encoding stream.	thread=23
2024-04-24T11:11:40.126918Z	debug	envoy pool external/envoy/source/common/http/http1/conn_pool.cc:53	[Tags: "ConnectionId":"46"] response completethread=23
2024-04-24T11:11:40.126934Z	debug	envoy pool external/envoy/source/common/conn_pool/conn_pool_base.cc:215	[Tags: "ConnectionId":"46"] destroying stream: 0 remaining	thread=23
2024-04-24T11:11:40.127165Z	debug	envoy connection external/envoy/source/common/network/connection_impl.cc:714	[Tags: "ConnectionId":"45"] remote close	thread=23
2024-04-24T11:11:40.127181Z	debug	envoy connection external/envoy/source/common/network/connection_impl.cc:278	[Tags: "ConnectionId":"45"] closing socket: 0	thread=23
2024-04-24T11:11:40.127222Z	debug	envoy conn_handler external/envoy/source/common/listener_manager/active_stream_listener_base.cc:136	[Tags: "ConnectionId":"45"] adding to cleanup list	thread=23
2024-04-24T11:11:40.237315Z	debug	envoy conn_handler external/envoy/source/common/listener_manager/active_tcp_listener.cc:160	[Tags: "ConnectionId":"47"] new connection from 10.102.4.182:40972	thread=23
2024-04-24T11:11:40.237382Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:393	[Tags: "ConnectionId":"47"] new stream	thread=23
2024-04-24T11:11:40.237444Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1192	[Tags: "ConnectionId":"47","StreamId":"13351226591452247748"] request headers complete (end_stream=true):
':authority', '10.102.4.254:15021'
':path', '/healthz/ready'
':method', 'GET'
'user-agent', 'kube-probe/1.29+'
'accept', '*/*'
'connection', 'close'
	thread=23
2024-04-24T11:11:40.237460Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1175	[Tags: "ConnectionId":"47","StreamId":"13351226591452247748"] request end stream	thread=23
2024-04-24T11:11:40.237484Z	debug	envoy connection external/envoy/source/common/network/connection_impl.h:98	[Tags: "ConnectionId":"47"] current connecting state: false	thread=23
2024-04-24T11:11:40.237513Z	debug	envoy router external/envoy/source/common/router/router.cc:514	[Tags: "ConnectionId":"47","StreamId":"13351226591452247748"] cluster 'agent' match for URL '/healthz/ready'	thread=23
2024-04-24T11:11:40.237556Z	debug	envoy router external/envoy/source/common/router/router.cc:731	[Tags: "ConnectionId":"47","StreamId":"13351226591452247748"] router decoding headers:
':authority', '10.102.4.254:15021'
':path', '/healthz/ready'
':method', 'GET'
':scheme', 'http'
'user-agent', 'kube-probe/1.29+'
'accept', '*/*'
'x-forwarded-proto', 'http'
'x-request-id', '0d13815c-ecdc-446a-96d7-10af2b535cef'
'x-envoy-expected-rq-timeout-ms', '15000'
	thread=23
2024-04-24T11:11:40.237582Z	debug	envoy pool external/envoy/source/common/conn_pool/conn_pool_base.cc:265	[Tags: "ConnectionId":"8"] using existing fully connected connection	thread=23
2024-04-24T11:11:40.237591Z	debug	envoy pool external/envoy/source/common/conn_pool/conn_pool_base.cc:182	[Tags: "ConnectionId":"8"] creating stream	thread=23
2024-04-24T11:11:40.237605Z	debug	envoy router external/envoy/source/common/router/upstream_request.cc:578	[Tags: "ConnectionId":"47","StreamId":"13351226591452247748"] pool ready	thread=23
2024-04-24T11:11:40.237630Z	debug	envoy client external/envoy/source/common/http/codec_client.cc:141	[Tags: "ConnectionId":"8"] encode complete	thread=23
2024-04-24T11:11:40.238010Z	debug	envoy client external/envoy/source/common/http/codec_client.cc:128	[Tags: "ConnectionId":"8"] response complete	thread=23
2024-04-24T11:11:40.238048Z	debug	envoy router external/envoy/source/common/router/router.cc:1506	[Tags: "ConnectionId":"47","StreamId":"13351226591452247748"] upstream headers complete: end_stream=true	thread=23
2024-04-24T11:11:40.238095Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1803	[Tags: "ConnectionId":"47","StreamId":"13351226591452247748"] closing connection due to connection close header	thread=23
2024-04-24T11:11:40.238112Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1869	[Tags: "ConnectionId":"47","StreamId":"13351226591452247748"] encoding headers via codec (end_stream=true):
':status', '200'
'date', 'Wed, 24 Apr 2024 11:11:40 GMT'
'content-length', '0'
'x-envoy-upstream-service-time', '0'
'server', 'envoy'
'connection', 'close'
	thread=23
2024-04-24T11:11:40.238134Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1974	[Tags: "ConnectionId":"47","StreamId":"13351226591452247748"] Codec completed encoding stream.	thread=23
2024-04-24T11:11:40.238151Z	debug	envoy connection external/envoy/source/common/network/connection_impl.cc:146	[Tags: "ConnectionId":"47"] closing data_to_write=143 type=0	thread=23
2024-04-24T11:11:40.238158Z	debug	envoy connection external/envoy/source/common/network/connection_impl_base.cc:47	[Tags: "ConnectionId":"47"] setting delayed close timer with timeout 1000 ms	thread=23
2024-04-24T11:11:40.238172Z	debug	envoy pool external/envoy/source/common/http/http1/conn_pool.cc:53	[Tags: "ConnectionId":"8"] response complete	thread=23
2024-04-24T11:11:40.238184Z	debug	envoy pool external/envoy/source/common/conn_pool/conn_pool_base.cc:215	[Tags: "ConnectionId":"8"] destroying stream: 0 remaining	thread=23
2024-04-24T11:11:40.238250Z	debug	envoy connection external/envoy/source/common/network/connection_impl.cc:788	[Tags: "ConnectionId":"47"] write flush complete	thread=23
2024-04-24T11:11:40.238258Z	debug	envoy connection external/envoy/source/common/network/connection_impl.cc:278	[Tags: "ConnectionId":"47"] closing socket: 1	thread=23
2024-04-24T11:11:40.238285Z	debug	envoy conn_handler external/envoy/source/common/listener_manager/active_stream_listener_base.cc:136	[Tags: "ConnectionId":"47"] adding to cleanup list	thread=23
2024-04-24T11:11:40.424953Z	debug	envoy dns external/envoy/source/extensions/network/dns_resolver/cares/dns_impl.cc:365	dns resolution for zipkin.istio-system started	thread=14
2024-04-24T11:11:40.425943Z	debug	envoy dns external/envoy/source/extensions/network/dns_resolver/cares/dns_impl.cc:285	dns resolution for zipkin.istio-system completed with status 0	thread=14
2024-04-24T11:11:40.425983Z	debug	envoy upstream external/envoy/source/common/upstream/upstream_impl.cc:437	transport socket match, socket default selected for host with address 172.20.11.80:9411	thread=14
2024-04-24T11:11:40.425999Z	debug	envoy upstream external/envoy/source/extensions/clusters/strict_dns/strict_dns_cluster.cc:177	DNS refresh rate reset for zipkin.istio-system, refresh rate 5000 ms	thread=14
{"route_name":null,"response_flags":"-","requested_server_name":null,"user_agent":"curl/7.88.1","request_id":"f679d60e-4d87-4cd9-a711-a7b4fb1dab43","method":"GET","bytes_sent":1090,"upstream_local_address":"10.102.4.254:47252","response_code":200,"downstream_remote_address":"10.102.4.254:47248","upstream_host":"10.102.4.14:8080","protocol":"HTTP/1.1","downstream_local_address":"10.102.4.14:8080","upstream_service_time":"4","response_code_details":"via_upstream","start_time":"2024-04-24T11:11:40.119Z","bytes_received":0,"upstream_transport_failure_reason":null,"duration":5,"authority":"diamant-workers.diamant.svc.cluster.local:8080","x_forwarded_for":null,"connection_termination_details":null,"path":"/DiaManT/admin/state","upstream_cluster":"outbound|8080||diamant-workers.diamant.svc.cluster.local"}

on the contrary, I can see the set-cookie in the istioingressgateway pod

2024-04-24T11:15:21.572184Z	debug	envoy main external/envoy/source/server/server.cc:239	flushing stats	thread=14
2024-04-24T11:15:22.916128Z	debug	envoy conn_handler external/envoy/source/common/listener_manager/active_tcp_listener.cc:160	[Tags: "ConnectionId":"8921"] new connection from 10.102.4.135:14236	thread=22
2024-04-24T11:15:22.918473Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:393	[Tags: "ConnectionId":"8921"] new stream	thread=22
2024-04-24T11:15:22.918539Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1192	[Tags: "ConnectionId":"8921","StreamId":"17682034051785138785"] request headers complete (end_stream=true):
':authority', 'my.gateway.fqdn'
':path', '/DiaManT/admin/state'
':method', 'GET'
'x-forwarded-for', '10.102.4.254'
'x-forwarded-proto', 'https'
'x-forwarded-port', '443'
'x-amzn-trace-id', 'Root=1-6628e9ca-5c2573f86e1efda34fe6f46a'
'user-agent', 'curl/7.88.1'
'accept', '*/*'
	thread=22
2024-04-24T11:15:22.918554Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1175	[Tags: "ConnectionId":"8921","StreamId":"17682034051785138785"] request end stream	thread=22
2024-04-24T11:15:22.918588Z	debug	envoy connection external/envoy/source/common/network/connection_impl.h:98	[Tags: "ConnectionId":"8921"] current connecting state: false	thread=22
2024-04-24T11:15:22.918681Z	debug	envoy filter source/extensions/filters/http/alpn/alpn_filter.cc:92	override with 3 ALPNs	thread=22
2024-04-24T11:15:22.918699Z	debug	envoy router external/envoy/source/common/router/router.cc:514	[Tags: "ConnectionId":"8921","StreamId":"17682034051785138785"] cluster 'outbound|8080||diamant-workers.diamant.svc.cluster.local' match for URL '/DiaManT/admin/state'	thread=22
2024-04-24T11:15:22.918737Z	debug	envoy router external/envoy/source/common/router/router.cc:731	[Tags: "ConnectionId":"8921","StreamId":"17682034051785138785"] router decoding headers:
':authority', 'my.gateway.fqdn'
':path', '/DiaManT/admin/state'
':method', 'GET'
':scheme', 'https'
'x-forwarded-for', '10.102.4.254,10.102.4.135'
'x-forwarded-proto', 'https'
'x-forwarded-port', '443'
'x-amzn-trace-id', 'Root=1-6628e9ca-5c2573f86e1efda34fe6f46a'
'user-agent', 'curl/7.88.1'
'accept', '*/*'
'x-envoy-external-address', '10.102.4.135'
'x-request-id', 'd143f4d6-2aec-4611-a454-ea05c6ec5e4a'
'x-envoy-decorator-operation', 'diamant-workers.diamant.svc.cluster.local:8080/DiaManT*'
'x-envoy-peer-metadata-id', 'router~10.102.3.221~istio-ingressgateway-internal-584b5fb44d-7l5wz.istio-ingress-internal~istio-ingress-internal.svc.cluster.local'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKHgoMSU5TVEFOQ0VfSVBTEg4aDDEwLjEwMi4zLjIyMQoZCg1JU1RJT19WRVJTSU9OEggaBjEuMjEuMQryAQoGTEFCRUxTEucBKuQBCiYKA2FwcBIfGh1pc3Rpby1pbmdyZXNzZ2F0ZXdheS1pbnRlcm5hbAoiCgVpc3RpbxIZGhdpbmdyZXNzZ2F0ZXdheS1pbnRlcm5hbApCCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEh8aHWlzdGlvLWluZ3Jlc3NnYXRld2F5LWludGVybmFsCi8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAohChdzaWRlY2FyLmlzdGlvLmlvL2luamVjdBIGGgR0cnVlChoKB01FU0hfSUQSDxoNY2x1c3Rlci5sb2NhbAo4CgROQU1FEjAaLmlzdGlvLWluZ3Jlc3NnYXRld2F5LWludGVybmFsLTU4NGI1ZmI0NGQtN2w1d3oKJQoJTkFNRVNQQUNFEhgaFmlzdGlvLWluZ3Jlc3MtaW50ZXJuYWwKcAoFT1dORVISZxpla3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2lzdGlvLWluZ3Jlc3MtaW50ZXJuYWwvZGVwbG95bWVudHMvaXN0aW8taW5ncmVzc2dhdGV3YXktaW50ZXJuYWwKgwEKEVBMQVRGT1JNX01FVEFEQVRBEm4qbAolChVhd3NfYXZhaWxhYmlsaXR5X3pvbmUSDBoKZXUtd2VzdC0xYQooCg9hd3NfaW5zdGFuY2VfaWQSFRoTaS0wMmE2OWRkMzcyYjYyYTcxMAoZCgphd3NfcmVnaW9uEgsaCWV1LXdlc3QtMQowCg1XT1JLTE9BRF9OQU1FEh8aHWlzdGlvLWluZ3Jlc3NnYXRld2F5LWludGVybmFs'
'x-envoy-attempt-count', '1'
	thread=22
2024-04-24T11:15:22.918766Z	debug	envoy pool external/envoy/source/common/conn_pool/conn_pool_base.cc:265	[Tags: "ConnectionId":"8919"] using existing fully connected connection	thread=22
2024-04-24T11:15:22.918774Z	debug	envoy pool external/envoy/source/common/conn_pool/conn_pool_base.cc:182	[Tags: "ConnectionId":"8919"] creating streamthread=22
2024-04-24T11:15:22.918789Z	debug	envoy router external/envoy/source/common/router/upstream_request.cc:578	[Tags: "ConnectionId":"8921","StreamId":"17682034051785138785"] pool ready	thread=22
2024-04-24T11:15:22.918824Z	debug	envoy client external/envoy/source/common/http/codec_client.cc:141	[Tags: "ConnectionId":"8919"] encode completethread=22
2024-04-24T11:15:22.921980Z	debug	envoy router external/envoy/source/common/router/router.cc:1506	[Tags: "ConnectionId":"8921","StreamId":"17682034051785138785"] upstream headers complete: end_stream=false	thread=22
2024-04-24T11:15:22.922059Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1869	[Tags: "ConnectionId":"8921","StreamId":"17682034051785138785"] encoding headers via codec (end_stream=false):
':status', '200'
'x-content-type-options', 'nosniff'
'x-xss-protection', '1; mode=block'
'cache-control', 'no-cache, no-store, max-age=0, must-revalidate'
'pragma', 'no-cache'
'expires', '0'
'x-frame-options', 'DENY'
'content-security-policy', 'script-src 'self''
'referrer-policy', 'no-referrer'
'content-type', 'application/json'
'date', 'Wed, 24 Apr 2024 11:15:22 GMT'
'x-envoy-upstream-service-time', '3'
'server', 'istio-envoy'
'set-cookie', 'istio="2c4c0519ea6f18d5"; HttpOnly'
	thread=22
2024-04-24T11:15:22.922285Z	debug	envoy client external/envoy/source/common/http/codec_client.cc:128	[Tags: "ConnectionId":"8919"] response complete	thread=22
2024-04-24T11:15:22.922315Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1974	[Tags: "ConnectionId":"8921","StreamId":"17682034051785138785"] Codec completed encoding stream.	thread=22
2024-04-24T11:15:22.922521Z	debug	envoy pool external/envoy/source/common/http/http1/conn_pool.cc:53	[Tags: "ConnectionId":"8919"] response complete	thread=22
2024-04-24T11:15:22.922536Z	debug	envoy pool external/envoy/source/common/conn_pool/conn_pool_base.cc:215	[Tags: "ConnectionId":"8919"] destroying stream: 0 remaining	thread=22
{"response_code_details":"via_upstream","upstream_local_address":"10.102.3.221:43466","requested_server_name":null,"x_forwarded_for":"10.102.4.254,10.102.4.135","duration":3,"start_time":"2024-04-24T11:15:22.918Z","method":"GET","user_agent":"curl/7.88.1","response_flags":"-","upstream_transport_failure_reason":null,"upstream_host":"10.102.4.14:8080","upstream_service_time":"3","response_code":200,"bytes_sent":1090,"connection_termination_details":null,"request_id":"d143f4d6-2aec-4611-a454-ea05c6ec5e4a","path":"/DiaManT/admin/state","protocol":"HTTP/1.1","route_name":null,"downstream_remote_address":"10.102.4.135:14236","authority":"my.gateway.fqdn","bytes_received":0,"upstream_cluster":"outbound|8080||diamant-workers.diamant.svc.cluster.local","downstream_local_address":"10.102.3.221:443"}

@Raven6681
Copy link
Author

The destination service is a headless service (StatefulSet). Is this perhaps the issue? Because if I change the kind, from StatefulSet to Deployment, I can see the cookie in curl.

@j2gg0s
Copy link
Contributor

j2gg0s commented Apr 24, 2024

The destination service is a headless service (StatefulSet). Is this perhaps the issue? Because if I change the kind, from StatefulSet to Deployment, I can see the cookie in curl.

I'm not familiar with the conversion rules in this area.
I suggest checking whether your request hits the relevant route through LDS/RDS.

@Raven6681
Copy link
Author

Raven6681 commented Apr 25, 2024 •
edited

By far I am no expert and I am not familiar with LDS/RDS, but as I understand (please correct me if I am wrong) if the request was relevant with LDS, I should be able to see the destination service, in the output of istioctl proxy-config listener <source_pod>.

On the contrary, for RDS, I should be able to see the destination service in the output of istioctl proxy-config route <source_pod>.

In the first scenario, I see nothing. I

In the second, when the destination service is headless (StatefulSet) I get this:

istioctl proxy-config route agi-connector-ffb658b7d-f4kpf.agi-connector -o json | jq '.[].virtualHosts[] | select(.name == "diamant-workers.diamant.svc.cluster.local:8080")'
{
  "name": "diamant-workers.diamant.svc.cluster.local:8080",
  "domains": [
    "diamant-workers.diamant.svc.cluster.local",
    "diamant-workers.diamant",
    "diamant-workers.diamant.svc",
    "*.diamant-workers.diamant.svc.cluster.local",
    "*.diamant-workers.diamant",
    "*.diamant-workers.diamant.svc"
  ],
  "routes": [
    {
      "match": {
        "prefix": "/DiaManT",
        "caseSensitive": true
      },
      "route": {
        "cluster": "outbound|8080||diamant-workers.diamant.svc.cluster.local",
        "timeout": "0s",
        "retryPolicy": {
          "retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
          "numRetries": 2,
          "retryHostPredicate": [
            {
              "name": "envoy.retry_host_predicates.previous_hosts",
              "typedConfig": {
                "@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
              }
            }
          ],
          "hostSelectionRetryMaxAttempts": "5",
          "retriableStatusCodes": [
            503
          ]
        },
        "hashPolicy": [
          {
            "cookie": {
              "name": "istio",
              "ttl": "0s"
            }
          }
        ],
        "maxGrpcTimeout": "0s"
      },
      "metadata": {
        "filterMetadata": {
          "istio": {
            "config": "/apis/networking.istio.io/v1alpha3/namespaces/diamant/virtual-service/diamant-workers-sticky-vs"
          }
        }
      },
      "decorator": {
        "operation": "diamant-workers.diamant.svc.cluster.local:8080/DiaManT*"
      }
    }
  ],
  "includeRequestAttemptCount": true
}

and when the destination is a Deployment, the only thing that changes is the domains which changes to:

{
    "name": "diamant-workers.diamant.svc.cluster.local:8080",
    "domains": [
      "diamant-workers.diamant.svc.cluster.local",
      "diamant-workers.diamant",
      "diamant-workers.diamant.svc",
      "172.20.218.161"
    ],
...

with the IP 172.20.218.161 being the ClusterIP of the service. Hash policy is there in both situations but cookie can be seen only when destination is a Deployment

@j2gg0s
Copy link
Contributor

j2gg0s commented Apr 29, 2024

The destination service is a headless service (StatefulSet). Is this perhaps the issue? Because if I change the kind, from StatefulSet to Deployment, I can see the cookie in curl.

I think headless might be the key.

A headless service will be transformed into a Cluster of type ORIGINAL_DST,
which is not configured with the HashPolicy of Route.

istioctl pc cluster ping-746bf5cdd5-2lkhr --fqdn "outbound|8080||pong.test-ohai.svc.cluster.local" -o json | jq '.[].type'
"ORIGINAL_DST"

When node is gateway rather than sidecar, headless service is still being translated into EDS, not ORIGINAL_DST.
So, hash policy is still work.

istioctl pc cluster test-ohai-default-88d7987b9-dwcdc.test-ingress-ohai --fqdn "outbound|8080||pong.test-ohai.svc.cluster.local" -o json |
 jq '.[].type'
"EDS"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Raven6681 @j2gg0s