Outbound Traffic Policy: REGISTRY_ONLY , sidecar ALLOW_ANY works for various ports but fails for port 80 (http)

[ceastman-r7]

Bug Description

I have the global mesh set to: Outbound Traffic Policy: REGISTRY_ONLY but I have a sidecar entry as follows:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: curl
  namespace: apps
spec:
  outboundTrafficPolicy:
    mode: ALLOW_ANY
  workloadSelector:
    labels:
      app: curl

No service entries specified in the apps namespace.

works as expected:
curl https://www.cnn.com

returns a 502 bad gateway:
curl http://www.cnn.com

when the sidecar entry is deleted
curl https://www.cnn.com
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.cnn.com:443

Version

istioctl version
client version: 1.12.4
control plane version: 1.12.4
data plane version: 1.12.4-tetratefips-v0 (79 proxies)

kubectl version --short
Client Version: v1.22.5
Server Version: v1.21.12-eks-a64ea69

Additional Information

No response

[howardjohn]

A bit confused since you said there is no ServiceEntry but then you deleted the Service Entry?

[ceastman-r7]

I deleted the sidecar crd:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  annotations:
    meta.helm.sh/release-name: cluster-sidecar
    meta.helm.sh/release-namespace: istio-system
  creationTimestamp: "2022-07-06T15:32:58Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: curl
  namespace: apps
  resourceVersion: "41667507"
  uid: 0990d03f-f0bf-41df-9811-6586d2022fd7
spec:
  outboundTrafficPolicy:
    mode: ALLOW_ANY
  workloadSelector:
    labels:
      app: curl

[ceastman-r7]

I turned on the istio proxy debug logs and found this:

{"level":"debug","time":"2022-07-11T20:25:23.897445Z","scope":"envoy http","msg":"[C827202] new stream"}
{"level":"debug","time":"2022-07-11T20:25:23.897475Z","scope":"envoy http","msg":"[C827202][S11150850561230200979] request headers complete (end_stream=true):\n':authority', 'google.com'\n':path', '/'\n':method', 'GET'\n'user-agent', 'curl/7.29.0'\n'accept', '*/*'\n"}
{"level":"debug","time":"2022-07-11T20:25:23.897482Z","scope":"envoy http","msg":"[C827202][S11150850561230200979] request end stream"}
{"level":"debug","time":"2022-07-11T20:25:23.897548Z","scope":"envoy filter","msg":"cannot find route entry"}
{"level":"debug","time":"2022-07-11T20:25:23.897567Z","scope":"envoy http","msg":"[C827202][S11150850561230200979] Sending local reply with details direct_response"}
{"level":"debug","time":"2022-07-11T20:25:23.897593Z","scope":"envoy http","msg":"[C827202][S11150850561230200979] encoding headers via codec (end_stream=true):\n':status', '502'\n'date', 'Mon, 11 Jul 2022 20:25:23 GMT'\n'server', 'envoy'\n"}

[ceastman-r7]

This seems specific to port 80 traffic (http). When I try other protocols they work as expected when the outbound traffic policy is set to registry_only but I have the sidecar resource set to override it for a specific microservice.

it seems that for port 80 (http) traffic it is not getting routed to the PassthroughCluster

works:
whois www.google.com
{"upstream_cluster":"PassthroughCluster","downstream_local_address":"192.0.47.59:43","response_tx_duration":null,"method":null,"upstream_local_address":"10.0.27.204:34358","request_duration":null,"upstream_transport_failure_reason":null,"protocol":null,"upstream_host":"192.0.47.59:43","duration":7,"start_time":"2022-07-19T14:38:36.958Z","bytes_received":16,"connection_termination_details":null,"bytes_sent":2140,"x_forwarded_for":null,"downstream_remote_address":"10.0.27.204:34356","authority":null,"requested_server_name":null,"request_id":null,"user_agent":null,"response_flags":"-","response_duration":null,"path":null,"response_code_details":null,"response_code":0}

curl https://www.google.com
{"upstream_cluster":"PassthroughCluster","response_tx_duration":null,"authority":null,"start_time":"2022-07-19T14:35:18.878Z","upstream_transport_failure_reason":null,"duration":73,"response_duration":null,"method":null,"response_code_details":null,"path":null,"bytes_received":848,"upstream_local_address":"10.0.27.204:40948","downstream_local_address":"172.253.122.147:443","upstream_host":"172.253.122.147:443","downstream_remote_address":"10.0.27.204:40946","protocol":null,"response_flags":"-","connection_termination_details":null,"requested_server_name":null,"request_duration":null,"user_agent":null,"x_forwarded_for":null,"request_id":null,"bytes_sent":20798,"response_code":0}

doesn't work:
curl http://www.google.com
{"method":"GET","start_time":"2022-07-19T14:34:26.853Z","downstream_remote_address":"10.0.27.204:53848","x_forwarded_for":null,"request_duration":0,"response_tx_duration":null,"response_code":502,"request_id":null,"bytes_received":0,"response_code_details":"direct_response","upstream_host":null,"user_agent":"curl/7.84.0-DEV","authority":"www.google.com","downstream_local_address":"172.253.63.99:80","bytes_sent":0,"connection_termination_details":null,"upstream_local_address":null,"path":"/","protocol":"HTTP/1.1","requested_server_name":null,"upstream_cluster":null,"duration":0,"response_flags":"-","response_duration":null,"upstream_transport_failure_reason":null}

[ceastman-r7]

@robertpanzer Why would port 80 (http) traffic get routed differently from other traffic?

[ceastman-r7]

Related to this request from 2019: #12873