Internal listener access logging in Envoy

[howardjohn]

When we use HBONE in envoy, we get access logs showing unexpected info

Without hbone:

[2024-04-08T20:53:05.142Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 1648 0 0 "10.244.0.1" "curl/8.7.1" "d68f23d1-02a2-4999-8572-6f154573c74c" "172.18.0.101" "10.244.0.10:80" outbound|80||echo.default.svc.cluster.local 10.244.0.7:53060 10.244.0.7:8080 10.244.0.1:59040 - 

With hbone:

[2024-04-08T20:44:12.363Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 1648 3 2 "10.244.0.1" "curl/8.7.1" "66dc2115-b7f5-45eb-96c2-2a2b221559f7" "172.18.0.101" "envoy://connect_originate/10.244.0.10:80" outbound|80||echo.default.svc.cluster.local envoy://internal_client_address/ 10.244.0.7:8080 10.244.0.1:27113 -

Probably we can tweak these to avoid exposing internals

[zirain]

I recall istio only print listener access log when 404 in the past?

[howardjohn]

This isn't about listener access logs, but for hbone the destination endpoint is an internal listener instead of ip

[zirain]

you can either change the hardcode log format or just remove it(this option may cause difficult to debug).

[howardjohn]

I think we want to log the real source and dest IP. This may just be changing the format to access some other info, but we might need to make changes in Envoy/istio-proxy to expose that info if its not available (not sure what info we have)

[howardjohn]

There are two parts here:

• Source ip is envoy://internal_client_address/. This I think requires us to either change envoy core, or pass "real source ip" as metadata across the internal listener
• Destination IP is "envoy://connect_originate/10.244.0.10:80". This seems easier, since the original IP is known to be in EDS metadata and access logs can access it. However, we would want to do something like metadata.ip OR upstream_addr for the non HBONE cases; I don't see how to do that in envoy. %CEL(...) seemed promising but has a lot of bugs

[zirain]

envoy://connect_originate/10.244.0.10:80 is from %UPSTREAM_HOST% which related to https://github.com/envoyproxy/envoy/blob/main/source/common/network/address_impl.cc#L470.
envoy://internal_client_address/ is from %UPSTREAM_LOCAL_ADDRESS% which related to https://github.com/envoyproxy/envoy/blob/main/source/extensions/bootstrap/internal_listener/client_connection_factory.cc#L30