Enable envoy access logging per pod

[rvennam]

This is a feature/doc request to enable envoy access logging per pod. For example, enabling access logs for ingress gateway pod or user pod is vital for debugging many issues.

The current tasks requires users to enable it for the entire mesh and requires a rerunning the istio installer with an additional flag.
https://istio.io/latest/docs/tasks/observability/logs/access-log/#enable-envoy-s-access-logging

Users do not necessarily have the ability or want to enable mesh wide settings to debug a single application issue.

[GregHanson]

Would it be worth posting an example EnvoyFilter for gateway and sidecar? For instance the following EnvoyFilter works for ingress gateway:

kubectl apply -n istio-system -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: test-filter
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        config:
          access_log:
          - name: envoy.file_access_log
            typed_config:
              "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
EOF

[rvennam]

@esnible istioctl proxy-config log seems close but it only sets envoy logging level..not access log. Should this be extended to support access as well? https://istio.io/latest/docs/reference/commands/istioctl/#istioctl-proxy-config-log

[rvennam]

@GregHanson Your EnvoyFilter example works great and I think it's worth adding to the Istio documentation.

[uc-dang-tiki]

It dose not work on istio 1.7.3. Our cluster config disable accesslog, and only want to open accesslog for some workload. I try this envoyfilter (base on @GregHanson example), but it dose not work for me

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: test-enable-accesslog
spec:
  workloadSelector:
    labels:
      workload.user.cattle.io/workloadselector: deployment-platform-prod-prod-eris
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        config:
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

[GregHanson]

@uc-dang-tiki your EnvoyFilter is missing a typed_config in your patch and the filterchain name is incorrect:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: test-enable-accesslog
spec:
  workloadSelector:
    labels:
      workload.user.cattle.io/workloadselector: deployment-platform-prod-prod-eris
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

Also, make sure the EnvoyFilter is created in the same namespace as your target workload. Also, the EnvoyFilter is only for HTTP traffic - TCP, TLS, and HTTPS traffic won't appear in the logs.

Always check to make sure istioctl proxy-status does not show STALE for the workload you are attempting to apply the EnvoyFilter to. If nothing is stale, you can check the istioctl proxy-config listener|route|cluster <pod_name>.<pod_namespace> -o json to confirm the right names and structures are used in the EnvoyFilter

[uc-dang-tiki]

Thanks you ! it work like a charm :) . But I also have to remove context

[PatrickDuncan]

Hey, it would great if this capability was added to the the localhost:15000 API in the istio-proxy. Example you can change the log level with it: curl -XPOST http://localhost:15000/logging?level=debug. We'd rather not create/delete Kube resources to change the access logging for a single Pod
@GregHanson

[tariq1890]

@PatrickDuncan You would need to file a feature request to Envoy instead of Istio. The link you refer to is the envoy admin endpoint.

@GregHanson @esnible I do, however, think it's reasonable for istioctl to provide a way to enable access-logging. Isn't it supposed to be a wrapper over all the envoy internal workings? Right now istioctl seems to be 1:1 mapped with the envoy admin endpoint. I think it would be useful for the EnvoyFilter abstraction to be added as a feature to istioctl.

[esnible]

@tariq1890 The fact that @GregHanson needed 26 lines of break-glass YAML to show how to do this, but @uc-dang-tiki didn't have success using Greg's example without additional help, shows the current difficulty of this.

Greg's solution requires a WorkloadSelector, rather than a pod ID. Perhaps an istioctl syntax like this:

kubectl label pod httpbin-5b5bdcdf6b-vrdn4 use-custom-logformat-123=true
istioctl pc access-log set-format --workload use-custom-logformat-123=true --format "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

The implementation of istioctl pc access-log set-format would be to create an EnvoyFilter.

Thoughts?

[tariq1890]

@esnible Sounds good.

I think we should have a default format. Ideally, I wouldn't want to specify a wordy format when I look to enable access logging

I was thinking something like this

istioctl pc access-log -l app=foo

We can just use the existing labels instead of adding a new one

[esnible]

@tariq1890 Would it make sense to have both a default format, or default short and long formats, plus custom?

Users could say istioctl pc access log ... -verbosity=high or istioctl pc access log ... -custom="..."?

[olliezhu]

Sorry in advance for jumping in this thread with another troubleshooting question. I tried to implement @GregHanson's suggestion but was not able to reproduce per-pod access logging. (I'm running Istio 1.5.10.)

$ kubectl apply --namespace myspace -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: test-filter
spec:
  workloadSelector:
    labels:
      myspace-service: ollies-service
      myspace-service-env: sandbox
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        config:
          access_log:
          - name: envoy.file_access_log
            typed_config:
              "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
EOF

For context, the pod I'd like to target contains metadata labels that include

metadata:
  labels:
    myspace-role: service-instance
    myspace-service: ollies-service
    myspace-service-env: sandbox
    myspace-version: git_196a3545
    security.istio.io/tlsMode: istio
    service.istio.io/canonical-name: ollies-service
    ...

I ran wget from this pod

$ kubectl exec --namespace myspace "$OS_POD" --container ollies-service -- wget example.com

and looked for any GET requests to example.com in the logs but could not find any.

$ kubectl logs --namespace myspace --selector myspace-service=ollies-service --container=ollies-service
$ kubectl logs --namespace myspace --selector myspace-service=ollies-service --container=istio-proxy

Also, a meta-ask: As a newbie to Istio, I'd appreciate some guidance on how to try to troubleshoot these kinds of issues myself. Currently my best bet is to create the EnvoyFilter as described and cross my fingers that it works as I expect it, and I have no visibility into what exactly is not working as expected and why.

[esnible]

@olliezhu for troubleshooting, the first thing I do is istioctl ps to see if any of the pods aren't SYNCED. If they are not, it usually means Envoy rejected what Pilot gave it.

The next thing is istioctl dashboard envoy <mypod>, and click on config_dump. That brings up Envoy's config. I then use the browser's "Find" to search that giant JSON config for something in my EnvoyFilter added, to see if Pilot pushed the change to Envoy.