verify_outgoing not supported on gRPC listener; verify_incoming breaks Consul Connect envoy sidecars

[quinndiggitypolymath]

Configuration that has worked sans issues from as early as consul 1.8.0 to 1.11.4 + 1.11.5, and has no issues with envoy 1.18.4, 1.18.6, or 1.20.2 (but lacks support for envoy 1.22.0), is broken in consul 1.12.0.

It seems consul 1.12.0 is no longer properly configuring envoy's gRPC client's TLS keypair causing: SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE for proxies managed by nomad (which also worked without issue for many version) or run with:

/usr/bin/consul connect envoy \
                -admin-bind=localhost:19000 \
                -grpc-addr=https://127.0.0.1:8502 \
                -ca-file=/etc/consul.d/tls/tls.ca.d/ca.chain.pem \
                -ca-path=/etc/consul.d/tls/tls.ca.d \
                -client-cert=/etc/consul.d/tls/tls.crt.pem \
                -client-key=/etc/consul.d/tls/tls.key.pem \
                -http-addr=https://127.0.0.1:8501 \
                -tls-server-name=localhost \
                -token=... \
                -envoy-version=1.20.2 \
                -sidecar-for ...

regardless of the following (redundant) env vars being set:

 export CONSUL_HTTP_SSL=true
 export CONSUL_HTTP_ADDR=https://127.0.0.1:8501
 export CONSUL_GRPC_ADDR=https://127.0.0.1:8502
 export CONSUL_CACERT="/etc/consul.d/tls/tls.ca.d/ca.chain.pem"
 export CONSUL_CAPATH="/etc/consul.d/tls/tls.ca.d"
 export CONSUL_CLIENT_CERT="/etc/consul.d/tls/tls.crt.pem"
 export CONSUL_CLIENT_KEY="/etc/consul.d/tls/tls.key.pem"

Is likely due to regressions caused by the recent changes made to decouple the grpc/https/internal_rpc configurations: https://www.consul.io/docs/upgrading/upgrade-specific#tls-configuration

It seems consul is no longer honouring verify_outgoing for envoy's gRPC client outbound connections

With or without the following previously fully functional, but now deprecated, options set...:

# deprecated - https://www.consul.io/docs/agent/config/config-files#tls_deprecated_options
ca_path   = "/etc/consul.d/tls/tls.ca.d/"
cert_file = "/etc/consul.d/tls/tls.crt.pem"
key_file  = "/etc/consul.d/tls/tls.key.pem"

# deprecated - https://www.consul.io/docs/agent/config/config-files#tls_deprecated_options
verify_incoming        = true
verify_incoming_rpc    = true
verify_incoming_https  = true
verify_outgoing        = true
verify_server_hostname = true

...all of the following forms all fail to have envoy utilize the configured client keypair:

tls {
  defaults {
    ca_file           = "/etc/consul.d/tls/tls.ca.d/ca.chain.pem"
    ca_path           = "/etc/consul.d/tls/tls.ca.d/"
    cert_file         = "/etc/consul.d/tls/tls.crt.pem"
    key_file          = "/etc/consul.d/tls/tls.key.pem"
    tls_min_version   = "TLSv1_3"
    #tls_cipher_suites = "" # https://github.com/hashicorp/consul/search?q=goTLSCipherSuites+%3D+map
    verify_incoming   = true
    verify_outgoing   = true
  }
  #grpc {
  #
  #}
  https {
    tls_min_version = "TLSv1_2" # terraform consul provider
  }
  internal_rpc {
    verify_server_hostname = true
  }
}

tls {
  defaults {
    ca_file           = "/etc/consul.d/tls/tls.ca.d/ca.chain.pem"
    ca_path           = "/etc/consul.d/tls/tls.ca.d/"
    cert_file         = "/etc/consul.d/tls/tls.crt.pem"
    key_file          = "/etc/consul.d/tls/tls.key.pem"
    tls_min_version   = "TLSv1_2" # terraform consul provider
    #tls_min_version   = "TLSv1_3"
    #tls_cipher_suites = "" # https://github.com/hashicorp/consul/search?q=goTLSCipherSuites+%3D+map
    verify_incoming   = true
    verify_outgoing   = true
  }
  grpc {
    ca_file           = "/etc/consul.d/tls/tls.ca.d/ca.chain.pem"
    ca_path           = "/etc/consul.d/tls/tls.ca.d/"
    cert_file         = "/etc/consul.d/tls/tls.crt.pem"
    key_file          = "/etc/consul.d/tls/tls.key.pem"
    tls_min_version   = "TLSv1_2" # terraform consul provider
    #tls_min_version   = "TLSv1_3"
    #tls_cipher_suites = "" # https://github.com/hashicorp/consul/search?q=goTLSCipherSuites+%3D+map
    verify_incoming   = true
  }
  https {
    ca_file           = "/etc/consul.d/tls/tls.ca.d/ca.chain.pem"
    ca_path           = "/etc/consul.d/tls/tls.ca.d/"
    cert_file         = "/etc/consul.d/tls/tls.crt.pem"
    key_file          = "/etc/consul.d/tls/tls.key.pem"
    tls_min_version   = "TLSv1_2" # terraform consul provider
    #tls_min_version   = "TLSv1_3"
    #tls_cipher_suites = "" # https://github.com/hashicorp/consul/search?q=goTLSCipherSuites+%3D+map
    verify_incoming   = true
    verify_outgoing   = true
  }
  internal_rpc {
    ca_file                = "/etc/consul.d/tls/tls.ca.d/ca.chain.pem"
    ca_path                = "/etc/consul.d/tls/tls.ca.d/"
    cert_file              = "/etc/consul.d/tls/tls.crt.pem"
    key_file               = "/etc/consul.d/tls/tls.key.pem"
    tls_min_version        = "TLSv1_2" # terraform consul provider
    #tls_min_version        = "TLSv1_3"
    #tls_cipher_suites      = "" # https://github.com/hashicorp/consul/search?q=goTLSCipherSuites+%3D+map
    verify_incoming        = true
    verify_outgoing        = true
    verify_server_hostname = true
  }
}

as well as any of the above utilizing TLSv1_3

[quinndiggitypolymath]

the same issue applies to consul 1.12.0 against envoy 1.22.0; nomad 1.2.6as well is no longer able to start connect sidecars, failing with the exact same SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE, but is perfectly fine when consul 1.12.0 is ripped out and reverted to 1.11.5

[quinndiggitypolymath]

just another note, the same issue applies in all combinations of consul connect envoy ... with any combination of -envoy-version=... or -envoy-binary=... for either of envoy 1.22.0 or 1.20.2

consul 1.12.0 is completely unable to interop with envoy, with or without nomad managing the sidecar configuration

[mikemorris]

The PR that changed this config parsing was #12504, which may be helpful in tracking down a possible regression - specifically the change in agent/xds/server.go from tlsConfigurator.Cert() != nil to tlsConfigurator.GRPCTLSConfigured() (and the implementation of those two functions) looks worth investigating more closely.

[shoenig]

Reports of this issue have been showing up in a few Nomad channels. I just want to clarify that this bug is not related to Nomad and can be reproduced using only Consul / envoy.

➜ consul version 
Consul v1.12.0
Revision 09a8cdb4a
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

➜ cat consul.hcl 
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-server-consul-0.pem"
key_file = "dc1-server-consul-0-key.pem"

ports {
  https = 8501
}

➜ cat redis.hcl 
services {
  name = "redis"
  id = "redis"
  port = 6789
  address = "127.0.0.1"

  connect {
    sidecar_service {
      name = "redis-sidecar-proxy"
      port = 6789
      address = "127.0.0.1"    
    }
  }
}

➜ cat envoy.sh 
#!/usr/bin/env bash

set -euo pipefail

consul connect envoy \
       -admin-bind=localhost:19000 \
       -grpc-addr=https://127.0.0.1:8502 \
       -ca-file=/tmp/repro/consul-agent-ca.pem \
       -client-cert=/tmp/repro/dc1-client-consul-0.pem \
       -client-key=/tmp/repro/dc1-client-consul-0-key.pem \
       -http-addr=https://127.0.0.1:8501 \
       -tls-server-name=localhost \
       -envoy-version=1.20.2 \
       -sidecar-for redis

consul agent -dev -config-file=consul.hcl -config-file=redis.hcl

➜ ./envoy.sh

[2022-05-17 13:56:37.251][156368][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:188] cm init: initializing cds
[2022-05-17 13:56:37.251][156368][warning][main] [external/envoy/source/server/server.cc:642] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[2022-05-17 13:56:37.251][156368][info][main] [external/envoy/source/server/server.cc:764] starting main dispatch loop
[2022-05-17 13:56:37.252][156368][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] DeltaAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
[2022-05-17 13:56:37.557][156368][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] DeltaAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
[2022-05-17 13:56:37.960][156368][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] DeltaAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE

[quinndiggitypolymath]

indeed - with or without nomad managing the sidecar configuration, consul 1.12.0 is completely unable to interop with envoy with the introduction of this regression

[eculver]

We (Consul Core) are looking into this and should have some feedback for you shortly.

[boxofrad]

Hi @quinndiggitypolymath 👋🏻

Thanks for the report! It looks like this is a regression in the way we handle the (now deprecated) top-level verify_incoming option.

Prior to 1.12, it wasn't possible to enable TLS client-auth on Consul's gRPC port. From 1.12 onwards, you can enable it with tls.grpc.verify_incoming and it inherits tls.defaults.verify_incoming.

We should be maintaining the old behavior when translating from the deprecated verify_incoming field (here) but it looks like I missed it and we're assuming verify_incoming applies to the gRPC port too, sorry! We'll get that fixed in a patch release.

If you're using the new-style TLS configuration, you can explicitly disable it by setting tls.grpc.verify_incoming to false.

In the future, we may want to update the consul connect envoy command to configure client certificates.

[quinndiggitypolymath]

@boxofrad, while I appreciate the suggested workaround, the preference would really be to correctly leverage mTLS on envoy's gRPC client connections, so as to not require the disabling of this rather important validation; hopefully the following is a relevant/useful overview of the method envoy exposes in order to do so: https://www.envoyproxy.io/docs/envoy/latest/start/quick-start/securing#use-mutual-tls-mtls-to-connect-with-client-certificates

...
46              private_key:
47                filename: certs/serverkey.pem
48
49  clusters:
50  - name: service_envoyproxy_io
51    type: LOGICAL_DNS
52    # Comment out the following line to test on v6 networks
53    dns_lookup_family: V4_ONLY
54    load_assignment:
55      cluster_name: service_envoyproxy_io
56      endpoints:
57      - lb_endpoints:
58        - endpoint:
59            address:
60              socket_address:
61                address: www.envoyproxy.io
62                port_value: 443
63    transport_socket:
64      name: envoy.transport_sockets.tls
65      typed_config:
66        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
67        common_tls_context:
68          tls_certificates:
69          - certificate_chain:
70              filename: certs/clientcert.pem
...

[quinndiggitypolymath]

A viable workaround I believe would be leveraging --config-yaml with envoy ( https://www.envoyproxy.io/docs/envoy/latest/operations/cli#cmdoption-config-yaml ), however I lack expertise with envoy and haven't fully connected how to make this work well with the client tls certificates injected via a json string passed to the envoy binary via nomad/consul; would be amazing if there's someone who can mention a proof of concept of this working that I can extend upon for our use cases.

[quinndiggitypolymath]

A viable workaround I believe would be leveraging --config-yaml with envoy ( https://www.envoyproxy.io/docs/envoy/latest/operations/cli#cmdoption-config-yaml ), however I lack expertise with envoy and haven't fully connected how to make this work well with the client tls certificates injected via a json string passed to the envoy binary via nomad/consul; would be amazing if there's someone who can mention a proof of concept of this working that I can extend upon for our use cases.

specifically how to configure the gRPC tls client keypair in envoy's specific implementation, as the --config-yaml parameter itself is not tough:

consul connect envoy ... -- --config-yaml '{"...":"..."}'

[quinndiggitypolymath]

and on the nomad side:

      connect {
        sidecar_task {
          shutdown_delay = "5s"
          config {
            ...

[quinndiggitypolymath]

last note until I can revisit this issue, as we use vault as the CA for consul/connect, the keypair itself on the nomad side is straightforward via:

template {
        change_mode   = "restart"
        #change_mode   = "signal"
        #change_signal = "SIGHUP"
        destination   = "secrets/consul.tls.key.pem"
        data          = <<-EOF
        {{- $common_name := (print "common_name=client." (env "meta.NNNNNN.hashi_datacenter_name") ".consul") -}}
        {{- with secret "pki/issue/consul-client" $common_name "ip_sans=127.0.0.1" -}}
        {{ .Data.private_key }}
        {{- end }}
        EOF
      }
      template {
        change_mode   = "restart"
        #change_mode   = "signal"
        #change_signal = "SIGHUP"
        destination   = "secrets/consul.tls.crt.pem"
        data          = <<-EOF
        {{- $common_name := (print "common_name=client." (env "meta.NNNNNN.hashi_datacenter_name") ".consul") -}}
        {{- with secret "pki/issue/consul-client" $common_name "ip_sans=127.0.0.1" -}}
        {{ .Data.certificate }}
        {{- end }}
        EOF
      }
      ```

[quinndiggitypolymath]

The envoy specifics of this seem to be:
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/grpc_service.proto#config-core-v3-grpcservice

google_grpc
...
Precisely one of envoy_grpc, google_grpc must be set.

envoy_grpc doesn't support tls client creds, but google_grpc does:
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/grpc_service.proto#config-core-v3-grpcservice-googlegrpc

channel_credentials

config.core.v3.GrpcService.GoogleGrpc.ChannelCredentials

https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/grpc_service.proto#envoy-v3-api-msg-config-core-v3-grpcservice-googlegrpc-channelcredentials

ssl_credentials

config.core.v3.GrpcService.GoogleGrpc.SslCredentials

Precisely one of ssl_credentials, google_default, local_credentials must be set.

https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/grpc_service.proto#envoy-v3-api-msg-config-core-v3-grpcservice-googlegrpc-sslcredentials

{
  "root_certs": "{...}",
  "private_key": "{...}",
  "cert_chain": "{...}"
}

[quinndiggitypolymath]

and seems to be leveraged in other areas within consul:

consul/agent/checks/grpc.go

Line 11 in 21e855d

"google.golang.org/grpc/credentials"

package checks

import (
	...
	"google.golang.org/grpc/credentials"
	...
)

[quinndiggitypolymath]

the BootstrapConfig:
https://github.com/hashicorp/consul/blob/v1.12.0/command/connect/envoy/bootstrap_config.go#L24

consumed by generateConfig():
https://github.com/hashicorp/consul/blob/v1.12.0/command/connect/envoy/envoy.go#L496

called by run():
https://github.com/hashicorp/consul/blob/v1.12.0/command/connect/envoy/envoy.go#L381

(just thinking out loud as I poke around)

[quinndiggitypolymath]

There's the template that'd likely need to be adjusted: https://github.com/hashicorp/consul/blob/v1.12.0/command/connect/envoy/bootstrap_tpl.go#L129

[quinndiggitypolymath]

Seems envoy_grpc:
https://github.com/hashicorp/consul/blob/v1.12.0/command/connect/envoy/bootstrap_tpl.go#L238

    "ads_config": {
      "api_type": "DELTA_GRPC",
      "transport_api_version": "V3",
      "grpc_services": {
        "initial_metadata": [
          {
            "key": "x-consul-token",
            "value": "{{ .Token }}"
          }
        ],
        "envoy_grpc": {
          "cluster_name": "{{ .LocalAgentClusterName }}"
        }
      }
    }

would need to leverage google_grpc instead:
https://github.com/envoyproxy/envoy/blob/3170f64f8c83682d148469cd078fe3cec2717ca0/test/server/server_corpus/clusterfuzz-testcase-minimized-config_fuzz_test-5762646786179072#L272

  ads_config {
    api_type: GRPC
    transport_api_version: V3
    grpc_services {
      google_grpc {
        target_uri: "*"
        stat_prefix: "                                        "
      }
    }
  }

[quinndiggitypolymath]

So, potentially:

    "ads_config": {
      "api_type": "DELTA_GRPC",
      "transport_api_version": "V3",
      "grpc_services": {
        "initial_metadata": [
          {
            "key": "x-consul-token",
            "value": "{{ .Token }}"
          }
        ],
        "google_grpc": {
          "cluster_name": "{{ .LocalAgentClusterName }}",
          "channel_credentials": {
            "ssl_credentials": {
              "root_certs": "{...}",
              "private_key": "{...}",
              "cert_chain": "{...}"
            }
          }
        }
      }
    }

plus the templating logic

[quinndiggitypolymath]

Ah, rather:
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/grpc_service.proto#envoy-v3-api-msg-config-core-v3-grpcservice-envoygrpc

cluster_name

    (string, REQUIRED) The name of the upstream gRPC cluster. SSL credentials will be supplied in the Cluster transport_socket.

https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto#envoy-v3-api-field-config-cluster-v3-cluster-transport-socket

transport_socket

    (config.core.v3.TransportSocket) Optional custom transport socket implementation to use for upstream connections. To setup TLS, set a transport socket with name envoy.transport_sockets.tls and UpstreamTlsContexts in the typed_config. If no transport socket configuration is specified, new connections will be set up with plaintext.

https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/tls/v3/tls.proto#envoy-v3-api-msg-extensions-transport-sockets-tls-v3-upstreamtlscontext

[quinndiggitypolymath]

https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/tls/v3/tls.proto#envoy-v3-api-msg-extensions-transport-sockets-tls-v3-upstreamtlscontext

common_tls_context

    (extensions.transport_sockets.tls.v3.CommonTlsContext) Common TLS context settings.

https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/tls/v3/tls.proto#envoy-v3-api-msg-extensions-transport-sockets-tls-v3-commontlscontext

tls_certificates

    (repeated extensions.transport_sockets.tls.v3.TlsCertificate)

https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/tls/v3/common.proto#envoy-v3-api-msg-extensions-transport-sockets-tls-v3-tlscertificate

{
  "certificate_chain": "{...}",
  "private_key": "{...}"
}

[quinndiggitypolymath]

So, potentially:
https://github.com/hashicorp/consul/blob/v1.12.0/command/connect/envoy/bootstrap_tpl.go#L159

        {{- if .AgentTLS -}}
        "transport_socket": {
          "name": "tls",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
            "common_tls_context": {
              "validation_context": {
                "trusted_ca": {
                  "inline_string": "{{ .AgentCAPEM }}"
                }
              },
              "tls_certificates": [
                {
                  "certificate_chain": "{...}",
                  "private_key": "{...}"
                }
              ]
            }
          }
        },
        {{- end }}

plus the templating logic

[quinndiggitypolymath]

BootstrapTplArgs https://github.com/hashicorp/consul/blob/v1.12.0/command/connect/envoy/bootstrap_tpl.go#L5 would need to be extended with the credentials, but I'll give this a try with a static keypair in the interim

[quinndiggitypolymath]

yeah, with:

consul connect envoy \
                -admin-bind=localhost:19000 \
                -grpc-addr=https://127.0.0.1:8502 \
                -ca-file=/etc/consul.d/tls/tls.ca.d/ca.chain.pem \
                -ca-path=/etc/consul.d/tls/tls.ca.d \
                -client-cert=/etc/consul.d/tls/tls.crt.pem \
                -client-key=/etc/consul.d/tls/tls.key.pem \
                -http-addr=https://127.0.0.1:8501 \
                -tls-server-name=localhost \
                -token=... \
                -envoy-version=1.20.2 \
                -sidecar-for ... \
                -bootstrap

altering the config to include:

"tls_certificates": [
  {
    "certificate_chain": {
      "inline_string": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n"
    },
    "private_key": {
      "inline_string": "-----BEGIN EC PRIVATE KEY-----\n...\n-----END EC PRIVATE KEY-----\n"
    }
  }
]

and starting with:

 envoy --disable-hot-restart --config-path ...

envoy is starting up without complaints

so that's half the issue sorted

[quinndiggitypolymath]

not a great discovery that gRPC hasn't been validating incoming connections' tls credentials to-date 😑 at least there is "key": "x-consul-token",

[quinndiggitypolymath]

just until the fix is ready (@boxofrad, pleeeease don't make the default both insecure + unconfigurable; let's fix this vulnerability, not suppress it), a quick means of altering/fixing the envoy bootstrap config is:

consul connect envoy ... -bootstrap | jq \
  --arg certificate_chain "$(awk 'NF {sub(/\r/, ""); printf "%s\n",$0;}' tls.crt.pem)" \
  --arg private_key "$(awk 'NF {sub(/\r/, ""); printf "%s\n",$0;}' tls.key.pem)" \
'
.static_resources.clusters[] | select(.name=="local_agent").transport_socket.typed_config.common_tls_context.tls_certificates = [{
  certificate_chain: {
    inline_string: ($certificate_chain + "\n")
  },
  private_key: {
    inline_string: ($private_key + "\n")
  }
}]
' \
> bootstrap.json
 envoy --disable-hot-restart --config-path bootstrap.json

where tls.crt.pem + tls.key.pem are allocated like so: #13088 (comment)

a little more involved on the nomad side, as it requires a prestart task to generate the bootstrap.json for the sidecar task; as well, the reloading of the credentials on reissuance hasn't been verified/etc - the hot restart variation should take care of that

ultimately this needs the BootstrapConfig https://github.com/hashicorp/consul/blob/v1.12.0/command/connect/envoy/bootstrap_config.go#L24 and relevant codepaths that consume it to be altered; I will do this in my spare time, but it would be massively appreciated if the hashi team could consider taking it on, as I can't guarantee the PR will be quick (as tests need to be altered as well, etc)

lastly, rather than inline_string in the envoy json, this could be any of:

• filename
• inline_string
• environment_variable
  https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/base.proto#config-core-v3-datasource

[quinndiggitypolymath]

edit: meant BootstrapTplArgs and relevant codepaths that consume it + bootstrapTemplate to be altered

https://github.com/hashicorp/consul/blob/v1.12.0/command/connect/envoy/bootstrap_tpl.go#L5
https://github.com/hashicorp/consul/blob/v1.12.0/command/connect/envoy/bootstrap_tpl.go#L159

[quinndiggitypolymath]

and jq would not be necessary in the short term fix, if envoy ... --config-yaml "node: {id: 'node1'}" is capable of more complex alterations/overrides (specifically, altering the first .static_resources.clusters[] array entry), but I'm not aware if that is possible
https://www.envoyproxy.io/docs/envoy/latest/operations/cli#cmdoption-config-yaml

[boxofrad]

Thanks for taking the time to research this so thoroughly, @quinndiggitypolymath!

The approach you mentioned is actually very similar to the way in which we configure mutual-TLS on connections between sidecar proxies in the service mesh.

As for Envoy's xDS (gRPC) connections to the Consul agent, these are currently authenticated with an ACL token via the x-consul-token metadata field you pointed out.

Though it would be possible to have Envoy present a client certificate and for the Consul agent to verify it (your code snippets are on the right tracks) it wouldn't necessarily improve your security posture and would require quite a bit of PKI maintenance overhead.

To expand on this: it wouldn't be advisable to give your Envoy sidecars access to the agent's private certificate/key (as this is trusted by Consul servers to secure internal RPC traffic) so you'd need to generate and distribute new client certificates for Envoy, signed by a separate CA.

In any case, we'd still rely on ACL tokens for authentication/authorization, requiring client certificates would just provide an additional layer of security by preventing random clients from connecting - which isn't a huge concern as, by default, Consul agents will only bind the gRPC port to the loopback interface.

[quinndiggitypolymath]

@boxofrad, would you be opposed to keeping this issue to track future work on implementing verify_outgoing for gRPC connections initiated by envoy? At the moment, based on your confirmation that the only authentication measure is x-consul-token, verify_incoming on the gRPC listener only serves to break Consul Connect envoy sidecar functionality - I believe there could be a use-case in which verify_incoming would be useful, particularly when the gRPC listener is not bound to localhost.

[quinndiggitypolymath]

To expand on this: it wouldn't be advisable to give your Envoy sidecars access to the agent's private certificate/key (as this is trusted by Consul servers to secure internal RPC traffic) so you'd need to generate and distribute new client certificates for Envoy, signed by a separate CA.

Regarding this, envoy doesn't drop permissions as a child process of consul, so it already has access to the agent's private certificate/key

[quinndiggitypolymath]

As well, since verify_incoming prevents a significant number of codepaths from being reachable by an attacker with local access, there is benefit from being able to stop incoming connections without relying solely on x-consul-token

[quinndiggitypolymath]

Defense in depth, etc, etc

[fred-gb]

Hi 👋🏻

Help @quinndiggitypolymath ! 😅

I think, i'm in similar situation: #16617

I tried to create a separated CA and cert, but I have always an error.

What is the solution you find?

Thanks