Add support for transparent connect proxies

[apollo13]

Hi, I know that https://www.consul.io/docs/connect/transparent-proxy is still beta, but it would be sooooo great if Nomad could support that in the next releases :)

[tgross]

Hi @apollo13! Much of what transparent proxy provides is for filling gaps in k8s (because it's a bit more "kit of parts" than combining Nomad and Consul). There's a conflict here in the security model between the two: Consul transparent proxy assumes and requires 1 service per network namespace, whereas Nomad supports n services per network namespace. We already provide most of what transparent proxies support, with the major exception of preventing service mesh circumvention.

So we're still brainstorming what exactly transparent proxy support would look like in Nomad. It's likely to be implemented a set of features that together make up all the same things as the Consul tproxy, rather than a standalone "support Consul tproxy" feature. Some ideas we've considered, not all of which are compatible with each other:

• Implement a network.mode = "transparent" that updates the routing table in the allocation's network NS to prevent circumvention.
• Expose Consul DNS to allocations.
• Create a separate net NS for each service. This could complicate a lot of use cases for task groups with multiple tasks that communicate over localhost rather than Unix Domain Sockets.

In any case, just a heads up that this isn't on our very near-term roadmap.

[apollo13]

Ok, my main goal is to get rid of the manual declaration of all the connect services that I need. If there are ideas on how to do that without manually specifying all, then it doesn't have to be the transparent proxy. Thank you for the clarifications!

[shoenig]

@apollo13 if you're talking about no longer needing to define upstreams, then transparent proxy only helps in that regard in that upstreams are inherited from intentions, so that you only need to declare them once. (Unless intentions are disabled, in which case the default behavior is a free-for-all). Making services connect-native is another way to eliminate the need for upstreams, but of course that only helps if you own the code. (And you'd still need to declare intentions)

[apollo13]

Declaring them only once would imo already be a win. connect-native is certainly one way of doing things but often not easily possible for existing code (For example changing a database connection in Django to be connect native is probably close to not possible). So in that sense the proxy would still be a massive improvement (at least for me). I usually do not care about inbound connections that much (they are solved differently). But it would be really great if I could limit outgoing connections of a group to whatever the intentions allow (and not more).

…

On Thu, May 20, 2021, at 20:53, Seth Hoenig wrote: @apollo13 <https://github.com/apollo13> if you're talking about no longer needing to define `upstreams`, then transparent proxy only helps in that regard in that upstreams are inherited from `intentions`, so that you only need to declare them once. (Unless intentions are disabled, in which case the default behavior is a free-for-all). Making services connect-native <https://www.consul.io/docs/connect/native#connect-native-app-integration> is another way to eliminate the need for upstreams, but of course that only helps if you own the code. (And you'd still need to declare intentions) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10628 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAT5CZWSPUIV3OF5BR3YIDTOVLDZANCNFSM45GVLYYA>.

[komapa]

We would like to see this implemented for Nomad as well. Mainly echoing @apollo13's comment from above, 100%.

[tgross]

I'll be picking this work up after the new year. The rough plan of action is going to be:

1. Implement CNI_ARGS support in Nomad, which we'll need for better-supporting third party service meshes anyways.
2. Put together a hacky first pass at figuring out what updates we'd need to add to consul-cni (probably allowing the cni-proxy-config annotation to be passed in as a CNI arg and that causing it to bypass the bits that make k8s API calls.)
3. Use that hacked CNI plugin to prototype out some ideas of what configuration we want to expose to job authors.
4. Work up a design doc to share with our colleagues over on Consul to get their buy in on our approach.
5. Finalize that implementation.

[apollo13]

That sounds great!

Work up a design doc to share with our colleagues over on Consul to get their buy in on our approach.

Assuming this is not ending up as an enterprise only feature, do you mind sharing the design doc publicly (once you have written it ;))? Maybe the community can spot some errors etc in there?

[tgross]

Assuming this is not ending up as an enterprise only feature, do you mind sharing the design doc publicly (once you have written it ;))? Maybe the community can spot some errors etc in there?

This is planned for Nomad CE. We don't make a practice of sharing the design docs directly because they usually have a bunch of private background info in there like specific customer requests or internal business requirements. But I'd love to extract what I can and share it ahead of implementation. That'd be a great habit for us to get into!

[tgross]

As mentioned above, I wanted to surface our internal design doc ("RFC") here. Note that this is an excerpt with internal discussions removed. Please note this is a design document that we use to build rough consensus within the team and with our sibling teams (like the Consul folks). It may not 100% match up with the final implementation as I progress through it. No promises here! So if you're reading this in the future after the feature has shipped, please instead refer to the documentation! 😀

---

Background

On Linux, both Kubernetes and Nomad typically run workloads in network namespaces. Both orchestrators use Container Network Interface (CNI) plugins to configure the network namespace by creating network bridges, iptables rules, etc. In Kubernetes, containers within a pod share the same network namespace. In Nomad, tasks within an allocation share the same network namespace. This arrangement is what allows Consul Connect to run an Envoy proxy in one task while the user's application runs in another task in the same allocation.

Consul Connect provides for secure communication between workloads in the service mesh. However, it does not by default enforce that workloads only communicate over the mesh. Transparent proxy mode directs all inbound and outbound traffic for a workload through the Envoy sidecar via iptables. This forces workloads to use only the service mesh, with the option to configure exceptions for specific CIDRs, ports, and non-mesh destinations (or to enforce mesh destinations only).

The primary benefit of transparent proxy is the ability for applications to use Consul DNS URLs to access upstreams, e.g. http://foo.virtual.consul without having to manually configure upstreams and local listener ports, ex. http://localhost:1234.

The diagram below shows a typical Connect allocation with two tasks. The group has a network.port mapping configured for port 8000 only.

Nomad clients run a series of "allocrunner hooks" for the allocation as a whole before running a task runner for each task within the allocation, and the task runner has its own set of per-task "taskrunner hooks". Since Nomad 0.10, network configuration is setup in one of the allocrunner hooks. Nomad network blocks can have one of 4 modes: none, host, bridge, and cni. Only bridge networking is supported for Connect workloads, even though the cni mode is configured in mostly the same way as bridge networking (so in theory it could support Connect but users would have to provide the correct CNI config themselves).

Nomad clients have a hard-coded CNI configuration template that includes calls to the loopback, bridge, firewall, and portmap plugins, for use in bridge networking mode.

When an allocation with bridge networking starts, the network_hook executes the following steps:

• Call a platform-specific network manager's CreateNetwork method. On Linux this creates the network namespace and returns the path to the file that represents that namespace.
• Call a platform-specific network configurator's Setup method. On Linux this:
  • Idempotently creates required iptables chains.
  • Renders the hard-coded CNI configuration template with the subnet and bridge name. (With mode="cni/*", the network configurator uses the named CNI configuration from disk.)
  • Creates an argument for the CNI portmap plugin to configure the port mappings specified in the allocation's group.network block.
  • Invokes CNI plugins with the rendered CNI plugin configs against the network namespace.

On success, the resulting structs.AllocNetworkStatus is optionally used by task drivers to set labels on the container, etc.

Note that Connect is not involved in this existing networking workflow at any point. When the job is submitted, the server automatically adds an Envoy proxy sidecar task to each group that requires one for Connect. The taskrunner for that task invokes Consul's envoy bootstrap command to configure the sidecar, and the Envoy task is in the same allocation as the user's application so they share the same network namespace. To support tproxy, Nomad networking configuration will need to be aware of this Connect-specific configuration.

Under Kubernetes, Consul CNI plugin receives its configuration from the k8s API. The consul-k8s control plane annotates pods with the expected iptables configuration. When the CNI plugin is run, it makes requests to the k8s API to determine how to configure iptables rules for the pod, and to update the k8s control plane with status of that work. The configuration is a JSON-encoded blob that deserializes into the Consul SDK’s iptables.Config struct.

Proposal

Nomad will support Connect transparent proxy mode by invoking Consul CNI during network setup when requested by the user. This will require changes to both Nomad and the Consul CNI plugin.

Updates to Nomad Job Spec

The Nomad job’s service.connect.proxy block will be updated to add a new transparent_proxy block. As much as possible, Nomad will configure a sensible iptables configuration for tproxy without additional job spec changes. But the tproxy block will also allow further customization.

A minimum transparent_proxy block configuration will be as follows. This configuration will direct all incoming and output communication through the Envoy sidecar.

proxy {
  transparent_proxy {}
}

Applications may need to expose additional ports for health checks or other external traffic. A transparent_proxy.exclude_inbound_ports field will allow for a user-supplied list of ports. Nomad will automatically add the ports for the existing expose.path block, and the port of any health check where check.expose=true (see Health Checks below). The exclude_inbound_ports field will not configure the Envoy proxy itself, only the iptables rules used for exclusion. A complete transparent_proxy configuration might look like the following (only the transparent_proxy section is new). This example configuration allows inbound access to the ports labeled http and metrics w/o going through the Envoy proxy.

proxy {
  transparent_proxy {
    uid                    = 101   # default, see iptables.Config
    outbound_port          = 15001 # default
    exclude_inbound_ports  = []    # default, can be set with a name
                                   # that matches a network.port
                                   # label or a port number.
    exclude_outbound_ports = []    # default
    exclude_outbound_cidrs = []    # default
    exclude_uids           = []    # default
  }

  expose {

    path {
      path            = "/metrics"
      protocol        = "http"
      local_path_port = 9001

      # Any expose.path.listener_port will be automatically
      # added to the exclude_inbound_ports set.
      listener_port   = "metrics"
    }
  }

  # Note that when using tproxy, upstreams blocks are no longer
  # required. But a user might want to have both while migrating
  # their services to use tproxy. Nomad does not automatically create 
  # Consul intentions from the upstream blocks.
  upstreams {
    destination_name = "count-api"
    local_bind_port  = 8080
  }

}

Any task group that includes a transparent_proxy block for any service will have an implicit constraint added during job registration that the node has the fingerprinted attribute plugins.cni.version.consul-cni, which indicates that the node has the Consul CNI plugin available.

Updates to Consul CNI

The Consul CNI plugin currently accepts K8S_POD_NAMESPACE and K8S_POD_INFRA_CONTAINER_ID arguments from the CNI command line arguments. It uses these values to make API calls to k8s to determine which CNI configuration to read from disk, and to update k8s with status updates of the process. This workflow poses two problems in Nomad:

• First, Nomad doesn't have a facility to provide API access to CNI plugins. Although Workload Identity tokens are available at the time we run the networking hook, we'd need to sign a token with a policy specific to this use case. Additionally, the Task API socket is created by the task runner which is specific to a single task (not the whole allocation), and happens much later. This would mean the CNI plugin would need to communicate with the Nomad HTTP server, which is likely protected with mTLS.
• Second, Nomad doesn't currently have an API that the CNI plugin could use to derive the iptables configuration. In k8s, the iptables configuration is added as a JSON blob to a pod annotation.

Nomad has no need for "pod annotations" from the CNI plugin, so the workflow where the CNI plugin updates its status is unnecessary.

This allows for a much simpler implementation for Nomad and the Consul CNI plugin. The Nomad client will provide the JSON-encoded iptables.Config as a CNI command-line argument. The Consul CNI plugin will need to be updated as follows:

• Look for a IPTABLES_CONFIG argument in the CNI_ARGS. If present:
  • Decode the iptables configuration from that value
  • Skip over the remaining k8s API calls
  • Setup iptables in the network namespace as usual

Updates to Nomad bridge networking and CNI configuration

The network_hook will now have two hard-coded CNI configuration templates; the new configuration will be identical to the original but with the addition of consul-cni to the list of plugins. When an allocation with bridge networking starts on Linux, the network_hook will execute the following steps:

• Call the platform-specific network manager's CreateNetwork method to create the network namespace, just as before.
• Call the platform-specific network configurator's Setup method. This will:
  • Idempotently create the required iptables chains.
  • Determine if any service.connect.proxy has tproxy enabled, to select which of the CNI configuration templates to use. Render that template.
  • Create an argument for the CNI portmap plugin to configure the port mappings specified in the allocation's group.network block, just as before.
  • Create a IPTABLES_CONFIG argument for the Consul CNI plugin (see iptables.Config below).
  • Invoke CNI plugins with the rendered CNI plugins against the network namespace.

iptables.Config

The IPTABLES_CONFIG argument passed to the Consul CNI plugin will be a JSON-encoded version of the Consul SDK's iptables.Config struct. The fields of this will be set as follows:

• ConsulDNSIP: Set by Nomad from the fingerprinted Consul agent. See the DNS section below.
• ConsulDNSPort: Set by Nomad from the fingerprinted Consul DNS port.
• ProxyUserID: defaults to 101, which is the UID that the Envoy container image runs as (in the Envoy container’s user namespace, if any). Can be set in the job spec by tproxy.uid
• ProxyInboundPort: set to the dynamic port set by the Nomad client.
• ProxyOutboundPort: defaults to 15001, the hard-coded value copied from k8s connect-inject, which matches what the Envoy bootstrap command sets. Can be set in the job spec by the tproxy.outbound_port field.
• ExcludeInboundPorts: derived from combining the proxy.exclude_inbound_ports field and the proxy.path block as described above.
• ExcludeOutboundPorts: defaults to empty, user-configurable via tproxy.exclude_outbound_ports block
• ExcludeOutboundCIDRs: defaults to empty, user-configurable via tproxy.exclude_outbound_cidrs block
• ExcludeUIDs: defaults to empty, user-configurable via tproxy.exclude_uids block
• NetNS: set from the drivers.NetworkIsolationSpec.Path field, which is the network namespace path that'll be used for the CNI configuration; Nomad will also set the CNI_NETNS environment variable.

DNS

As noted above, tasks using transparent proxy should be able to use Consul DNS URLs to access upstreams. In the http://foo.virtual.consul example, Consul will resolve this hostname to the Consul-issued virtual IP of the foo service. When the request gets made using that IP it is redirected via IPtables config to the Envoy proxy. The Envoy proxy matches on the request IP to direct the request to the foo service.

In order for the application container to resolve foo.virtual.consul the DNS lookup must be made to the local Consul client (or Consul server but that’s not applicable for Nomad). The local Consul client is listening on port 8600 and is accessible to the application container through the host IP (assuming the Consul agent’s bind_addr is set to a host IP and not localhost). Nomad can write a new nameserver to /etc/resolv.conf but resolv.conf does not support ports. Therefore we must use a combination of iptables rules and resolv.conf rules to get the DNS lookup directed to the local Consul client.

1. Nomad will fingerprint the address that the Consul agent has bound on, and write an additional nameserver with that address to the top of /etc/resolv.conf. This will ensure DNS lookups are first made to <Consul agent IP>:53. We must add an additional nameserver so that if the DNS lookup fails (e.g. it wasn’t to a Consul service), the next nameserver in the list will be used. Consul does support doing its own recursive lookup using the recursors config but that would require users to configure which is not wanted.

2. Consul’s iptables sdk will handle writing the iptables rule to redirect udp and tcp requests to <Consul agent IP>:53 to the configured ConsulDNSIP and ConsulDNSPort. Which will be the address of the node-local Consul client from within the application container and Consul’s DNS port respectively. The DNS port will be determined via fingerprinting as it is exposed in the /agent/self endpoint.

Note: if users enable DialedDirectly they will also be able to use regular Consul DNS hostnames, e.g. foo.service.consul., but mesh functionality is limited when using these URLs.

Health Checks

Consul health checks of type grpc, tcp, and http require that the service being checked can be reached from the Consul agent running on Nomad and that it can be reached without going through the Envoy proxy, because the Consul agent doesn’t have the required TLS. 

Nomad has an existing check.expose field that will automatically generate an expose.path block for http and grpc check types at the time of job submission. We’ll extend this to allow for check.expose to be set on tcp checks, but only when a transparent_proxy block is in use. For these task groups, we’ll generate a proxy.exclude_inbound_ports that includes the check port in the same job mutating hook.

[tgross]

my rough implementation checklist:

• fingerprint Consul DNS fingerprint: add DNS address and port to Consul fingerprint #19969
• update my hacking branch with the block definition described in the RFC and make sure that works end-to-end
• transparent_proxy block for jobspec: transparent proxy: add jobspec support #20144
• CNI_ARGS generation and adding to CNI networking config
• consul-cni PR (concurrent with the above) to accept CNI_ARGS
• docs
• E2E testing

[apollo13]

Hi tgross, this all sounds very exciting. A few small questions/remarks:

Only bridge networking is supported for Connect workloads, even though the cni mode is configured in mostly the same way as bridge networking (so in theory it could support Connect but users would have to provide the correct CNI config themselves).

Probably the only thing limiting theory from becoming reality here is that nomad hardcodes and checks for bridge in a few places?

Nomad will fingerprint the address that the Consul agent has bound on, and write an additional nameserver with that address to the top of /etc/resolv.conf.

It would be great if it would be possible to disable this behavior (or at least also have it for normal allocations as well?). As it currently stands I am already pushing a DNS server that is aware of the .consul domain into my allocations so they can use consul service lookups.

Can you expand a little bit on how the transparent proxy works (I guess I haven't understood it fully yet) and especially what the virtual addresses do? Am I correct that it works somewhat like this:

• All application outbound traffic is redirected to the tproxy process (probably by checking the originating uid in the namespace?)
• As far as I understood it all the traffic is routed to a single port in the tproxy (namely OutboundListenerPort?)
• How does the tproxy now determine which IP:PORT was requested?

Assuming I am using virtual services where I guess each service gets it's own IP: Can my application now access foo.virtual.consul:any_port and it ends up at the correct service (the virtual IP docs in consul seem rather sparse)?

What would DialDirectly change? One would use the actual IPs from foo.service.consul (or rather foo.connect.consul or is that resolved implicitly?) and have the tproxy connect to that exact instance?

Sorry if the questions about tproxy itself are kinda out of scope, but I am trying hard to understand what is happening here :)

[tgross]

It would be great if it would be possible to disable this behavior (or at least also have it for normal allocations as well?). As it currently stands I am already pushing a DNS server that is aware of the .consul domain into my allocations so they can use consul service lookups.

Good call. We discussed that a bit but didn't have a reasonable use case in mind, so having your example is valuable.

Can you expand a little bit on how the transparent proxy works (I guess I haven't understood it fully yet)
...
All application outbound traffic is redirected to the tproxy process (probably by checking the originating uid in the namespace?)

Right. From a high-level view, it's adding some iptables rules to the existing Connect implementation. Those rules are applied internal to the network namespace, so that outbound traffic from the task flows through the Envoy proxy we've configured for Connect, instead of just wherever the task wants.

How does the tproxy now determine which IP:PORT was requested?

The virtual IP address in this case is just Envoy load balancing between the real IP addresses that Nomad is advertising for the service to Consul. The DialDirectly would be for "I want to reach that allocation", but I'm not sure I've worked out how Consul exposes that via DNS.

[apollo13]

The virtual IP address in this case is just Envoy load balancing between the real IP addresses that Nomad is advertising for the service to Consul.

Understood, the missing link for me is if I request foo.virtual.consul and bar.virtual.consul the iptables rules redirect both requests to 127.0.0.1:tproxy_outbound_port (so far so good). Now how does the tproxy reconstruct the original target IP so it can figure out which service was requested?

[apollo13]

Okay, learned something. There is SO_ORIGINAL_DST: https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/listener_filters/original_dst_filter

EDIT:// And even more interesting with the TPROXY target https://blog.cloudflare.com/how-we-built-spectrum

[blake]

The virtual IP address in this case is just Envoy load balancing between the real IP addresses that Nomad is advertising for the service to Consul. The DialDirectly would be for "I want to reach that allocation", but I'm not sure I've worked out how Consul exposes that via DNS.

When transparent proxy is enabled for a service, Consul allocates a virtual IP (VIP) for that service from the 240.0.0.0/4 address range.

The outbound listener on the downstream service's Envoy proxy has a series of filter chains that match destination VIP addresses to the corresponding Envoy cluster for the target upstream service. For example,

{
  "filter_chains": [
    {
      "filter_chain_match": {
        "prefix_ranges": [
          {
            # Consul assigned virtual IP for the service `fake-service`
            "address_prefix": "240.0.0.1",
            "prefix_len": 32
          }
        ]
      },
      "filters": [
        {
          "name": "envoy.filters.network.tcp_proxy",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
            "stat_prefix": "upstream.fake-service.default.default.dc2",
            "cluster": "fake-service.default.dc2.internal.fabb7415-0d8e-5230-5455-d31c35d0ddd9.consul"
          }
        }
      ]
    }
  ]
}

The cluster contains a list of each endpoint / service instance for the logical service and their actual addresses in the cluster.

{
  "dynamic_endpoint_configs": [
    {
      "endpoint_config": {
        "@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
        "cluster_name": "fake-service.default.dc2.internal.fabb7415-0d8e-5230-5455-d31c35d0ddd9.consul",
        "endpoints": [
          {
            "locality": {},
            "lb_endpoints": [
              {
                "endpoint": {
                  "address": {
                    "socket_address": {
                       # IP address of the upstream service instance
                      "address": "10.42.1.78",
                      "port_value": 20000
                    }
                  },
                  "health_check_config": {}
                },
                ...snip...

In this configuration, downstream applications need to use the <name>.virtual.consul hostname when addressing upstreams so that Envoy to correctly route the connection over the mesh to the upstream service. Traffic is load balanced across all healthy endpoints/instances in the upstream cluster.

When TransparentProxy.DialedDirectly is enabled on an upstream service, the filter chain on the downstream proxy is configured to match directly on the IP addresses of the upstream service instances instead of the virtual IP.

{
  "filter_chains": [
    {
      "filter_chain_match": {
        "prefix_ranges": [
          {
            # upstream instance 1
            "address_prefix": "10.42.1.78",
            "prefix_len": 32
          },
          {
            # upstream instance 2
            "address_prefix": "10.42.1.79",
            "prefix_len": 32
          }
        ]
      },
      "filters": [
        {
          "name": "envoy.filters.network.tcp_proxy",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
            "stat_prefix": "upstream.fake-service.default.default.dc2",
            "cluster": "passthrough~fake-service.default.dc2.internal.fabb7415-0d8e-5230-5455-d31c35d0ddd9.consul"
          }
        }
      ]
      ...snip...

Instead of load balancing traffic across available upstream instances, Envoy is configured to send connections to a passthrough / original destination cluster that forwards the connection directly to the original destination IP and port.

In this configuration, downstream applications need to use the <name>.service.consul hostname when addressing the upstream.

[tgross]

Thanks for the assist @blake! 😀

[tgross]

This has been merged to main and will ship in Nomad 1.8.0. I'm working on Tutorial updates now in the private repository for those.

[apollo13]

Nice work, and I thought 1.8 would be a stabilization release 😜

…

On Wed, Apr 10, 2024, at 17:04, Tim Gross wrote: This has been merged to `main` and will ship in Nomad 1.8.0. I'm working on Tutorial updates now in the private repository for those. — Reply to this email directly, view it on GitHub <#10628 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAT5CYOTJ2WSZGMBYQ75LLY4VIGRAVCNFSM45GVLYYKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBUG44DAMZRGQZQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[suikast42]

Awesome

[koder29406]

This has been merged to main and will ship in Nomad 1.8.0. I'm working on Tutorial updates now in the private repository for those.

Hello, are there any plans for detailed examples on how to use the transparent proxy in a Nomad job?
A full example would be good, in addition to the minimal one here: https://developer.hashicorp.com/nomad/docs/job-specification/transparent_proxy#minimal-example

[tgross]

@koder29406 the Service Mesh tutorial https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh has been updated to use transparent proxy now. (Small warning that there are a minor issues int the setup in that Tutorial which I've got a PR up to fix... that should land later today.)

[rahadiangg]

Hi @tgross, I use nomad 1.8.0 and try to use transparent_proxy{}. Based on nomad documentation that you provide, nomad required consul-cni 1.5.4 or above, but currently the latest consul-cni is 1.4.3.

I'm also try to validate my job that required consul-cni grather than 1.4.2. So, I think the documentation needs to be updated?

[tgross]

Hi @rahadiangg! By documentation you're talking about here: https://developer.hashicorp.com/nomad/tutorials/integrate-consul/consul-service-mesh#verify-nomad-client-consul-configuration right? That's definitely just a typo, which I'll fix. But is there another place in the docs I'm missing?