Nomad job with envoy escape hatch has no dynamic port interpolation available

[johnalotoski]

Nomad version

Nomad v1.2.9, v1.3.2

Issue

Envoy uses a default stream_idle_timeout of 5 minutes which is in conflict with a few long running APIs we are using. There are a few direct override knobs for envoy timeouts in Consul, but there is no direct knob for this stream_idle_timeout. Advanced Consul Escape hatches are available, and in this case, configuring an escape hatch override for envoy_public_listener_json appears that it would solve the problem, as we could add a route public listener idle_timeout which will override envoy's default stream_idle_timeout, or directly modify the stream_idle_timeout itself.

However, when Nomad sets up the consul job, the dynamic port the envoy listener will use ahead of time is not known, so some Nomad interpolation appears to be necessary to be able to declare this snippet properly. Taking the example of the Consul envoy_public_listener_json from the Consul link above, with slight modifications, I believe we'd need to be able to set the connect.sidecar_service.proxy.config.envoy_public_listener_json stanza in the Nomad job declaration to something like the following where the Nomad assigned port is interpolated and passed to the Consul escape hatch json override:

{
  "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
  "name": "public_listener:0.0.0.0:${NOMAD_PORT_connect_proxy_<SERVICE_NAME>}",
  "address": {
    "socket_address": {
      "address": "0.0.0.0",
      "port_value": ${NOMAD_PORT_connect_proxy_<SERVICE_NAME>}
    }
  },
  "filter_chains": [
    {
      "filters": [
        {
          "name": "envoy.filters.network.http_connection_manager",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
            "stat_prefix": "public_listener",
            "http_filters": [
              {
                "name": "envoy.filters.http.router"
              }
            ],
            "route_config": {
              "name": "public_listener",
              "virtual_hosts": [
                {
                  "name": "public_listener",
                  "domains": ["*"],
                  "routes": [
                    {
                      "match": {
                        "prefix": "/"
                      },
                      "route": {
                        "cluster": "local-app",
                        "idle_timeout": "10m"
                      }
                    }
                  ]
                }
              ]
            }
          }
        }
      ]
    }
  ],
 "traffic_direction": "INBOUND"
}

However, interpolation of the Nomad assigned dynamic port to the Consul connect service doesn't appear available to the passed json escape hatch override in the testing I've done, which seems to eliminate the possibility of using envoy escape hatches to override parameters there aren't already direct Consul overrides for.

I've tested the above approach by deploying the job with a random hardcoded port substituted in the parameterization above for the escape hatch which will be incorrect. Then, once the job is deployed, adjusting the above job definition for the escape hatch snippet and assigning the correct port that Nomad has utilized for the connect proxy and re-deploying/updating the job, at which point the escape hatch override does work as intended. So it appears the only remaining issue is the ability to interpolate a Nomad dynamic port and pass it to the escape hatch snippet.

Perhaps Nomad interpolation can done in this snippet and I'm not aware of it?

[lgfa29]

Hi @johnalotoski 👋

#14445 adds interpolation to configuration values, so I think what you described would then be possible.

I'm building a binary from that PR here: https://github.com/hashicorp/nomad/actions/runs/2974808759

Once that's done, would you be able to try and validate if this provides what you need?

Thanks!

[johnalotoski]

Hi @lgfa29, thanks very much for the fix! I took the diff from here and compiled it into our Nomad 1.3.2 version currently being used, and tested it works!

I did find that the notation I needed to use to get the variable substitution to work correctly was (modifying the example snippet from above slightly):

...
      "port_value": "$${NOMAD_PORT_connect_proxy_<SERVICE_NAME>}"
...

Where double $ was required and also string quotes which were then properly removed and substituted with a JSON integer once interpolation was done. Presumably the double $$ is for HCL escaping of the "$" char?

Thanks!

[lgfa29]

Nice! Thanks for test it. I will update the docs and get that PR merged so it's available in the next release 🙂

Where double $ was required and also string quotes which were then properly removed and substituted with a JSON integer once interpolation was done. Presumably the double $$ is for HCL escaping of the "$" char?

Ah yeah. The interpolation is done within a string right? ${...} would start an HCL2 expression, so it needs to be escaped:
https://www.nomadproject.io/docs/job-specification/hcl2/expressions#string-templates

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.