Consul Connect service port definition confusion

[Oloremo]

Hi,

So I just noticed that if I define a port in network stanza and when re-use it in service as a label - it doesn't work with Consul Connect integration.

By "doesn't work" I mean that when I set that service as an upstream to other Nomad task it creates an Envoy endpoint but no traffic could pass through it. It instantly closes the connection.

Example of non-working configuration:

    network {
      mode = "bridge"
      port "main" {
        to = 10000
      }
    }

    service {
      name = "lb-server"
      port = "main"
      connect {
        sidecar_service {}
      }
    }

Example of working configuration:

    network {
      mode = "bridge"
      #port "main" {
      #  to = 10000
      #}
    }

    service {
      name = "lb-server"
      port = "10000"
      connect {
        sidecar_service {}
      }
    }

Is it something expected that I missed in docs?

Nomad version: 1.0.2
Consul version: 1.9.2

[tgross]

The working example you have here matches the docs for the sidecar_service example. IIRC the issue you're running into with trying to configure the port at the network block is that's the configuration that's used for the main task and not being passed to the sidecar.

[Oloremo]

IIRC the issue you're running into with trying to configure the port at the network block is that's the configuration that's used for the main task and not being passed to the sidecar.

Is that expected behavior?

[tgross]

Yes. Another way to look at it is that the default sidecar_task has no port mapping configuration, so having a port mapping in the network block won't route traffic anywhere the sidecar task is listening.

[Oloremo]

For me it's confusing. Could be a doc issue?

For example yes, if we look at sidecar stanza doc we see a clear example with the port as an int. But that page doesn't tell us anything about that port has to be an int, not a label.

If we go to the service stanza doc we see the service example with the port as a label.

So my question is - how should the end-user understand that I have to use int and not a label for Consul Connect cases?

[tgross]

Yeah the docs definitely could use some improvement here, especially because in the case of Consul Connect we're always in address_mode = "alloc" but that mode doesn't restrict the port specification:

alloc - Advertise the mapped to value of the labeled port and the allocation address. If a to value is not set, the port falls back to using the allocated host port. The port field may be a numeric port or a port label specified in the same group's network stanza.

Going to mark this as a docs issue to fix.

[Oloremo]

Could it be also a validation issue? Like it would be great if plan\run would highlight it as an issue.

[tgross]

I see that @krishicks has a PR open in #9987 which should fix the docs, and there's a discussion there about how this could be better handled. (Assigning to @krishicks only so that we don't have something marked "in progress" that has no owner... this will close out once that PR is merged. We can keep #9907 open which has a bit more detail.)

[AndrewChubatiuk]

@tgross I'm using now named ports for sidecars to spin up sidecar on a certain interface. It works since Nomad v1.0.4 and was introduced by #9975

job "celery_exporter" {
  type        = "system"
  group "celery_exporter" {

    network {
      mode = "bridge"

      port "exporter" {
        to           = 8888
        host_network = "private"
      }

      port "sidecar" {
        host_network = "private"
      }

      port "stats" {
        host_network = "private"
      }
    }
    service {
      name = "celery-exporter"
      port = "exporter"

      meta {
        envoy_address = "${NOMAD_ADDR_stats}"
      }

      check {
        type     = "http"
        path     = "/metrics"
        interval = "5s"
        timeout  = "2s"

        check_restart {
          limit = 3
          grace = "30s"
          ignore_warnings = false
        }
      }

      connect {
        sidecar_service {
          port = "sidecar"
          proxy {
            expose {
              path {
                path            = "/metrics"
                protocol        = "http"
                local_path_port = 9102
                listener_port   = "stats"
              }
            }
            config {
              envoy_prometheus_bind_addr = "0.0.0.0:9102"
            }
            upstreams {
              destination_name = "redis"
              local_bind_port  = 6379
            }
          }
        }
      }
    }

    task "exporter" {
      driver = "docker"

      config {
        image = "ovalmoney/celery-exporter"
        force_pull = true
        args = [
          "--listen-address=0.0.0.0:8888"
        ]
      }

      env {
        CELERY_EXPORTER_BROKER_URL = "redis://${NOMAD_UPSTREAM_ADDR_redis}/0"
      }

      resources {
        cpu    = 50
        memory = 64
      }
    }
  }
}

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.