[FEATURE] Expose host port services in Cilium

[protochron]

Is your feature request related to a problem? Please describe.
Right now Netreap doesn't have a great way to support host services. The easy way to enable that is to simply add the portmap CNI plugin to the configuration chain, ex.

{
  "name": "cilium",
  "cniVersion": "1.0.0",
  "plugins": [
     {
       "type": "cilium-cni",
       "enable-debug": false
     },
     {
      "type": "portmap",
      "capabilities": {"portMappings": true}
     }
  ]
}

Describe the solution you'd like
Cilium has an API for managing services. Normally these are derived from Kubernetes services, but it looks like we should be able to create services with that API to map host ports in Nomad jobs that request them.

Describe alternatives you've considered

Additional context

[deverton]

https://github.com/cilium/cilium/blob/52c383566e7f22882c0e9c0f038cef110ad5c4cf/pkg/k8s/watchers/pod.go#L644 seems to be the related code within Cilium.

[deverton]

As a workaround for now I've got this hacky way of using a pre and post start task to add and remove service definitions. It requires that the Nomad client's are configured to have the Cilium socket directory declared as a host volume. You can define a task that needs host port mapping like the example below.

You will also need to manually manage the service IDs to ensure that they are unique as the Cilium API doesn't seem to provide a way to generate those.

With this configuration you no longer need the portmap plugin in the CNI config.

The way this works is that when Nomad creates group network via the Cilium CNI plugin, that call includes a unique ID for the allocation. We can then use that unique ID to pull the Cilium endpoint ID out and the assigned IPv4 address. From there the Cilium service command can be used to create the actual port mapping.

job "fabio" {
  type      = "system"

  group "fabio" {

    network {
      mode = "cni/cilium"
      port "lb" {
        static = 8080
      }
      port "ui" {
        static = 8081
      }
    }

    volume "cilium" {
      type      = "host"
      read_only = false
      source    = "cilium"
    }

    task "create-service" {
      driver = "exec"

      config {
        command = "/bin/sh"
        args = [
          "$${NOMAD_TASK_DIR}/create-service.sh"
        ]
      }

      lifecycle {
        hook    = "poststart"
        sidecar = false
      }

      template {
        destination = "local/create-service.sh"
        data        = <<-EOH
          #!/bin/sh

          ENDPOINT_ID=$(cilium endpoint list -o jsonpath='{$[?(@.status.external-identifiers.container-id=="{{ env "NOMAD_ALLOC_ID" }}")].id}')
          BACKEND_IP=$(cilium endpoint get "$ENDPOINT_ID" -o jsonpath='{$[0].status.networking.addressing[0].ipv4}')

          cilium service update \
            --k8s-host-port \
            --id 1 \
            --backends "$BACKEND_IP:{{ env "NOMAD_ALLOC_PORT_lb" }}" \
            --frontend "{{ env "NOMAD_HOST_ADDR_lb" }}"

          cilium service update \
            --k8s-host-port \
            --id 2 \
            --backends "$BACKEND_IP:{{ env "NOMAD_ALLOC_PORT_ui" }}" \
            --frontend "{{ env "NOMAD_HOST_ADDR_ui" }}"
        EOH
        perms = "0755"
      }

      volume_mount {
        volume           = "cilium"
        destination      = "/var/run/cilium"
        propagation_mode = "private"
      }
    }

    task "delete-service" {
      driver = "exec"

      config {
        command = "/bin/sh"
        args = [
          "$${NOMAD_TASK_DIR}/delete-service.sh"
        ]
      }

      lifecycle {
        hook    = "poststop"
        sidecar = false
      }

      template {
        destination = "local/delete-service.sh"
        data        = <<-EOH
          #!/bin/sh

          set -x 

          cilium service delete 2
          cilium service delete 1
        EOH
        perms = "0755"
      }

      volume_mount {
        volume           = "cilium"
        destination      = "/var/run/cilium"
        propagation_mode = "private"
      }
    }

# .... rest of the task definition
}