IPv6 metrics are not scrapped from brupop-controller-server service

[jpmcb]

Image I'm using:

image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v1.0.0

Issue or Feature Request:

Metrics from the brupop-controller-server.brupop-bottlerocket-aws service on the /metrics route are not scrapable on IPv6 clusters. I would expect this endpoint to also work on IPv6 clusters as it does in IPv4 clusters. This prevents our recommended prometheus scrape config from working on IPv6 and blocks users on IPv6 clusters from getting brupop metrics.

In the happy path, getting metrics on an IPv4 cluster:

$ curl brupop-controller-server.brupop-bottlerocket-aws/metrics
# HELP brupop_hosts_state Brupop host's state
# TYPE brupop_hosts_state gauge
brupop_hosts_state{state="Idle"} 2
brupop_hosts_state{state="StagedAndPerformedUpdate"} 1
# HELP brupop_hosts_version Brupop host's bottlerocket version
# TYPE brupop_hosts_version gauge
brupop_hosts_version{bottlerocket_version="1.9.0"} 3

Replication steps:

1. On an IPv6 cluster running bottlerocket nodes, deploy the brupop stack:

$ kubectl apply -f bottlerocket-update-operator.yaml

2. Deploy a simple debug pod to the brupop-bottlerocket-aws namespace:

apiVersion: v1
kind: Pod
metadata:
  namespace: brupop-bottlerocket-aws
  name: ubuntu
  labels:
    app: ubuntu
spec:
  containers:
  - image: ubuntu
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: ubuntu
  restartPolicy: Always

$ kubectl apply -f debug.yaml

3. Get a forward shell to that debug pod:

$ kubectl exec -n brupop-bottlerocket-aws --stdin --tty ubuntu -- /bin/bash

4. Get curl

# apt update
# apt install curl
# apt install dnsutils

5. And attempt to manually scrape the metrics:

# curl brupop-controller-server.brupop-bottlerocket-aws/metrics
curl: (7) Failed to connect to brupop-controller-server.brupop-bottlerocket-aws port 80 after 1 ms: Connection refused

The fully qualified service also doesn't work:

# curl brupop-controller-server.brupop-bottlerocket-aws.svc.cluster.local/metrics
curl: (7) Failed to connect to brupop-controller-server.brupop-bottlerocket-aws.svc.cluster.local port 80 after 4 ms: Connection refused

And it doesn't seem to be a problem with routing to the service: the nslookup to the service resolves correctly:

# nslookup brupop-controller-server.brupop-bottlerocket-aws
Server:         fdc1:cad0:d971::a
Address:        fdc1:cad0:d971::a#53

Name:   brupop-controller-server.brupop-bottlerocket-aws.svc.cluster.local
Address: fdc1:cad0:d971::9870