CORS-4334: Konnectivity #10344

tthvo · 2026-03-14T00:43:28Z

Just curious: Any reasons to choose HTTPConnect over gRPC, which is should be (theoretically) faster? From docs, it said 👇

# This controls the protocol between the API Server and the Konnectivity # server. Supported values are "GRPC" and "HTTPConnect". There is no # end user visible difference between the two modes. You need to set the # Konnectivity server to work in the same mode. proxyProtocol: GRPC

My guess is that it's fine to use gRPC? If so, we need to adjust --mode=grpc in konnectivity-server-pod.yaml

tthvo · 2026-03-14T05:27:41Z

nit: these pods only run during bootstrap and will "never?" get updated so we can just ignore this setting, right 🤔?

Besides, I guess 10% of 3 control plane node is ~ 1 node; thus, it is equivalent to maxUnavailable: 1, which is already the default that k8s set (according to docs).

tthvo · 2026-03-14T05:21:45Z

nit: we should give this agent container a resource request so that it won't be the first get evicted if node is under pressure (theoretically).

As reference, Hypershift sets the following values 👀

tthvo · 2026-03-14T05:23:24Z

nit: we should give this server container a resource request so that it won't be the first get evicted if node is under pressure (theoretically).

As reference, Hypershift sets the following values 👀

tthvo · 2026-03-14T03:54:28Z

We should also honour the field .BootstrapNodeIP if set via the environment variable OPENSHIFT_INSTALL_BOOTSTRAP_NODE_IP, right?

Tracing back to the commit, it may be necessary for assisted installer 🤔?

installer/pkg/asset/ignition/bootstrap/common.go

Lines 347 to 351 in 30c4271

bootstrapNodeIP := os.Getenv("OPENSHIFT_INSTALL_BOOTSTRAP_NODE_IP")

if bootstrapNodeIP != "" && net.ParseIP(bootstrapNodeIP) == nil {

logrus.Warnf("OPENSHIFT_INSTALL_BOOTSTRAP_NODE_IP must have valid ip address, given %s. Skipping it", bootstrapNodeIP)

bootstrapNodeIP = ""

}

installer/pkg/asset/ignition/bootstrap/common.go

Line 95 in 30c4271

BootstrapNodeIP string

May we can do something like 👇 WDYT?

{{- if .BootstrapNodeIP }} # Use explicitly configured bootstrap node IP BOOTSTRAP_NODE_IP="{{.BootstrapNodeIP}}" echo "Using configured bootstrap node IP: ${BOOTSTRAP_NODE_IP}" {{- else }} # Detect bootstrap node IP at runtime using the default route source address. # Konnectivity agents use this to connect back to the bootstrap server. {{- if .UseIPv6ForNodeIP }} BOOTSTRAP_NODE_IP=$(ip -6 -j route get 2001:4860:4860::8888 | jq -r '.[0].prefsrc') {{- else }} BOOTSTRAP_NODE_IP=$(ip -j route get 1.1.1.1 | jq -r '.[0].prefsrc') {{- end }} echo "Detected bootstrap node IP: ${BOOTSTRAP_NODE_IP}" {{- end }}

tthvo · 2026-03-14T02:46:29Z

Suggested change

oc delete namespace openshift-bootstrap-konnectivity \

--kubeconfig=/opt/openshift/auth/kubeconfig \

--ignore-not-found=true || true

oc delete namespace openshift-bootstrap-konnectivity \

--kubeconfig=/opt/openshift/auth/kubeconfig \

--ignore-not-found=true

I guess we should fail if the cleanup somehow failed (except not-found) right? Otherwise, resources will be left behind and can potentially "break" the end openshift cluster?

tthvo · 2026-03-13T23:45:59Z

We need to add remove this rule when destroying bootstrap, right? This probably means patching the awscluster CR and waiting for the rule to disappear...

💡 Another idea: since this is scoped to only bootstrap node, the installer may pre-create a security group specifically for bootstrap with this rule? This SG can be attached via AdditionalSecurityGroups.

Whoops, I just saw #10344 (comment) so we do need to clean up the rule :D

-Original file line number
+Diff line change
@@ -0,0 +1,15 @@
+    apiVersion: apiserver.k8s.io/v1beta1
+    kind: EgressSelectorConfiguration
+    egressSelections:
+    - name: "cluster"
+      connection:
+        proxyProtocol: "HTTPConnect"
+        transport:
+          uds:
+            udsName: "/etc/kubernetes/config/konnectivity-server.socket"
+    - name: "controlplane"
+      connection:
+        proxyProtocol: "Direct"
+    - name: "etcd"
+      connection:
+        proxyProtocol: "Direct"

-Original file line number
+Diff line change
@@ -0,0 +1,13 @@
+    apiVersion: v1
+    kind: Secret
+    metadata:
+      name: konnectivity-agent-certs
+      namespace: openshift-bootstrap-konnectivity
+      labels:
+        app: konnectivity-agent
+        openshift.io/bootstrap-only: "true"
+    type: Opaque
+    data:
+      tls.crt: ${KONNECTIVITY_AGENT_CERT_BASE64}
+      tls.key: ${KONNECTIVITY_AGENT_KEY_BASE64}
+      ca.crt: ${KONNECTIVITY_CA_CERT_BASE64}

-Original file line number
+Diff line change
@@ -0,0 +1,58 @@
+    apiVersion: apps/v1
+    kind: DaemonSet
+    metadata:
+      name: konnectivity-agent
+      namespace: openshift-bootstrap-konnectivity
+      labels:
+        app: konnectivity-agent
+        openshift.io/bootstrap-only: "true"
+    spec:
+      selector:
+        matchLabels:
+          app: konnectivity-agent
+      updateStrategy:
+        type: RollingUpdate
+        rollingUpdate:
+          maxUnavailable: 10%
+      template:
+        metadata:
+          labels:
+            app: konnectivity-agent
+        spec:
+          hostNetwork: true
+          dnsPolicy: Default
+          priorityClassName: system-node-critical
+          tolerations:
+          - operator: Exists
+          containers:
+          - name: konnectivity-agent
+            image: ${KONNECTIVITY_IMAGE}
+            command:
+            - /usr/bin/proxy-agent
+            args:
+            - --logtostderr=true
+            - --ca-cert=/etc/konnectivity/ca.crt
+            - --agent-cert=/etc/konnectivity/tls.crt
+            - --agent-key=/etc/konnectivity/tls.key
+            - --proxy-server-host=${BOOTSTRAP_NODE_IP}
+            - --proxy-server-port=8091
+            - --health-server-port=2041
+            - --agent-identifiers=default-route=true
+            - --keepalive-time=30s
+            - --probe-interval=5s
+            - --sync-interval=5s
+            - --sync-interval-cap=30s
+            livenessProbe:
+              httpGet:
+                path: /healthz
+                port: 2041
+              initialDelaySeconds: 10
+              periodSeconds: 10
+            volumeMounts:
+            - name: konnectivity-certs
+              mountPath: /etc/konnectivity
+              readOnly: true
+          volumes:
+          - name: konnectivity-certs
+            secret:
+              secretName: konnectivity-agent-certs

-Original file line number
+Diff line change
@@ -0,0 +1,5 @@
+    apiVersion: kubecontrolplane.config.openshift.io/v1
+    kind: KubeAPIServerConfig
+    apiServerArguments:
+      egress-selector-config-file:
+      - "/etc/kubernetes/config/egress-selector-config.yaml"

-Original file line number
+Diff line change
@@ -0,0 +1,6 @@
+    apiVersion: v1
+    kind: Namespace
+    metadata:
+      name: openshift-bootstrap-konnectivity
+      labels:
+        openshift.io/bootstrap-only: "true"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CORS-4334: Konnectivity #10344

Uh oh!

Diff view

Diff view

There are no files selected for viewing

tthvo Mar 14, 2026

Uh oh!

tthvo Mar 14, 2026

Uh oh!

tthvo Mar 14, 2026

Uh oh!

tthvo Mar 14, 2026

Uh oh!

tthvo Mar 14, 2026 •

edited

Loading

Uh oh!

tthvo Mar 14, 2026

Uh oh!

tthvo Mar 14, 2026

Uh oh!

tthvo Mar 13, 2026

Uh oh!

tthvo Mar 14, 2026

Uh oh!

Uh oh!

-Original file line number
+Diff line change
@@ Expand Up @@
     . /usr/local/bin/bootstrap-cluster-gather.sh
     # shellcheck source=bootstrap-verify-api-server-urls.sh
     . /usr/local/bin/bootstrap-verify-api-server-urls.sh
+    # shellcheck source=konnectivity.sh.template
+    . /usr/local/bin/konnectivity.sh
     mkdir --parents /etc/kubernetes/{manifests,bootstrap-configs,bootstrap-manifests}
@@ Expand Down Expand Up / @@ -245,6 +247,8 @@ then @@
         record_service_stage_success
     fi
+    konnectivity_setup
     if [ ! -f kube-apiserver-bootstrap.done ]
     then
         record_service_stage_start "kube-apiserver-bootstrap"
@@ Expand All / @@ -269,9 +273,12 @@ then @@
     		--infra-config-file=/assets/manifests/cluster-infrastructure-02-config.yml \
     		--rendered-manifest-files=/assets/manifests \
     		--payload-version=$VERSION \
-    		--operand-kubernetes-version="${KUBERNETES_VERSION}"
+    		--operand-kubernetes-version="${KUBERNETES_VERSION}" \
+    		--config-override-files=/assets/konnectivity-config-override.yaml
     	cp kube-apiserver-bootstrap/config /etc/kubernetes/bootstrap-configs/kube-apiserver-config.yaml
+    	# Copy egress selector config to bootstrap-configs where KAS can read it
+    	cp /opt/openshift/egress-selector-config.yaml /etc/kubernetes/bootstrap-configs/egress-selector-config.yaml
     	cp kube-apiserver-bootstrap/bootstrap-manifests/* bootstrap-manifests/
     	cp kube-apiserver-bootstrap/manifests/* manifests/
@@ Expand Down Expand Up / @@ -566,6 +573,8 @@ then @@
         record_service_stage_success
     fi
+    konnectivity_manifests
     REQUIRED_PODS="openshift-kube-apiserver/kube-apiserver,openshift-kube-scheduler/openshift-kube-scheduler,openshift-kube-controller-manager/kube-controller-manager,openshift-cluster-version/cluster-version-operator"
     if [ "$BOOTSTRAP_INPLACE" = true ]
     then
@@ Expand Down Expand Up / @@ -651,6 +660,8 @@ if [ ! -f api-int-dns-check.done ]; then @@
       fi
     fi
+    konnectivity_cleanup
     # Workaround for https://github.com/opencontainers/runc/pull/1807
     touch /opt/openshift/.bootkube.done
     echo "bootkube.service complete"

-Original file line number
+Diff line change
@@ -0,0 +1,55 @@
+    #!/usr/bin/env bash
+    set -euo pipefail
+    # Generate Konnectivity certificates with a self-signed CA (1-day validity).
+    # These are needed for mTLS between the Konnectivity server and agents
+    # during the bootstrap phase.
+    #
+    # Usage: konnectivity-certs.sh <bootstrap-node-ip>
+    BOOTSTRAP_NODE_IP="${1:?Usage: konnectivity-certs.sh <bootstrap-node-ip>}"
+    KONNECTIVITY_CERT_DIR=/opt/openshift/tls/konnectivity
+    mkdir -p "${KONNECTIVITY_CERT_DIR}"
+    echo "Generating Konnectivity certificates in ${KONNECTIVITY_CERT_DIR}..."
+    # Generate self-signed Konnectivity CA
+    openssl req -x509 -newkey rsa:2048 -nodes \
+        -keyout "${KONNECTIVITY_CERT_DIR}/ca.key" \
+        -out "${KONNECTIVITY_CERT_DIR}/ca.crt" \
+        -days 1 \
+        -subj "/CN=konnectivity-signer/O=openshift"
+    # Server certificate for agent endpoint (needs bootstrap IP as SAN)
+    openssl req -new -newkey rsa:2048 -nodes \
+        -keyout "${KONNECTIVITY_CERT_DIR}/server.key" \
+        -out "${KONNECTIVITY_CERT_DIR}/server.csr" \
+        -subj "/CN=konnectivity-server/O=openshift"
+    openssl x509 -req -in "${KONNECTIVITY_CERT_DIR}/server.csr" \
+        -CA "${KONNECTIVITY_CERT_DIR}/ca.crt" \
+        -CAkey "${KONNECTIVITY_CERT_DIR}/ca.key" \
+        -CAcreateserial \
+        -out "${KONNECTIVITY_CERT_DIR}/server.crt" \
+        -days 1 \
+        -extfile <(printf "extendedKeyUsage=serverAuth\nsubjectAltName=IP:%s" "${BOOTSTRAP_NODE_IP}")
+    # Agent client certificate (shared by all agents)
+    openssl req -new -newkey rsa:2048 -nodes \
+        -keyout "${KONNECTIVITY_CERT_DIR}/agent.key" \
+        -out "${KONNECTIVITY_CERT_DIR}/agent.csr" \
+        -subj "/CN=konnectivity-agent/O=openshift"
+    openssl x509 -req -in "${KONNECTIVITY_CERT_DIR}/agent.csr" \
+        -CA "${KONNECTIVITY_CERT_DIR}/ca.crt" \
+        -CAkey "${KONNECTIVITY_CERT_DIR}/ca.key" \
+        -CAcreateserial \
+        -out "${KONNECTIVITY_CERT_DIR}/agent.crt" \
+        -days 1 \
+        -extfile <(printf "extendedKeyUsage=clientAuth")
+    # Clean up CSR files
+    rm -f "${KONNECTIVITY_CERT_DIR}"/*.csr
+    echo "Konnectivity certificates generated successfully."

-Original file line number
+Diff line change
@@ Expand Up @@
     						ToPort:                   10259,
     						SourceSecurityGroupRoles: []capa.SecurityGroupRole{"controlplane", "node"},
     					},
+    					{
+    						Description:              "Konnectivity agent traffic from cluster nodes",
+    						Protocol:                 capa.SecurityGroupProtocolTCP,
+    						FromPort:                 8091,
+    						ToPort:                   8091,
+    						SourceSecurityGroupRoles: []capa.SecurityGroupRole{"controlplane", "node"},
+    					},
     					{
     						Description: BootstrapSSHDescription,
     						Protocol:    capa.SecurityGroupProtocolTCP,
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up @@
     					Destination:      ptr.To("*"),
     					Action:           capz.SecurityRuleActionAllow,
     				},
+    				{
+    					Name:             "konnectivity_in",
+    					Protocol:         capz.SecurityGroupProtocolTCP,
+    					Direction:        capz.SecurityRuleDirectionInbound,
+    					Priority:         103,
+    					SourcePorts:      ptr.To("*"),
+    					DestinationPorts: ptr.To("8091"),
+    					Source:           ptr.To(source),
+    					Destination:      ptr.To("*"),
+    					Action:           capz.SecurityRuleActionAllow,
+    				},
     				{
     					Name:             fmt.Sprintf("%s_ssh_in", clusterID.InfraID),
     					Protocol:         capz.SecurityGroupProtocolTCP,
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up @@
     					},
     				},
     			},
+    			{
+    				// Konnectivity
+    				Action:    capibmcloud.VPCSecurityGroupRuleActionAllow,
+    				Direction: capibmcloud.VPCSecurityGroupRuleDirectionInbound,
+    				Source: &capibmcloud.VPCSecurityGroupRulePrototype{
+    					PortRange: &capibmcloud.VPCSecurityGroupPortRange{
+    						MaximumPort: 8091,
+    						MinimumPort: 8091,
+    					},
+    					Protocol: capibmcloud.VPCSecurityGroupRuleProtocolTCP,
+    					Remotes: []capibmcloud.VPCSecurityGroupRuleRemote{
+    						{
+    							RemoteType:        capibmcloud.VPCSecurityGroupRuleRemoteTypeSG,
+    							SecurityGroupName: clusterWideSGNamePtr,
+    						},
+    					},
+    				},
+    			},
     		},
     	}
     }
@@ Expand Down @@

CORS-4334: Konnectivity #10344

Are you sure you want to change the base?

Uh oh!

CORS-4334: Konnectivity #10344

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tthvo Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tthvo Mar 14, 2026 •

edited

Loading