[Design Discussion] Bi-directional communication channel for private Data Plane to Control Plane communication

[JKAUSHALYA]

Related Feature Issue

#4247

Problem Summary

Data Plane instances deployed in private networks or corporate VPCs cannot accept inbound traffic from the Control Plane due to strict firewall and NAT restrictions. This prevents the Control Plane from pushing real-time updates and operational commands without forcing users to expose endpoints or configure complex, insecure network rules. To resolve this, we need an outbound "phone-home" architecture where the Data Plane initiates a persistent, bi-directional channel (such as WebSockets or gRPC streaming) back to the Control Plane.

High-Level Approach

To solve the private network connectivity problem, we propose implementing an outbound agent model ("Phone-Home") where the Data Plane (DP) initiates and maintains a long-lived outbound connection to the Control Plane (CP).

Primary Recommendation: WebSocket Agent Architecture

We lean toward a WebSocket (WSS over Port 443) implementation due to its minimal network requirements and ease of integration across standard cloud infrastructure.

+------------------------------------+             +----------------------------------+
|           DATA PLANE               |             |          CONTROL PLANE           |
|         (Private Network)          |             |         (Public Network)         |
|                                    |             |                                  |
|  +------------------------------+  |  Outbound   |  +----------------------------+  |
|  |  WebSocket Client (Agent)    |==================>|  WebSocket Server          |  |
|  +------------------------------+  |  WSS (443)  |  +----------------------------+  |
|                 |                  |             |                |                 |
|  Executes local commands & returns |             |  Dispatches commands via RPC   |
|  tagged responses                  |             |  over established socket       |
+------------------------------------+             +----------------------------------+

• Connection Lifecycle:

  • The DP agent boots up and establishes an encrypted outbound WebSocket connection (wss://cp.thunderid.io/connect) authenticated via mTLS or scoped API tokens.
  • The DP maintains continuous connection health via standard ping/pong heartbeats and automatic backoff reconnection logic.

• Reverse RPC Protocol:

  • CP issues commands down the open socket as structured JSON-RPC messages (with standard request_id, method, and params).
  • DP receives the frame, executes the request locally, and responds over the same socket tagged with the matching request_id.

Alternative Approaches Under Consideration

To ensure we choose the optimal long-term architecture for ThunderID, we want to evaluate three alternative patterns alongside WebSockets:

gRPC Bidirectional Streaming (HTTP/2)

Overview: Similar to WebSockets, but uses persistent HTTP/2 streams and binary Protocol Buffers.

Pros/Cons: Offers superior performance and strict schema contracts, but requires intermediate load balancers and reverse proxies to fully support end-to-end HTTP/2 multiplexing.

Centralized Message Broker (e.g., NATS / RabbitMQ)

Overview: Both CP and DP connect as clients to a central message broker using Request-Reply patterns.

Pros/Cons: Completely decouples CP and DP state management and handles offline command queuing automatically, but introduces an extra stateful infrastructure dependency to operate.

Reverse Network Tunneling (e.g., WireGuard / Cloudflare Tunnel / frp)

Overview: DP runs a tunnel agent establishing an encrypted network bridge, allowing CP to make standard HTTP/REST calls as if DP were locally accessible.

Pros/Cons: Allows us to keep plain REST APIs on the DP without custom socket wrappers, but increases operational complexity by requiring external tunnel management or sidecars.

Architecture Overview

1. High-Level Architecture Concept

ThunderID utilizes an Agent-Driven, Outbound WebSocket Architecture (commonly known as the "Phone-Home" pattern) to bridge the communication gap between the Control Plane (CP) and Data Plane (DP).

In enterprise environments, Data Planes are frequently deployed within isolated private networks, VPCs, or edge environments protected by strict firewalls that disallow inbound traffic. To circumvent these restrictions without introducing security vulnerabilities or requiring custom network configurations, the Data Plane acts as an active client that establishes a persistent outbound WebSocket connection (wss://) to the Control Plane.

Once established, this continuous full-duplex stream enables bi-directional communication, allowing the CP to push real-time commands, configuration updates, and operational tasks down to the DP via a structured JSON-RPC 2.0 protocol over WebSocket frames.

---

2. System Components & Topology

+-------------------------------------------------------+       +-------------------------------------------------------+
|                     DATA PLANE                        |       |                    CONTROL PLANE                      |
|                 (Private Network / VPC)               |       |                   (Public / Cloud)                    |
|                                                       |       |                                                       |
|  +--------------------+      +---------------------+  |       |  +--------------------+      +---------------------+  |
|  |  Data Plane Engine | <--> |  WS Agent (Client)  |================>|   API Gateway / LB  | <--> |  CP WS Connection   |  |
|  +--------------------+      +---------------------+  | Outbound   +--------------------+      |       Manager       |  |
|                                                       | WSS (443)                                +---------------------+  |
|                                                       |                                                     |             |
|                                                       |                                          +---------------------+  |
|                                                       |                                          |    Redis / Store    |  |
|                                                       |                                          |  (Connection Map)   |  |
|                                                       |                                          +---------------------+  |
+-------------------------------------------------------+       +-------------------------------------------------------+

Component Breakdown

Data Plane WS Agent (Client):

• Embedded alongside or within the Data Plane process.
• Initiates the outbound TLS handshake (wss://cp.thunderid.io/connect) over port 443 upon startup.
• Authenticates using mTLS certificates or scoped token exchange headers.
• Receives incoming RPC commands from the CP, executes them locally via DP service methods, and returns tagged responses.
• Handles lifecycle management: health checks, connection keep-alives (Ping/Pong), and automatic backoff reconnection logic.

Control Plane WS Gateway (Server):

• Terminates inbound WebSocket connections at the edge load balancer / API Gateway.
• Validates and authenticates connected DP agents.
• Maintains an in-memory registry mapping active DataPlaneID instances to their respective socket descriptor.
• Leverages a shared state store (e.g., Redis Pub/Sub) to route commands across horizontally scaled CP instances.
• Serializes and dispatches RPC payloads directly to the target DP's persistent channel.

3. Communication Workflows

Phase 1: Connection & Authentication Handshake

Data Plane Agent                                      Control Plane Gateway
       |                                                        |
       | ----- 1. Outbound HTTP Upgrade (WSS + Auth Headers) --> |
       |                                                        | [Validate Credentials / mTLS]
       |                                                        | [Register DataPlaneID in Registry]
       | <---- 2. HTTP 101 Switching Protocols ---------------- |
       |                                                        |
       | === Persistent Bi-Directional Pipe Established ======= |

Phase 2: Reverse RPC Command Dispatch

When the Control Plane needs to trigger an action on a specific Data Plane (e.g., updating access rules):

Control Plane Core                   CP Connection Manager                 Data Plane Agent
        |                                     |                                   |
        | -- 1. Dispatch(DataPlaneID) ------> |                                   |
        |                                     | [Lookup Active Socket]            |
        |                                     | -- 2. Send JSON-RPC Request ----> |
        |                                     |     {"jsonrpc": "2.0",            |
        |                                     |      "id": "req-9876",             |
        |                                     |      "method": "Policy.Sync",      |
        |                                     |      "params": {...}}             |
        |                                     |                                   | [Execute Action]
        |                                     | <-- 3. Send JSON-RPC Response --- |
        |                                     |     {"jsonrpc": "2.0",            |
        |                                     |      "id": "req-9876",             |
        |                                     |      "result": {"status": "ok"}}  |
        | <-- 4. Return Execution Status ---- |                                   |

Security Considerations

Because the Data Plane operates inside a private network with administrative capabilities over local workloads, the persistent WebSocket channel between the Control Plane (CP) and Data Plane (DP) must be secured against unauthorized access, command injection, and session hijacking.

---

1. Authentication & Identity Verification

• Mutual TLS (mTLS) / Client Certificates:
  • Primary Recommendation: Enforce mTLS during the initial TLS handshake. Each DP agent is provisioned with a unique, short-lived X.509 client certificate issued by a private ThunderID Certificate Authority (CA).
  • The certificate's Common Name (CN) or Subject Alternative Name (SAN) must explicitly contain the DataPlaneID.
• Token-Based Authentication (Alternative):
  • If mTLS is not viable, DP agents must present a securely stored, cryptographically signed Bearer Token (or JWT) in the Authorization header during the initial HTTP Upgrade request.
  • CP verifies token validity, signature, and scope before switching protocols to WebSocket.

---

2. Transport Layer Security (TLS / WSS)

• Strict HTTPS/WSS: Plain WebSocket (ws://) connections are strictly prohibited. All communication must occur over wss:// on port 443.
• TLS Configuration: Enforce modern TLS cipher suites (TLS 1.3 preferred, TLS 1.2 minimum) and disable weak or deprecated ciphers.
• Certificate Pinning: The DP agent should pin the root CA or server certificate of the Control Plane to prevent Man-in-the-Middle (MitM) attacks caused by compromised local certificate stores or proxies.

---

3. Command Validation & Least Privilege

• Strict RPC Method Whitelisting:
  • The DP agent must maintain a strict whitelist of executable JSON-RPC methods (Policy.Sync, Agent.Ping, etc.).
  • Any message invoking an unregistered method must be rejected immediately with a standard JSON-RPC MethodNotFound (-32601) error.
• Schema Validation & Sanitization:
  • Input parameters (params) must be validated against a strict JSON Schema before execution to protect against command injection, memory corruption, or path traversal attacks.
• Scope Enforcement:
  • The CP connection manager must verify that dispatched requests match the specific DataPlaneID bound to the socket, preventing cross-tenant command leakage.

---

4. Replay Attack & Tamper Prevention

• Cryptographic Request Nonces:
  • Every JSON-RPC payload includes a unique, cryptographically generated request_id along with a timestamp.
• Replay Protection Window:
  • The DP agent drops any command received outside an acceptable time drift window (e.g., ±30 seconds) or with a duplicated request_id within a given cache TTL.

---

5. Connection Hijacking & Session Management

• Single Active Socket per Data Plane:
  • The Control Plane registry enforces a strict policy of one active WebSocket connection per DataPlaneID.
  • If a new handshake request arrives for an already connected DataPlaneID, the CP must authenticate the new request, forcefully terminate the legacy socket, and log a security audit event.
• Automatic Revocation:
  • CP provides an emergency kill-switch mechanism to immediately terminate the WebSocket connection and invalidate the DP agent's credentials in case of compromise.

---

6. Denial of Service (DoS) & Resource Isolation

• Handshake Rate Limiting:
  • CP API Gateways must rate-limit outbound socket connection attempts per IP/DataPlaneID to mitigate connection flooding attacks.
• Max Frame & Payload Size Limits:
  • Enforce maximum WebSocket frame size limits (e.g., 1 MB per frame) on both CP and DP to prevent memory exhaustion attacks from oversized payloads.
• Execution Timeouts:
  • DP sets aggressive execution timeouts on all incoming RPC tasks to prevent malicious or malformed commands from hanging local execution threads.

---

7. Audit Logging & Observability

• Signed Audit Trails:
  • Both CP and DP must generate structured audit logs for every command received, executed, or failed, including request_id, method, origin_user_id, and timestamp.
• Sensitive Data Redaction:
  • Secrets, private keys, or credentials passed within RPC payload parameters must be masked or redacted prior to log output.

Impacted Areas

Implementing the outbound WebSocket communication channel affects several core components across the ThunderID codebase, deployment infrastructure, and operational tooling.

---

1. Control Plane (CP) Components

• WebSocket Gateway & Connection Manager (/pkg/cp/gateway or equivalent):
  • New Module: Needs a high-concurrency WebSocket server module to handle incoming HTTP Upgrade requests, authenticate DP agents, manage active connection states, and handle TCP keep-alives/heartbeats.
• RPC Command Dispatcher (/pkg/cp/dispatcher):
  • Update: Refactor existing CP API dispatching logic. Instead of executing direct inbound HTTP/REST calls to Data Planes, the dispatcher must route RPC requests to the local Connection Manager or over the internal pub/sub bus.
• Authentication & Identity Service (/pkg/cp/auth):
  • Update: Extend authentication handlers to validate mTLS client certificates or DP bearer tokens during the initial WebSocket handshake.

---

2. Data Plane (DP) Components

• Outbound WS Agent (/pkg/dp/agent):
  • New Module: Implement an agent lifecycle manager responsible for initiating outbound wss:// connections, managing reconnection backoff timers with full jitter, and responding to ping/pong keep-alives.
• JSON-RPC Command Handler & Router (/pkg/dp/rpc):
  • New Module: Add a reverse-RPC dispatcher on the DP that unpacks JSON-RPC payloads, routes commands (Policy.Sync, Key.Rotate, etc.) to local DP services, and formats success/error responses back into frame payloads.
• DP Configuration & Credentials Manager (/pkg/dp/config):
  • Update: Update DP configuration schemas to support new settings: Control Plane WebSocket endpoint URL, mTLS certificate paths/tokens, ping intervals, and timeout thresholds.

---

3. Shared Infrastructure & Storage

• State Store & Connection Registry (e.g., Redis / Etcd):
  • Update: Introduce a distributed connection mapping schema (e.g., DataPlaneID -> ControlPlaneNodeID) in Redis/Etcd. This enables any scaled CP instance to locate which specific node holds the open socket for a given Data Plane and publish messages to it via Pub/Sub.

---

4. Deployment, Packaging & DevOps

• Helm Charts & Terraform Modules:
  • Update: Simplify Data Plane deployment configurations. Inbound Ingress resources, Load Balancer rules, and open firewall ports previously required for the DP can now be removed from deployment templates.
• Certificate Authority & Provisioning Pipeline:
  • Update: Ensure automated provisioning and rotation of DP client certificates (mTLS) or API keys during initial boot/registration.

---

5. Observability & Telemetry

• Metrics & Monitoring (/pkg/telemetry):
  • Update: Expose new Prometheus metrics for connection health:
    • thunderid_cp_active_websocket_connections
    • thunderid_dp_reconnection_attempts_total
    • thunderid_rpc_roundtrip_latency_seconds
• Structured Logging & Tracing:
  • Update: Inject trace IDs (e.g., OpenTelemetry span contexts) into JSON-RPC frame headers to maintain end-to-end distributed tracing across CP and DP boundaries.

Alternatives Considered

During the architectural design phase for ThunderID's Control Plane (CP) to Data Plane (DP) communication channel, several architectural patterns were evaluated. While the primary objective was solving the private network firewall traversal issue, factors such as operational overhead, network compatibility, performance, and alignment across WSO2 teams were heavily weighed.

---

1. Decision Matrix & Evaluation Summary

Evaluation Criterion	WebSocket	gRPC Streaming	Reverse Tunnel	NATS / Broker	Weightage
Ease of Setup & Developer Experience (3 = Easy/Code-only, 1 = Complex/Infra-heavy)	3	2	1	2	50%
Network & Firewall Compatibility (3 = Works everywhere, 1 = Strict requirements)	3	2	3	3	75%
Performance & Latency (3 = Ultra-low latency, 1 = High overhead)	2	3	2	3	25%
Built-in Messaging & Framing (3 = Full RPC/Pattern support, 1 = None)	2	3	1	3	75%
Compatibility with other WSO2 teams	3	0	0	0	50%
Total Weighted Score	7.25	5.50	4.00	6.25	—

---

High - 3
Medium - 2
Low - 1
Cannot consider - 0

2. Detailed Breakdown of Alternatives

Alternative 1: gRPC Bidirectional Streaming (HTTP/2)

• Overview: Utilizes persistent HTTP/2 streams with Protocol Buffer binary serialization for CP-DP communication.
• Why it wasn't selected: Although gRPC excels in performance (score: 3) and strong RPC typing, it poses network compatibility challenges (score: 2). Many enterprise firewalls, legacy corporate proxies, and L7 load balancers restrict end-to-end HTTP/2 gRPC streaming. Furthermore, it lacks immediate compatibility with established patterns across other WSO2 product teams.

Alternative 2: Centralized Message Broker (NATS / RabbitMQ / Apache Kafka)

• Overview: Both CP and DP connect as clients to an intermediary message broker, using Request-Reply pub/sub subjects.
• Why it wasn't selected: NATS scored high overall (6.25) due to its built-in request-reply messaging and resilience. However, it introduces an additional stateful infrastructure dependency that must be deployed, monitored, and maintained alongside ThunderID. For lightweight DP deployments, requiring a dedicated broker cluster increases deployment friction and TCO (Total Cost of Ownership).

Alternative 3: Reverse Network Tunneling (WireGuard / Cloudflare Tunnel / frp)

• Overview: Runs a tunnel daemon alongside the Data Plane to establish an encrypted virtual network bridge, allowing CP to make direct REST/HTTP calls to the DP.
• Why it wasn't selected: Scored lowest overall (4.00). Reverse tunnels add significant infrastructure complexity, require running extra sidecars/daemons, and complicate zero-trust security audits by exposing network-layer connectivity rather than application-scoped RPC endpoints.

---

3. Rationale for Selecting WebSockets

WebSocket emerged as the clear winner with a Total Weighted Score of 7.25, driven by key decisive factors:

1. Maximum Network Compatibility: Runs over standard wss:// (Port 443), seamlessly bypassing enterprise firewalls, NATs, and legacy proxies without requiring special L7 infrastructure rules.
2. Zero Extra Infrastructure: Can be implemented entirely in application code without forcing operators to manage third-party message brokers or tunnel daemons.
3. Cross-Team Alignment: Provides strong compatibility with existing patterns and shared libraries used across other WSO2 engineering teams, streamlining code reuse, security reviews, and internal adoption.

Questions for Community Input

To finalize the design and implementation plan for the Control Plane (CP) to Data Plane (DP) communication channel, we would love input from the community on the following technical and operational questions:

---

1. Protocol & Framing

• JSON-RPC 2.0 vs. Custom Binary/Protobuf Frames: Is JSON-RPC 2.0 sufficient for our expected DP command throughput and payload sizes, or should we consider Protocol Buffer binary frames over WebSockets for higher serialization efficiency?
• Offline Command Handling: How should the Control Plane handle commands dispatched while a Data Plane is temporarily disconnected? Should CP fail fast (synchronous error), or buffer requests in Redis/DB with a Time-To-Live (TTL)?

---

2. Authentication & Security

• Authentication Defaults: Should Mutual TLS (mTLS) be enforced as a hard requirement for all DP connections, or should token-based authentication (Bearer Tokens / API Keys) be supported as an alternative for local/development environments?
• Egress Proxy Support: Are there specific enterprise HTTP proxy configurations (e.g., authenticated forward proxies) in your target deployment environments that could interfere with long-lived WebSocket connections?

---

3. Resilience & Connection Lifecycle

• Heartbeat & Timeout Tuning: Is a 30-second ping/pong heartbeat interval sufficient to prevent aggressive enterprise firewalls and cloud load balancers (e.g., AWS ALB, NGINX) from dropping idle TCP connections?
• Reconnection Strategy: Are there specific backoff and jitter algorithms preferred to prevent "thundering herd" reconnection issues when a Control Plane cluster node restarts?

---

4. Scalability & State Management

• Distributed Routing: Is Redis Pub/Sub the ideal choice for routing RPC calls across horizontally scaled Control Plane instances, or should we consider alternative state synchronization backends (e.g., NATS, Etcd, or gRPC mesh between CP nodes)?

---

💬 How to contribute: Please leave a comment below with your thoughts, use cases, or preferred approaches on any of the points above!

[bhagyasakalanka]

Pros and Cons of Using WebSockets vs. VPN for Connecting a Private Data Plane (PDP)

Option 1: WebSocket

Pros

• Easy for end users to configure and get started.
• No additional infrastructure or connectivity costs.
• Firewall-friendly, as outbound WebSocket connections are typically allowed over HTTPS (443).
• Suitable for quick onboarding and evaluation environments.

Cons

• Higher implementation complexity.

  • WebSocket connections are established between individual CP and DP pods. In a multi-pod deployment, one CP pod may not be aware of WebSocket connections established by another CP pod to a DP.
  • Addressing this requires additional coordination between CP instances, introducing significant implementation and maintenance complexity. Alternatively, simplifying the design may require compromising high availability.

• More complex connection lifecycle management, including reconnects, failover, and stale connection handling.

• Scaling and load balancing WebSocket connections is more challenging than stateless REST communication.

• Long-lived connections require additional monitoring and operational handling.

---

Option 2: VPN

Pros

• Enables communication between the CP and PDP using standard REST APIs.
• Provides a consistent communication model for both public and private Data Planes.
• Thunder does not need to manage the underlying communication channel or high availability of persistent connections.
• Stateless REST communication simplifies scaling, load balancing, and failover.
• Leverages mature networking technologies with well-established operational practices.
• Easier to troubleshoot using standard networking and HTTP tools.

Cons

• Additional infrastructure and operational cost.
• Initial VPN setup is more involved compared to a simple WebSocket connection.
• Users evaluating Thunder PDP must provision and configure a VPN before getting started, increasing the onboarding effort.
• Requires VPN management, including certificate/key rotation, network configuration, and access management.
• VPN connectivity may be restricted by some enterprise networking policies or require additional approvals.

[JKAUSHALYA]

Architectural Decision Matrix: In-App WebSocket vs. Product-Managed Tunnel/VPN Sidecar

By having the product team package and manage an automated network tunnel or overlay VPN (e.g., a bundled WireGuard, Tailscale, or Cloudflare Tunnel sidecar/agent), we maintain a strict separation of concerns.

This keeps the core ThunderID Identity Provider (IdP) codebase focused purely on identity services (using standard REST/gRPC endpoints) while offloading complex connection state and firewall traversal to a dedicated, battle-tested networking component.

---

Scoring Scale

• High (3): Optimal outcome / Minimal effort / Low operational risk
• Medium (2): Acceptable trade-off / Moderate effort
• Low (1): High friction / Disproportionate development overhead / High operational risk

---

Decision Matrix Table

Evaluation Criterion	In-App WebSocket Agent (Custom Code)	Product-Managed Tunnel / VPN Sidecar	Weightage
Domain Separation & IdP Code Cleanliness (3 = Zero socket/transport code in IdP app, 1 = IdP app manages socket state)	1 (Pollutes IdP codebase with reverse RPC, ping/pong, and socket session state)	3 (IdP code stays 100% focused on identity using standard HTTP/gRPC)	25%
Engineering Effort & Time-to-Market (3 = Uses battle-tested networking tools, 1 = Reinventing socket HA in app)	1 (High initial & ongoing effort writing reconnect logic, heartbeats, & Redis registries)	3 (Low effort; leverages existing tools like WireGuard/frp/Tailscale as sidecars)	25%
Customer Onboarding & Experience (3 = Automated/zero-touch deploy, 1 = Manual customer NetOps setup)	3 (Zero network setup; outbound connection over port 443)	3 (Zero customer NetOps setup; sidecar automatically connects on container boot)	20%
Connection Reliability & HA (3 = Handled by mature networking kernel/daemon, 1 = Custom app reconnection logic)	2 (Must handle edge cases, dropped frames, and memory leaks in application code)	3 (Offloaded to battle-tested networking daemons built specifically for tunnel HA)	20%
Security & Port Scoping (3 = Restrictive L7 app scoping, 1 = Broad L3 network exposure)	3 (L7 application-scoped RPC only)	2 (L3/L4 tunnel restricted strictly to CP and DP IP/port bindings)	10%
Total Weighted Score (Out of 3.0)	1.80	2.80	100%

---

Key Architectural Takeaways

1. Clean Separation of Concerns (Score: 2.80): Packaging a product-managed tunnel agent (e.g., WireGuard / frp / Cloudflare Tunnel sidecar) is the clear winner. The Control Plane can make standard HTTP/REST or gRPC calls (POST /v1/sync) directly to the Data Plane over the secure virtual interface without adding transport-layer logic to the identity application.
2. Zero Customer Friction Preserved: Because the tunnel agent is configured and shipped automatically by the ThunderID team (e.g., as part of the Helm chart or Docker Compose manifest), customers do not need to involve their internal network operations teams.
3. Reduced Maintenance Burden: The product team avoids inventing, testing, and supporting custom WebSocket connection state, heartbeat loops, and distributed session maps—relying instead on battle-tested networking utilities maintained by the broader open-source community.

[senthalan]

As ThunderID also supports VM runtime, how the sidecar is going to work in that case?

[bhagyasakalanka]

CP↔DP transport — decision matrix

Options

• A — In-app WebSocket agent · DP dials CP over WSS/443, JSON-RPC push
• B — Product-managed tunnel sidecar · WireGuard / frp / cloudflared / Tailscale
• C — DP-initiated polling · HTTPS pull, versioned desired state, optional SSE upgrade

Scale assumption: 10k DPs × 3 pods = 30k pods

Scoring: 3 = High (optimal outcome, minimal effort, low operational risk) · 2 = Medium (acceptable trade-off, moderate effort) · 1 = Low (high friction, disproportionate overhead, high operational risk). Half-points where merged criteria disagreed.

#	Criterion	Description (3 = best · 1 = worst)	Wt	A: WebSocket	B: Tunnel	C: Polling
1	Engineering effort & time-to-market	How much new code is needed before first release, and how much of it is transport plumbing rather than product. 3 = builds on the existing REST stack · 1 = a new transport layer plus per-connection state inside the IdP app	13%	1	2	3
2	Maintenance, operations & run cost	Total ongoing cost to own: engineering time, infrastructure to run, binaries to patch, support load. 3 = nothing extra to run, patch or support · 1 = standing infra, a second binary to CVE-patch, new on-call surface	20%	2	1	3
3	Reliability, HA & scale at 30k pods	How the channel behaves at fleet size and under failure. 3 = stateless, self-healing, scales horizontally with no registry · 1 = per-DP connection state plus a distributed registry to shard	17%	1.5	1.5	3
4	Security posture	Blast radius if the CP is compromised, plus how hard the customer's security team is to get past. 3 = outbound-only, L7-scoped, approves like any SaaS API call · 1 = L3 reachability into the customer VPC, formal security review	18%	2.5	1	3
5	Network compatibility & zero-touch install	Whether it connects unaided in real enterprise networks and restricted clusters. 3 = plain HTTPS through any proxy, allowlist or cluster policy · 1 = UDP or non-proxy-traversable, blocked by restricted PSA / OpenShift SCCs	9%	2	1	3
6	Convergence latency & bounded staleness	How fast a change reaches every pod, and whether there is a guaranteed upper bound. 3 = sub-second delivery with confirmation · 1 = slow, or no upper bound on how stale a DP can silently get	9%	2	3	2
7	Cross-pod consistency & synchronized activation	What multi-pod DPs cost us: coordination we have to build ourselves. 3 = all pods flip together with per-pod ack · 1 = requires leader election or N-way dedup that we build and maintain	8%	1	2	2
8	Operation-model fit	Whether an operator can issue a command and get a live result back covering the whole DP. 3 = any CP replica can reach any DP pod directly and collect per-pod acks · 1 = eventual, fire-and-forget only. Note that with multiple CP replicas a WSS socket is visible only to the pod holding it, so responses must be routed back via a bus or registry hop — async infrastructure is required either way	6%	2	3	1
	Total weighted (out of 3.0)		100%	1.80	1.60	2.71