Coder WebSocket and DERP Connections Failing When Deployed Behind Teleport Proxy #17036

spiffaz · 2025-03-21T08:37:47Z

spiffaz
Mar 21, 2025

Environment

Coder deployed in Kubernetes via Helm
Coder chart version: coder-2.20.2
Deployed behind Teleport Application Service (not using direct ingress)
Coder is deployed as an internal ClusterIP service at http://coder.coder.svc.cluster.local
External access through Teleport: https://coder.org.teleport.sh

Problem Description

We're experiencing two connectivity issues with Coder when accessed through Teleport:

WebSocket Connection Failures:
- Coder's WebSocket connections fail with "expected handshake response status code 101 but got 200"
- The /health/websocket health check endpoint fails
- Normal HTTP requests to Coder UI work fine, but any WebSocket-dependent functionality fails
DERP Relay Connection Failures:
- DERP relay connections are being redirected to Teleport's authentication page
- The /health/derp health check endpoint fails
- This prevents workspace connections from functioning correctly

These issues only occur when Coder is accessed through Teleport. The basic UI loads correctly, but workspace connections and other WebSocket/DERP dependent functionality fail.

Error Details

WebSocket Error

EWS01: websocket dial: failed to WebSocket dial: expected handshake response status code 101 but got 200

HTML Response Received (instead of WebSocket upgrade)

<!DOCTYPE html>
<html lang="en">
<head>
<script src="/web/config.js"></script>
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="referrer" content="no-referrer" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="grv_csrf_token" content="449fc56af0c97e20e5f415a7955f57bbb08d1c5af64e1199c383bcaf6a2f832d" />
<meta name="grv_bearer_token" content="" />
<meta name="robots" content="noindex" />
<link href="/web/app/favicon-light.png" rel="icon" type="image/png" media="(prefers-color-scheme: light)" />
<link href="/web/app/favicon-dark.png" rel="icon" type="image/png" media="(prefers-color-scheme: dark)" />
<title></title>
<script type="module" crossorigin src="/web/app/app.js"></script>
<link rel="stylesheet" crossorigin href="/web/app/index.css">
</head>
<body>
<div id="app"></div>
</body>
</html>

DERP Connection Error

connect to derp: derphttp.Client.Connect connect to https://coder.org.teleport.sh/derp: GET failed: <nil>: <a href="https://org.teleport.sh:443/web/launch/coder.org.teleport.sh?path=%2Fderp">Found</a>.

DERP Connection Logs

derphttp.Client.Connect: connecting to https://coder.org.teleport.sh/derp
derphttp.Client.Connect: TLS version 0x304
derpclient: got cert org.teleport.sh
derpclient: got cert R10
derphttp.Client.Connect: not using fast start
derphttp.Client.Connect: DERP server returned status 302

Health Check Failures

Current Coder Configuration

We haven't made any special configuration changes to Coder besides deploying it in Kubernetes. We're investigating whether we need specific environment variables or configuration for proxy deployments.

URL Configuration Challenge

A key challenge we're facing is related to the CODER_ACCESS_URL configuration. We're specifically concerned about how Coder handles internal communication between components.

Current Limitation:
We're currently using a single CODER_ACCESS_URL which points to our Teleport-proxied external URL (https://coder.org.teleport.sh). This creates a problem because:

When users access Coder, they go through this URL and authenticate with Teleport, which works fine for basic UI access.
However, Coder's internal components (including WebSocket connections and DERP relay) also attempt to use this same URL for their communication.
Our Teleport proxy requires authentication for all requests, which creates a conflict for these internal communications that aren't designed to handle Teleport's authentication.

We've researched CODER_DERP_SERVER_RELAY_URL but understand this is specifically for inter-node communication in high-availability setups, not for separating user access from internal component communication.

Troubleshooting Steps Taken

Confirmed Coder UI access works through Teleport
Added WebSocket headers to the Teleport proxy configuration
Attempted to force WebSocket relay with CODER_FORCE_WEBSOCKET_RELAY=true, but this alone didn't resolve the issue
Checked the health check endpoints that confirm both issues
Verified we can access Coder directly (bypassing Teleport) without these issues

Questions

Is Coder's embedded DERP relay compatible with being accessed through a reverse proxy like Teleport?
What is the recommended configuration for Coder when deployed behind a proxy?
Would disabling the embedded DERP relay (CODER_DERP_SERVER_ENABLED=false) resolve the DERP issue? What functionality would be impacted if we disable it?
Are there specific environment variables we should set for Coder when behind a proxy?
Does the CODER_ACCESS_URL need to be set to our external Teleport URL?
Are there known compatibility issues between reverse proxies and Coder's WebSocket/DERP requirements?
Is there a way to configure Coder to use different URLs for different types of communication? Specifically, can we configure a separate URL for internal component communication that bypasses our authentication proxy?
Are there any undocumented settings that separate the "user access URL" from "internal communication URLs" within Coder?

Requested Assistance

We need guidance on the correct configuration to make Coder work properly when accessed through a proxy like Teleport. Specifically:

Recommended settings or environment variables for proxy deployments
Whether to disable the DERP relay or how to configure it properly
Any way to separate internal communication paths from user-facing paths
Any other configuration needed for WebSocket support through a proxy

We're very limited by the single CODER_ACCESS_URL configuration because our Teleport proxy needs authentication for all requests, which Coder's internal components can't provide. Any solution that allows components to communicate directly while still allowing user access through our proxy would resolve our issues.

Thank you for your assistance.

juliaogris · 2026-04-26T23:40:20Z

juliaogris
Apr 26, 2026

Hi @spiffaz, I work on Teleport app access. I ran into the same two symptoms
while setting up Coder behind Teleport (auth, proxy, and app access), so I
wanted to share what ended up working for me in case it helps.

IIUC what you are hitting is two separate problems, and each needs its own fix.

1. Browser WebSocket upgrades returning Teleport HTML

Coder validates the Origin header on WebSocket upgrades. By default
Teleport rewrites Host to the upstream URI, which trips Coder's
validation. When the upgrade fails, the browser surfaces whatever bytes
end up in the response - in your case what looks like a Teleport proxy
page (the grv_csrf_token meta tag suggests the response came from
Teleport rather than Coder). I have not traced the exact path that
produces those bytes, but adding rewrite.headers so Host and Origin
reach Coder as the public Teleport URL fixed it for me:

app_service:
  apps:
    - name: coder
      uri: http://coder.coder.svc.cluster.local
      public_addr: coder.org.teleport.sh
      rewrite:
        headers:
          - "Host: coder.org.teleport.sh"
          - "Origin: https://coder.org.teleport.sh"

This is the same pattern the Teleport docs recommend for Grafana and
ArgoCD, which have similar WebSocket / Origin
requirements.

2. DERP and agent traffic being redirected to `/web/launch`

Point CODER_ACCESS_URL at an internal address and let the browser
use the public Teleport URL on its own. On a single-VM Docker Compose
setup:

CODER_ACCESS_URL=http://172.17.0.1:3000

172.17.0.1 is the Docker bridge gateway IP, reachable from sibling
workspace containers but not exposed publicly. For Kubernetes, the
ClusterIP DNS name should play the same role, for example
http://coder.coder.svc.cluster.local.

The /web/launch/... 302 you saw in the DERP log is Teleport gating
Coder's internal callers (agent <-> coderd, DERP relay, self-health
probes), which use Coder-native auth that Teleport cannot validate.
Keeping ACCESS_URL inward keeps those on a path Teleport is not in
front of, while the browser continues to hit Teleport via
location.origin.

I have this running as a single-VM Docker Compose demo with both
fixes applied. Browser WebSockets, build log streaming, the
workspace terminal, and the VS Code / Cursor Desktop flows all work
through Teleport; the IDE side uses the mTLS client cert flags that
shipped in the stock CLI in v2.33.0-rc.3.
No VNet, this is plain HTTP app access. Happy to share the
Terraform and Docker Compose files if useful.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Coder WebSocket and DERP Connections Failing When Deployed Behind Teleport Proxy #17036

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Coder WebSocket and DERP Connections Failing When Deployed Behind Teleport Proxy #17036

Uh oh!

spiffaz Mar 21, 2025

Environment

Problem Description

Error Details

WebSocket Error

HTML Response Received (instead of WebSocket upgrade)

DERP Connection Error

DERP Connection Logs

Health Check Failures

Current Coder Configuration

URL Configuration Challenge

Troubleshooting Steps Taken

Questions

Requested Assistance

Replies: 1 comment

Uh oh!

juliaogris Apr 26, 2026

1. Browser WebSocket upgrades returning Teleport HTML

2. DERP and agent traffic being redirected to /web/launch

spiffaz
Mar 21, 2025

juliaogris
Apr 26, 2026

2. DERP and agent traffic being redirected to `/web/launch`