Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[caddy] proxy requests to port locations through ws proxy #4419

Merged
merged 2 commits into from
Jun 9, 2021
Merged

[caddy] proxy requests to port locations through ws proxy #4419

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7b6d137
[caddy] pump port traffic through ws-proxy
akosyakov Jun 9, 2021
de8cb0a
[ws-proxy] correct websocket headers
akosyakov Jun 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitpod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,3 +47,4 @@ vscode:
- ms-kubernetes-tools.vscode-kubernetes-tools
- stkb.rewrap
- zxh404.vscode-proto3
- matthewpi.caddyfile-support
6 changes: 5 additions & 1 deletion components/proxy/conf/Caddyfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,11 @@ https://*.*.{$GITPOD_DOMAIN} {

@workspace_port header_regexp host Host ^(webview-|browser-|extensions-)?(?P<workspacePort>[0-9]{2,5})-(?P<workspaceID>[a-z0-9][0-9a-z\-]+).ws(?P<location>-[a-z0-9]+)?.{$GITPOD_DOMAIN}
handle @workspace_port {
reverse_proxy ws-{re.host.workspaceID}-ports.{$KUBE_NAMESPACE}.{$KUBE_DOMAIN}:{re.host.workspacePort} {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are you removing the mapping for workspace ports?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ws-proxy does it after adjusting headers: https://github.com/gitpod-io/gitpod/pull/4419/files#diff-39f2e37e657f269523bb53adff2f549f05b1af3db45432b89daa1c2068847ef3R269

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as i understood without this change it does not go through ws-proxy at least in prev env?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct.

Just in case, I copied this configuration from what we had in NGINX previously.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in nginx we also delegated to ws-proxy:

gitpod/chart/config/proxy/vhost.server.conf

Lines 160 to 192 in a0a5017

server {
listen {{ $listen }};
# Matches:
# - (webview-|browser-|extensions-)?+ for now, we only support Theia webviews here! (TODO is there a - meaningful - way to generalize this?)
# - (?<port>[0-9]{2,5})- port to forward to
# - (?<wsid>[a-z][0-9a-z\-]+) workspace Id
# - \.ws(-[a-z0-9]+)? workspace base domain
# "" needed because of {} (nginx syntax wart)
server_name "~^(webview-|browser-|extensions-)?+(?<port>[0-9]{2,5})-(?<wsid>[a-z0-9][0-9a-z\-]+)\.ws-{{ .Values.components.proxy.withWsCluster.shortname }}\.${PROXY_DOMAIN_REGEX}$";
{{- if $useHttps }}
include lib.ssl.conf;
include lib.https_redirect.conf;
{{- end }}
# include lib.workspace-port-locations.conf;
include lib.region-headers.conf;
include lib.resolver.conf;
location / {
include lib.proxy.conf;
include lib.ws-sse.conf;
# Increase connect timeout
proxy_connect_timeout 10s;
# Set max body size to make big uploads work
client_max_body_size 2048m;
# disable the error log to not spam our logs when the kube DNS doesn't know about the service yet
error_log off;
proxy_pass http://ws-proxy.{{ .Values.components.proxy.withWsCluster.namespace }}.svc.cluster.local:8080$request_uri;
but moved it to caddy while switching to nginx on purpose. I am not sure.

reverse_proxy https://ws-proxy.{$KUBE_NAMESPACE}.{$KUBE_DOMAIN}:9090 {
transport http {
tls_insecure_skip_verify
}

import upstream_headers

header_up X-Gitpod-WorkspaceId {re.host.workspaceID}
Expand Down
2 changes: 1 addition & 1 deletion components/ws-proxy/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ require (
github.com/golang/mock v1.5.0
github.com/google/go-cmp v0.5.5
github.com/google/tcpproxy v0.0.0-20200125044825-b6bb9b5b8252
github.com/gorilla/handlers v1.4.2
github.com/gorilla/handlers v1.5.1
github.com/gorilla/mux v1.7.4
github.com/prometheus/client_golang v1.9.0
github.com/sirupsen/logrus v1.7.0
Expand Down
4 changes: 4 additions & 0 deletions components/ws-proxy/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.m
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
github.com/felixge/httpsnoop v1.0.1 h1:lvB5Jl89CsZtGIWuTcDM1E/vkVs49/Ml7JJe07l8SPQ=
github.com/felixge/httpsnoop v1.0.1/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/fogleman/gg v1.2.1-0.20190220221249-0403632d5b90/go.mod h1:R/bRT+9gY/C5z7JzPU0zXsXHKM4/ayA+zqcVNZzPa1k=
github.com/franela/goblin v0.0.0-20200105215937-c9ffbefa60db/go.mod h1:7dvUGVsVBjqR7JHJk0brhHOZYGmfBYOrK0ZhYMEtBr4=
github.com/franela/goreq v0.0.0-20171204163338-bcd34c9993f8/go.mod h1:ZhphrRTfi2rbfLwlschooIH4+wKKDR4Pdxhh+TRoA20=
Expand Down Expand Up @@ -154,6 +156,8 @@ github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORR
github.com/gorilla/context v1.1.1/go.mod h1:kBGZzfjB9CEq2AlWe17Uuf7NDRt0dE0s8S51q0aT7Yg=
github.com/gorilla/handlers v1.4.2 h1:0QniY0USkHQ1RGCLfKxeNHK9bkDHGRYGNDFBCS+YARg=
github.com/gorilla/handlers v1.4.2/go.mod h1:Qkdc/uu4tH4g6mTK6auzZ766c4CA0Ng8+o/OAirnOIQ=
github.com/gorilla/handlers v1.5.1 h1:9lRY6j8DEeeBT10CvO9hGW0gmky0BprnvDI5vfhUHH4=
github.com/gorilla/handlers v1.5.1/go.mod h1:t8XrUpc4KVXb7HGyJ4/cEnwQiaxrX/hz1Zv/4g96P1Q=
github.com/gorilla/mux v1.6.2/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs=
github.com/gorilla/mux v1.7.3/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs=
github.com/gorilla/mux v1.7.4 h1:VuZ8uybHlWmqV03+zRzdwKL4tUnIp1MAQtp1mIFE1bc=
Expand Down
8 changes: 8 additions & 0 deletions components/ws-proxy/pkg/proxy/routes.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,14 @@ func installWorkspacePortRoutes(r *mux.Router, config *RouteHandlerConfig) error
// forward request to workspace port
r.NewRoute().HandlerFunc(
func(rw http.ResponseWriter, r *http.Request) {
// a work-around for servers which does not respect case-insensitive headers, see https://github.com/gitpod-io/gitpod/issues/4047#issuecomment-856566526
for _, name := range []string{"Key", "Extensions", "Accept", "Protocol", "Version"} {
values := r.Header["Sec-Websocket-"+name]
if len(values) != 0 {
r.Header.Del("Sec-Websocket-" + name)
r.Header["Sec-WebSocket-"+name] = values
}
}
r.Header.Add("X-Forwarded-Proto", "https")
r.Header.Add("X-Forwarded-Host", r.Host+":443")
proxyPass(
Expand Down
20 changes: 20 additions & 0 deletions components/ws-proxy/pkg/proxy/routes_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,7 @@ func TestRoutes(t *testing.T) {
"Location": {
"https://blobserve.ws.test-domain.com/gitpod-io/supervisor:latest/__files__/favicon.ico",
},
"Vary": {"Accept-Encoding"},
},
Body: "<a href=\"https://blobserve.ws.test-domain.com/gitpod-io/supervisor:latest/__files__/favicon.ico\">See Other</a>.\n\n",
},
Expand All @@ -222,6 +223,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Type": {"text/html; charset=utf-8"},
"Location": {"https://blobserve.ws.test-domain.com/gitpod-io/ide:latest/__files__/"},
"Vary": {"Accept-Encoding"},
},
Body: "<a href=\"https://blobserve.ws.test-domain.com/gitpod-io/ide:latest/__files__/\">See Other</a>.\n\n",
},
Expand All @@ -238,6 +240,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Length": {"38"},
"Content-Type": {"text/plain; charset=utf-8"},
"Vary": {"Accept-Encoding"},
},
Body: "blobserve hit: /gitpod-io/ide:latest/\n",
},
Expand All @@ -254,6 +257,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Length": {"38"},
"Content-Type": {"text/plain; charset=utf-8"},
"Vary": {"Accept-Encoding"},
},
Body: "blobserve hit: /gitpod-io/ide:latest/\n",
},
Expand All @@ -271,6 +275,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Length": {"24"},
"Content-Type": {"text/plain; charset=utf-8"},
"Vary": {"Accept-Encoding"},
},
Body: "workspace hit: /?foobar\n",
},
Expand All @@ -288,6 +293,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Length": {"35"},
"Content-Type": {"text/plain; charset=utf-8"},
"Vary": {"Accept-Encoding"},
},
Body: "workspace hit: /not-from-blobserve\n",
},
Expand All @@ -305,6 +311,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Length": {"42"},
"Content-Type": {"text/plain; charset=utf-8"},
"Vary": {"Accept-Encoding"},
},
Body: "workspace hit: /not-from-failed-blobserve\n",
},
Expand All @@ -327,6 +334,7 @@ func TestRoutes(t *testing.T) {
"Access-Control-Expose-Headers": {"Authorization"},
"Content-Length": {"37"},
"Content-Type": {"text/plain; charset=utf-8"},
"Vary": {"Accept-Encoding"},
},
Body: "workspace hit: /somewhere/in/the/ide\n",
},
Expand All @@ -341,6 +349,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Length": {"50"},
"Content-Type": {"text/plain; charset=utf-8"},
"Vary": {"Accept-Encoding"},
},
Body: "supervisor hit: /_supervisor/v1/status/supervisor\n",
},
Expand All @@ -355,6 +364,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Length": {"43"},
"Content-Type": {"text/plain; charset=utf-8"},
"Vary": {"Accept-Encoding"},
},
Body: "supervisor hit: /_supervisor/v1/status/ide\n",
},
Expand All @@ -366,6 +376,9 @@ func TestRoutes(t *testing.T) {
),
Expectation: Expectation{
Status: http.StatusUnauthorized,
Header: http.Header{
"Vary": {"Accept-Encoding"},
},
},
},
{
Expand All @@ -379,6 +392,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Length": {"47"},
"Content-Type": {"text/plain; charset=utf-8"},
"Vary": {"Accept-Encoding"},
},
Body: "supervisor hit: /_supervisor/v1/status/content\n",
},
Expand All @@ -394,6 +408,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Type": {"text/html; charset=utf-8"},
"Location": {"https://test-domain.com/start/#blabla-smelt-9ba20cc1"},
"Vary": {"Accept-Encoding"},
},
Body: ("<a href=\"https://test-domain.com/start/#blabla-smelt-9ba20cc1\">Found</a>.\n\n"),
},
Expand All @@ -408,6 +423,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Type": {"text/html; charset=utf-8"},
"Location": {"https://test-domain.com/start/#blabla-smelt-9ba20cc1"},
"Vary": {"Accept-Encoding"},
},
Body: ("<a href=\"https://test-domain.com/start/#blabla-smelt-9ba20cc1\">Found</a>.\n\n"),
},
Expand All @@ -423,6 +439,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Length": {"60"},
"Content-Type": {"text/plain; charset=utf-8"},
"Vary": {"Accept-Encoding"},
},
Body: "blobserve hit: /gitpod-io/supervisor:latest/worker-proxy.js\n",
},
Expand All @@ -438,6 +455,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Type": {"text/html; charset=utf-8"},
"Location": {"https://blobserve.ws.test-domain.com/gitpod-io/supervisor:latest/__files__/main.js"},
"Vary": {"Accept-Encoding"},
},
Body: "<a href=\"https://blobserve.ws.test-domain.com/gitpod-io/supervisor:latest/__files__/main.js\">See Other</a>.\n\n",
},
Expand All @@ -463,6 +481,7 @@ func TestRoutes(t *testing.T) {
Header: http.Header{
"Content-Type": {"text/html; charset=utf-8"},
"Location": {"https://blobserve.ws.test-domain.com/gitpod-io/supervisor:latest/__files__/main.js"},
"Vary": {"Accept-Encoding"},
},
Body: "<a href=\"https://blobserve.ws.test-domain.com/gitpod-io/supervisor:latest/__files__/main.js\">See Other</a>.\n\n",
},
Expand Down Expand Up @@ -542,6 +561,7 @@ func TestRoutes(t *testing.T) {
"Cache-Control": {"public, max-age=31536000"},
"Content-Length": {"62"},
"Content-Type": {"text/plain; charset=utf-8"},
"Vary": {"Accept-Encoding"},
},
Status: http.StatusOK,
Body: "blobserve hit: /blobserve/gitpod-io/supervisor:latest/main.js\n",
Expand Down