Websocket establishment does not work with Supermicro X9 to X11 HTML5 console #3935

bluecmd · 2023-02-02T13:04:19Z

What happened?

I am configuring Pomerium to front the IPMI/BMC HTML5 console for a couple of Supermicro servers.
Firmware Revision : 03.94

What did you expect to happen?

When I use the console it can connect. If I copy Websocket request from Firefox as cURL and manually run it, I expect it to print the RFB/VNC data channel.

How'd it happen?

Using the web interface, activating HTML5 fails with "Server disconnected (code: 1006)".

Also with cURL:

Ran:

curl 'https://foobar.my.domain/' \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0' \
-H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Sec-WebSocket-Version: 13' \
-H 'Origin: https://foobar.my.domain' \
-H 'Sec-WebSocket-Extensions: permessage-deflate' \
-H 'Sec-WebSocket-Key: redacted==' \
-H 'Connection: keep-alive, Upgrade' \
-H 'Cookie: _pomerium=redacted.redacted.redacted; langSetFlag=0; language=English; SID=redacted; mainpage=remote; subpage=man_ikvm_html5' \
-H 'Sec-Fetch-Dest: websocket' \
-H 'Sec-Fetch-Mode: websocket' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Pragma: no-cache' \
-H 'Cache-Control: no-cache' \
-H 'Upgrade: websocket'  \
-vvvv

I receive HTTP/2 200 and the index page, and not the expected 101 Switching Protocols and the data channel.

What's your environment like?

pomerium: 0.21.0-rc1-1674048966+e8004bbb
envoy: 1.24.0+05e1343657086268edaa16a1e8a1429d9c8f6e50a44f492cf629fdc71da2ecdc

Hardware: SYS-2028TP-HTR, BMC version 03.94 (latest as of this writing)

What's your config.yaml?

  - from: https://foorbar.my.domain
    to: https://10.199.100.248
    tls_skip_verify: true
    allow_websockets: true
    allowed_users:
       - ...

What did you see in the logs?

Nothing that stands out

However, I noticed that if I inspect the headers that are sent from Pomerium to the backend it appears that the 'Upgrade: websocket' and 'Conneciton: upgrade' headers are not set as I would expect.

Additional context

Proxying this service works just fine with nginx instead of Pomerium using the following configuration:

  location / {
    proxy_set_header Host $http_host;
    proxy_set_header Upgrade $mapped_http_upgrade;
    proxy_set_header Connection $mapped_connection_upgrade;
    proxy_pass https://10.199.100.248;
    proxy_http_version 1.1;
    client_max_body_size 2G;
  }

Since it works fine with nginx I believe this to be a bug in Pomerium and/or Envoy.

The text was updated successfully, but these errors were encountered:

calebdoxsey · 2023-02-08T20:43:50Z

I was not able to reproduce this issue. Testing with a simple websocket server and client its working for me:

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"os"

	"github.com/gorilla/websocket"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/ws", func(w http.ResponseWriter, r *http.Request) {
		ws, err := (&websocket.Upgrader{}).Upgrade(w, r, nil)
		if err != nil {
			log.Println(err)
			return
		}
		defer ws.Close()
		for {
			// Write
			err := ws.WriteMessage(websocket.TextMessage, []byte("Hello, Client!"))
			if err != nil {
				return
			}

			// Read
			_, msg, err := ws.ReadMessage()
			if err != nil {
				return
			}
			fmt.Printf("%s\n", msg)
		}
	})
	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		_, _ = io.WriteString(w, `<!doctype html>
<html lang="en">
<body>
	<p id="output"></p>

	<script>
	var loc = window.location;
	var uri = 'ws:';

	if (loc.protocol === 'https:') {
		uri = 'wss:';
	}
	uri += '//' + loc.host;
	uri += loc.pathname + 'ws';

	ws = new WebSocket(uri)

	ws.onopen = function() {
		console.log('Connected')
	}

	ws.onmessage = function(evt) {
		var out = document.getElementById('output');
		out.innerHTML += evt.data + '<br>';
	}

	setInterval(function() {
		ws.send('Hello, Server!');
	}, 1000);
	</script>
</body>

</html>
`)
	})
	err := http.ListenAndServeTLS("127.0.0.1:20001", "_wildcard.localhost.pomerium.io.pem", "_wildcard.localhost.pomerium.io-key.pem", mux)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%v\n", err)
		os.Exit(1)
	}
}

I am not familiar with Supermicro X9, X11 or IPMI/BMC. Perhaps they use a variation of Websockets that's different from the example I'm using.

Based on the error message above (which doesn't happen for me), it sounds like HTTP/2 is being used. Could you try forcing the codec type to http1? https://www.pomerium.com/docs/reference/codec-type

bluecmd · 2023-02-09T07:28:59Z

No change with codec_type: http1.

I replaced the target with a openssl s_server -key /etc/ssl/private/ssl-cert-snakeoil.key -cert /etc/ssl/certs/ssl-cert-snakeoil.pem -accept 8880 process to dump the headers that Pomerium sends to the backend when the websocket is established. This is the result:

GET / HTTP/1.1
host: xxxx
user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0
accept: */*
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br
sec-websocket-version: 13
origin: https://xxxx
sec-websocket-extensions: permessage-deflate
sec-websocket-key: xxxx==
sec-fetch-dest: websocket
sec-fetch-mode: websocket
sec-fetch-site: same-origin
pragma: no-cache
cache-control: no-cache
x-forwarded-for: 2a07:5xxx
x-forwarded-proto: https
x-envoy-external-address: 2a07:5xxx
x-request-id: d5dad481-7042-45eb-aedf-0ff8c1b693b2
cookie: SID=xxxx

Shouldn't Pomerium include Upgrade: websocket and Connection: upgrade headers?

EDIT: These headers are included if the client sends a HTTP1.1 (as opposed to the the default HTTP/2) request it seems:

GET / HTTP/1.1
host: xxx
user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0
accept: */*
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br
sec-websocket-version: 13
origin: https://xxxx
sec-websocket-extensions: permessage-deflate
sec-websocket-key: xxx==
connection: keep-alive, Upgrade
sec-fetch-dest: websocket
sec-fetch-mode: websocket
sec-fetch-site: same-origin
pragma: no-cache
cache-control: no-cache
upgrade: websocket
x-forwarded-for: xxxx
x-forwarded-proto: https
x-envoy-external-address: 2a07:xxxxx
x-request-id: xxxx
cookie: SID=xxxx

bluecmd · 2023-02-09T08:09:24Z

Ok, so I spent some more time digging into the particulars for the Supermicro BMC console application.

For it to be happy to accept a websocket connection the request needs to:

Have CamelCase capitalization on the header names (!!)
Include Upgrade and Connection headers.

This is the minimum request that I have managed to get working:

 curl --http1.1 'https:/10.x.y.z/' -k  -H 'Sec-WebSocket-Version: 13' -H 'Origin: https://foobar' -H 'Sec-WebSocket-Key: something' -H 'Connection: keep-alive, Upgrade'  -H 'Upgrade: websocket'

If I change e.g. Upgrade to upgrade the request is rejected.

calebdoxsey · 2023-02-09T14:28:03Z

Envoy's default behavior is to convert all headers to lowercase. It looks like there is a way we can change this when using websockets which may fix the issue: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/header_casing#config-http-conn-man-header-casing.

bluecmd · 2023-02-18T15:09:30Z

I tested the main branch just now with the #3956 PR merged, and can happily confirm it appears to be working!

The configuration that works for me:

  - from: https://xyz-bmc.local.my.domain
    to: https://10.1.2.3
    tls_skip_verify: true
    allow_websockets: true
    allowed_users: *bmc_admins

🥳

desimone added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. bug Something isn't working labels Feb 2, 2023

calebdoxsey self-assigned this Feb 8, 2023

calebdoxsey removed their assignment Feb 8, 2023

calebdoxsey added the blocked PR/ISSUE is blocked by third party label Feb 8, 2023

calebdoxsey added bug Something isn't working and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsMoreData Waiting for additional user feedback or case studies labels Feb 9, 2023

calebdoxsey self-assigned this Feb 10, 2023

calebdoxsey mentioned this issue Feb 10, 2023

envoyconfig: preserve case of HTTP headers when using HTTP/1 #3956

Merged

4 tasks

desimone closed this as completed Feb 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Websocket establishment does not work with Supermicro X9 to X11 HTML5 console #3935

Websocket establishment does not work with Supermicro X9 to X11 HTML5 console #3935

bluecmd commented Feb 2, 2023 •

edited

calebdoxsey commented Feb 8, 2023

bluecmd commented Feb 9, 2023 •

edited

bluecmd commented Feb 9, 2023

calebdoxsey commented Feb 9, 2023

bluecmd commented Feb 18, 2023

Websocket establishment does not work with Supermicro X9 to X11 HTML5 console #3935

Websocket establishment does not work with Supermicro X9 to X11 HTML5 console #3935

Comments

bluecmd commented Feb 2, 2023 • edited

What happened?

What did you expect to happen?

How'd it happen?

What's your environment like?

What's your config.yaml?

What did you see in the logs?

Additional context

calebdoxsey commented Feb 8, 2023

bluecmd commented Feb 9, 2023 • edited

bluecmd commented Feb 9, 2023

calebdoxsey commented Feb 9, 2023

bluecmd commented Feb 18, 2023

bluecmd commented Feb 2, 2023 •

edited

bluecmd commented Feb 9, 2023 •

edited