🐛 [Kasm] 400 Bad Request when exposing via RunTipi.io using Cloudflare tunnel

[jasonpearce]

Store Application

Kasm Workspaces

App version

1.120.20221218

Description

Describe the bug
When accessing a new Kasm installation on a RunTipi.io server via a Cloudflare tunnel, the browser response is:

400 Bad Request
The plain HTTP request was sent to HTTPS port
nginx

Expected behavior
I'd like to be able to access https://kasm.example.com, running as a RunTipi.io app, via a Cloudflare tunnel remotely/externally.

Screenshots

Workspaces Version
Latest Ubuntu Server, RunTipi.io, and Kasm app

Workspaces Installation Method
Physical Lenovo Tiny PC, Ubuntu Server 22.04, Docker, RunTipi.io, Cloudflare Tunnel, Kasm App via the RunTipi app store

Client Browser (please complete the following information):

• OS: Windows 11
• Browser Firefox 122

Workspace Server Information (please provide the output of the following commands):

• uname -a
• cat /etc/os-release
• sudo docker info
• sudo docker ps | grep kasm

Additional context
I can use Cloudflare tunnels to externally expose other apps installed from the RunTipi.io app store. I've done so for about eight apps. This issue occurs only within the Kasm app. The Kasm app works fine inside my home network via IP address and port. Thank you.

Steps to reproduce

To Reproduce
Steps to reproduce the behavior:

1. Build a new Ubuntu 22.04 server on a refubished PC at home behind your firewall
2. Install RunTipi (https://runtipi.io/docs/getting-started/installation)
3. Install app Hello World as a test baseline (https://runtipi.io/docs/apps-available)
4. Expose the Hello World app using Cloudflare tunnels (https://runtipi.io/docs/guides/expose-apps-with-cloudflare-tunnels)
5. Validate that https://helloworld.example.com works externally via the Cloudflare tunnel (it does)
6. Install app Kasm Workspaces (https://runtipi.io/docs/apps-available)
7. Validate that https://:port works locally (it does)
8. Expose the Kasm app using Cloudflare tunnels (https://runtipi.io/docs/guides/expose-apps-with-cloudflare-tunnels)
9. Validate that https://kasm.example.com works externally via the Cloudflare tunnel (error 400 bad request)

App logs

I'm not finding a way to view logs in the Tipi dashboard.

Browser

Browser Agnostic

Browser logs

No response

User-Config changes

No changes.

Other

Posted this on the Kasm github:
kasmtech/workspaces-issues#509

One response was...

The error is indicating that the system is trying to access (presumably the kasm server) over HTTP instead of HTTPS , so somewhere in your stack you need to ensure your reverse proxy style system (Cloudflare Tunnel) is proxying to Kasm via HTTPS and not HTTP. I'm not familiar with Tipi so check there as well

Please confirm the following

• I believe this issue is a bug that affects all users of RunTipi, not something specific to my installation.
• I have already searched for relevant existing issues and discussions before opening this report.
• I have updated the title field above with a concise description.

[steveiliop56]

Hello @jasonpearce,

Can you please make sure this is set to https?

[jasonpearce]

Thank you for responding @steveiliop56. It is. Here are my Public Hostname settings in Cloudflare (with some information obscured):

Basic Information
Public hostname: (https://kasm.example.com/)
Path: *
Service: https://192.168.xxx.xxx
Origin configurations:

• http2Origin:
• noTLSVerify:

[steveiliop56]

Hello @jasonpearce,

Can you confirm that you can access kasm normally via the ip address and port?

[jasonpearce]

Yes. Locally I can access Kasm via https://:port.

I can also confirm that four other Cloudflare access tunnels to other Runtipi apps work internally and externally. The only difference is that Kasm is the only one using the Cloudflare origin configuration of "http2Origin" in addition to the "noTLSVerify." All others use only "noTLSVerify." If I remove "http2Origin" from the Cloudflare setting for Kasm, I do not observe a difference.

This weekend, I'll attempt to capture and provide ample screen shots if that would be helpful. Again, thank you for assisting.

[jasonpearce]

Here are some screenshots I said I would provide. I hope they are helpful.

Cloudflare settings

Tipi on local network

Working Hello World Settings for LAN and WAN

Working Kasm Settings for LAN

Not Working Kasm Setting for WAN

Closing

Please let me know what additional information I can provide to help you or others identify if this is only an issue on my end or if this is a bug. I did use Tipi to uninstall and reinstall the Kasm app (same results).

[meienberger]

It seems there is an issue on how the tipi reverse-proxy operates and forwards the request to kasm. I cannot find anything useful in the linuxserver environments that could help with it. Probably some tweaking with the nginx headers could help. I will do some testing

[jasonpearce]

This weekend, I upgraded from Tipi v2.5.x to v3.0.3. After doing so, Kasm Workspaces 1.120.20221218 no longer worked via the local IP address. A few other apps also stopped working. I installed some new apps, and some of them would work, others would not.

Worked by local IP before and after upgrade:
Cloudflared, Hello World, IT-Tools, Jellyfin, Linkwarden, SearXNG, Stirling-PDF, ViewTube

Worked by local IP before upgrade, but 400 Bad Request after upgrade:
Kasm Workspaces

Installed after upgrade and works by local IP:
Nextcloud, Uptime Kuma

Installed after upgrade, but 400 Bad Request:
Netboot.xyz, Tailscale

My priority is to have Kasm Workspaces working via local IP and via a Cloudflared tunnel. To provide you some more information, I built a new virtual machine to do some testing.

Ubuntu 22.04.04 LTS Desktop:
Built a new virtual machine. Fully patched it. Rebooted.

Runtipi v3.0.3:
Installed Tipi v 3.0.3. Can always access via local IP. This always worked.

Hello World vLatest:
Installed Hello World. This always worked via local IP.

Rebooted.

Kasm Workspaces v1.120.20221218:
Installed Kasm Workspaces. In Kasm, installed some virtual browsers (Brave, Chromium, Edge, Firefox, Tor). Tested them all. They all worked via local IP.

Rebooted.

Kasm Workspaces:
After reboot, I tested them all again via Local IP. Everything worked.

Cloudflared v2024.2.1:
Installed Cloudflared. Went to cloudflare.com and successfully setup a new Connector to my new VM. Both cloudflare.com and Cloudflared looked good (connected, healthy).

Rebooted.

Kasm 400 Bad Request:
After installing Cloudflared and rebooting, Kasm stopped working via an IP address. All attempts resulted in error "400 Bad Request". My attempts:

• Stop Cloudflared
• Restart Kasm (still error 400)
• Stop all apps but Hello World (my baseline test app, always worked)
• Reboot
• After reboot, only Hello World is running
• Start Kasm (still error 400)
• Stop Kasm
• Uninstall Cloudflared
• Reboot
• Start Kasm (still error 400)
• Stop Kasm
• Uninstall Kasm
• Only Hello World remains (still working)
• Reboot
• Reinstall Kasm v1.120.20221218
• Within Kasm, reinstall some browsers for testing via port IP:8743 (setup interface, required first step)
• Stop Kasm
• Reboot
• Start Kasm to again attempt to visit IP:8744 (user interface)
• Kasm works via IP:8744
• Stop Kasm
• Start Kasm again and test again via IP:8744 (error 400 returns)
• Stop Kasm
• Reboot
• Start Kasm
• Test Kasm via IP:8744 (error 400 remains)
• Note that while IP:8744 provides an error 400, IP:8743 (the setup Kasm port) continues to work just fine

Summary:
With Tipi v3.0.3, once Cloudflared is installed I can no longer get Kasm Workspaces to work via a local IP address -- even after Stopping or Uninstalling Cloudflared. Only after reinstalling Kasm Workspaces can I get Kasm to work again, but it will work only until it is stopped or the server reboots. After 1 stop or 1 reboot, Kasm will no longer work by local IP again. Some other apps also appear to have issues via the local IP address if Cloudflared is/was installed (Tailscale, Netboot.xyz); but I did not test them as extensively.

I hope some of this information helps you troubleshoot and is not wasting your time.

[jasonpearce]

I'm not experienced enough to help solve this problem. But in an attempt to help, I asked a LLM to review the code on the Kasm repositories and suggest some changes that might fix the issue. Here's what one AI proposed. I understand this may very well be misleading or incorrect.

LLM AI:

After reviewing the Kasm repositories, I found a potential cause for the issue. The problem might be related to the NGINX reverse proxy configuration in the Kasm Workspaces source code. Specifically, the nginx.conf file in the kasm-workspaces/src/kasm_deployments/nginx/ directory.

To fix the issue, you can try the following steps:

1. Modify the nginx.conf file to include a new server block that listens on the HTTP protocol and redirects to HTTPS. This will ensure that all requests are using HTTPS, which might be the root cause of the 400 Bad Request error.

Add the following server block within the http block in the nginx.conf file:

server {
    listen 80;
    server_name kasm.example.com;
    return 301 https://$host$request_uri;
}

2. Make sure that the nginx.conf file includes the http2 directive in the listen directive for the HTTPS server block. This ensures that the connection uses HTTP/2, which is required for Cloudflare tunnels.

Change the existing HTTPS server block to:

server {
    listen 443 ssl http2;
    server_name kasm.example.com;

    # ... (other configurations)
}

3. After making these changes, rebuild the Docker image and restart the Kasm app to apply the modifications.

Please note that these changes are based on the assumption that the issue is related to HTTP/HTTPS configuration. If the problem persists, further investigation might be required to identify other potential causes.

If you need more assistance or have any questions, please let me know.

[jasonpearce]

After upgrading to...

• RunTipi v3.1.3
• Cloudflared v2024.3.0
• Kasm Workspaces v1.120.20221218

My results are unchanged:

• Hello World still works by local IP address and Cloudflare Tunnel
• Kasm Workspace is an error "400 Bad Request" by local IP address (http)
• Kasm Workspace works via local IP address (https)
• Kasm Workspace is an error "400 Bad Request" by Cloudflare Tunnel (https)

[steveiliop56]

I might be able to solve it by fixing some labels. But if that doesn't work I unfortunately will have to disable the expose feature.

[jasonpearce]

Ok. I understand. I agree. If the bug/conflict with Cloudflared cannot be resolved, then removing the expose feature would be the best option to "resolve" this bug. Thank you.

[steveiliop56]

So I unfortunately cannot solve the traefik issue so the best solution for now is to completely disable traefik on kasm because neither local domains nor exposed work (I get the same issue as you) #3048