[Bug]: Terminal not accessible

[dickermoshe]

Error Message and Logs

I've installed coolify on a spare laptop.
I used Cloudflare tunnels to access it remotely.
I set the correct environmental vars so that the the realtime and terminal api should work on the frontend.

REALTIME websockets work, But TERMINAL websockets do NOT!

Steps to Reproduce

1. Install Coolify on a Server
2. Navigate to Terminal Page
   ERROR: Terminal websocket connection lost.

Ok, let's use curl on localhost to see what's going on:

root@oldlaptop:/home/mick# curl localhost:6002/ready
OK

Seems like the server for the realtime API is running, that's good

Let's get those logs:

root@oldlaptop:/hodocker logs 4bb1497b23c9 -tv
2024-12-15T02:28:25.311313459Z 2024-12-15 02:28:25 [TERMINAL] Coolify realtime terminal server listening on port 6002. Let the hacking begin!
2024-12-15T02:28:25.576541725Z 2024-12-15 02:28:25 [SOKETI]
2024-12-15T02:28:25.584539981Z 2024-12-15 02:28:25 [SOKETI] 🕵️‍♂️ Initiating metrics endpoints...
2024-12-15T02:28:25.585333414Z 2024-12-15 02:28:25 [SOKETI]
2024-12-15T02:28:25.586187631Z 2024-12-15 02:28:25 [SOKETI] 🎉 Server is up and running!
2024-12-15T02:28:25.587015263Z 2024-12-15 02:28:25 [SOKETI] 📡 The Websockets server is available at 127.0.0.1:6001
2024-12-15T02:28:25.587567900Z 2024-12-15 02:28:25 [SOKETI] 🔗 The HTTP API server is available at http://127.0.0.1:6001
2024-12-15T02:28:25.588409286Z 2024-12-15 02:28:25 [SOKETI] 🎊 The /usage endpoint is available on port 9601.
2024-12-15T02:28:25.588987941Z 2024-12-15 02:28:25 [SOKETI]
2024-12-15T02:28:25.589690367Z 2024-12-15 02:28:25 [SOKETI] (node:8) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-15T02:28:25.590229087Z 2024-12-15 02:28:25 [SOKETI]
2024-12-15T02:28:25.590810387Z 2024-12-15 02:28:25 [SOKETI] Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-15T02:28:25.591389923Z 2024-12-15 02:28:25 [SOKETI] For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-15T02:28:25.592023949Z 2024-12-15 02:28:25 [SOKETI] (Use `node --trace-warnings ...` to show where the warning was created)

Hmmm, looks like everything is ok.
Let's try copying the request from chrome and seeing if it works from the CLI

We can use a tool like websocat to test if this works locally:

root@oldlaptop:/home/mick# websocat 'ws://localhost:6002'   -H 'Upgrade: websocket'   -H 'Origin: https://coolify.dickersystems.com'   -H 'Cache-Control: no-cache'   -H 'Accept-Language: en-US,en;q=0.9'   -H 'Pragma: no-cache'   -H 'Connection: Upgrade'   -H 'Sec-WebSocket-Key: RB72t/uvO8ejOMLhMQljUA=='   -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'   -H 'Sec-WebSocket-Version: 13'   -H 'Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits'
websocat: WebSocketError: WebSocketError: Received unexpected status code (400 Bad Request)
websocat: error running

Hmmm, it doesn't. No logs in docker logs either. Seems like we need better logging!!

We can confirm that copything from curl SHOULD work by copying the request from the realtime websocket that is working.

root@oldlaptop:/home/mick# websocat 'ws://localhost:6001/app/REDACTED?protocol=7&client=js&version=8.3.0&flash=false'   -H 'Upgrade: websocket'   -H 'Origin: https://coolify.dickersystems.com'   -H 'Cache-Control: no-cache'   -H 'Accept-Language: en-US,en;q=0.9'   -H 'Pragma: no-cache'   -H 'Connection: Upgrade'   -H 'Sec-WebSocket-Key: +has5XTnkQrT1iRzO6Lx3g=='   -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'   -H 'Sec-WebSocket-Version: 13'   -H 'Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits'
{"event":"pusher:connection_established","data":"{\"socket_id\":\"REDACTED\",\"activity_timeout\":30}"}

Example Repository URL

No response

Coolify Version

v4.0.0-beta.379

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Ubuntu 24.04.1 LTS

Additional Information

No response

[ricky-cc]

+1 got the same issue with custom ssh port

[gitsacha]

+1, I've been able to fix the ws connection error for the realtime service using this thread but I wasn't able to find a solution for the terminal ws.

Using Cloudflare Tunnels (cloudflared) with the following config on an ec2:

credentials-file: /root/.cloudflared/_TUNNEL_ID_.json
ingress:
  - hostname: "ssh.mydomain.com"
    service: ssh://localhost:22
  - hostname: "realtime.mydomain.com"
    service: http://localhost:6001
  - hostname: "coolify.mydomain.com"
    service: http://localhost:8000
  - hostname: "*.mydomain.com"
    service: http://localhost:80
  - service: http_status:404

[KwaminaWhyte]

Having same issue

[nvo87]

Hi! I've solved these issue by setting path to /terminal/ws in cloudflare tunnel.

We can see that browser tries to request wss://yourdomain.com/termial/ws

So we need to connect it with 6002 port of coolify-realtime container

P.S.
I think the instruction terminal.coolify.io mapped to localhost:6002 from documentation is misleading. To make requests to terminal.coolify.io you need to add TERMINAL_HOST=terminal.coolify.io to /data/coolify/source/.env. But this will take longer and won't work anyway, because /terminal/ws endpoint is bound to app.coolify.io.

[rber474]

Hi!

Had the same issue and applied @nvo87 solution. Now it is working.
Please, note host order is important, the host <yourdomain.ext>/terminal/ws should be ordered before the host pointing to port 8000.

[gitsacha]

Thanks! It worked for me too.

For those using cloudflared with a config file directly on the server (typically at /etc/cloudflared/config.yml or ~/.cloudflared/config.yml), here's the fix:

tunnel: YOUR_TUNNEL_ID
credentials-file: /root/.cloudflared/YOUR_TUNNEL_ID.json
ingress:
  - hostname: "ssh.yourdomain.com"
    service: ssh://localhost:22
  - hostname: "realtime.yourdomain.com"
    service: http://localhost:6001
  # FIX HERE
  - hostname: "coolify.yourdomain.com"
    service: http://localhost:6002
    path: "/terminal/ws"
  - hostname: "coolify.yourdomain.com"
    service: http://localhost:8000
  - hostname: "*.yourdomain.com"
    service: http://localhost:80
  - service: http_status:404

[cesarhfborges]

Hello, I'm having the same problem, the socket is not working, neither using a local ip nor using a domain, it doesn't connect to port 6001 or 6002, I can't say if it happened after I updated to version v4.0.0-beta.390 but it was working both via local ip and domain but I needed to connect to a certain container and that's when I noticed.

I've already tried to point cloudflared directly to the address + port or just to the address without a port, anyway, but it seems to me something happened with the coolify socket service and I believe that it itself is not responding.