Unable to establish websocket connection when using a reverse proxy (selfhosted setup)

[czouche]

Bug report

Describe the bug

I'm trying to establish a realtime subscription with a self-hosted Supabase instance through Caddy as a reverse proxy but I'm getting the following error:

WebSocket connection to 'wss://db.mydomain.com/realtime/v1/websocket?apikey=my.jwt.key&vsn=1.0.0' failed

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

1. Create a self-hosted setup as described in the Supabase docs.
2. Setup Caddy as a reverse proxy

https://db.mydomain.com {
        reverse_proxy /rest/v1/* localhost:8000
        reverse_proxy /auth/v1/* localhost:8000
        reverse_proxy /realtime/v1/* localhost:8000
        reverse_proxy /storage/v1/* localhost:8000
        reverse_proxy * localhost:3000
}

4. Create a test table and setup a subscription to it.

const supabase = createClient(
  "https://db.mydomain.com",
  "my.jwt.key"
);

supabase
    .channel("*")
    .on(
      "postgres_changes",
      {
        event: "*",
        schema: "public",
        table: "test_table",
      },
      (payload) => {
        console.log(payload)
      }
    )
    .subscribe();

Expected behavior

I should be able to establish a websocket connection and listen to changes in the table, just like the SaaS version of Supabase.

System information

• OS: Linux
• Browser (if applies) Firefox
• Version of supabase-js: 2.7.0
• Version of Node.js: 18

Additional context

• I can insert records with supabase-js with no issues using this setup.
• Kong container logs:

supabase-kong                   | 172.18.0.1 - - [03/Feb/2023:17:04:51 +0000] "GET /realtime/v1/websocket?apikey=my.jwt.key&vsn=1.0.0 HTTP/1.1" 500 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"

• Realtime container logs:

realtime-dev.supabase-realtime  | Request: GET /socket/websocket?apikey=my.jwt.key&vsn=1.0.0
realtime-dev.supabase-realtime  | ** (exit) an exception was raised:
realtime-dev.supabase-realtime  |     ** (MatchError) no match of right hand side value: {:error, :tenant_not_found_in_host}
realtime-dev.supabase-realtime  |         (realtime 2.0.0) lib/realtime_web/channels/user_socket.ex:22: RealtimeWeb.UserSocket.connect/3
realtime-dev.supabase-realtime  |         (phoenix 1.6.13) lib/phoenix/socket.ex:550: Phoenix.Socket.user_connect/6
realtime-dev.supabase-realtime  |         (phoenix 1.6.13) lib/phoenix/socket.ex:438: Phoenix.Socket.__connect__/3
realtime-dev.supabase-realtime  |         (phoenix 1.6.13) lib/phoenix/transports/websocket.ex:33: Phoenix.Transports.WebSocket.connect/4
realtime-dev.supabase-realtime  |         (phoenix 1.6.13) lib/phoenix/endpoint/cowboy2_handler.ex:22: Phoenix.Endpoint.Cowboy2Handler.init/4
realtime-dev.supabase-realtime  |         (cowboy 2.9.0) /app/deps/cowboy/src/cowboy_handler.erl:37: :cowboy_handler.execute/2
realtime-dev.supabase-realtime  |         (cowboy 2.9.0) /app/deps/cowboy/src/cowboy_stream_h.erl:306: :cowboy_stream_h.execute/3
realtime-dev.supabase-realtime  |         (cowboy 2.9.0) /app/deps/cowboy/src/cowboy_stream_h.erl:295: :cowboy_stream_h.request_process/3

[w3b6x9]

realtime-dev.supabase-realtime | ** (MatchError) no match of right hand side value: {:error, :tenant_not_found_in_host}

@Zedo9 did you set up a custom tenant? see https://github.com/supabase/realtime#server-setup.

the default tenant is realtime-dev but if your domain is something like https://db.mydomain.com then Realtime is going to look at the subdomain, db, and use that as the tenant. If that doesn't exist then you'll get that error.

[czouche]

@Zedo9 did you set up a custom tenant? see https://github.com/supabase/realtime#server-setup.

I see! Actually I didn't setup the custom tenant since it wasn't mentioned in the supabase self hosting guide.

Edit: Even after creating the custom tenant with an external_id = db, I'm still getting the same error.

Here is the request that I made:

curl -X POST \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer my.jwt.token' \
-d $'{
    "tenant" : {
      "name": "db",
      "external_id": "db",
      "jwt_secret": "my.jwt.secret",
      "extensions": [
        {
          "type": "postgres_cdc_rls",
          "settings": {
            "db_name": "postgres",
            "db_host": "host.docker.internal",
            "db_user": "supabase_admin",
            "db_password": "mydbpassword",
            "db_port": "5432",
            "region": "us-west-1",
            "poll_interval_ms": 100,
            "poll_max_record_bytes": 1048576,
            "ip_version": 4
          }
        }
      ]
    }
  }' \
  http://localhost:4000/api/tenants

It succeeded and I can see that there is a new record under _realtime.tenants and _realtime.extensions.

[w3b6x9]

@Zedo9 are you getting the same error? if not, can you share whatever error you're getting now?

[czouche]

@Zedo9 are you getting the same error? if not, can you share whatever error you're getting now?

Exactly, I'm still getting the same error with the same realtime container logs tenant_not_found_in_host.

[w3b6x9]

@Zedo9 I think this might be an issue on our end and how we're stripping the subdomain. I'll pass this on to the rest of the team and we'll investigate. As a quick test, do you mind creating another tenant via the same POST request where the tenant is named https://db or wss://db?

[czouche]

Okay, so I did add the https://db and wss://db tenants, but I was still getting the same error. Although, I noticed that the db_host hash in the default tenant was different from the ones I was creating (It is equivalent to db).

I'm using host.docker.internal as a db_host which requires some additional configuration on Linux apparently to work (as mentioned here, so it could be the problem?

Btw, I also tried the following db_host values: db, localhost, db.mydomain.com and had the same output.

[w3b6x9]

@Zedo9 I don't think that's the issue b/c {:error, :tenant_not_found_in_host} happens before Realtime server is trying to connect to your db with host db_host.

Matter of fact, {:error, :tenant_not_found_in_host} is hit even before it starts looking inside tenants table.

I just took another look at your setup so you're using Caddy in front of Kong which sits in front of Realtime. There must be something about what Caddy is doing that's disrupting b/c it works just fine without it.

It's weird b/c realtime-dev is meant to be the tenant inside of host realtime-dev.supabase-realtime in realtime-dev.supabase-realtime | ** (MatchError) no match of right hand side value: {:error, :tenant_not_found_in_host}.

@abc3 can you take a quick look at this and see if anything jumps out?

[w3b6x9]

@abc3 let's keep this open until folks have access to this via realtime image. will need to do a release to get this out.

[corasan]

@w3b6x9 is there an estimate of when this will be released to use in local dev? I just pulled and downloaded the latest images but realtime is still v2.4.0 (I noticed the lastest release is v2.5.1)

[w3b6x9]

@corasan just opened up PRs to version bump it up to v2.5.1: supabase/cli#874 and supabase/supabase#12518.

[w3b6x9]

@corasan it's available in Realtime image v1.38.5. feel free to reopen if it's still not working.