Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP 431 & Realtime Subscriptions Failing #761

Closed
2 tasks done
ghost opened this issue Dec 16, 2023 · 11 comments
Closed
2 tasks done

HTTP 431 & Realtime Subscriptions Failing #761

ghost opened this issue Dec 16, 2023 · 11 comments
Labels
bug Something isn't working

Comments

@ghost
Copy link

ghost commented Dec 16, 2023

Bug report

  • I confirm this is a bug with Supabase, not with my own application.
  • I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

When running Realtime using the latest supabase CLI (1.123.4), using a web-address of localhost is causing an HTTP 431 Request header fields too large error. However, if I change nothing else but change the browser's url from localhost to 127.0.0.1 then everything works fine.

I am now seeing this issue on 127.0.0.1 as well.

Update 1

From an incognito browser (no cookies whatsoever) here is the request that is failing:

GET /realtime/v1/websocket?apikey=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZS1kZW1vIiwicm9sZSI6ImFub24iLCJleHAiOjE5ODM4MTI5OTZ9.CRXP1A7WOeoJeXxjNni43kdQwgnWNReilDMblYTn_I0&vsn=1.0.0 HTTP/1.1
Host: 127.0.0.1:54321
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Sec-WebSocket-Version: 13
Origin: http://127.0.0.1:3000
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: Go4trD/Gc2EevJRrcr9Zwg==
DNT: 1
Sec-GPC: 1
Connection: keep-alive, Upgrade
Cookie: sb-127-auth-token.0=%7B%22access_token%22%3A%22eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJhdXRoZW50aWNhdGVkIiwiZXhwIjoxNzAyNzg0Mzk2LCJpYXQiOjE3MDI3ODA3OTYsImlzcyI6Imh0dHA6Ly8xMjcuMC4wLjE6NTQzMjEvYXV0aC92MSIsInN1YiI6IjFhOGZkZDcyLWNlN2EtNDk0Ny04YmNkLTdmYzY5NDRhMTBjYSIsImVtYWlsIjoibmJhcnJvd0BpbnNwaXJldG1zdGVjaC5jb20iLCJwaG9uZSI6IiIsImFwcF9tZXRhZGF0YSI6eyJwcm92aWRlciI6ImtleWNsb2FrIiwicHJvdmlkZXJzIjpbImtleWNsb2FrIl19LCJ1c2VyX21ldGFkYXRhIjp7ImVtYWlsIjoibmJhcnJvd0BpbnNwaXJldG1zdGVjaC5jb20iLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwiZnVsbF9uYW1lIjoiTmljaG9sYXMgQmFycm93IiwiaXNzIjoiaHR0cHM6Ly9vbmUtaWQuaW5zcGlyZXRtc2Nvbm5lY3QuY29tL3JlYWxtcy9pbnNwaXJlIiwibmFtZSI6Ik5pY2hvbGFzIEJhcnJvdyIsInBob25lX3ZlcmlmaWVkIjpmYWxzZSwicHJvdmlkZXJfaWQiOiI5M2I3ZmJiYi02Y2I4LTQ1MmMtYWJlNC01NTcwYjUxNjM4ZGUiLCJzdWIiOiI5M2I3ZmJiYi02Y2I4LTQ1MmMtYWJlNC01NTcwYjUxNjM4ZGUifSwicm9sZSI6ImF1dGhlbnRpY2F0ZWQiLCJhYWwiOiJhYWwxIiwiYW1yIjpbeyJtZXRob2QiOiJvYXV0aCIsInRpbWVzdGFtcCI6MTcwMjc4MDc5Nn1dLCJzZXNzaW9uX2lkIjoiNjBiNGM1ZDEtYTQyMi00NTIxLTg2ZTEtODlmZDE1OWZkOWJlIn0.nSE5ev-JZnbb2RLZyLRKVAUik5Gs_B8eEaJVXhO_p3Y%22%2C%22token_type%22%3A%22bearer%22%2C%22expires_in%22%3A3600%2C%22expires_at%22%3A1702784396%2C%22refresh_token%22%3A%22XdVS-LJkbPnbjBgA4DbIxQ%22%2C%22user%22%3A%7B%22id%22%3A%221a8fdd72-ce7a-4947-8bcd-7fc6944a10ca%22%2C%22aud%22%3A%22authenticated%22%2C%22role%22%3A%22authenticated%22%2C%22email%22%3A%22nbarrow%40inspiretmstech.com%22%2C%22email_confirmed_at%22%3A%222023-11-21T18%3A24%3A00.389495Z%22%2C%22phone%22%3A%22%22%2C%22confirmed_at%22%3A%222023-11-21T18%3A24%3A00.389495Z%22%2C%22last_sign_in_at%22%3A%222023-12-17T02%3A39%3A56.631000962Z%22%2C%22app_metadata%22%3A%7B%22provider%22%3A%22keycloak%22%2C%22providers%22%3A%5B%22keycloak%22%5D%7D%2C%22user_metadata%22%3A%7B%22email%22%3A%22nbarrow%40inspiretmstech.com%22%2C%22email_verified%22%3Atrue%2C%22full_name%22%3A%22Nicholas%20Barrow%22%2C%22iss%22%3A%22https%3A%2F%2Fone-id.inspiretmsconnect.com%2Frealms%2Finspire%22%2C%22name%22%3A%22Nicholas%20Barrow%22%2C%22phone_verified%22%3Afalse%2C%22provider_id%22%3A%2293b7fbbb-6cb8-452c-abe4-5570b51638de%22%2C%22sub%22%3A%2293b7fbbb-6cb8-452c-abe4-5570b51638de%22%7D%2C%22identities%22%3A%5B%7B%22identity_id%22%3A%2286357ebb-499f-45da-8591-69388747683d%22%2C%22id%22%3A%2293b7fbbb-6cb8-452c-abe4-5570b51638de%22%2C%22user_id%22%3A%221a8fdd72-ce7a-4947-8bcd-7fc6944a10ca%22%2C%22identity_data%22%3A%7B%22email%22%3A%22nbarrow%40inspiretmstech.com%22%2C%22email_verified%22%3Atrue%2C%22full_name%22%3A%22Nicholas%20Barrow%22%2C%22iss%22%3A%22https%3A%2F%2Fone-id.inspiretmsconnect.com%2Frealms%2Finspire%22%2C%22name%22%3A%22Nicholas%20Barrow%22%2C%22phone_verified%22%3Afalse%2C%22provider_id%22%3A%2293b7fbbb-6cb8-452c-abe4-5570b51638de%22%2C%22sub%22%3A%2293b7fbbb-6cb8-452c-abe4-5570b51638de%22%7D%2C%22provider%22%3A%22keycloak%22%2C%22last_sign_in_at%22%3A%222023-11-21T18%3A24%3A00.386993Z%22%2C%22created_at%22%3A%222023-11-21T18%3A24%3A00.387015Z%22%2C%22updated_at%22%3A%222023-12-17T02%3A39%3A55.902206Z%22%2C%22email%22%3A%22nbarrow%40inspiretmstech.com%22%7D%5D%2C%22created_at%22%3A%222023-11-21T18%3A24%3A00.383339Z%22%2C%22updated_at%22%3A%222023-12-17T02%3A39%3A56.641683Z%22%7D%2C%22provider_token%22%3A%22eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJXZTlwZ0pJMDJKT3BqeGtfSnBBRW43VVNpOGtCRDBPTDhrLXVhUTRPTFJZIn0.eyJleHAiOjE3MDI3ODEwOTUsImlhdCI6MTcwMjc4MDc5NSwiYXV0aF90aW1lIjoxNzAyNzgwNzk1LCJqdGkiOiI1Njc4YzZlOS05YmFjLTRkZjYtYjc1OS1lYTliMDRmMjM4OGQiLCJpc3MiOiJodHRwczovL29uZS1pZC5pbnNwaXJldG1zY29ubmVjdC5jb20vcmVhbG1zL2luc3BpcmUiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiOTNiN2ZiYmItNmNiOC00NTJjLWFiZTQtNTU3MGI1MTYzOGRlIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiaW5zcGlyZS10bXMiLCJzZXNzaW9uX3N0YXRlIjoiZWQzMzMwM2UtODhmMS00ZTI2LTlhZDUtOTY5Mjg3MzVkZTFlIiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyIiXSwicmVhbG1fYWNjZXNzI; sb-127-auth-token.1=jp7InJvbGVzIjpbImRlZmF1bHQtcm9sZXMtaW5zcGlyZSIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIiwic2lkIjoiZWQzMzMwM2UtODhmMS00ZTI2LTlhZDUtOTY5Mjg3MzVkZTFlIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsIm5hbWUiOiJOaWNob2xhcyBCYXJyb3ciLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJiYXJyb3duaWNob2xhcyIsImdpdmVuX25hbWUiOiJOaWNob2xhcyIsImZhbWlseV9uYW1lIjoiQmFycm93IiwiZW1haWwiOiJuYmFycm93QGluc3BpcmV0bXN0ZWNoLmNvbSJ9.pyCaCsLDzug5m2CnBUMjTtpP7pdu-cY-HhoJSUpVcGjrHBs7NiXC3m3ALXjORyiVFlwdJd3UN_DHbBr_TkSFK5MtlAA0SrCwYb0XjeGmGG8heoj3Hrg4X7dPjmr-Hx0J7u0QcCr3LGybKFJvSE8iCxr5RnaXgmwkzvJrBq0dapWjUzzf-DL3NB5bj7-Rb6sdMeIg8dmQBy_yWAgSDwuj80lT7r7H9mF-n0gj2F4gSj94cTmt6R4-_6VqnXBJeNR6AxiXN1A1DV1_ud5o5E7dLSCOuaddJEiEsVHLz_Sb32KQ19edCif_XUz_yeteslj3_EqhgbqipRGzaVxe6GUF3w%22%2C%22provider_refresh_token%22%3A%22eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI0N2Y1NTRmMS04MmM0LTRmNDUtOGFmNi1hMjQ1MmNkMjhkMzQifQ.eyJleHAiOjE3MDI3ODI1OTUsImlhdCI6MTcwMjc4MDc5NSwianRpIjoiN2Y2NjU1YWEtZjkwYy00M2E4LTg2NjctOGIwYzg4OTIxOGQ3IiwiaXNzIjoiaHR0cHM6Ly9vbmUtaWQuaW5zcGlyZXRtc2Nvbm5lY3QuY29tL3JlYWxtcy9pbnNwaXJlIiwiYXVkIjoiaHR0cHM6Ly9vbmUtaWQuaW5zcGlyZXRtc2Nvbm5lY3QuY29tL3JlYWxtcy9pbnNwaXJlIiwic3ViIjoiOTNiN2ZiYmItNmNiOC00NTJjLWFiZTQtNTU3MGI1MTYzOGRlIiwidHlwIjoiUmVmcmVzaCIsImF6cCI6Imluc3BpcmUtdG1zIiwic2Vzc2lvbl9zdGF0ZSI6ImVkMzMzMDNlLTg4ZjEtNGUyNi05YWQ1LTk2OTI4NzM1ZGUxZSIsInNjb3BlIjoib3BlbmlkIGVtYWlsIHByb2ZpbGUiLCJzaWQiOiJlZDMzMzAzZS04OGYxLTRlMjYtOWFkNS05NjkyODczNWRlMWUifQ.m4DpYqjx-ILSiCg3I460pzhlAQsK3VVsRjjK1NKaDLU%22%7D
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-site
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket

Here is the response:

HTTP/1.1 431 Request Header Fields Too Large
Content-Length: 0
Connection: keep-alive
Access-Control-Allow-Origin: *
X-Kong-Upstream-Latency: 1
X-Kong-Proxy-Latency: 1
Via: kong/2.8.1

Update 2

This looks related to the NGINX instance that runs Kong. Kong has an nginx_http_client_header_buffer_size variable (https://legacy-gateway--kongdocs.netlify.app/enterprise/2.4.x/property-reference/?_ga=2.40455847.426939276.1702781538-1754633255.1702781538#nginx_http_client_header_buffer_size). From the NGINX docs, (http://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_buffer_size), the default value os 1k. However, my request above (per Firefox) is aroung 6k.

Update 3

Based on this it looks like KONG_NGINX_HTTP_CLIENT_HEADER_BUFFER_SIZE needs to be increased for SSR and cookies, as the cookies are much larger than the default 1k request size for Realtime.


To Reproduce

  1. Try connecting to realtime from a Next.js (14) website from 127.0.0.1, and everything works fine
  2. Change absolutely nothing else, other than switching the 127.0.0.1 in the URL to localhost and the websockets fail

NOTE: Chrome does not report websocket errors well; I had to use Firefox to see that this was a 431 error

Expected behavior

There should be no functional difference between localhost and 127.0.0.1

Screenshots

Screenshot 2023-12-16 at 6 51 24 PM

System information

  • OS: macOS 13.3 (22E252)
  • Version of supabase-js: ^2.38.4
  • Version of Node.js: v18.17.1
@ghost ghost added the bug Something isn't working label Dec 16, 2023
@ghost ghost changed the title HTTP 431 () & Realtime Subscriptions Failing When Using localhost Instead of 127.0.0.1 HTTP 431 & Realtime Subscriptions Failing When Using localhost Instead of 127.0.0.1 Dec 16, 2023
@ghost
Copy link
Author

ghost commented Dec 17, 2023

Also, if this helps, I know there should not be a difference to using 127.0.0.1 versus localhost. The issue is that the API I'm trying to integrate with uses OAuth 2.0 and they do not allow IP Addresses for valid/approved redirect URIs, so I can only use localhost and not 127.0.0.1.

The idea is my app needs to be running on the same URL for Supabase Auth to work, but that URL also needs to work with my external OAuth tool (which restricts me to just localhost).

@ghost
Copy link
Author

ghost commented Dec 17, 2023

In my `` logs, I can confirm:

2023-12-16 21:32:32 192.168.48.1 - - [17/Dec/2023:02:32:32 +0000] "GET /realtime/v1/websocket?apikey=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZS1kZW1vIiwicm9sZSI6ImFub24iLCJleHAiOjE5ODM4MTI5OTZ9.CRXP1A7WOeoJeXxjNni43kdQwgnWNReilDMblYTn_I0&vsn=1.0.0 HTTP/1.1" 431 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
2023-12-16 21:32:36 192.168.48.1 - - [17/Dec/2023:02:32:36 +0000] "GET /realtime/v1/websocket?apikey=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZS1kZW1vIiwicm9sZSI6ImFub24iLCJleHAiOjE5ODM4MTI5OTZ9.CRXP1A7WOeoJeXxjNni43kdQwgnWNReilDMblYTn_I0&vsn=1.0.0 HTTP/1.1" 431 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
2023-12-16 21:32:39 192.168.48.1 - - [17/Dec/2023:02:32:39 +0000] "GET /realtime/v1/websocket?apikey=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZS1kZW1vIiwicm9sZSI6ImFub24iLCJleHAiOjE5ODM4MTI5OTZ9.CRXP1A7WOeoJeXxjNni43kdQwgnWNReilDMblYTn_I0&vsn=1.0.0 HTTP/1.1" 431 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
2023-12-16 21:32:42 192.168.48.1 - - [17/Dec/2023:02:32:42 +0000] "GET /realtime/v1/websocket?apikey=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZS1kZW1vIiwicm9sZSI6ImFub24iLCJleHAiOjE5ODM4MTI5OTZ9.CRXP1A7WOeoJeXxjNni43kdQwgnWNReilDMblYTn_I0&vsn=1.0.0 HTTP/1.1" 431 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
2023-12-16 21:32:46 192.168.48.1 - - [17/Dec/2023:02:32:46 +0000] "GET /realtime/v1/websocket?apikey=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZS1kZW1vIiwicm9sZSI6ImFub24iLCJleHAiOjE5ODM4MTI5OTZ9.CRXP1A7WOeoJeXxjNni43kdQwgnWNReilDMblYTn_I0&vsn=1.0.0 HTTP/1.1" 431 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"

@ghost ghost changed the title HTTP 431 & Realtime Subscriptions Failing When Using localhost Instead of 127.0.0.1 HTTP 431 & Realtime Subscriptions Failing Dec 17, 2023
@ghost
Copy link
Author

ghost commented Dec 17, 2023

Additional Context and my best guess of what is going on:

  • this only started when I moved from local storage to cookie-based storage of access tokens for SSR and the @supabase/ssr package
  • it looks like this caused the cookies to now be sent in the websocket request, greatly increasing the payload size (compared to no cookies)
  • the cookies increase size so much that the websocket fails (due to Kong)

@gabrielperales
Copy link

gabrielperales commented Dec 18, 2023

Hi @nbarrow-inspire-labs, by default, Phoenix limits the size of the headers to 4KB (reference), which is not a big size if you have several cookies. You will likely reach that limit if several services run on the same domain and the path is not scoping your cookies. If you are working locally on different projects, all those cookies will be stored at 127.0.0.1 and it is easy to reach that limit.

Can you share a screenshot of the size of all the cookies you have on the domain you are having trouble with?

You can check it in the Chrome DevTools under the application tab
CleanShot 2023-12-18 at 12 15 16

@gabrielperales
Copy link

gabrielperales commented Dec 18, 2023

Maybe you can change the configuration to allow bigger headers in your app. I think you just have to change the config/config.exs file with these changes:

# Configures the endpoint
config :realtime, RealtimeWeb.Endpoint,
  # ...
  http: [
    port: System.get_env("PORT", "4000"),
    protocol_options: [max_header_value_length: 8192] # <-- this will duplicate the size of the header's limit
  ]

@ghost
Copy link
Author

ghost commented Dec 18, 2023

@gabrielperales thanks for getting back to me. I think you're right, the size is just larger than the 4kb maximum (see screenshot below). I'm experiencing this issue using the supabase CLI but also the self-hosted docker compose example. Do you know if setting an environment variable will configure max_header_value_length? Edit: would something like this do: #762 ?

Screenshot 2023-12-18 at 6 45 14 PM

@ghost
Copy link
Author

ghost commented Dec 19, 2023

@gabrielperales would something like this work: #762

@gabrielperales
Copy link

gabrielperales commented Dec 19, 2023

@gabrielperales would something like this work: #762

It worked for you, so yes :). But I would check why those cookies are so big and why you have those two rather than change the configuration. Anyway, I don't see any problem with having the option to overwrite that limit, but first, check what you are storing in those cookies and if everything is needed.

If the answer is yes, then you will need to overwrite.

You probably store the whole user profile in the cookie session and don't need to do that.

@ghost
Copy link
Author

ghost commented Dec 19, 2023

But I would check why those cookies are so big and why you have those two rather than change the configuration. Anyway, I don't see any problem with having the option to overwrite that limit, but first, check what you are storing in those cookies and if everything is needed.

@gabrielperales I did look at the cookies; there are two of them, both set by Supabase. One seems to be from my auth provider configured through Supabase (Keycloak) and the second seems to be a direct Supabase token. I.e., one is to authenticate between the client and Supabase, and one seems to be for Supabase to refresh itself against my Keycloak instance.

@gabrielperales
Copy link

I've been researching and you are right. It seems like Keycloak cookies are pretty big.

https://keycloak.discourse.group/t/keycloak-cookies-are-too-large/15872
react-keycloak/react-keycloak#139
https://stackoverflow.com/questions/55955926/how-to-reduce-the-size-of-the-access-refresh-tokens-in-keycloak

I don't know if there is a way to make them smaller, but seems legit to increase the header limit in this case. Let's see what the members of the Supabase team thinks about #762 :)

filipecabaco added a commit that referenced this issue Dec 26, 2023
)

Update runtime.exs to allow a `MAX_HEADER_LENGTH` env variable

Solves #761 by introducing an environment variable to override config -> http -> protocol_options -> max_header_value_length or leaves it at the default 4096 if no other value is specified


---------

Co-authored-by: Filipe Cabaço <filipe@supabase.io>
@filipecabaco
Copy link
Contributor

PR was merged! thank you @nbarrow-inspire-labs for the contribution 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants