🔄 Sync with upstream changes #9
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them
cowboy 2.13.0 set the default active_n=1
Currently all text frames as handled only with JSON which already requires UTF-8
This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition
Verify that replication connection is able to reconnect when faced with WAL bloat issues
A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now.
Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size`
--------- Co-authored-by: Eduardo Gurgel Pinho <eduardo.gurgel@supabase.io>
* fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it
Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node.
Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc
|
|
Fudster
added a commit
that referenced
this pull request
Sep 25, 2025
* 🔄 Sync with upstream changes (#2) * chore: fix couple of flaky tests (supabase#1517) * fix: Improve runtime setup logic (supabase#1511) Cleanup runtime.exs logic to be more organized and easier to mantain * fix: runtime setup error (supabase#1520) --------- Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Filipe Cabaço <filipe@supabase.io> * 🔄 Sync with upstream changes (#4) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> * 🔄 Sync with upstream changes (#6) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> * 🔄 Sync with upstream changes (#7) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <eduardo.gurgel@supabase.io> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> * 🔄 Sync with upstream changes (#9) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <eduardo.gurgel@supabase.io> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> --------- Co-authored-by: Al @h0lybyte <5599058+h0lybyte@users.noreply.github.com> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com>
Pull Request Test Coverage Report for Build 39ddca24682fcf258363793ba78eb607e45bfde7-PR-9Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
Fudster
added a commit
that referenced
this pull request
Oct 4, 2025
* 🔄 Sync with upstream changes (#2) * chore: fix couple of flaky tests (supabase#1517) * fix: Improve runtime setup logic (supabase#1511) Cleanup runtime.exs logic to be more organized and easier to mantain * fix: runtime setup error (supabase#1520) --------- Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Filipe Cabaço <filipe@supabase.io> * 🔄 Sync with upstream changes (#4) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> * 🔄 Sync with upstream changes (#6) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> * 🔄 Sync with upstream changes (#7) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <eduardo.gurgel@supabase.io> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> * 🔄 Sync with upstream changes (#9) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <eduardo.gurgel@supabase.io> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> * 🔄 Sync with upstream changes (#11) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <eduardo.gurgel@supabase.io> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) * fix: specify that only private channels are allowed when replaying (supabase#1543) messages * fix: rate limit connect module (supabase#1541) On bad connection, we rate limit the Connect module so we prevent abuses and too much logging of errors --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> --------- Co-authored-by: Al @h0lybyte <5599058+h0lybyte@users.noreply.github.com> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com>
Fudster
added a commit
that referenced
this pull request
Oct 7, 2025
* 🔄 Sync with upstream changes (#2) * chore: fix couple of flaky tests (supabase#1517) * fix: Improve runtime setup logic (supabase#1511) Cleanup runtime.exs logic to be more organized and easier to mantain * fix: runtime setup error (supabase#1520) --------- Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Filipe Cabaço <filipe@supabase.io> * 🔄 Sync with upstream changes (#4) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> * 🔄 Sync with upstream changes (#6) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> * 🔄 Sync with upstream changes (#7) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <eduardo.gurgel@supabase.io> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> * 🔄 Sync with upstream changes (#9) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <eduardo.gurgel@supabase.io> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> * 🔄 Sync with upstream changes (#11) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <eduardo.gurgel@supabase.io> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) * fix: specify that only private channels are allowed when replaying (supabase#1543) messages * fix: rate limit connect module (supabase#1541) On bad connection, we rate limit the Connect module so we prevent abuses and too much logging of errors --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> * 🔄 Sync with upstream changes (#13) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <eduardo.gurgel@supabase.io> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) * fix: specify that only private channels are allowed when replaying (supabase#1543) messages * fix: rate limit connect module (supabase#1541) On bad connection, we rate limit the Connect module so we prevent abuses and too much logging of errors * build: automatically cancel old tests/build on new push (supabase#1545) Currently, whenever you push any commit to your branch, the old builds are still running and a new build is started. Once a new commit is added, the old test results no longer matter and it's just a waste of CI resources. Also reduces confusion with multiple builds running in parallel for the same branch/possibly blocking any merges. With this little change, we ensure that whenever a new commit is added, the previous build is immediately canceled/stopped and only the build (latest commit) runs. * fix: move message queue data to off-heap for gen_rpc pub sub workers (supabase#1548) * fix: rate limit Connect.lookup_or_start_connection on error only (supabase#1549) * fix: increase connect error rate window to 30 seconds (supabase#1550) * fix: set a lower fullsweep_after flag for GenRpcPubSub workers (supabase#1551) * fix: hardcode presence limit (supabase#1552) * fix: further decrease limit on presence events (supabase#1553) * fix: bump up realtime (supabase#1554) * fix: lower rate limit to 100 events per second (supabase#1556) * fix: move connect rate limit to socket (supabase#1555) * fix: reduce max_frame_size to 5MB * fix: fullsweep_after=100 on gen rpc pub sub workers --------- Co-authored-by: Eduardo Gurgel Pinho <eduardo.gurgel@supabase.io> * fix: collect global metrics without tenant tagging (supabase#1557) * feat: presence payload size (supabase#1559) * Also tweak buckets to account all the way to 3000KB * Start tagging the payload size metrics with message_type. message_type can be presence, broadcast or postgres_changes * fix: use GenRpc for Realtime.Latency pings (supabase#1560) * Fastlane for phoenix presence_diff (supabase#1558) It uses a fork of Phoenix for time being * fix: count presence_diff events on MessageDispatcher * fix: remove traces from console during development --------- Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Kevin Grüneberg <k.grueneberg1994@gmail.com> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> --------- Co-authored-by: Al @h0lybyte <5599058+h0lybyte@users.noreply.github.com> Co-authored-by: Eduardo Gurgel <eduardo.gurgel@supabase.io> Co-authored-by: Filipe Cabaço <filipe@supabase.io> Co-authored-by: Bradley Haljendi <5642609+Fudster@users.noreply.github.com> Co-authored-by: Kevin Grüneberg <k.grueneberg1994@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Upstream Sync
This PR contains the latest changes from the upstream repository.
Changes included:
Review checklist:
This PR was automatically created by the upstream sync workflow