Bandit + Phoenix LiveView on Safari iOS Fails to Reconnect

[aarroisi]

Environment

Elixir version: 1.16.2
Phoenix version: 1.7.12
Phoenix LiveView version: 0.20.14
Bandit version: 1.5.2
Thousand Island version: 1.3.5
Operating system: iOS 17.0.3
Browser: Safari

Actual behavior

I open Safari, viewing a LiveView page that's connected via websocket. Everything works normal. Then I return to home screen, wait for about a minute, then open Safari again. I saw the client-error with the usual "We can't find the internet" for about 10 seconds, then server-error with the usual "Something went wrong" for about 5 seconds, then the page refreshes automatically.

This error shows in the Safari console (I open it using Developer mode, connected to a Mac):
WebSocket connection to 'wss://example-aarroisi.fly.dev/live/websocket?_csrf_token=SihGLVAoM2hsVUo-PDwLHgcbJnU1LxItzB0GfBt93x8HyOjPVA_AQFDW&_track_static%5B0%5D=https%3A%2F%2Fexample-aarroisi.fly.dev%2Fassets%2Fapp-a5fe870326318b359c192ce86b220d13.css%3Fvsn%3Dd&_track_static%5B1%5D=https%3A%2F%2Fexample-aarroisi.fly.dev%2Fassets%2Fapp-a7b7a1bb2570182b93179aee6863d1bb.js%3Fvsn%3Dd&_mounts=0&_live_referer=undefined&vsn=2.0.0' failed: The operation couldn’t be completed. Operation timed out

I also saw the app is trying to connect using longpoll, but eventually failed and refreshed the page.

You can view the issue at play here: https://youtu.be/MxfFKilcA_k.

It's a very vanilla project generated with mix phx.new command, and using generated live view from generic mix phx.gen.live command.
The source code is here: https://github.com/aarroisi/example-aarroisi.
You can try it yourself at https://example-aarroisi.fly.dev/posts.

After I switched it from Bandit to Cowboy, the problem vanishes completely and working as expected. So I guess the problem is at Bandit on iOS Safari.

Expected behavior

When I try it in Chrome iOS, this problem doesn't show up. In Chrome, after waking up from sleep, the client error message only show up for about half a second, then it reconnects without refresh. I expect this should also happen in Safari.

[mtrudel]

Thanks for the report! I'll get a chance to look at this later this week.

Having a repro case provided makes my job about 1000x easier. Thanks for that!

[mtrudel]

I can repro this on your instance, but I'm unable to repro this locally or via a new fly instance.

In the cases where I can't repro, this seems to be happening:

• When Safari goes into the background, one or two future heartbeats go back and forth via the websocket
• When Safari comes back, a connection close frame is received from the server, and Safari starts a new websocket without issue

When it goes wrong, what seems to happen is that Safari doesn't get a response back when trying to create that new websocket connection, falls back to long polling, which also fails. At this point, signs would point to something funny with your fly config, or perhaps fly itself. As I'm not able to repro this myself, I wonder if you have any special fly config in that respect, or if you can provide both your fly logs and screenshots of your safari inspector (along with the 'preview' panes for any of the websocket connections displaying the back and forth websocket messages with timestamps).

[mtrudel]

Any progress on getting the above debug info @aarroisi ?

[aarroisi]

@mtrudel Hi, so sorry for the late reply. Here is the log from fly. It started with me opening https://example-aarroisi.fly.dev/posts, put it in background for about a minute, then opened Safari again. It showed error reconnecting message, then eventually refreshed the whole page.

2024-06-08T01:44:09.175 app[e82de11b774628] sin [info] 01:44:09.174 request_id=F9bkDEFCzjzEQjUAAAHB [info] GET /posts

2024-06-08T01:44:09.178 app[e82de11b774628] sin [info] 01:44:09.177 request_id=F9bkDEFCzjzEQjUAAAHB [info] Sent 200 in 2ms

2024-06-08T01:44:09.841 app[e82de11b774628] sin [info] 01:44:09.840 [info] CONNECTED TO Phoenix.LiveView.Socket in 22µs

2024-06-08T01:44:09.841 app[e82de11b774628] sin [info] Transport: :websocket

2024-06-08T01:44:09.841 app[e82de11b774628] sin [info] Serializer: Phoenix.Socket.V2.JSONSerializer

2024-06-08T01:44:09.841 app[e82de11b774628] sin [info] Parameters: %{"_csrf_token" => "bTBzIQ4nAR8ZOCA5ZippAk45MgETKiB52dAYlAhfAMqx0EDAvjsuuaZ5", "_live_referer" => "undefined", "_mounts" => "0", "_track_static" => %{"0" => "https://example-aarroisi.fly.dev/assets/app-a5fe870326318b359c192ce86b220d13.css?vsn=d", "1" => "https://example-aarroisi.fly.dev/assets/app-a7b7a1bb2570182b93179aee6863d1bb.js?vsn=d"}, "vsn" => "2.0.0"}

2024-06-08T01:45:24.299 app[e82de11b774628] sin [info] 01:45:24.298 [info] CONNECTED TO Phoenix.LiveView.Socket in 20µs

2024-06-08T01:45:24.299 app[e82de11b774628] sin [info] Transport: :websocket

2024-06-08T01:45:24.299 app[e82de11b774628] sin [info] Serializer: Phoenix.Socket.V2.JSONSerializer

2024-06-08T01:45:24.299 app[e82de11b774628] sin [info] Parameters: %{"_csrf_token" => "bTBzIQ4nAR8ZOCA5ZippAk45MgETKiB52dAYlAhfAMqx0EDAvjsuuaZ5", "_live_referer" => "undefined", "_mounts" => "0", "_track_static" => %{"0" => "https://example-aarroisi.fly.dev/assets/app-a5fe870326318b359c192ce86b220d13.css?vsn=d", "1" => "https://example-aarroisi.fly.dev/assets/app-a7b7a1bb2570182b93179aee6863d1bb.js?vsn=d"}, "vsn" => "2.0.0"}

2024-06-08T01:45:27.297 app[e82de11b774628] sin [info] 01:45:27.296 [info] CONNECTED TO Phoenix.LiveView.Socket in 43µs

2024-06-08T01:45:27.297 app[e82de11b774628] sin [info] Transport: :longpoll

2024-06-08T01:45:27.297 app[e82de11b774628] sin [info] Serializer: Phoenix.Socket.V2.JSONSerializer

2024-06-08T01:45:27.297 app[e82de11b774628] sin [info] Parameters: %{"_csrf_token" => "bTBzIQ4nAR8ZOCA5ZippAk45MgETKiB52dAYlAhfAMqx0EDAvjsuuaZ5", "_live_referer" => "undefined", "_mounts" => "1", "_track_static" => %{"0" => "https://example-aarroisi.fly.dev/assets/app-a5fe870326318b359c192ce86b220d13.css?vsn=d", "1" => "https://example-aarroisi.fly.dev/assets/app-a7b7a1bb2570182b93179aee6863d1bb.js?vsn=d"}, "vsn" => "2.0.0"}

2024-06-08T01:45:44.570 app[e82de11b774628] sin [info] 01:45:44.569 request_id=F9bkIndFAjzbCJMAAAHx [info] GET /posts

2024-06-08T01:45:44.572 app[e82de11b774628] sin [info] 01:45:44.571 request_id=F9bkIndFAjzbCJMAAAHx [info] Sent 200 in 1ms

2024-06-08T01:45:45.520 app[e82de11b774628] sin [info] 01:45:45.519 [info] CONNECTED TO Phoenix.LiveView.Socket in 20µs

2024-06-08T01:45:45.520 app[e82de11b774628] sin [info] Transport: :websocket

2024-06-08T01:45:45.520 app[e82de11b774628] sin [info] Serializer: Phoenix.Socket.V2.JSONSerializer

2024-06-08T01:45:45.520 app[e82de11b774628] sin [info] Parameters: %{"_csrf_token" => "B2BBIS5LAlQsQhAqZAZ_K2A0OSUADAMAX4sYL-k-t7Ak2iRhXgxQfGyL", "_live_referer" => "undefined", "_mounts" => "0", "_track_static" => %{"0" => "https://example-aarroisi.fly.dev/assets/app-a5fe870326318b359c192ce86b220d13.css?vsn=d", "1" => "https://example-aarroisi.fly.dev/assets/app-a7b7a1bb2570182b93179aee6863d1bb.js?vsn=d"}, "vsn" => "2.0.0"}

And here is the fly config file:

# fly.toml app configuration file generated for example-aarroisi on 2024-05-11T07:54:18+07:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = 'example-aarroisi'
primary_region = 'sin'
kill_signal = 'SIGTERM'

[build]

[deploy]
  release_command = '/app/bin/migrate'

[env]
  PHX_HOST = 'example-aarroisi.fly.dev'
  PORT = '8080'

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

  [http_service.concurrency]
    type = 'connections'
    hard_limit = 1000
    soft_limit = 1000

[[vm]]
  size = 'shared-cpu-1x'

I don't think it has any special configuration since it's what generated by fly automatically. Probably related to sin region?

If it's related to fly, I'll try to replicate this with a DigitalOcean droplet (vm) later today.

[feld]

Fly.io is known to have aggressive TCP timeout settings that break websockets, which is what I expect is happening here if this isn't reproducible elsewhere

https://community.fly.io/t/long-lived-tcp-connections-are-dropped/4074

[aarroisi]

Fly.io is known to have aggressive TCP timeout settings that break websockets, which is what I expect is happening here if this isn't reproducible elsewhere

https://community.fly.io/t/long-lived-tcp-connections-are-dropped/4074

Hi @feld, thanks for the info. However, @mtrudel said that it doesn't happen on his freshly made Fly app on his account. So, it's probably not a general Fly behavior.