improv (flashblocks): implement ping/pong on websocket connection + set deadline on reads

[0x416e746f6e]

• presently the flashblocks websocket connection is one-way (rollup-boost reads from builder, but never sends any data back).

• rollup-boost does not verify liveness of the above connection in any way.

  • meaning that if builder would vanish w/o explicitly closing the connection (which is more likely to happen if the builder is truly remote), then rollup-boost would have no idea about that and keep trying to read from the socket that is long since has no remote party to push the flashblocks into it.

• normally, the above issue can be mitigated by 2 means:

  • both sides implement websocket ping-pong protocol, with certain deadlines on read/write operations (meaning that if either of the messages if failed to be consumed by remote party, then the connection is considered corrupt, and is closed)

  • and additionally, the "meaningful" messages (i.e. not ping/pong ones) are also deadlined (e.g. in case of flashblocks we could say that if the websocket connection didn't deliver any payload for longer than a second, then it's broken, should be close - and we should try to set up a new one).

[0x416e746f6e]

example of what can happen.

on builder side:

...

on rollup-boost side:

note that:

• from 2025-06-08T18:11:10.761984Z (last flashblock of the last successfully delivered fb payload id 0x039e6a2badc5980e) the builder (still thinking the connection is up) keeps streaming flashblocks

  • until at 2025-06-08T18:11:28.262460Z it times-out, and closes its side of the network connection

• from 2025-06-08T18:11:11.951612771Z keeps logging "Error getting fb best payload" / "Missing payload" (even for the payloads 0x0316538cd9a5d625..0x035d7d55ec462265 that were pushed by the builder into the ws connection)

  • being completely ignorant of the fact that the connection is already gone:

    rollup-boost-58bb9bbfd8-7qk9k:/# netstat -plant | grep 1111
    tcp        0      0 0.0.0.0:1111            0.0.0.0:*               LISTEN      -
    tcp        0      0 10.79.70.16:54614       10.78.16.54:1111        ESTABLISHED -
    

[ferranbt]

Do you feel this is not covered in this test? #298
It checks that if the websocket connection gets dropped from the builder it gets restarted.

[0x416e746f6e]

no, it's not covered there.

#298 addresses different problem, namely making sure that rollup-boost does reconnect if it detects that the connection is gone.

present issue is about making sure that rollup-boost can detect that the connection is gone when it happens abnormally, i.e. when it was just "cut" w/o rollup-boost's OS receiving FIN or RST packet.

the only reliable way of ensuring that is to regularly poll the ws-connection with ping/pong messages. in http world, that's what keep-alive packets do w/o the developer having to worry about them. but this is websockets - no keep-alives here.

see also: https://datatracker.ietf.org/doc/html/rfc6455#section-5.5.2

[ferranbt]

Is it enough to send some data from rollup-boost to detect that the websocket is gone? Or we have to receive data back?

Also, do you know if there is any way to unit test this?

[0x416e746f6e]

Is it enough to send some data from rollup-boost to detect that the websocket is gone? Or we have to receive data back?

if there is data to be sent from rollup-boost we could do that, but then it's hard to tell whether it's really received (b/c there's no ack in websocket, unless explicitly implemented by the protocol).

ping/pong with some meaningful payload bytes could do that

[0x416e746f6e]

do you know if there is any way to unit test this?

I am not that proficient in rust. I guess some sort of a mock?

[karankurbur]

Same thing is needed in websocket proxy <-> rollup boost. subscriber will hang and not reconnect if read.next() doesn't resolve.

[ferranbt]

Closing this and opening a new issue for websocket proxy <-> rollup-boost.