Widget state not updated after network disconnection and recovery

[colinl]

If the network connection with the browser fails, and then a widget state (eg ui-text node) is updated by sending it a message, then when the network recovers the new state is not reflected in the browser.

To replicate, import the flow below, or recreate it.
Run a browser showing the editor on the machine running node red and run a browser on another machine showing the dashboard.
Check that clicking the inject nodes updates the text in the browser showing the dashboard.
Disconnect the network from the machine showing the dashboard and wait for the connection error messages to appear in the developer console.
On the machine showing the editor click the other inject node.
Reconnect the network to the machine running the browser. The text shown in the dashboard does not update to show the new value. Refreshing the browser page does update it correctly.

With D1, running the equivalent test, the text does update to the new value automatically on network recovery.

Edit: I am still seeing this with dashboard version 1.8.0

[{"id":"7235e7106295e0eb","type":"ui-text","z":"318b3c8445d47122","group":"d8734fb40940c231","order":0,"width":0,"height":0,"name":"ui-text test","label":"The label","format":"{{msg.payload}}","layout":"row-spread","style":false,"font":"","fontSize":16,"color":"#717171","className":"","x":460,"y":740,"wires":[]},{"id":"153ccf4715b87682","type":"inject","z":"318b3c8445d47122","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"Text One","payloadType":"str","x":220,"y":720,"wires":[["7235e7106295e0eb"]]},{"id":"302c888d24db31a3","type":"inject","z":"318b3c8445d47122","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"Text Two","payloadType":"str","x":220,"y":780,"wires":[["7235e7106295e0eb"]]},{"id":"d8734fb40940c231","type":"ui-group","name":"Charger","page":"f33c25a3d1e68ab9","width":"6","height":"1","order":-1,"showTitle":false,"className":""},{"id":"f33c25a3d1e68ab9","type":"ui-page","name":"Charger","ui":"04ee189a49c54f22","path":"/charger","layout":"flex","theme":"23aa79f4e489b044","order":8,"className":""},{"id":"04ee189a49c54f22","type":"ui-base","name":"Dashboard","path":"/dashboard"},{"id":"23aa79f4e489b044","type":"ui-theme","name":"Theme 1","colors":{"surface":"#ffffff","primary":"#514fba","bgPage":"#eeeeee","groupBg":"#ffffff","groupOutline":"#cccccc"},"sizes":{"pagePadding":"1px","groupGap":"1px","groupBorderRadius":"4px","widgetGap":"10px"}}]

[colinl]

I am still seeing this issue with version 1.8.0. The Connection Lost and Connection restored popups appear, and in the browser console I see
index-CUItCJGa.js:236 

GET http://owl.local:1880/dashboard/socket.io/?EIO=4&transport=polling&t=OyPSHxc net::ERR_ADDRESS_UNREACHABLE
index-CUItCJGa.js:244 
SIO connect error: Error: xhr poll error
   at Polling.onError (index-CUItCJGa.js:236:22311)
   at Request.<anonymous> (index-CUItCJGa.js:236:25856)
   at Emitter.emit (index-CUItCJGa.js:236:20532)
   at Request.onError (index-CUItCJGa.js:236:27307)
   at index-CUItCJGa.js:236:27070 undefined

index-CUItCJGa.js:244 SIO connected

But the dashboard still does not interact with the server in either direction, as far as I can tell.

[joepavitt]

Okay, I can now reliably reproduce this at least (a slightly different error, but I do now see the lack of interaction on the sockets) - investigating now

[joepavitt]

So, this is a fun investigation.

SocketIO deletes all custom event handlers when reconnecting, because our event handlers are run inside the mounted(), the refresh fixes it, but when a standard reconnect() takes place, the socket has cleared all the custom event handlers, and then doesn't resurrect them automatically.

I've asked https://stackoverflow.com/questions/78389767/reinstate-custom-event-handlers-after-socketio-re-connects which hopefully someone can assist with for my own understanding, but it seems a very odd architectural decision.

[colinl]

Have you looked at how D1 handles this, or is the situation different? D1 successfully reconnects.

[joepavitt]

@colinl I've managed to narrow this down to not being a SocketIO problem, but in fact a VueJS problem (and therefore Dashboard 1.0 doesn't help here unfortunately). More specifically, it's linked to how we're rendering our widgets. I've updated my SO question with the extra details if you're interested.

[joepavitt]

Okay, I'm still no nearer to a solution, but a summary of the past 5 hours of debugging this:

• Running socket.connect() after a socket.on('disconnect') event does resurrect event handlers and a connection successfully for all core nodes (except for template nodes) and does not work for custom nodes, but only if the client is the cause of the disconnect, if the server is killed and restarts, the socket.connect() is never successful
• A solution to force a reconnect even when the server is killed is to instatiate a whole new SocketIO manager as part of the reconnect logic - this works in restoring the connection for all widget types, but then kills all custom, and widget-scoped event handlers (e.g. on('msg-input'), which we cannot restore.
• There are also issues around how we're rendering a widget using <component :is="" />. This is meant to run mounted() and unmounted() when the is component is modified/updated - it does not, which means our event handlers, contained inside mounted () are lost
• Killing (and then reviving) the internet connection on a device that is running a Dashboard, results in SocketIO refusing to connect back to the server again until a full browser refresh takes place, or we delete our manager entirely and consequently clear all event handlers in our widgets.

[joepavitt]

I think I have a breakthrough...

Looking at the SocketIO client source, I found:

https://github.com/socketio/socket.io-client/blob/4f6030f2c4394bc51d63fe27ed3310d95b9e0a74/lib/socket.ts#L360-L361

the _reconnecting variable was set as true when the server disconnected, or when internet was lost, meaning that SocketIO was trying to handle the reconnection, but, for whatever reason I couldn't decipher, was never able to resurrect it.

To bypass this, we can pass reconnection: false to override the default behaviour (docs). Then, the _reconnecting check always passes, and forces SocketIO to reconnect when we tell it to.

Note, this still doesn't solve the problem for ui-template and custom nodes, that's all a different (to-be-resolved) Vue problem, but this will at least fix for core nodes.