Server list syncing

[Liareth]

The ability to sync player lists between servers (and allow cross-server tells as if they were native) would be very useful to many major servers.

• Server list must be shared between servers. Cross server tells and shouts must be allowed. Cross server party invites (or appearing on DM list) don't matter.

This will probably require a separate server binary which handled synchronising the player list and communicating cross-server tells and shouts. The flow being:

• User connects to server.
• Server notifies sync server.
• Sync server notifies other servers.
• Other servers fake a login.

Rinse and repeat for disconnected users and for sending tell/shout messages.

Sync server must receive a heartbeat every seconds or it will trigger disconnect notification for that server's players to all other servers.

It's pretty complicated but up for grabs if anyone wants to work on something like this.

[GoLoT]

After working on this for a bit I have a simple prototype running that does (mostly) what it's expected to do.

• A server acts as a master server while other servers connect to that master server and share their player lists. The master server then replicates that information to any new servers. Each server has a unique ID (currently their socket fd as reported by the master server). Connections are TCP so there is no need for a custom hearbeat, servers will be removed if the socket is closed. This also makes it a lot easier to keep a consistent state as events are received in the same order they are sent.
  Optionally there is an env var that can be used to set a password. It doesn't make much sense unless encryption is added but it's there just in case.

• Any new player connections, disconnections and chat messages are sent to the master servers and replicated to all connected servers. This is implemented through a base struct Event that has derived structs for each of the possible events.
  The structs are serialized (using the header-only library cereal) and sent through a plain socket, but adding TLS support shouldn't be a problem. All the network stuff runs on a separate thread and pushes events to a receive queue. This queue is processed from the main thread, making any changes to the nwserver state.
  There is also a send queue where events generated by nwserver (chat messages, disconnection events, etc.) are pushed. This queue is read by the network thread and the events are sent to the master server accordingly.

• Each server keeps a list of remote servers and their player lists. The fake clients are assigned player IDs based on their actual player IDs (as reported by their servers) and the server IDs (socket fd) left-shifted, using a simple OR operation. This cross-server ID is used by nwmain clients to avoid collisions on player lists and translated back to server ID - player ID pairs at the server whenever an event happens. Tell events are captured and if the target player ID is a cross-server ID, the event is pushed to the send queue and sent to the master server, which then relays it to the relevant server.

• Fake connections are handled through a custom function that sends a fake player login to the connected clients. Sadly, this makes any fake connections show up in the combat log possibly spamming the combat log with connection messages whenever a new server joins the mesh (if it has players connected already) or whenever a new player joins the server and receives the cross-server player list. The only way to avoid this would be to use the SendServerToPlayerPlayerList_All() function which updates the player list without generating any messages, but it has a hardcoded limit of 255 players because the number of players is sent as a char. Using it would require more precise player ID mappings to avoid collisions and would impose a limit on cross-server player connections.

• The system can be extended easily to support any other kind of messaging between servers by defining a new struct for that event and writing the relevant code on the event handler function. As an idea, it could be used to send commands to run specific scripts or remotely manage the server.

So far only one function has to be hooked exclusively and rewritten: HandlePlayerToServerChatMessage. The rest can be done with shared hooks: SendServerToPlayerPlayerList_All, SendServerToPlayerPlayerList_Add, SendServerToPlayerPlayerList_Delete and CServerExoApp::MainLoop.

The main issues I found so far:

• The hardcoded limit of 255 players for the SendServerToPlayerPlayerList_All event, which forces the use of SendServerToPlayerPlayerList_Add for cross-server player connections and the lack of a way to hide player connections (without .2da editing). Any ideas to get around it would be appreciated.

• Lack of click-to-reply support for cross-server messages, possibly because I made a mistake somewhere (I'm debugging it at the moment, will update with my findings). There is also the possibility that I'm missing something to make it work, maybe sending a CGameObject/CNWSCreature update to the clients so they can find the object attached to the player. Can't really tell without access to the nwclient code. The rest of the functionality works so far, being able to send tells to players through the player list.

• There is no way to sort the player list. New players are added to the top and I imagine it can get pretty crazy with high population servers, having local and cross-server players mixed. To tell them apart the cross-server player names are grey.

It's far from being ready to be released but I thought I should inform of the progress and hopefully get some feedback. Mainly regarding what other functionality could be useful and how to deal with the current issues.