Implement support for dedicated server export

[reduz]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

With the rise of popularity of monetizable multiplayer games, more users want to work on this. These types of games, however, require dedicated authoritative servers (hosted on the cloud via Kubernets, Agones, etc), to avoid users from cheating (And hence ruining the monetization).

Godot seems to work well for this due to its Linux friendly and Open Source nature, and the fact it can run in headless mode. There are some hurdles to overcome, however to make this process smoother for users:

• Exports still include all graphics and audio assets, which get included even if exported to server.
• Exports still include animation (and skeleton) data, which can be expensive to process even if there are no graphics.
• Export still includes all scripts, and its not possible to discern what is not needed on the server (or in exchange, what is, since you may want to include information to connect to a back-end that is not present in the player side).
• Several nodes may simply be undesired in the server (as an example, Sprite, MeshInstance), so at least it should be possible to run logic to strip nodes on server export by marking them for this end.
• Running a game at lower FPS to maximize CPU usage in a container (this is handled by Add physics step interpolation in 3D (implemented in 2D since 4.3) #2753 though).

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Server export should take care automatically (for the most part) of all the points above.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

The server export should be modified to include the following:

• Some nodes in the scene should be possible to mark as client-only (will be erased on server export, and subsequently also their animation data or references to them).
• Animations should be able to mark tracks as server-only, so the rest are stripped.
• Skeletons should be able to mark bones as server only, so the rest are stripped (remember, these can still be useful if root motion is used, or bones control physics objects via BoneAttachment).
• Graphics and Audio asset stripping, replaced by dummy assets:
  • Textures will be replaced by a DummyTexture that does nothing except report their correct size (if still used, remember some nodes may require this size for something).
  • Same for meshes, a DummyMesh will report the right AABB but will contain no data.
  • Streams will be replaced by DummyStream, which will do nothing, only report the correct length.
• An Engine.is_dedicated_server() method should be added for scripts to check.
• It should be possible to mark server assets before exporting as:
  • Strip: Graphics and Audio assets are stripped on server. (default)
  • Keep: Graphics and Audio assets are kept on server.
  • Remove: Asset will simply be removed on server.
  • RemoveOnClient: Asset will be removed on non-server exports.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Is there a reason why this should be core and not an add-on in the asset library?

Users are already using hacks to do this, but its far from easy to use and unoptimal, or require source code tweaking at engine level. This feature needs to be core.

Short FAQ:

Q: I still need access to the pixels of an image on the server, what do I do here?
A: Remember that in Godot images can be imported as Image rather than Texture, so by default those will not be stripped, as there is not much reason to. In the end you will still be able to mark the asset as "Keep" so its not stripped.

[WolfgangSenff]

This would be really, really useful for @BearDooks and us GodotNuts generally, because he maintains a Docker script that can point to a git repo and automatically exports the headless option and sets up multiplayer for it on a server, opening up ports and such. As such, having the extra option would make that script create far better server-side exports.

[Whimfoome]

So are we moving to one game project for both server and client, similarly to UE4? Because before we had to make 2 separate projects (at least in Game Development Center's tutorial series approach).

[reduz]

@Whimfoome Officially dedicated server exports were never really supported. The Godot server build was mostly for doing CI exports and tools, not to run as dedicated multiplayer game server. This is something Godot users discovered they could do pretty well so from our side the idea is to support it as best as possible.

[aRestless]

This is great news!

I have a background in game server hosting and would like to add a few topics to look out for. Some of them might already be on your radar or already solved by Godot. Some might also not fit the scope of this issue and could warrant separate ones:

• Allow IP binding and port binding for all allocated network sockets, meaning exposing these values as configuration parameters; Many game server applications do not support this, which can force game server providers to build custom hooks for the binaries to overwrite socket allocation
• For any functionality related to game server lists, e.g. the dedicated server registering itself towards some backend, do not rely on the server listening on the same IP that the request comes from (the default/gateway IP might be different to the one the server is bound to)
• Add an option to limit server FPS while no player is connected; this allows overcommitting CPU resources
• Make sure the engine properly handles environments with typical server specs, such as 48 physical CPU cores across two CPUs; e.g. Unity has some standard functionality that spawns one thread per core, which is not desirable in these environments.
• Support a standardized "server query" method for information like connected player numbers, current map, version, and so forth; Steam's A2S Query is widely used even outside Steam's ecosystem. A very bare-bones reference implementation can be found here: https://github.com/nitrado/libSteamProt

I hope this is helpful.

[YuriSizov]

Related #991

[reduz]

@pycbouh Not technically related I guess since one thing does not depend on the other, but definitely it needs to work for everything to make sense.

[reduz]

@aRestless I have the feeling that most of these can be done with add-ons (either official or not) since they are mostly game specific. Regarding hosting, Godot seems to have some things that are different than other engines, such as being very efficient resource wise, very quick to spawn and load, and you have very precise control on how threads are used. I guess as more users use it for this, we will understand better what can be optimized.

[Calinou]

Allow IP binding and port binding for all allocated network sockets, meaning exposing these values as configuration parameters; Many game server applications do not support this, which can force game server providers to build custom hooks for the binaries to overwrite socket allocation

Setting the server port can already be done, but it's up to you to expose it with a CLI argument (OS.get_cmdline_args()) or configuration file (ConfigFile class).

I think the listening IP can be set too, but I'm not sure (cc @Faless).

Add an option to limit server FPS while no player is connected; this allows overcommitting CPU resources

This can already be done by setting Engine.target_fps and Engine.iterations_per_second to a low value (try 1). Once a player connects, set those to 0 and 60 respectively (or whichever values you were previously using).

[aRestless]

@reduz I absolutely agree that some points in the list are more additional features and could be suited for add-ons.

@Calinou Thank you for those insights. IP/Port binding is probably more a documentation topic then. I have supported a number of game launches where, quite frankly, game developers were not even aware of the fact that a system can have more than one IP address and the missing support for IP/Port binding resulted in significant road blocks for going into production.

Speaking from a provider perspective, I'd love for IP/Port binding always being something explicit and configurable from the outside (instead of "0.0.0.0" from a code sample being hardcoded and no one ever going back to it). But such an approach might not fit into Godot's design.

[BearDooks]

So are we moving to one game project for both server and client, similarly to UE4? Because before we had to make 2 separate projects (at least in Game Development Center's tutorial series approach).

@Whimfoome I was able to use one project for this and then used the 'features tags' so I could make functions on work if it was running as the 'client' or the 'server'. You still need to export the project twice, once with each tag.

@reduz I think this would be a wonderful addition to the engine. As you have stated before it is not always easy to run a multiplayer headless server without some hacks to it. In my personal opinion I think it would allow more people to create multiplayer experiences in a less frustrating manner.

[nonunknown]

@reduz I have a question also, My job requires me to do a multiplayer game using godot, Currently I'm not using the godot itself to do server stuff, but instead only using WebSocket do pass data, and protobuffers to reduce its size (which a guy made a plugin to support protobuffers in godot) https://github.com/oniksan/godobuf . Can this be official?

The second question is, what about having access to godot's mainloop via API, I mean My server uses GoLang (this can be true for every language) So I could have godot's mainloop running in the server and call stuff from there, and replicate to clients, its possible?

PS: Currently I'm using raylib on the server to simulate the game there, and the client (godot instance) will only replicate whats happening on the server.

[LillyByte]

I would say, make sure this also supports Windows, as I only see Linux mentioned.

I know when people think about "dedicated hosting" they often think of that little cloud server out there that's running Linux-- but that's not how it works most of the time.

The bulk of gamers are Windows users; and a lot of users will launch dedicated servers on their home machines so they can play with family/friends without having to rent/manage servers.

[Calinou]

I would say, make sure this also supports Windows, as I only see Linux mentioned.

This will be supported once #991 is implemented.

Note that you can already run dedicated servers on Windows by passing the --no-window command line argument, even on a Windows Server machine (since it always has a GUI subsystem available, unlike Linux).

[aaronfranke]

@reduz I know you already said that it's not related, but really this proposal does not make sense to do until after #991. The ability to run a server headless seems more important than the ability to optimize the build by removing textures etc.

EDIT: #991 is now implemented.

[Calinou]

@reduz I know you already said that it's not related, but really this proposal does not make sense to do until after #991. The ability to run a server headless seems more important than the ability to optimize the build by removing textures etc.

While this proposal is not very useful without #991, these proposals can be implemented independently from each other (see above). Rest assured that #991 will be implemented 🙂

See discussion starting from here: https://chat.godotengine.org/channel/networking?msg=4SuqcdnjRfY83NhSk

[reduz]

@aaronfranke It's unrelated, because you will be able to run a server export on any target. It's also easier to do this proposal first, as the second requires APIs to be closer to final to do best.

[AlexDarigan]

So are we moving to one game project for both server and client, similarly to UE4? Because before we had to make 2 separate projects (at least in Game Development Center's tutorial series approach).

@Whimfoome
Are you talking explicitly about creating two seperate projects for the purpose of exporting or more generally about desiging the server and client at the same time? Because the latter became possible in one of the 3.2 updates. You can find an example here https://github.com/AlexDarigan/CustomMultiplayerGodot. (Though having a node that defaults this behaviour including to its children might be a good seperate proposal).

* Allow IP binding and port binding for all allocated network sockets, meaning exposing these values as configuration parameters; Many game server applications do not support this, which can force game server providers to build custom hooks for the binaries to overwrite socket allocation

* For any functionality related to game server lists, e.g. the dedicated server registering itself towards some backend, do not rely on the server listening on the same IP that the request comes from (the default/gateway IP might be different to the one the server is bound to)

* Support a standardized "server query" method for information like connected player numbers, current map, version, and so forth; Steam's A2S Query is widely used even outside Steam's ecosystem. A very bare-bones reference implementation can be found here: https://github.com/nitrado/libSteamProt

@aRestless
I'm not an Industry Professional but I've worked with Godot Multiplayer a bit.

On your first point, I don't think this would be hugely difficult to implement as a user. If we're talking about just taking an IP or Port we know and having godot server point to it. Godot also has the IP Class that has the useful get_local_addresses function.

On your second point, for high level multiplayer, you can change the listening port using the set_bind_ip function.

And I do support the idea of a standarized servery query. Connected Players is technially achievable using get_peers().size()

[vybr]

Would there also be the possibility to give the headless console a command line? This is so users can input server commands to affect the game (e.g. starting/stopping the server, spawning items, banning users). Not sure how useful that is for dedicated servers, but for player-hosted ones (e.g. Minecraft, Terraria) where servers are commonly hosted on the player's own machines, it's nice to have.

[Calinou]

Would there also be the possibility to give the headless console a command line? This is so users can input server commands to affect the game (e.g. starting/stopping the server, spawning items, banning users). Not sure how useful that is for dedicated servers, but for player-hosted ones (e.g. Minecraft, Terraria) where servers are commonly hosted on the player's own machines, it's nice to have.

This needs #2322 to be implemented. Once that proposal is implemented, this command input feature can be implemented by an add-on 🙂

In the meantime, you could make the server read from a text file every so often (every 10-30 seconds, possibly more often if a command was recently input). If the file isn't empty, execute its contents as a command and remove the file's contents (but do not remove the file entirely). With modern text editors reloading files automatically when they change on the filesystem, the user experience won't be so bad (while still not being great).

Another alternative is to use the WebSocketServer class and send command requests using a WebSockets client. Or you can use a REST API by hosting a HTTP server from the game server with a GDScript HTTP server implementation.

[dsnopek]

In PR #69546, I attempted to implement a small part of this proposal. However, after discussing with @Faless, I've created an alternative proposal that is a little more restricted in scope - see: #5970

[akien-mga]

Superseded by #5970.