How to make your mod feel lagless in multiplayer

• Please make sure to read Modding prerequisites page in order to use new ACS properties listed below

Introduction

As we all know, playing with or against other players is fun. But multiplayer also brings network related issues that you won't face when playing alone. Those issues are network latency (e.g. a time it takes for signal to go from server/client to client/server and back, also known as ping) and packet loss (e.g. when a command sent from server/client to client/server is lost). Luckily, there are ways to partially negate these issues and make multiplayer gameplay smoother.

Client Prediction

First thing that should be solved, when making a multiplayer game, is delayed movement. Imagine that you pressed a button to move forward. Your client will then send a command to the server, the server will calculate your movement, then send back a response, and only then your character will start moving on your screen. And the higher your ping is, the more delayed your movement will be.

There is a solution to that called Client Prediction. Now, imagine the same scenario as above. When you press a button, your client will send a command to the server, but it will also start moving your character immediately so that it doesn't feel delayed. It will also remember your input, the character state (whether it crouched or jumped or did something else) for each tick that the game run until it received a response from the server. It other words, it will try to predict your position, but will also remember your input to adjust your position if the prediction doesn't match what the server tells.

Zandronum (and Q-Zandronum) already does client prediction, so nothing to worry about. However, if you want to implement custom movement features for your mod, you'll need to make it client predictable so that it feels lagless. But more on that later.

Server Unlagged

Second thing is shooting with network latency. Imagine that you are about to shoot a moving target. You see the enemy, move your cursor perfectly, and shoot. But by the time your shooting command will reach the server, your target will already move from there. In fact, he is already not there, because information about his movement came to you with a delay. So you are shooting into the air.

There is a solution to that as well, called Unlagged. Since all damage calculation is done on server (or else it will be a huge security hole for cheaters to abuse), the server has to compensate for your ping. It is done in a few simple steps. First the server will remember every player's position for last 70 ticks (2 seconds). Then, when someone shoots, it will look at that player's ping and move everyone except the shooter back to where they were according to the shooter's ping. Then the server will calculate the attack, damage the victims if needed and bring everyone back to their current position.

Zandronum (and Q-Zandronum) already supports this for hitscan attacks. However, unlagged projectiles were only introduced in Q-Zandronum 1.3.

Q-Zandronum 1.3 changes

Before 1.3, most Decorate functions would not execute on client at all, or will only do some very basic stuff like play a sound and leave. In 1.3 most Decorate functions (full list is here) will execute on client, but with a limitation. The client is only allowed to do that if the function is called by this client's player pawn, or if the calling actor is marked as clientside (i.e. belongs to that client and does not exist on the server and other clients).

This behaviour is disabled by default and has to be enabled via a compat_predictfunctions cvar. Warning! This feature doesn't work out of the box, it requires support from the mod side. Turning it on without the support can break the mod. It is safe to turn the flag on when playing vanilla doom, heretic, hexen and strife games they support it out of the box.

How Q-Zandronum 1.3 unlags projectiles

When the compat_predictfunctions cvar is on, the A_FireCustomMissile, A_CustomMissile, A_ThrowGrenade and A_SpawnItemEx functions can be executed on client. That means that the client will spawn those actors before respected command reaches the server, and the server has to consider that. In short - the solution is a combination of client prediction and server unlagged.

So, let's imagine that we are using the A_FireCustomMissile to shoot a rocket. The client will spawn the respected rocket and mark it as clientside (because the server hasn't spawned it yet). That rocket will go thru it's decorate states as normal and play respected animations. It can also explode if it hits a monster, a player or a wall just like you expect a rocket to do. However, it will not damage other players and monsters, because only the server has authority to do that. The only exception to that are actors marked as clientside, because they are handled by this client and don't exist for others.

After a while, the player attack signal will reach the server and it will call the A_FireCustomMissile function too and spawn the respected rocket. Then, the server will look at player's ping and will start compensating it (see Server Unlagged). The projectile unlagged process, however, is a bit more complicated than unlagging hitscan attacks, because the rocket is not instant, but moves at a certain speed. So the server will move all other players back to where they were according to shooter's ping, and will see if the rocket hit anyone. But it won't restore everyone to their current positions yet. Instead the server will move players and the rocket one tick further, and check if it hit anyone again. Then again, and again, until it reaches current tick, or until the rocket explodes.

If the rocket exploded during unlagged calculations, the server will send a signal about it to the clients so that they play the explosion animations. If the rocket is still active, the server will send a signal to clients to spawn that rocket on their end. Keep in mind that the rocket already moved forward on server during unlagged calculations, so the clients will spawn it already shifted. The shift distance depends on rocket speed and shooter's ping and can range from right where the shooter is up to meters away and it can look like it spawned in thin air. The server will send this signal to the shooter as well, and then the shooter will replace it's clientside rocket with the server one to stay in sync. That means that the rocket will no longer be marked as clientside and this client no longer has direct control over it.

• IMPORTANT NOTE! If your projectile is supposed to also damage or thrust the player, for example, when rocket jumping, then make sure that projectile actor exists for multiple ticks after it exploded. Simply add TNT1 A 99 in the end. The reason is that player self damage and thrust is delayed on server to account for ping properly, and the exploding actor needs to exist when the server decideds that it is time to damage and thrust the player.

New flag name	Function	Effect
FPF_NOUNLAGGED CMF_NOUNLAGGED TGR_NOUNLAGGED SXF_NOUNLAGGED	A_FireCustomMissile A_CustomMissile A_ThrowGrenade A_SpawnItemEx	The server will not run unlagged calculations for the spawned actor, so the actor will spawn with no latency compensation.
FPF_UNLAGDEATH CMF_UNLAGDEATH TGR_UNLAGDEATH SXF_UNLAGDEATH	A_FireCustomMissile A_CustomMissile A_ThrowGrenade A_SpawnItemEx	By default, the server stops unlagging when an actor dies so that the actor could play it's death animation fully on clients. This flag will make it continue unlagging after the actor died.
FPF_SKIPOWNER CMF_SKIPOWNER TGR_SKIPOWNER SXF_SKIPOWNER	A_FireCustomMissile A_CustomMissile A_ThrowGrenade A_SpawnItemEx	The server will not send data about the spawned actor to the shooter's client, so the client will still have it's clientside actor. Keep in mind that in this case the clientside actor will not be synched with the server.
FPF_FORCESERVERSIDE CMF_FORCESERVERSIDE TGR_FORCESERVERSIDE SXF_FORCESERVERSIDE	A_FireCustomMissile A_CustomMissile A_ThrowGrenade A_SpawnItemEx	The actor will only spawn on the server and then replicate to clients. The shooter client will not try to predict spawn this actor at all.
JLOSF_SKIPOWNER	A_JumpIfTargetInLos	The server will not send position and frame update to this actor's owner.

Net ID changes

When a server spawns an actor and wants to replicate it to clients, it gives the actor a so called "Net ID". Later on, when the servers wants to send an update about that actor, for example, when it changes position, the server says "The actor with this Net ID is now at this position".

In Q-Zandronum 1.3 clients have more freedom to spawn actors on their own. Because of that they need a mechanism to synchronize client spawned actors with their server spawned counterparts. Now, each client has a pool of Net IDs that is bound to this client. When the client spawns an actor that should be in sync with server, it gives the actor a Net ID from that pool, and the server does the same. Then, when a command from the server arrives, the command will contain that Net ID and the client will simply replace the actor with that Net ID with the server one.

Each player has a pool size of 1000 Net IDs, and actors receive Net IDs in incremental order. That said, if the player 1 spawns an actor, it will receive a Net ID of 1. The next actor's Net ID will be 2, then 3 and so on, up to 1000, and then it will go back to 1 and start again. Player 2's IDs will start with 1001, then 1002 and so on. If the actor spawn is not triggered by a player, then the actor's Net ID will be outside all player's pool, starting with 65000 and going up.

You can use the sv_showspawnnames cvar on both server and client to see what Net IDs they give to the actors they spawn. You can also use cl_showspawnnames cvar on client to see what actors the server tells it to spawn and what Net IDs they have.

• If the sv_showspawnnames value is 1, the server console will print only actors that belong to a player. If the value is 2, then the server console will print all actors with Net ID spawned by the server.

Unlagging your mod

General tips

Most actors can belong to one of these categories:

• Gameplay actors (player pawns, monsters, projectiles, inventory items) that affect the world. You want these actors to stay in sync with the server. The server unlags and syncronizes actors by default as long as the compat_predictfunctions cvar is true. These actors should work just fine by default, although you may need some adjustments here and there.
• Effect actors (exploding projectile particles, other special effects) that are simply there for fancy graphics and don't affect the world directly. These actors don't have to stay in sync with the server, so you can safely spawn them using the combination of NOUNLAGGED and SKIPOWNER flags inside the function that spawns is. Then, you can also use the +NONETID or +SERVERNETID actor flag to not use a Net ID from player's pool for this actor. This will greatly reduce the probability of a desync.
• Actors that shouldn't be predicted by clients. For example, an actor that spawns, deals damage in an area, and disappears. The damage is always calculated on server, so clients don't have to worry about this actor. You can use the FORCESERVERSIDE function flag to prevent clients from spawning it completely.

Client predictable map ACS scripts

Please go to Client predictable map ACS scripts page for details.

Client predictable Action Assigned ACS scripts

Please go to Action assigned ACS scripts page for details.

Synchronizing random

There is one big problem that appears when a client tries to predict it's actions. That is random, or in particular random number generation on client not being in sync with the server. In case of client prediction that can result in clientside rocket spawning at a different angle than a serverside one, which can lead to a very non-smooth gameplay.

Luckily, there is a solution to that too. Q-Zandronum 1.3 introduces random synchronization per actor, or actor random in short, or arandom in very short. The Decorate and ACS functions that utilize random in some way or another now try to use the same random seed so that random returns the same value on both server and client. There is also a way to use actor random when calling the random(), random2() and frandom() Decorate expressions. They now have their synchronized counterparts arandom(), arandom2() and afrandom() respectively.

In order to debug the random synchronization, you need to use a new sv_showactorrandom cvar. It will print the actor name, the function that used Random, and the debug value. You need to set sv_showactorrandom 1 on both server and client and make sure the debug values are the same for all actors you want to stay synchronized.

List of Decorate and ACS functions that utilize Actor Random

A_BulletAttack, A_Jump, A_CustomBulletAttack, A_CustomFireBullets, A_CustomPunch, A_RailAttack, A_CustomRailgun, A_SpawnItemEx, A_ThrowGrenade, Thing_Projectile, A_FireCustomMissile

When using sv_showactorrandom you may also see the engine mentioning internal functions like P_CheckMissileSpawn or P_SpawnPlayerMissile. Those functions are called internally by Decorate and ACS functions that spawn actors, like A_FireCustomMissile. You need to look for such Decorate functions when debugging your mod instead of the internal ones.

UnlaggedActor

The server unlagged feature is only applied to players. That is intentional for performance reasons, because doom maps can have thousands of monsters and hundreds of active projectiles. Q-Zandronum 1.3 introduces an UnlaggedActor for cases when you want non-player actors to be unlagged too. To use it, you need to make your actors inherit from UnlaggedActor like below:

ACTOR ExampleActor : UnlaggedActor
{
	...
}

With this the actor will be processed during server unlagged just like the players are. This feature is useful, for example, to make shootable projectiles that players can shoot no matter how bad their ping is. You can make unlagged monsters as well, but keep in mind that having too many unlagged actors will slow down server performance.

Also, if an UnlaggedActor dies during unlagged calculations, it's position will not be restored to current position, so that it's death state is processed where it died.

Unlagging ACS scripts

Unlike Decorate actors, whose states are calculated and functions are executed on both server and client simultaneously, the ACS scripts can run on only server, only client or both. Thus, if you want to create an actor using ACS and you want the client to predict it and the server to unlag it's spawn, make sure to call the corresponding spawn function on both server and client.

You can use a new int GetNetworkState() ACS function to tell whether the script runs in a single player game, a single player botmatch, a multi player client or a multi player server. Possible returns are NETSTATE_SINGLE, NETSTATE_SINGLE_MULTIPLAYER, NETSTATE_CLIENT, NETSTATE_SERVER respectively.

You can call a new void SetNetworkReplicationFlags(int) function to achieve the same behavior you get in decorate when using NOUNLAGGED, UNLAGDEATH and SKIPOWNER flags. In ACS the values are NETREP_NOUNLAGGED, NETREP_UNLAGDEATH and NETREP_SKIPOWNER respectively. After spawning the actor it is highly advised to execute SetNetworkReplicationFlags( 0 ) on the very same tick to reset default behavior, otherwise you might get unpredictable issues somewhere else in your ACS code.

Use a new int UnlaggedReconcile(int player_tid) when you want to reconcile actors back in time to do some unlagged calculations in your ACS scripts. The function returns the number of ticks it went back. Make sure to run void UnlaggedRestore(int player_tid) on the very same tick AND for the same player_tid to restore actors to their proper positions.

Dealing with desyncs

If you shoot a projectile and suddenly see two projectiles instead of one, then you probably have a desync. That happens when the client spawned a projectile with certain Net ID, but the server assigned a different Net ID to that projectile. Then, when a server command about it arrived to the client, it did not replace it's projectile with the server one, but spawned a new one alongside it.

To resolve that issue you need to make sure that whenever a player shoots or does something else the server and client spawn the same actors, in same order and with same Net IDs. You can use the sv_showspawnnames cvar on both server and client to see what actors they spawn and what are their Net IDs.

You can also use the void SyncPlayerNetwork(int player_tid) function to force resync a player with the server when you think that the player most likely desynched. This command is affected by ping and it may actually desync a synched player if used improperly. It is recommended to call this when a player isn't doing anything, like one a level change, or if the player is frozen.

Recap on unlagging the mod

• Make sure all important actors have the same Net ID on both server and client. Use sv_showspawnnames on both server and client to see what net IDs they got. Mark effect actors as clientside and +NONETID so that they don't receive Net ID from player's pool to reduce a desync probability and simplify debugging.
• Use +SERVERNETID when you want an actor to spawn both on server and client, but not receive a Net ID from player's pool.
• Make sure random is synchronized. Use sv_showactorrandom on both server and client to see if their random is in sync. Also replace the random(), random2() and frandom() expressions with their synched counterparts arandom(), arandom2() and afrandom() everywhere where it matters.
• Unlag custom character movement using Action assigned ACS scripts
• Unlag custom acs map scripts that affect player movement using Client predictable map ACS scripts

New flags and function parameters

Function	Function parameters	Description
A_GiveInventory and A_GiveToTarget	(string type [, int amount [, pointer giveto [, int flags]]])	Extra int flags parameter was added.
A_PlaySound	[(sound whattoplay [, int slot [, double volume [, bool looping [, double attenuation [, int ptr_activator]]]]])]	Extra int ptr_activator parameter was added. It is used to specify which player is responsible for making this sound, so that the client will do prediction properly.
A_StopSound	[(int slot) [, int ptr_activator]]	Extra int ptr_activator parameter was added. It is used to specify which player is responsible for stopping a sound, so that the client will do prediction properly.
A_ChangeFlag	(string flagname, bool value [, int flags])	Extra int flags parameter was added.
A_ThrowGrenade	(string spawntype [float zheight [, float xyvel [, float zvel [, bool useammo [, int flags]]]]])	Extra int flags parameter was added.

New flag name	Function	Effect
FPF_NOUNLAGGED CMF_NOUNLAGGED TGR_NOUNLAGGED SXF_NOUNLAGGED	A_FireCustomMissile A_CustomMissile A_ThrowGrenade A_SpawnItemEx	The server will not run unlagged calculations for the spawned actor, so the actor will spawn with no latency compensation.
FPF_UNLAGDEATH CMF_UNLAGDEATH TGR_UNLAGDEATH SXF_UNLAGDEATH	A_FireCustomMissile A_CustomMissile A_ThrowGrenade A_SpawnItemEx	By default, the server stops unlagging when an actor dies so that the actor could play it's death animation fully on clients. This flag will make it continue unlagging after the actor died.
FPF_SKIPOWNER CMF_SKIPOWNER TGR_SKIPOWNER SXF_SKIPOWNER	A_FireCustomMissile A_CustomMissile A_ThrowGrenade A_SpawnItemEx	The server will not send data about the spawned actor to the shooter's client, so the client will still have it's clientside actor. Keep in mind that in this case the clientside actor will not be synched with the server.
RGF_SKIPOWNER GIF_SKIPOWNER TIF_SKIPOWNER	A_RadiusGive A_GiveInventory and A_GiveToTarget A_TakeInventory and A_TakeFromTarget	The server will not send data about the spawned actor to the shooter's client, so the client will still have it's clientside actor. Keep in mind that in this case the clientside actor will not be synched with the server.
WARPF_SKIPOWNER CF_SKIPOWNER	A_Warp A_ChangeFlag	The server will not send data about the affected actor to the owner's client.
FPF_FORCESERVERSIDE CMF_FORCESERVERSIDE TGR_FORCESERVERSIDE SXF_FORCESERVERSIDE	A_FireCustomMissile A_CustomMissile A_ThrowGrenade A_SpawnItemEx	The actor will only spawn on the server and then replicate to clients. The shooter client will not try to predict spawn this actor at all.
RGF_FORCESERVERSIDE GIF_FORCESERVERSIDE TIF_FORCESERVERSIDE	A_RadiusGive A_GiveInventory and A_GiveToTarget A_TakeInventory and A_TakeFromTarget	The item will only be spawned and put in the inventory on server and then replicated to client. The client will not try to predict spawn this item at all.
JLOSF_SKIPOWNER	A_JumpIfTargetInLos	The server will send position and frame update to this actor's owner.

Actor flag	Description
+SERVERNETID	The actor with this flag will not receive a Net ID from player's pool and instead will get a Net ID from server's pool.

List of Decorate functions that are can now execute on clients

Please note that the below functions only execute on clients when the compat_predictfunctions cvar is ON

A_Jump, A_JumpIf, A_JumpIfTargetLOS, A_JumpIfHealthLower, A_JumpIfCloser, A_JumpIfInventory, A_JumpIfInTargetInventory, A_GiveInventory, A_GiveToTarget, A_TakeInventory, A_TakeFromTarget, A_DropInventory, A_RadiusGive, A_ChangeFlag, A_FireCustomMissile, A_CustomMissile, A_ThrowGrenade, A_CustomMeleeAttack, A_CustomComboAttack, A_CustomRailgun, A_MonsterRail, A_RailAttack, A_CustomPunch, A_BulletAttack, A_CustomBulletAttack, A_MeleeAttack, A_MissileAttack, A_ComboAttack, A_BasicAttack, A_ThrowGrenade, A_Explode, A_RadiusThrust, A_Detonate, A_SpawnItem, A_SpawnItemEx, A_Teleport, A_ChangeVelocity, A_ScaleVelocity, A_SetScale, A_Countdown, A_PlaySound, A_PlaySoundEx, A_StopSound, A_StopSoundEx, A_FadeIn, A_FadeOut, A_FadeTo, A_SetBlend, A_Recoil, A_Unblock, ACS_NamedExecute, ACS_NamedLockedExecute, ACS_NamedLockedExecuteDoor, ACS_NamedExecuteWithResult, ACS_NamedExecuteAlways, A_Chase, A_FastChase, A_VileChase, A_ExtChase, A_FaceTarget, A_FaceMaster, A_FaceTracer, A_RaiseMaster, A_RaiseChildren, A_RaiseSiblings, A_Raise, A_Respawn, A_Wander, A_MonsterRefire, A_GetHurt, A_BossDeath, A_Warp, A_Tracer, A_Fire, A_Die

List of Specials that can be called on client from both Decorate and ACS

ThrustThing, ThrustThingZ, ChangeCamera, SetPlayerProperty, ACS_Execute, ACS_ExecuteAlways, ACS_ExecuteWithResult

New ACS functions

• Please make sure to read Modding prerequisites page in order to use new ACS functions listed below.
• Unlike the Decorate functions, the ACS functions can be executed on clients regardless of the compat_predictfunctions flag. However, they will not run clientside prediction without that flag being ON.
• When using compat_predictfunctions, the PlaySound(), PlayActorSound() and StopSound() functions don't replicate to the player that called this function, if it is executed on server side. To work around that you have to call the function on client as well.

Function	Description
int GetNetworkState()	Tells whether this is a single player game, a single player botmatch, a multi player client or a multi player server. Possible returns are NETSTATE_SINGLE, NETSTATE_SINGLE_MULTIPLAYER, NETSTATE_CLIENT, NETSTATE_SERVER.
void SetNetworkReplicationFlags(int)	This is the ACS counterpart to the Decorate NOUNLAGGED, UNLAGDEATH, SKIPOWNER and FORCESERVERSIDE flags listed above. Possible values are NETREP_NOUNLAGGED, NETREP_UNLAGDEATH and NETREP_SKIPOWNER. You can combine them using the `
int UnlaggedReconcile()	This function exposes the Server Unlagged functionality to ACS scripts. The function will reconcile all players except for activator and all unlagged actors back to where they were based on the activator's ping. Make sure to run UnlaggedRestore() on the very same tick or else it will break the game! The function returns the number on ticks it went back for.
void UnlaggedRestore()	Restores the reconciled players and actors positions to where they were before using UnlaggedReconcile().

List of ACS functions affected by NETREP flags

NETREP flag	Functions
NETREP_NOUNLAGGED	Thing_Projectile, Thing_Projectile2, SpawnProjectile
NETREP_UNLAGDEATH	Thing_Projectile, Thing_Projectile2, SpawnProjectile
NETREP_SKIPOWNER	Spawn, SpawnDirect, SpawnSpot, SpawnSpotDirect, SpawnSpotFacing, SpawnForced, SpawnSpotForced, SpawnSpotFacingForced, Thing_Projectile, Thing_Projectile2, SpawnProjectile, GiveInventory, GiveActorInventory, TakeInventory, TakeActorInventory, ThrustThing, ThrustThingZ, Thing_Stop, SetActorAngle, ChangeActorAngle, SetActorPitch, ChangeActorPitch, SetActorPosition, SetActorProperty, SetWeapon, HudMessage, HudMessageBold, FadeTo, FadeRange, Warp, PlaySound, PlayActorSound, StopSound, SectorSound, AmbientSound, LocalAmbientSound, SoundSequence
NETREP_DELAYTHRUST	SetActorVelocity

• When the NETREP_DELAYTHRUST flag is used, the server will try to smooth velocity change for the player by notifying him immediately and delaying the thrust on server side by player ping.

Default Doom, Heretic, Hexen and Strife weapons are unlagged out of the box!

It's as simple as the title says! Just set the compat_predictfunctions to true and play!

• While Hexen weapons are unlagged, Hexen levels use ACS scripts a lot and need to be unlagged manually. Because of that, stuff like the initial poly door in the first level would not play sound when using compat_predictfunctions.