Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Server side demos #265

Open
wants to merge 9 commits into
base: master
from

Conversation

Projects
None yet
6 participants
@lrq3000
Copy link

lrq3000 commented Mar 4, 2017

This patch implements server-side demos in a mod-agnostic way, since it only relies on server-side modification and data.

This patch works by storing all entities states info and then replaying them like "ghost" entities. Both servers and clients can replay these demos (servers are also accessible for spectators, making this patch a great replacement for GTV - it could also be extended to allow realtime rebroadcasting of games).

This patch was already tested on several competitions servers for years (eg, this current ExcessivePlus competition), so it is quite stable, and memory issues have been resolved using Valgrind.

TODO:

  • Add in ioq3 readme the new commands (and delete the server-side demos readme, it's provided for your info, as I don't know how to format the ioq3 readme).
  • In code/qcommon/msg.c there is an unclean include of g_public.h, which I don't know how to workaround, any help would be greatly appreciated:
#include "../game/g_public.h" // FIXME: necessary for entityShared_t management to work (since we need the definitions...), which is a very necessary function for server-side demos recording. It would be better if this functionality would be separated in an _ext.c file, but I could not find a way to make it work (because it also needs the definitions in msg.c, and since it's not a header, these are being redefined when included, producing a lot of recursive declarations errors...)

lrq3000 added some commits Mar 3, 2017

Fix Windows 32bits compilation
Signed-off-by: Stephen L. <lrq3000@gmail.com>
Port of server-side demo patch v0.9.9.5 by Amanieu and LRQ3000
Signed-off-by: Stephen L. <lrq3000@gmail.com>
Fix mover state + bump v1.0.0
Signed-off-by: Stephen L. <lrq3000@gmail.com>
Add readme for server-side demos
Signed-off-by: Stephen L. <lrq3000@gmail.com>
@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 4, 2017

This patch is quite a lot of new code for this new functionality (but not so much compared to other similar patchs). Also, a maximum of code has been separated into separate modules sv_demo.c and sv_demo_ext.c.

If you guys choose not to merge it, would it be possible to add this patch in the patchs list on the website? Also, it could be useful to add these patchs, which are not mine but also implement server-side demos but in an alternative way, by replaying network snapshot packets:

They both work BTW, although they offer slightly different functionalities (my PR implements full server-side demos replay, so it allows freeflying for example, whereas these two patchs implement something more like a multiview demo, without freeflying).

@ensiform

This comment has been minimized.

Copy link

ensiform commented Mar 4, 2017

IMO server creating client demos that can be stored in a web archive or sent off is a better approach especially since this technically adds a bunch of hacks that snoop the gamecode.

We have a simpler solution that just generates client demos on the server in OpenJK but it spans several commits otherwise I would list them here directly.

Here was the initial first commit for starting: JACoders/OpenJK@6073f79

@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 4, 2017

@ensiform nice but are you recording a separate demo for each client, or is it all concatenated into one demo per game at the end?

@ensiform

This comment has been minimized.

Copy link

ensiform commented Mar 4, 2017

Its per client demos stored in folders. AFAIK the servers which use it send the demos off. It has other features like keep for so long etc..

@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 4, 2017

@ensiform Ah I understand, then this is a different purpose from this PR, your approach is similar to TheDoctor's patch. It's indeed the most lightweight in terms of code modification, but you don't have freeflying, players switching nor rebroadcast (ie, replaying a demo on a server so players can connect and spectate). I wanted these features so I implemented a more heavy, simulation oriented approach. But most of the code is separated so it's not that bad I think ;) But I'd love to get some help to modularize even more!

About deleting old demos and sending them off, yes that's also what I did, players love that ;)

@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 4, 2017

It's great to hear multiple projects tried to implement server-side demos. I hope we can converge towards a centralized approach. This PR and the patchs by other authors provided above are mod-agnostic, so normally it should work for any ioq3 based game!

@ec-

This comment has been minimized.

Copy link

ec- commented Mar 4, 2017

@lrq3000 in order to apply my patch you also need to fix those engine limitation ec-/Quake3e@ef47444
Also, as long as you're referring to game' gentity_t - it is not mod-agnostic because ent->health may have different offset in other mods - which potentially leads to read/write data corruption
Also, freefly is not an engine functionality - it is generally gamecode functionality because it may heavily depend from game/cgame spectator code

@ensiform per-client demos have at least 3 obvious disadvantages:

  1. you can't switch pov during playback
  2. you need +numClients storage to save them
  3. simultaneous +numClients demo record will be heavily fragmented

While my implementation:

  • trully mod-agnostic - because it is pure demo, not game replay
  • you can switch pov on the fly
  • you can record dm_68-compatible demo during that (for q3mme for example)
  • every map entity being rerorded so you can recover even things that was not visible to players - this is very useful for freefly
  • produces non-fragmented, signle-stream with efficient entity/playerstate compression (3-5x savings for 15+ players)
  • fast and efficient server command compression - up to 4x-8x for long messages like scores
@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 5, 2017

Very nice @ec-, thank you very much for releasing your patch and for the pointer, that's a really nice work you did there! I can see your implementation as a great replacement for GTV.

I did not try your patch yet nor did I look deeply in the code, so I did not know you were recording all entities. So it means you can also allow freeflying, why don't you enable this?

About my patch, yes gentity_t->health is used but not required, it's just to update health on the HUD. Also, it should not be needed, it's only so because of a mistake in ioq3 code which should be fixed (normally gentity_t->health should be totally removed and playerState_t->stats[STAT_HEALTH] should be used instead! At least that's what is written in the code comments...). So if this would be fixed, the patch would become totally agnostic.

About freeflying, this is mod-agnostic in this implementation, because this implementation allows it by design: we do a full simulation of entities, so it's just like freeflying in a normal game, except that the entities are simulated. So freefly is still managed by gamecode, the engine just allows this by providing all entities.

Anyway, this implementation consumes a lot more space than yours, but weirdly it seems it can allow for more players (I had no troubles with more than 20, and the number of spectators is no issue), whereas rebroadcasting messages seems to be more heavy from what I read in your forum? That's surprising, I would have thought your implementation to be much more efficient than mine (like GTV, supporting hundreds of spectators at once!).

BTW, can you validate my account on your forum, would be nice to talk there too :)

@ec-

This comment has been minimized.

Copy link

ec- commented Mar 5, 2017

@lrq3000 you're called your implementation properly: it is server-side demo while mine is multiview protocol + server-side multiview demo recording
IIRC you're relying on server/game code during playback - while I'm not and this is major design difference. Moreover: for potential GTV-proxy - there is no need to have any primary host content like mod,map,etc.
Freefly needs fake entity/playerstate emulation on clientside - which doesn't belongs to protocol by itself so I skipped that for now (as many other things like better event/command compression) to keep code clean.

There is no any players/spectators limits as long as resulting snapshots is shorter than 16kbytes, command/chat filtering is not a proplem too (tell commands are already filtered during ss-record).
Each multiview slot that can accept mvjoin/mvleave (not counting 1 already hardcoded for server demo) takes about 4x memory (6.5m vs 1.7m) and this is unaviodable because it required for correct delta compression - so for 128 multiview spectator slots you need about 900mb of RAM. You may argue that original GTV have less RAM requirements but it handles usual connection slots backed up by buggy mod-side multiview, not pure multiview

We have some troubles with site so I don't see your post atm, try later or goto ioq3' irc channel

@JockeTF

This comment has been minimized.

Copy link

JockeTF commented Mar 5, 2017

I've been a big fan of server-side demos ever since I used to run an Ultra Freeze Tag server. Having that built into the server across all mods would be very nice. I'm afraid I can't give any input regarding any particular implementation, but free-flying and POV-switching is definitely something I care about.

lrq3000 added some commits Mar 5, 2017

Better messages when recording + chat messages (in addition to consol…
…e) when recording

Signed-off-by: Stephen L. <lrq3000@gmail.com>
Fix type in var name
Signed-off-by: Stephen L. <lrq3000@gmail.com>
@ensiform

This comment has been minimized.

Copy link

ensiform commented Mar 5, 2017

For what it's worth, the demos are hosted here via the OJK method (though I think it may be modified with curl and other things): http://demos.jactf.com/match.html

lrq3000 added some commits Mar 7, 2017

Better autorecord hooks + partial fix for tournament < 2 players auto…
…join for E+

Signed-off-by: Stephen L. <lrq3000@gmail.com>
Update README
Signed-off-by: Stephen L. <lrq3000@gmail.com>
@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 7, 2017

Thank you guys for the feedback, your comments are very useful.

@ensiform Very nice webapp to browse demos, you even have a search feature, awesome! :D Thank you for sharing.

@ec- Yes we had different approaches, but both I think have merits and disadvantages. And, as you say, you can even implement freefly by adding some code similar to my approach, so maybe we could merge our patches :p But let's focus on what approach is better with minimal work (ie, what we have now).

I think the biggest disadvantage of your approach is that it takes a lot of RAM per client, are you sure there is no way to reduce that, for example by allowing for uncompressed delta?

On the other hand, is your patch is 100% compatible with any mod? (not just mod-agnostic, but also 100% compatible with any gamecode modification). If that's the case (and I guess so), then that's a GREAT advantage. My patch is mod-agnostic too but not 100%, as messing with the gamecode might produce buggy playback (eg, if the gamecode uses new variables to define players statuses, then the patch cannot replay them correctly as it has no way to get a hold of these new variables that only exist in the gamecode, so the record and playback WILL work, but will be missing these special playerstatus variables). There could be a way to fix this: by creating a generic trap call bridge from server code to game code to access and to set virtually any variable, but I'm not sure this level of genericity is possible with C (I'm not experienced enough with this language).

In my opinion, if your patch is 100% compatible with any mod, and if you can find a way to reduce RAM per client (even if it's optional with a cvar for example and even if it requires more storage/network bandwidth), then your patch is a better fit for a PR to the core ioq3 than mine.

But I do think that my PR should at least live as an official optional patch, I'm sure people can find useful usage (eg, for research by storing all entities and events to analyze later with another software).

@ensiform

This comment has been minimized.

Copy link

ensiform commented Mar 8, 2017

If you can get your hands on q3f beta 2 builds still, I would test with those too because it is a total conversion.

@ec-

This comment has been minimized.

Copy link

ec- commented Mar 8, 2017

@lrq3000 you need memory for (democlients * PACKET_BACKUP * 1024) instead of (democlients * PACKET_BACKUP * 256) for working delta-compression on lagging clients, you can (locally) reduce PACKET_BACKUP at cost of higher probability to drop delta compression for them - tradeoff is possible but I don't see any need in that atm

PACKET_BACKUP in a number of server frames you can buffer to keep delta compression work for lagging client, q3 value is 64 i.e. at sv_fps 40 (cpma) you have 1.6s ping/lag window, for PACKET_BACKUP=16 you will have 0.4s etc.

Of course, all this related to broadcasting multiview slots as server-side multiview demo recording costs nothing

You need only client binary to handle multiview stream and cgame to play demo, so for example - I can playback multiview cpma stream in my mod, server/qagame code is absolutely unused at playback

@Chomenor

This comment has been minimized.

Copy link

Chomenor commented Mar 8, 2017

@ec-

Isn't it just PACKET_BACKUP * 1024? Aren't the entities the same for every client?

@lrq3000

This comment has been minimized.

Copy link
Owner Author

lrq3000 commented on f218c6c Mar 8, 2017

This fixes a typo, not a type...

@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 9, 2017

@ec- Ok I understand now why you need 4x memory per democlient feed, but I don't understand why you need to multiply this amount per the number of real spectator. I don't have to dig into your code, but maybe you can provide a quick outline of your architecture, particularly how you feed the real spectators with democlient replays?

For example, my entities-oriented architecture is quite simple: I record every events/entities/playerStates at demo recording, and for playback it just hooks at the end of each server frame and overwrites with demo events. This way, demo events always take the upper hand on server's events.

For your architecture, how do you assign each feed to one client? Do you send all feeds to all clients, and the clients choose the feed?

When in the past I thought about possible ways to implement your approach, I thought that the feeds should be assigned beforehand to each client, and then the server should broadcast only the necessary feeds to the clients who want them. So with a /mview 1, the client would basically assign himself for feed 1, and at playback, the server would make a list of all clients assigned to feed 1 (=democlient 1) and broadcast at once to all of them the same data, then do the same for feed 2, etc. Do you use a different approach, and if so why?

@ec-

This comment has been minimized.

Copy link

ec- commented Mar 9, 2017

@Chomenor You should look and unserstand this first https://github.com/ioquake/ioq3/blob/master/code/server/sv_init.c#L350

@lrq3000 Everything is coming through network/message layer in my implementation, so (on multiview slots) in each snapshot I'm sending:

  • all linked entities (up to 1024 comparing to 256 in regular snapshots) not depending from their visibility
  • each playerstate (even spectators) with entity' visibility bitmask

On client side:

  • decompress all entities and playerStates on snapshot arrival

At demo playback, when cgame calls trap_GetSnapshot [ i.e. CL_GetSnapshot() ] - I'm just extract playerState I've selected (via \mvfollow or MOUSE2) and fill entities (up to 256 from playerState' visibility bitmask) from common linked entity set, i.e. no interaction with server/game code at all

So from each multiview slot you are receiving (and recording) ALL possible data you need to reconstruct ANY pov for later playback - this is my general concept and it works flawlessly

@Chomenor

This comment has been minimized.

Copy link

Chomenor commented Mar 9, 2017

@ec- My suspicion is that implementation is inefficient and you could optimize it to store the entities on a per backup snapshot basis, not per client basis. You would normally need to add a per client visibility mask, so you don't try to delta from a previously hidden entity, but since you say you are sending all entities without respect to visibility you might not even need that.

@ec-

This comment has been minimized.

Copy link

ec- commented Mar 9, 2017

@Chomenor I guess I understand what you mean but each client has its own shapshot timing
https://github.com/ioquake/ioq3/blob/master/code/server/sv_snapshot.c#L670 and it is not synced with actual GAME_RUN_FRAME call.
Also, GAME_CLIENT_THINK comes from multiple clients with much higher than sv_fps frequency so even client with +1ms shifted ->lastSnapshotTime may have completely different entity parameters in his snapshot
It may be optimized by scanning and not allocating space for equal entities but you need that space for worst case anyway

@Chomenor

This comment has been minimized.

Copy link

Chomenor commented Mar 9, 2017

@ec- Yes, you would need to compensate for clients that skip frames. You would maybe store a server entity frame number in client->frames, in place of num_entities and first_entity that is used for the old entity system. If the frame doesn't exist anymore you would fall back to a non-delta snapshot.

I'm not aware that lastSnapshotTime changes entity contents. I think at least in vanilla ioq3, SV_SendClientMessages is called, and it generates (or skips) a snapshot for every client with no entity changes in between. Other implementations may be different, and of course that would affect the entity storage requirements.

As for optimizing equal entities, I agree on both points.

@Chomenor

This comment has been minimized.

Copy link

Chomenor commented Mar 9, 2017

I'd also like to throw out another idea for a demo implementation. On the ioquake3 side, you write the relevant data (gamestate, entities, playerstates, commands, etc) to a standard binary stream format that can be sent either to a file or to a network port. Then you create a separate "demo client" application (based on the ioq3 dedicated server) that can read from either the file or real time network stream, and either write a standard demo from a specific client's perspective or emulate a server that people can connect to with client switching and perhaps freefly functionality.

The advantages of this approach:

  • Very minimal code and changes in main ioquake3 project, and nothing mod-dependant
  • Supports server admins with downloadable demo archives (the demo client works as a command line tool to generate standard format demos on the webserver)
  • Supports event live viewing with unlimited viewers (especially if you add a function to demo client to mirror to other demo clients) and no chance of messing with the main game server
  • Supports freefly replay for promotional videos and such, by running demo client in server emulation mode with a file input
  • If any mod-specific tweaks are necessary they can be done with a customized demo client without changing the main ioq3 code running on the server

The main downside is it could be trickier to implement, especially keeping the timing stuff straight for the real time stream. Nothing insurmountable, though.

Implementation detail: The demo client should just ignore a stream until it receives a gamestate. Once you set up a connection you need to change the map on the real server for the demo client to start functioning as a spectator server. This makes things a lot simpler.

@ec-

This comment has been minimized.

Copy link

ec- commented Mar 9, 2017

@Chomenor

store a server entity frame number in client->frames, in place of num_entities and first_entity

Looks like a whole snapshot system redesign on top of "full entity set + bitmask" approach, may be good if you're agree to trade some CPU time for RAM on server side

that can read from either the file or real time network stream

You need to provide delta-compression anyway to get reasonable bandwidth

@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 9, 2017

@Chomenor if I'm nod mistaken, this sounds a lot like my PR: you basically want to save the game events and replay them faithfully.

The problem for mod compatibility is not the same as being mod agnostic : by implementing server side demos in server code, you are mod agnostic, but not necessarily mod compatible. Indeed, the game code might define new variables and events that are not stored in server, so you cannot save them in a demo. In other words, saving the full server state does not guarantee that you save the full game state.

@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 9, 2017

@ec- ok i understand better, thank you for the outline. But i still do not understand why the RAM amount is dependent on the number of multiview spectators. Is it because you need the delta, so you store the previous state of each mv spec (like ioq3 does for real players in normal games)? If that's the case, can't you just store one state only and duplicate to all clients (just modifying the address), since anyway all mv spec receive the same data, and thus the same delta?

@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 9, 2017

(tell me if I'm wrong about delta, i guess that in case of a demo, the state at any point in time should be the same for everyone, and so calculating the delta between two states would yield the same result for everyone too, only the UDP datagram headers and encryption would change, but not the delta contained inside).

@ec-

This comment has been minimized.

Copy link

ec- commented Mar 9, 2017

@lrq3000

all mv spec receive the same data

no, because each client have its own lastSnapshotTime, rate and snaps settings

I can try to rework whole snapshot system in "full entity set + visibility bitmask" concept to save RAM even for regular slots. But I'm afraid that those significant and fragile changes will be never pushed to ioq3 and people will start loosing their mind trying to backport my multiview patch in their engines

You are trying to cover everything, don't do that, just keep it simple

@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 10, 2017

@ec- No no I understand, I'm not trying to cover everything, I was trying to simplify your approach, but it seems my idea of your implementation was a bit too naive, so there is no way around because of the varying snaps and rate among spectators :/ You should make a clean PR to ioquake3 :)

Aside note: there are other interesting implementations:

@ensiform

This comment has been minimized.

Copy link

ensiform commented Mar 10, 2017

ET:Legacy is GPLv3+ with Zenimax clauses, so you can't really use their code with GPLv2 unless it is solely someone's code that wasn't originally in the codebase which said author allows you to re-license.

And upgrading ioquake3 to GPLv3 nullifies OpenJK and jk2mv and any other jk2/jka projects from using further upstream ioq3 fixes because of the way things are licensed.

@lrq3000

This comment has been minimized.

Copy link
Author

lrq3000 commented Mar 10, 2017

@rmarquis

This comment has been minimized.

Copy link

rmarquis commented Mar 26, 2017

@lrq3000 I believe relicensing ET:Legacy modification shouldn't be an issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.