Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for seeking to arbitrary ticks #49

Merged
merged 21 commits into from
Feb 29, 2024
Merged

Add support for seeking to arbitrary ticks #49

merged 21 commits into from
Feb 29, 2024

Conversation

saul
Copy link
Owner

@saul saul commented Feb 23, 2024

This PR adds a new SeekToTickAsync method that can jump to arbitrary ticks within the demo. It supports seeking backwards and forwards, and makes use of Source 2 demo 'snapshot' ticks to do this efficiently.

  • Refactor the code out of DemoParser.cs
  • Add unit tests

Closes #47

@saul saul mentioned this pull request Feb 23, 2024
Copy link

github-actions bot commented Feb 24, 2024


BenchmarkDotNet v0.13.9+228a464e8be6c580ad9408e98f18813f6407fb5a, Ubuntu 22.04.4 LTS (Jammy Jellyfish)
AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
.NET SDK 8.0.200
  [Host]     : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
  Job-LFXIPO : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
  Job-KRGKAI : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2

InvocationCount=1  MaxIterationCount=16  UnrollFactor=1  
WarmupCount=1  

Method Job Arguments Mean Error StdDev Ratio RatioSD Gen0 Gen1 Allocated Alloc Ratio
ParseDemo Job-LFXIPO /p:Baseline=true 2.335 s 0.0972 s 0.0955 s 1.00 0.00 6000.0000 1000.0000 594.31 MB 1.00
ParseDemo Job-KRGKAI Default 2.195 s 0.0119 s 0.0106 s 0.95 0.04 5000.0000 2000.0000 489.9 MB 0.82

@saul saul changed the title [WIP] Add support for seeking to arbitrary ticks Add support for seeking to arbitrary ticks Feb 25, 2024
@@ -9,18 +9,25 @@ namespace DemoFile;

public sealed partial class DemoParser
{
private readonly ArrayPool<byte> _bytePool = ArrayPool<byte>.Create();

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use shared ArrayPool, or allow the user to specify it (in constructor, or as a property)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the reason is, shared ArrayPool (or custom provided) may already have ready buffers, so that way you avoid allocating new buffers

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the simplest way of doing this:

private readonly ArrayPool<byte> _bytePool = ArrayPool<byte>.Shared;
public ArrayPool<byte> BytePool { get => _bytePool; init => _bytePool = value; }

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason to not use the shared ArrayPool is that it is thread-safe, and the locking is unnecessary overhead. I can't think of other libraries that expose the ArrayPool that is used.

the reason is, shared ArrayPool (or custom provided) may already have ready buffers, so that way you avoid allocating new buffers

This is true, but as long as the same few arrays are used during parsing, it doesn't really matter. We're talking allocations of a handful of arrays here over the whole demo

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

afaik every ArrayPool is thread-safe : https://learn.microsoft.com/en-us/dotnet/api/system.buffers.arraypool-1?view=net-8.0#thread-safety

but, you're right that there will be only a few arrays alive at any given time, so I guess it's fine

Copy link

@in0finite in0finite Feb 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

speaking of which, I've seen some places where 16 length array is rented, this creates overhead, maybe it's better to use Span<byte> span = stackalloc byte[16]; ?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't stackalloc in async methods

@in0finite
Copy link

when i do this:

        await demo.StartReadingAsync(File.OpenRead(path), default);

        await demo.SeekToTickAsync(new DemoTick(500), default);

it gives me:

System.IndexOutOfRangeException
  HResult=0x80131508
  Message=Index was outside the bounds of the array.
  Source=DemoFile
  StackTrace:
   at DemoFile.DemoParser.OnPacketEntities(CSVCMsg_PacketEntities msg) in /_/src/DemoFile/DemoParser.Entities.cs:line 418
   at DemoFile.PacketEvents.ParseNetMessage(Int32 msgType, ReadOnlySpan`1 buf) in /_/src/DemoFile/PacketEvents.cs:line 136
   at DemoFile.DemoParser.OnDemoPacket(CDemoPacket msg) in /_/src/DemoFile/DemoParser.cs:line 134
   at DemoFile.DemoParser.OnDemoFullPacket(CDemoFullPacket fullPacket) in /_/src/DemoFile/DemoParser.FullPacket.cs:line 157
   at DemoFile.DemoEvents.ReadDemoCommand(EDemoCommands msgType, ReadOnlySpan`1 buffer) in /_/src/DemoFile/DemoEvents.cs:line 48
   at DemoFile.DemoParser.<MoveNextCoreAsync>d__67.MoveNext() in /_/src/DemoFile/DemoParser.cs:line 287
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at DemoFile.DemoParser.<SeekToTickAsync>d__125.MoveNext() in /_/src/DemoFile/DemoParser.FullPacket.cs:line 98
   at Program.<Main>d__0.MoveNext()
   at Program.<Main>(String[] args)

@saul
Copy link
Owner Author

saul commented Feb 25, 2024

Just pushed a fix for the IndexOutOfRangeException - can you try now?

@in0finite
Copy link

If I seek to 5000, it gives me 1 player death event, when I seek to 50000, there are no death events. Is this expected ?

@in0finite
Copy link

Also,
for match (map Overpass) : https://www.hltv.org/matches/2369507/faze-vs-spirit-iem-katowice-2024
I get this:

Unhandled exception. System.Exception: Delta on non-existent entity 1
   at DemoFile.DemoParser.OnPacketEntities(CSVCMsg_PacketEntities msg) in /_/src/DemoFile/DemoParser.Entities.cs:line 366
   at DemoFile.PacketEvents.ParseNetMessage(Int32 msgType, ReadOnlySpan`1 buf) in /_/src/DemoFile/PacketEvents.cs:line 136
   at DemoFile.DemoParser.OnDemoPacket(CDemoPacket msg) in /_/src/DemoFile/DemoParser.cs:line 134
   at DemoFile.DemoEvents.ReadDemoCommand(EDemoCommands msgType, ReadOnlySpan`1 buffer) in /_/src/DemoFile/DemoEvents.cs:line 33
   at DemoFile.DemoParser.MoveNextCoreAsync(EDemoCommands msgType, Boolean isCompressed, Int32 size, CancellationToken cancellationToken) in /_/src/DemoFile/DemoParser.cs:line 303
   at DemoFile.DemoParser.SeekToTickAsync(DemoTick targetTick, CancellationToken cancellationToken) in /_/src/DemoFile/DemoParser.FullPacket.cs:line 98
   at Program.Main(String[] args) in /_/examples/DemoFile.Example.Basic/Program.cs:line 19
   at Program.<Main>(String[] args)

@saul
Copy link
Owner Author

saul commented Feb 25, 2024

You can't depend on game events etc. occuring while seeking. This is the reason why Valve removed the round_start/end events as you can't rely on them to know when rounds are starting and finishing.

As for this:

Unhandled exception. System.Exception: Delta on non-existent entity 1

Please can you share the code that reproduces the exception?

@in0finite
Copy link

Please can you share the code that reproduces the exception?

var path = "test.dem";

var demo = new DemoParser();

demo.Source1GameEvents.PlayerDeath += e =>
{
    Console.WriteLine($"{e.Attacker?.PlayerName} [{e.Weapon}] {e.Player?.PlayerName}");
};

await demo.StartReadingAsync(File.OpenRead(path), default);

await demo.SeekToTickAsync(new DemoTick(50000), default);

@in0finite
Copy link

You can't depend on game events etc. occuring while seeking

But why are game events firing while seeking ? In my understanding, parser should ignore all events (and entity changes) while seeking.

@saul
Copy link
Owner Author

saul commented Feb 25, 2024

I've just pushed another change that should fix the seeking to tick 50,000 issue

@in0finite
Copy link

It works now, but game events are still being fired. Is it possible to seek without invoking any callbacks ? Or without parsing anything except StringTables ?

@in0finite
Copy link

I get some weird error:

Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
   at System.Collections.Generic.List`1.set_Item(Int32 index, T value)
   at DemoFile.StringTable.ReadUpdate(ReadOnlySpan`1 stringData, Int32 entries) in /_/src/DemoFile/StringTable.cs:line 126
   at DemoFile.DemoParser.OnUpdateStringTable(CSVCMsg_UpdateStringTable msg) in /_/src/DemoFile/DemoParser.StringTables.cs:line 61
   at DemoFile.PacketEvents.ParseNetMessage(Int32 msgType, ReadOnlySpan`1 buf) in /_/src/DemoFile/PacketEvents.cs:line 106
   at DemoFile.DemoParser.OnDemoPacket(CDemoPacket msg) in /_/src/DemoFile/DemoParser.cs:line 134
   at DemoFile.DemoEvents.ReadDemoCommandCore[T](Action`1 callback, MessageParser`1 parser, ReadOnlySpan`1 buffer, Boolean isCompressed) in /_/src/DemoFile/DemoEvents.cs:line 79
   at DemoFile.DemoEvents.ReadDemoCommand(EDemoCommands msgType, ReadOnlySpan`1 buffer, Boolean isCompressed) in /_/src/DemoFile/DemoEvents.cs:line 37
   at DemoFile.DemoParser.MoveNextCoreAsync(EDemoCommands msgType, Boolean isCompressed, Int32 size, CancellationToken cancellationToken) in /_/src/DemoFile/DemoParser.cs:line 301
   at Program.TestBug(String path) in /_/examples/DemoFile.Example.Basic/Program.cs:line 73

with this code :

public static async Task TestBug(string path)
{
    var stream = new MemoryStream(File.ReadAllBytes(path));

    var demo = new DemoParser();

    await demo.StartReadingAsync(stream, default);

    var tickCount = demo.TickCount.Value;

    const int FullPacketInterval = 64 * 60;

    for (int i = FullPacketInterval * 4; i <= tickCount; i += FullPacketInterval)
    {
        Console.WriteLine($"Seeking to tick {i}");

        demo = new DemoParser();

        //stream = new MemoryStream(File.ReadAllBytes(path));
        stream.Position = 0;
        await demo.StartReadingAsync(stream, default);

        await demo.SeekToTickAsync(new DemoTick(i), default);

        while (await demo.MoveNextAsync(default))
        {
        }
    }
}

@saul
Copy link
Owner Author

saul commented Feb 26, 2024

This is because DemFullPackets only contains the string tables that changed since the last DemFullPacket.

I'm going to have to rethink how stringtables are stored to allow seeking. Will let you know when I've pushed a fix.

@saul saul merged commit cad95be into main Feb 29, 2024
2 checks passed
@saul saul deleted the tick-step-seek branch February 29, 2024 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Seeking support
2 participants