Streaming interface

[dangmai]

Right now in order to parse the demo, we have to read the entire file first. I think it'd be really cool if we can also support Node streams, so that we can have this stream: download .dem.bz file from Valve server -> unzip -> parse. It'd be much faster than download the zip file, unzip it completely, then read the unzipped file and parse it.

I thought about trying demofile as is, to see if it'd just magically work, using the following code:

function streamingAnalyze(stream, done) {
  const demoFile = new demofile.DemoFile();
  stream.pipe(through2(function handle(chunk, enc, callback) {
    demoFile.parse(chunk);
    callback();
  }))
  .on('finish', () => done(result));
}

But it failed, throwing the following stack trace:

C:\Dev\demo\node_modules\bytebuffer\dist\bytebuffer-node.js:499
                throw RangeError("Illegal offset: 0 <= "+offset+" (+"+length+") <= "+this.buffer.length);
                ^

RangeError: Illegal offset: 0 <= 170 (+107899) <= 64464
    at ByteBuffer.Object.<anonymous>.module.exports.ByteBufferPrototype.readBytes (C:\Dev\demo\node_modules\bytebuffer\dist\bytebuffer-node.js:499:23)
    at ByteBuffer.Object.<anonymous>.ByteBuffer.readIBytes (C:\Dev\demo\node_modules\demofile\ext\bytebuffer.js:21:15)
    at DemoFile._handleDemoPacket (C:\Dev\demo\node_modules\demofile\demo.js:203:31)
    at DemoFile._parseRecurse (C:\Dev\demo\node_modules\demofile\demo.js:312:14)
    at ontimeout (timers.js:380:14)
    at tryOnTimeout (timers.js:244:5)
    at Timer.listOnTimeout (timers.js:214:5)

[saul]

Great idea. I don't think it's going to be particularly simple to implement though. At the moment the library expects the whole demo to be available in the buffer.

We'd need to make it aware that if the rest of the file isn't available, to hang back until we get more data in the stream.

[dangmai]

You're right, I read through the code and realize that demofile depends on binary-parser, which does not support streams. There's an outstanding ticket keichi/binary-parser#40 for that, but the project in general looks pretty inactive to me so I won't hold my breath.

Feel free to close the ticket or keep it open if you intend to implement this in the future.

[zoidyzoidzoid]

There are some newer binary parsers, or someone could fork and update the keichi parser to support streams, if someone ever really wants this.

I found this one in a quick search too: https://github.com/Pupix/binary-buffer-parser

[saul]

We'd just need to buffer enough until we had enough data to parse a tick, there isn't really any need to use a different binary parser.

[thecatontheflat]

As an easy win, maybe we could expose some controls over the parsing process via demofile.pause() / demofile.resume() methods? This way it would be possible to tell the library to wait until there's a new chunk of buffer available.

[PrimeGoose]

will this trigger anti cheat attention?

[saul]

No, this won't trigger anti cheat detection. Demos are buffered in memory for a few minutes before they're flushed to disk. So what's on disk is always a few minutes behind what's actually happening in the match. Valve thought of this :)

[saul]

I've raised a PR for this here: #328

[saul]

I'm having a reproducible issue with Node v12 on that PR. For some reason the stream is ending early, just after parsing the header. Does anybody have any ideas? It works fine on Node v14 and I can't reproduce the issue on my local Linux VM with Node v12 either.

[saul]

This will imminently be released as v2.5.0. Considering Node v12 will be end-of-life by April, I've decided to drop support for it. This library now requires Node v14+.

I've also updated the examples to use the streaming interface and it's tested as part of CI.

Apologies for the slow turnaround on this ticket!

[LeBesta]

No, this won't trigger anti cheat detection. Demos are buffered in memory for a few minutes before they're flushed to disk. So what's on disk is always a few minutes behind what's actually happening in the match. Valve thought of this :)

Sorry if this is not the place to ask, but does this mean the demoparser itself is a few minutes behind as well, or is it able to parse the game in real time? (as in at the same time as its livestream)

[saul]

HLTV streams are already delayed for precisely the same reason. There is no delay added by this library though.