Faster Parsing

[denniske]

Hi, I want to parse a lot recorded matches. At the moment parsing one match takes 1-10 seconds on my machine. I think I am already using the fast parsing option.

I am only interested in game duration and which player has won so I do not need to parse all data.

Is there a way to speed up the parsing process?

I have noticed that a large portion of the processing is parsing the map tiles. Can I somehow skip this?

[happyleavesaoc]

Most of the time is probably spent parsing the header. "fast mode" only affects the body right now. If you only need those two pieces of data, you could skip parsing the header entirely (read header length, then skip forward to read the body -- I can provide an example if you want).

[denniske]

Yes that sounds good. An example would be great :)

[happyleavesaoc]

Here's an example:

import struct
import os
from mgz import header, fast

with open('/path/to/file', 'rb') as data:
    eof = os.fstat(data.fileno()).st_size
    prefix_size = 8
    header_len, next_header = struct.unpack('<II', data.read(prefix_size))
    data.read(header_len - prefix_size) # read but don't parse header bytes
    fast.meta(data)
    while data.tell() < eof:
        fast.operation(data)

[denniske]

Thanks for the sample. It worked for me.

Though I assume it will not return the correct duration for restored matches, if I the header is skipped? Because the time where the restored match starts, is written in the header at self._header.initial.restore_time?

[happyleavesaoc]

@denniske Depends on if you want the duration of the rec or the match. If you want the match duration, yes, you will need to add the restore_time.

[denniske]

Thanks for the clarification. I assume for most matches skipping header will be okay then.

Btw: Have you been online on discord lately? I have some questions unrelated to this repository which I think you know something about. Would be cool if you could check them there.