Cannot parse 1.21 replays

[regexpressyourself]

Hi icza,

I am trying out your tool, and am having a problem parsing replays from the 1.21 update. I am pretty sure I was able to properly parse 1.20 update replays, so it must be a change with the most recent patch.

Here is a link to the replay file I am using.

I am running:

go run cmd/screp/screp.go my_replay_file.rep

The error message is:

Failed to parse replay: not a replay file
exit status 2

It looks like BWChart had a problem with 1.21 replays as well.

I'm not a Go dev, but I'm happy to help in any way I can.

Thanks in advance!

[icza]

You can download older versions of the screp binary here: https://github.com/icza/screp/releases

Can you try if you can parse these replays with the old version?

[regexpressyourself]

Absolutely. Both 1.0.0 and 1.0.1 return the same error: Failed to parse replay: not a replay file

[icza]

I looked into it more deeply, and the replay format changed. It will take unknown time to reverse engineer the changes and implement parsing them.

[regexpressyourself]

Understandable. Thanks for looking into it.

[laelbe]

I am also looking forward to a fixed version. Thanks.

[dragonattack]

So Blizzard changed replay format completely? Or it's just a few changes that need to be reverse engineered?

[icza]

No, it's not a "complete" change, but the replay file header changed and new commands have been added for sure.

[dragonattack]

jca2 has updated his BwChart to parse 1.21 replays
also there are some new actions HEX codes in that thread

[icza]

Well, in that thread jca2 himself claims it's still a work in progress, and at the end of that thread users claim it is broken again.

[regexpressyourself]

In the developer update at the end up 2017, they stated that one of the priorities for 2018 was to get the BWAPI working again. Hopefully that comes with a little more transparency about replay parsing?

[icza]

Yes, hopefully. The BWAPI project also contain(ed) a replay parser, but it hasn't been updated a while.

[dragonattack]

jca2 released Version 2.00C, Build 2.

This versions appears to have eliminated 99% of the corruption issues.

We could just ignore unknown actions until we get to know them. APM count will be not very accurate, but still.

Or if it's not the way it would be great if we could just read replay header at least (player names, races, etc). That would help much.

[icza]

Will look into it when I have time. BWChart is not open source though, so it's not that big of help...

[laelbe]

I tried something..

• repparser.go Line 204 :
  var repID = []byte("seRS")

• repdecoder.go Line 64:
  if _, err = f.Seek(32, 0); err != nil {

[an3uryzm]

Hello, what if i contact Bwchart creator and try to get a source code of some bwchart (of course i will tell him that i am willing to help out this project). Will it be helpful?

[icza]

Yes, it would be helpful.

Also, the source code is not even needed, if you can get the new replay format (or what has changed in the latest 1.21 version), that would be enough for me to update the library to support 1.21 replays.

[regexpressyourself]

For what it's worth, this is the creator of bwchart's TL page: http://www.teamliquid.net/forum/profile.php?user=jca2

[marianogappa]

Asked JCA to share the 1.21 changes here. Would you consider working on it once you have them? Alternatively could you provide some straightforward instructions? There's no Go source for replay parsing at the moment, and it would enable some interesting use cases.

[icza]

Yes, sure, if I know the changes, I will implement them. I just don't have the time to reverse engineer it myself.

[marianogappa]

@icza

The format seems to be as follows:

• 4 bytes checksum
• 4 bytes block
• 4 bytes encoded length
• data (compressed or not), if compressed starts with 789c
• in between blocks, 4 bytes for encoded length

Compression format is zlib.
Note 789c on 3rd line:

c219 c293 0100 0000 0400 0000 7365 5253
1d1f 0100 c84f 4b52 0100 0000 c200 0000
789c 63b4 0c65 0002 0fc9 cb9c 311c 5000
1248 0875 89f4 2ecd 740a 704a 2e62 4003
0d40 c8c8 c4c6 c0cf c0c8 2000 e4b3 00e9

Haven't gotten any reply from TL yet. Hopefully this is enough to salvage the parsing that already exists? Surely there's some new actions that will be unknown.

[icza]

@marianogappa I'm not sure this is enough, but better than nothing. I'll start working with it.

[icza]

Statud report: started working on 1.21 support a few days ago, development takes place on the dev branch.

So far parsing succeeds with the replay header, but new commands are not yet supported.

[marianogappa]

This is great news @icza! Thank you. Were the instructions enough? If not, can you please provide the steps somewhere?

[icza]

@marianogappa Actually the only change between the existing modern parser (1.16+) and what you've described for 1.21+ replays was the in-between block between the sections, the rest is the same for replays 1.16-1.20 and was already implemented. One change you didn't mention was the content of the replay ID section, it is changed from "reRS" to "seRS" in 1.21+ replays.

New commands are yet to be reverse engineered.

[marianogappa]

@icza Thank you. Are the changes you mention pushed to the dev branch? Because I've run go run screp.go (within /cmd/screp) on an 1.21 replay and it was still erroring out. This was 2 days ago; maybe it has changed since then.

[marianogappa]

I did the above within the latest dev branch from 2 days ago.

[icza]

@marianogappa I pushed another commit to dev yesterday, it does not fail now. When a new, unknown command is encountered, it stops parsing that block. Best that can be done without knowing anything more about them.

Besides the commands section, everything else can be parsed as with previous replays.

[icza]

Good news fellas! I've finished parsing new, 1.21 commands, working code is on dev.

I make some tests, and if all good, I will merge it into master and create a new binary release.

[marianogappa]

@icza sounds great!
I'm trying the code with my latest replays and it keeps returning this:

Failed to parse replay: not a replay file
exit status 2

I've just pulled dev, so I'm confused. My working copy is clean on this commit:

f833bc3 (HEAD -> dev, origin/dev) Andras Belicza Add logic to parse new 1.21 commands

I'm running:

~/Code/screp/cmd/screp dev ✔ $ go run screp.go <replay_path>

Games were played this week; tried many.
What replays are you trying with? Do you want me to send you some samples?
It's possible that it's happening because Blizzard has done 2 patches recently.

[icza]

I'm trying "old" 1.21 replays. If I'm not mistaken, current SC version is 1.22? Could it be Blizzard changed something again?

Yes, please send me 1.22 replays (e.g. to my email, same username as here, and it's a gmail address).

P.S.: Yes, current version is 1.22.2 (patch notes).

[marianogappa]

@icza I've sent you around 5 replays, each named by the date it was played, up to yesterday. This way you can test against different versions including the current one.

[icza]

Also just found a bug report, that even SC itself can't play replays: https://us.battle.net/forums/en/starcraft/topic/20770507009

[icza]

@marianogappa Also sorry, user name in my email is iczaaa (the rest is the same as detailed above). Please resend replays.

[marianogappa]

Email bounced. Are you sure about that email address? Note that your username is "icza".

[marianogappa]

Resent.

[icza]

@marianogappa Got the replays, thanks.

The version on the dev branch is capable of parsing all your replays without errors. Are you sure you used the dev branch for your tests?

[marianogappa]

I still think it's something else; probably something silly which just needs some log debugging, because I can't parse any replays spanning from September to date:

✔ ~/Code/screp/cmd/screp dev ✔ $ go run screp.go ~/Desktop/send/replays/20180922.rep
Failed to parse replay: not a replay file
exit status 2
✘1 ~/Code/screp/cmd/screp dev ✔ $ go run screp.go ~/Desktop/send/replays/20181014.rep
Failed to parse replay: not a replay file
exit status 2
✘1 ~/Code/screp/cmd/screp dev ✔ $ go run screp.go ~/Desktop/send/replays/20181125.rep
Failed to parse replay: not a replay file
exit status 2
✘1 ~/Code/screp/cmd/screp dev ✔ $ go run screp.go ~/Desktop/send/replays/20181207.rep
Failed to parse replay: not a replay file
exit status 2
✘1 ~/Code/screp/cmd/screp dev ✔ $ go run screp.go ~/Desktop/send/replays/20181214.rep
Failed to parse replay: not a replay file
exit status 2
✘1 ~/Code/screp/cmd/screp dev ✔ $ go run screp.go ~/Desktop/send/replays/20181219.rep
Failed to parse replay: not a replay file
exit status 2

[marianogappa]

The "dev" in the log I'm showing is the branch I'm on.

[icza]

Please include the output of go run screp.go -version.

[marianogappa]

$ go version
go version go1.11 darwin/amd64
$ go run screp.go -version
screp version: v1.1.0
Parser version: v1.1.0
Platform: darwin amd64
Built with: go1.11
Author: Andras Belicza
Home page: https://github.com/icza/screp

[marianogappa]

f833bc3 (HEAD -> dev, origin/dev) Andras Belicza Add logic to parse new 1.21 commands (27 minutes ago)

[icza]

You haven't updated everything I guess. It prints parser version v1.2.0 for me.

This is the full output for me:

screp version: v1.1.0
Parser version: v1.2.0
Platform: linux amd64
Built with: go1.11.4
Author: Andras Belicza
Home page: https://github.com/icza/screp

[marianogappa]

Ah, got it. I'm updated but the screp.go is using the library that I originally cloned and was compiled only the first time. Let me see.

[marianogappa]

There seems to be a compiler error on repparser.go#166? The repdecoder.Decoder interface doesn't have a NewSection method, but it's being invoked on that line.

[icza]

NewSection was added in one of my previous commits, a few days back. Please do a full repository clone, it looks your local version is "out-of-sync".

[marianogappa]

✔ ~/Code $ rm -rf screp/
✔ ~/Code $ g clone git@github.com:icza/screp.git
Cloning into 'screp'...
remote: Enumerating objects: 32, done.
remote: Counting objects: 100% (32/32), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 293 (delta 13), reused 21 (delta 7), pack-reused 261
Receiving objects: 100% (293/293), 77.54 KiB | 0 bytes/s, done.
Resolving deltas: 100% (135/135), done.
~/Code/screp/cmd/screp master ✔ $ g co dev
Branch dev set up to track remote branch dev from origin.
Switched to a new branch 'dev'
✔ ~/Code/screp/cmd/screp dev ✔ $ go run screp.go ~/Desktop/send/replays/20181214.rep
Failed to parse replay: not a replay file
exit status 2

The reason that doesn't work is because go is using the cached copy of the imports. Let me clean that.

[marianogappa]

FWIW this is the line to do that:

$ rm -rf $GOPATH/pkg/darwin_amd64/github.com/icza

[icza]

FYI, there is go clean and go clean -cache.

[marianogappa]

Thanks. I got it working! Awesome.

[icza]

Coolio. Then I proceed with merge and release. Thanks for all the feedback and help.

[marianogappa]

Ok from my point of view you're good to close this after you do the master merge.

[marianogappa]

Thank you! This is a very useful tool.

[icza]

Here it is, v1.1.0 binary releases:

https://github.com/icza/screp/releases/tag/v1.1.0

Merry Christmas and Happy New Year to Everyone!

[marianogappa]

@icza: awesome, nice job!

FYI I looked deeper into my issue because the pkg explanation didn't make sense to me. The root cause was that my cloned version of the repo wasn't in the GOPATH, and I also had an outdated version on the GOPATH. So the imports went off the outdated GOPATH version. Whoops, sorry.

[regexpressyourself]

Wow @icza I just want to say thanks, and great job!

I opened this issue over a year ago, and you're still rocking it over here. Really appreciate what you've done with this package and this problem!!

[icza]

@regexpressyourself Sorry it took so long to add support for 1.21+ replays, the changes were non-trivial, and I was hoping someone with the knowledge would come forward so I don't have to invest my time to reverse engineer the changes myself (unfortunately this wasn't the case, although I received some help).

[regexpressyourself]

No need to apologize -- definitely non-trivial. I'm just glad it was figured out!