This document is a work in progress. Please open an issue for any technical or other errors you find.
Fast, complete Dota 2 "demo" (aka "replay") parser written in cython. Cython is a Python-like language which is processed into C and then compiled for execution speed.
You can interact with the smoke library like a normal python library*.
Python 3 support might be possible, if our protobuf library is compatible. Figuring this out is not a priority for us, but feel free to conduct your own investigation. Happy to accept pull requests.
* The installation process is a hair more involved.
On a fast CPU, smoke parses pro game replays with spectators and commentators at ~103x game time. A full parse on a 57 minute-long TI game replay takes ~33 seconds.
For a normal "pub" game of 47 minutes, a full parse takes ~19 seconds, or around ~148x game time.
You can always omit data you don't need for faster parses. Voice data is a good place to start (see below). The numbers above are upper bounds.
smoke is heavily optimized, but if speed is of utmost concern for you, or if you prefer Java, check out clarity. It is 2-5x faster than smoke.
We've successfully compiled and run smoke on these platforms:
- gcc 4.8.1 (Fedora 19 64-bit, Ubuntu 13.10 32- and 64-bit)
- gcc 4.8.2 (Fedora 20 64-bit)
- clang-500.2.79 (Mac OS X 10.9)
It probably doesn't have any serious portability issues at this point.
Of course. Join us on quakenet IRC in #dota2replay. But do be patient--if we don't answer immediately, we're probably playing Dota 2.
smoke is authored using python 2.7.x*.
If you use a Unix-like operating system (Linux or Mac OS X), installating smoke should be pretty painless. Windows hackers, halp! If you figure out how to get it running on Windows, let us know. It should be possible.
First, you need a C compiler. OS X users will need to install the Xcode
"Command Line Tools" from
Apple and a package
manager like Homebrew or MacPorts. Ubuntu users may want to install the
build-essential package for a quick, standard compiler:
sudo apt-get install build-essential
You will also need
sudo apt-get install python-dev
You will need the
snappy development libraries. Mac OS X users can get this
easily with Homebrew or MacPorts. With Homebrew, for example:
$ brew install snappy $ brew install protobuf
@jptaylor helpfully points out that OS X Mavericks users need to set some compiler options via environment variables, like so:
export CFLAGS=-Qunused-arguments export CPPFLAGS=-Qunused-arguments
In Ubuntu, you might install dependencies thusly:
$ sudo apt-get install libsnappy-dev libprotobuf-dev python-dev
And the python libraries, preferably in your virtualenv:
$ pip install cython # http://bit.ly/1dd0JRI for problems with virtualenv $ pip install palm $ pip install python-snappy
Next, you must install
palm 0.1.9 from source--it's not in PyPI, so you can't
get it with pip:
$ git clone https://github.com/bumptech/palm.git && cd palm $ python setup.py install
smoke by cloning it:
$ git clone https://github.com/skadistats/smoke.git && cd smoke $ python setup.py install
That's it! You're good to go.
If you want to hack on smoke, you might consider doing this instead of the second line in the last section above:
$ python setup.py build_ext --inplace # no system install $ export PYTHONPATH=$PWD
If you hack on smoke, you might occasionally get persistent build failures that have nothing to do with your code (this only applies to --inplace). It's a bit kludgy, but you can reset the build thusly from within your project dir:
$ find . -name \*.so -delete $ find . -name \*.h -delete $ find . -name \*.c -delete $ find . -name \*.pyc -delete $ rm -rf build
If you have compile or runtime problems after this, it's not Cython.
smoke parses only the data you're interested in from a replay. Choose from:
- entities: in-game things like heroes, players, and creeps
- modifiers: auras and effects on in-game entities✝
- "temp" entities: fire-and-forget things the game server tells the client about*
- user messages: many different things, including spectator clicks, global chat messages, overhead events (like last-hit gold, and much more), etc.*✝
- game events: lower-level messages like Dota TV control (directed camera commands, for example), combat log messages, etc.*
- voice data: the protobuf-formatted binary data blobs that are somehow strung into voice--only really relevant to commentated pro matches*✝
- sounds: sounds that occur in the game*✝
- overview: end-of-game summary, including players, game winner, match id, duration, and often picks/bans
* transient: new dataset (i.e. list, dict) for each tick of the parse
✝ unprocessed: data is provided as original protobuf message object
Parsing Replay Data
By default, smoke parses everything. This is the slowest parsing option. Here is a simple example which parses a demo, doing nothing:
# entity_counter.py import io from smoke.io.wrap import demo as io_wrp_dm from smoke.replay import demo as rply_dm with io.open('37633163.dem', 'rb') as infile: # wrap a file IO as a "demo" demo_io = io_wrp_dm.Wrap(infile) # read the header that occurs at demo start demo_io.bootstrap() # create a demo with our IO object demo = rply_dm.Demo(demo_io) # read essential pre-match data from the demo demo.bootstrap() # this is the core loop for iterating over a game for match in demo.play(): # this is where you will do things! see smoke.replay.match count = len(match.entities) # parses game overview found at the end of the demo file demo.finish()
When run with
time python entity_counter.py, we get:
real 0m32.689s user 0m32.411s sys 0m0.242s
Perhaps you want to be more selective about parsing. We do this by bitmask. Here's code similar to the above, but more restrictive about what it parses. Consequently, it'll be tons faster:
# with_less_data.py import io from smoke.io.wrap import demo as io_wrp_dm from smoke.replay import demo as rply_dm from smoke.replay.const import Data with io.open('37633163.dem', 'rb') as infile: demo_io = io_wrp_dm.Wrap(infile) demo_io.bootstrap() # it's a bitmask -- see smoke.replay.demo for all options parse = Data.All ^ (Data.UserMessages | Data.GameEvents | Data.VoiceData | Data.TempEntities) demo = rply_dm.Demo(demo_io, parse=parse) demo.bootstrap() for match in demo.play(): count = len(match.entities) # parses game overview found at the end of the demo file demo.finish()
When run with
time python with_less_data.py:
real 0m20.116s user 0m19.904s sys 0m0.196s
Finally, if we just want an overview of the game:
# overview_only.py import io from smoke.io.wrap import demo as io_wrp_dm from smoke.replay import demo as rply_dm from smoke.replay.demo import Data with io.open('37633163.dem', 'rb') as infile: demo_io = io_wrp_dm.Wrap(infile) overview_offset = demo_io.bootstrap() # returns offset to overview # we can seek on the raw underlying IO instead of parsing everything infile.seek(overview_offset) demo = rply_dm.Demo(demo_io) demo.finish() print demo.match.overview
When run with `time python overview_only.py':
real 0m0.189s user 0m0.124s sys 0m0.034s
If you only need
GameEvents (for example), you end up
with 5 second parses. So parse as little as you can!
Take a look at
smoke.replay.match to see which properties you can access
playing a demo.
See LICENSE in the project root. The license for this project is a modified MIT with an additional clause requiring specifically worded hyperlink attribution in web properties using smoke.