New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Record/replay input logs #1096

Merged
merged 7 commits into from Feb 19, 2017

Conversation

Projects
None yet
3 participants
@scurest
Contributor

scurest commented Feb 4, 2017

This PR lets you record and replay a log of all the input given to Player.

This moves towards #646.


Record a log with

$ EasyRPG_Player --record-input input.log

and then replay it with

$ EasyRPG_Player --replay-input input.log

If you also use the same --seed, you should (AFAIK) get to watch the exact same playthrough you just recorded.

The intention is that this could eventually be used for automated regression testing, as bug reporting data, to confirming bugs still exist, etc. If Ghabry's event tracer can be rigged to give compatible output it could also be used for compatibility testing with RPG_RT. Right now, you could technically use it to look for changes between Player versions by comparing save files (ie. record a playthrough where you save, modify Player, replay the log, diff the .lsd files). But in order to do this fast, we'd need to add a way to run the Player "headless" (ie. don't emit graphics and run updates as fast as possible) or else you'd have to watch the whole playthrough every time you do it :p

Input Log Format

The input log is plain-text. There is one line for each call to Input::Update (this is roughly, but not exactly, one line per frame) that contains the InputButtons that were pressed when the call happened. The buttons are formatted as with operator<<(std::ostream&, const std::bitset<BUTTON_COUNT>&).

The format is really just proof-of-concept and we'll probably want something more efficient and robust. If the general idea of this PR looks good and there's no bugs, I can open an issue afterwards to talk about a good format for this.

scurest added some commits Feb 3, 2017

Introduce Input::Source
Presently Input reads data from the UI, sets its state, and supplies
its state to the engine. A Source abstracts over where the incoming
data comes from. The only currently available source is UiSource,
which reads data from the UI, making Input behave just as it did
before this commit.
Allow recording input to a log file
A command line arg now allows recording pressed_buttons to an
fstream.

Since Input::Init can now be affected by the CLI args, calling it
was moved into Player::Init (Graphics::Init was moved too because
it needs to be done before we use Output::Warning to signal that
the log file wasn't opened properly.) I also cleared out remaining
apparently-unused includes from main().
Allow replaying input logs
A new Input::Source is added that reads pressed_buttons from the
log file written by --record-input. Another command-line arg is added
to select the log file for replaying.

If --seed is also used, this *should* give perfectly reproducible
runs of a game (eg. for automated testing, etc.). Main caveat is that
resetting the game (with F12) does not go through the Input
module so it doesn't get reproduced.
Handle reset (F12) in Player instead of UI
This entails adding a new InputButton, RESET. Handling resetting
in Player lets it work with input recording/replaying.
Workaround missing move constructor on Android
The move-constructor and swap are both missing for ifstream on
Android (old stdlib version?), so there's not much choice but
to construct the ifstream in LogSource itself.
@Ghabry

This comment has been minimized.

Show comment
Hide comment
@Ghabry

Ghabry Feb 4, 2017

Member

Thx for the PR, will take a look later.

@scurest
Android ships with different stdc++ versions. A more up-to-date version of stdc++ is the clang-version which is provided by Android, too. But by now we were unable to switch the c++-library because this results in strange build errors while compiling our libraries. Android is a real can of worms.

Member

Ghabry commented Feb 4, 2017

Thx for the PR, will take a look later.

@scurest
Android ships with different stdc++ versions. A more up-to-date version of stdc++ is the clang-version which is provided by Android, too. But by now we were unable to switch the c++-library because this results in strange build errors while compiling our libraries. Android is a real can of worms.

@scurest

This comment has been minimized.

Show comment
Hide comment
@scurest

scurest Feb 4, 2017

Contributor

Thanks. Np on the Android thing; the workaround isn't really worse or anything.

Contributor

scurest commented Feb 4, 2017

Thanks. Np on the Android thing; the workaround isn't really worse or anything.

@Ghabry

This comment has been minimized.

Show comment
Hide comment
@Ghabry

Ghabry Feb 6, 2017

Member

Works for me.
There is one potential desync issue: When savegames are available the Title scene will highlight "Load". Maybe always focus "New game" when record/replay are enabled?

Member

Ghabry commented Feb 6, 2017

Works for me.
There is one potential desync issue: When savegames are available the Title scene will highlight "Load". Maybe always focus "New game" when record/replay are enabled?

@Ghabry Ghabry added this to the 0.5.1 milestone Feb 6, 2017

@Ghabry

This comment has been minimized.

Show comment
Hide comment
@Ghabry

Ghabry Feb 6, 2017

Member

And the replays will not sync on different platforms because "rand()" depends on the used libc. This can be resolved by replacing rand() everywhere with Utils::GetRandomNumber() which uses std::mt19937 which generates the same output on all platforms (at least the standard says so, untested).

Combined with a way to take screenshots during recording and to compare them during replay this feature could be also used to test parts of the Player that can't be tested using event tracing e.g. detecting of rendering regressions (only regression tests are possible because you need a screenshot of the correct state :D).

Member

Ghabry commented Feb 6, 2017

And the replays will not sync on different platforms because "rand()" depends on the used libc. This can be resolved by replacing rand() everywhere with Utils::GetRandomNumber() which uses std::mt19937 which generates the same output on all platforms (at least the standard says so, untested).

Combined with a way to take screenshots during recording and to compare them during replay this feature could be also used to test parts of the Player that can't be tested using event tracing e.g. detecting of rendering regressions (only regression tests are possible because you need a screenshot of the correct state :D).

@scurest

This comment has been minimized.

Show comment
Hide comment
@scurest

scurest Feb 6, 2017

Contributor

Yeah, you definitely need to restore the save files to reproduce a run. Even ignoring that desync, the saves have a save_count field that will ruin your diffs if you don't. I want to avoid having any actual Player logic depend on the input recording stuff though. Is it okay to just add that the save files need to be restored too to the docs for --replay-input?


I have a branch with the relevant fixes for rand. In small experiments, the save files do reproduce perfectly on it (except for the timestamp field). They still aren't portable though, because the algorithm for uniform_int_distribution isn't part of the standard ;) I was going to do a separate PR for that though.

Now that I think of it, there might also be an issue with the BGM ticks. Do you know how reproducible those are? It looks like they depend on the audio implementation.

edit: The screenshot button is recorded in the log too! Like with save files, you'd have to record, move them to a different folder, replay, then compare.

Contributor

scurest commented Feb 6, 2017

Yeah, you definitely need to restore the save files to reproduce a run. Even ignoring that desync, the saves have a save_count field that will ruin your diffs if you don't. I want to avoid having any actual Player logic depend on the input recording stuff though. Is it okay to just add that the save files need to be restored too to the docs for --replay-input?


I have a branch with the relevant fixes for rand. In small experiments, the save files do reproduce perfectly on it (except for the timestamp field). They still aren't portable though, because the algorithm for uniform_int_distribution isn't part of the standard ;) I was going to do a separate PR for that though.

Now that I think of it, there might also be an issue with the BGM ticks. Do you know how reproducible those are? It looks like they depend on the audio implementation.

edit: The screenshot button is recorded in the log too! Like with save files, you'd have to record, move them to a different folder, replay, then compare.

@Ghabry

This comment has been minimized.

Show comment
Hide comment
@Ghabry

Ghabry Feb 6, 2017

Member

For this prototype implementation it's good enough for me when you document that savegames must be part of the movie. Can be fixed later using a different movie format / movie folder which somehow stores the savegames together with the movie.

Okay, then I ignore rand for now.

The timestamp is a design flaw in liblcf. Not liblcf should set the timestamp but the caller of liblcf. This is a problem in lcf2xml, too because the timestamp gets updated when converting back and forth :/. If you want you can submit a patch to liblcf which removes the "GenerateTimestamp" call and put it at an appropriate spot in Player.

BGM_Ticks doesn't return the correct value by now. When BGM_Ticks ever gets fixed it should return the same output on any platform and the only game I know which depends on BGM_Ticks is one of the "Mother" games which has the combo system implemented in an acustic way (not working in EasyRPG yet).

Member

Ghabry commented Feb 6, 2017

For this prototype implementation it's good enough for me when you document that savegames must be part of the movie. Can be fixed later using a different movie format / movie folder which somehow stores the savegames together with the movie.

Okay, then I ignore rand for now.

The timestamp is a design flaw in liblcf. Not liblcf should set the timestamp but the caller of liblcf. This is a problem in lcf2xml, too because the timestamp gets updated when converting back and forth :/. If you want you can submit a patch to liblcf which removes the "GenerateTimestamp" call and put it at an appropriate spot in Player.

BGM_Ticks doesn't return the correct value by now. When BGM_Ticks ever gets fixed it should return the same output on any platform and the only game I know which depends on BGM_Ticks is one of the "Mother" games which has the combo system implemented in an acustic way (not working in EasyRPG yet).

@scurest

This comment has been minimized.

Show comment
Hide comment
@scurest

scurest Feb 6, 2017

Contributor

Oh, that's why my save files look the same after I run them through lcf2xml! :)

Contributor

scurest commented Feb 6, 2017

Oh, that's why my save files look the same after I run them through lcf2xml! :)

scurest added some commits Feb 6, 2017

Generate save file timestamp in Player
This allows the code that generates it in liblcf to be removed.
Soften claims about reproducibility
Document that the state of the save files affects replayed runs.
Clarify that --replay-input is only for reproducing button presses.

scurest added a commit to scurest/liblcf that referenced this pull request Feb 6, 2017

@Ghabry Ghabry self-requested a review Feb 6, 2017

@Ghabry

Ghabry approved these changes Feb 6, 2017

@Ghabry Ghabry merged commit b52b9cf into EasyRPG:master Feb 19, 2017

6 checks passed

Android (armeabi-v7a) Build finished.
Details
GNU/Linux Build finished.
Details
OSX Build finished.
Details
Windows (x64) Build finished.
Details
Windows (x86) Build finished.
Details
web Build finished.
Details

@scurest scurest deleted the scurest:input-log branch Feb 19, 2017

Ghabry added a commit to libretro/easyrpg-libretro that referenced this pull request May 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment