Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deltarune desync problem (again) #121

Open
BenLubar opened this Issue Dec 9, 2018 · 18 comments

Comments

Projects
None yet
3 participants
@BenLubar
Copy link
Contributor

commented Dec 9, 2018

This one doesn't seem to be related to save states. Something is causing the first fight with three Jigsawry to desync at random. Running the game again with the same movie generally "fixes" the issue, but I feel like there's either a race condition somewhere or Deltarune is accessing the clock or something like that before libTAS replaces the functions.

There does not seem to be any similar problem in Undertale, at least as far as I have reached in the game.

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 9, 2018

Ok, this appears to have been due to Deltarune calling RNG in an alternate thread and me having the clock settings wrong for that.

@BenLubar BenLubar closed this Dec 9, 2018

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 10, 2018

Nope, that doesn't seem to have been the root cause. It desynced again on the Jigsawry fight.

The two attacks I've seen desync are Jigsawry's only attack and the attack Hathy uses when she is alone.

@BenLubar BenLubar reopened this Dec 10, 2018

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 10, 2018

I am confused at this point - maybe the "correct" version of results for the button presses I'm entering is a desync and I was just getting lucky with how it broke. It seems to be breaking the same way 100% of the time now that I have the clock settings correct.

@BenLubar BenLubar closed this Dec 10, 2018

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 10, 2018

Sorry about closing and re-opening this issue so many times.

I just went in to fix the desync... and it didn't desync. Unless there's something weird going on with fast forward causing something that calls RNG to not be called, I have no idea what could be causing it to desync versus not.

@BenLubar BenLubar reopened this Dec 10, 2018

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 10, 2018

Ok, on the left is a full recording of the TAS with the first ten minutes chopped off. On the right is a recording of the TAS after fast forwarding to the ten minute mark without recording.

https://files.lubar.me/deltarune-desync.mp4

@clementgallet

This comment has been minimized.

Copy link
Owner

commented Dec 10, 2018

I don't see where the desync could come from. Is DR using libc rand functions ? If so, did you print random log to see if there is a shift in calls ?

If fast-forward is involved, you could try to force all frames to render, to see if it changes anything ? (you have to modify the code, but it's pretty straightforward, returns false in skipDraw() in library/frame.cpp).

Do you have a movie file so that I can look at it ?

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 10, 2018

I've invited you to the GitLab repo, which is the same as https://github.com/BenLubar/tobytas but also contains the LTM contents.

The desync happens between frames 20175 and 20250.

The two functions it calls for RNG are choose and random. I'm not sure how they're implemented. I tried using the random_get_seed function on every frame, but it always returns 0, so presumably it only knows about the initial seed.

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 10, 2018

Ok, I've done some research:

The random number generator seems to be custom-built. It's located at 00413f30 in the version of runner that the Deltarune for Linux Reddit post includes.

There's a single 32-bit integer argument at $rax-4.

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 10, 2018

Fast forward with all frames rendered does give the "correct" results, but I'm not sure whether that's because it slows down the game logic enough that the race condition doesn't happen or that the rendering code is somehow affecting RNG.

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 11, 2018

The last synchronized RNG call happens on frame 16062, and the first call that only happens on the non-fast-forward version is on frame 16065.

Here's the diff of frames where RNG is called: https://gist.github.com/BenLubar/698870e15e94f98105c7308d323d97d2

A bunch of new stuff appears around that frame:

  • The only "now playing" notification in the entire game
  • Enemies that chase you in the overworld
  • The grass is animated on that screen but not anywhere before then

Since there are a lot of RNG calls that the fast forward gains in addition to the (more) calls that it skips, I'm guessing this isn't due to rendering calling RNG but rather due to some other thread being starved of resources and not getting to call RNG as much as it normally would be able to.

Here's a video of the section of the game where the RNG desyncs: https://files.lubar.me/deltarune-desync-a.mp4

(Coincidentally, 16065 is a non-rendered frame during a room transition.)

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 11, 2018

Alright, to re-synchronize the TAS on frame 20000, this (very long) gdb command can set the RNG to the correct state:

sudo gdb -iex 'set pagination off' -p `pgrep deltarune` -ex 'p *(uint32_t*)0xb459e0 = 0x54d893f4' -ex 'p *(uint32_t*)0xb459e4 = 0x78a5607f' -ex 'p *(uint32_t*)0xb459e8 = 0x5b950dca' -ex 'p *(uint32_t*)0xb459ec = 0x6d401548' -ex 'p *(uint32_t*)0xb459f0 = 0x6b18d9bf' -ex 'p *(uint32_t*)0xb459f4 = 0x7ad08b25' -ex 'p *(uint32_t*)0xb459f8 = 0x15d14f51' -ex 'p *(uint32_t*)0xb459fc = 0x8f86587a' -ex 'p *(uint32_t*)0xb45a00 = 0x774c4628' -ex 'p *(uint32_t*)0xb45a04 = 0xa77b0fcc' -ex 'p *(uint32_t*)0xb45a08 = 0x9d24ca05' -ex 'p *(uint32_t*)0xb45a0c = 0xf0995dcb' -ex 'p *(uint32_t*)0xb45a10 = 0xbf9f96f1' -ex 'p *(uint32_t*)0xb45a14 = 0xf320a719' -ex 'p *(uint32_t*)0xb45a18 = 0x3ad475dd' -ex 'p *(uint32_t*)0xb45a1c = 0xc6fac141' -ex 'p *(uint32_t*)0xb45a20 = 0x9' -batch
@clementgallet

This comment has been minimized.

Copy link
Owner

commented Dec 16, 2018

It sounds like I don't have access to your gitlab repo files, even being invited as Guest (or I failed to find them, which is totally possible).

@noellekiq

This comment has been minimized.

Copy link
Contributor

commented Dec 16, 2018

https://gitlab.com/ben_lubar/tobytas ?

also, doesn't gitlab email you when you're invited to a repository?

@clementgallet

This comment has been minimized.

Copy link
Owner

commented Dec 16, 2018

I got the invitation, accepted it. Now I have access to the repo, but not the files ? This is all I can access: https://i.imgur.com/hITBNVB.png

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 16, 2018

Ah, weird, I thought Guest would give read-only access on a code sharing website, but I guess I was wrong.

You should have access to the files now.

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 16, 2018

At the current point in the TAS, the desync caused by fast forwarding will cause a game over followed by a "don't continue", so if that affects your ability to debug, the desync starts at 8:55 and is clearly visible in the fight at 11:11. The fight ends at 11:26 in the non-desynced version.

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Dec 23, 2018

This can be considered lower priority now, because I've finished the Deltarune portion of the TAS and Undertale does not appear to exhibit this behavior.

I'm replacing Deltarune with this program that just waits a certain number of frames and exits:

https://gist.github.com/BenLubar/08acceb7f319f9a104a38de4d8edc849

@BenLubar

This comment has been minimized.

Copy link
Contributor Author

commented Jan 7, 2019

This also causes the second half of Undertale (the song) to not play in New Home.

As far as I can tell, the underlying cause of this issue is how sleeps are handled when fast forwarding:

/* If the function was called from the main thread, transfer the wait to
* the timer and do not actually wait.
*/

If a secondary thread sleeps at all in fast forward mode and that thread's actions affect gameplay in any way, you're going to get a desync because the main thread doesn't ever sleep.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.