Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bitperfect copying of LSD #268

Merged
merged 5 commits into from Nov 4, 2018

Conversation

Projects
None yet
4 participants
@fmatthew5876
Copy link
Contributor

fmatthew5876 commented Oct 18, 2018

Fixes #246

This series of changes is towards doing perfect copies of LSD files.

So far I am able to copy a few local save games I have.

If anyone has lots of saved games, please test copying with this branch of send the saves to me.

@fmatthew5876 fmatthew5876 force-pushed the fmatthew5876:ldb_copy_lsd branch 3 times, most recently from 7a97ffb to 557e9a1 Oct 18, 2018

@fmatthew5876

This comment has been minimized.

Copy link
Contributor Author

fmatthew5876 commented Oct 23, 2018

Thanks for those saves, I was able to fix a few more things.

I checked them all and only these are remaining:

  • Der Brief für den König - Demo II (Ver1.4) - Kopie - This is in your 2k/ folder but it has 2k3 battle_commands. Copying it as a 2k game fails as liblcf won't write out those battle_commands. When processed as a 2k3 game the lsd's copy cleanly.
  • Dark Sword - Demo 2.1.2/Dark Sword II Remake/Save15.lsd - One picture has SavePicture::transparency = 96. We treat this as a bool so our code converts it to 1 and writes it back out as 1. The picture text is DynResource\MessageBox\EmptyPixel.png. Could this be a third party patch?

This test and PR doesn't mean we write LSD files correctly. I only tested copying them from one file to another, not actually saving with player. For my test, I had to manually hack Data::system.ldb_id = 2003 to make the the 2k3 games copy correctly.

@CherryDT

This comment has been minimized.

Copy link

CherryDT commented Oct 23, 2018

Yes looks like some dynrpg Plugin taking advantage of the fact that a bool is just saved and loaded as one byte in memory and puts more stuff into the same byte

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Oct 23, 2018

We can declare Dark Sword as "completely incompatible", uses too many DynRPG plugins. DynRPG was a mistake :P

The other one: Will check later if this is another case of wrong engine detection like .flow

@fdelapena fdelapena added this to the 0.6.0 milestone Oct 25, 2018

@fmatthew5876 fmatthew5876 force-pushed the fmatthew5876:ldb_copy_lsd branch from 557e9a1 to 2162651 Oct 27, 2018

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Oct 29, 2018

(feel free to put LMU fixes in this PR, makes the testing easier for me)

@fmatthew5876

This comment has been minimized.

Copy link
Contributor Author

fmatthew5876 commented Oct 29, 2018

(feel free to put LMU fixes in this PR, makes the testing easier for me)

Sure, let me know any games with LMU failures and I'll investigate and add here,

@fmatthew5876 fmatthew5876 force-pushed the fmatthew5876:ldb_copy_lsd branch 2 times, most recently from bc6b330 to dd501cd Oct 29, 2018

@fmatthew5876

This comment has been minimized.

Copy link
Contributor Author

fmatthew5876 commented Oct 29, 2018

I've also removed the save count dependency. So you don't need to worry about that messing up your test results here.

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Oct 29, 2018

Okay, I got a small amount of LDBs and LMTs more and had to delete ~30 LMUs because the LDB was missing for some reason which means the test wouldn't make sense.

The result is really impressive, great job!

LDB Test results for 1551 files

Summary:
2k:   1089/1089 (100.0%)
2k3:  416/416 (100.0%)
2k3e: 46/46 (100.0%)

LMT Test results for 1550 files

Summary:
2k:   1088/1088 (100.0%)
2k3:  416/416 (100.0%)
2k3e: 46/46 (100.0%)

LMU (Map0001) Test results for 1518 files

./2k/(Demo3)The Fuc. Schrank (V 3.0.3)
./2k/Aliminator -VV 1.02 - Kelven - 2k- 13.08.2003
./2k/DBL2
./2k/Der Feind (Demo 2)
./2k/The History of Planet Bryyo - First Conflict (v1.04)
./2k/Yoshis Maze Mania (v1.10)
./2k3/Colourway1.01
./2k3/Die letzte Grenze (Beta)
./2k3/Duckworld 2 [2k3] (v2.0)
./2k3/Duckworld 3 [2k3] (v0.60)
./2k3/Duckworld [2k3] (v1.1)
./2k3/Fog over Esidar (Demo)
./2k3/Raus aus dem Dungeon
./2k3/Super Peach RPG (Demo)
./2k3/The Legend of Phylandia (Demo)

Summary:
2k:   1062/1068 (99.4%)
2k3:  396/405 (97.8%)
2k3e: 45/45 (100.0%)

LSD Test results for 358 files

./2k3/Dark Sword - Demo 2.1.2/Dark Sword II Remake

Summary:
2k:   179/179 (100.0%)
2k3:  176/177 (99.4%)
2k3e: 2/2 (100.0%)

My script also collects the failing files now:
failed.zip

@fmatthew5876 fmatthew5876 force-pushed the fmatthew5876:ldb_copy_lsd branch from dd501cd to 8d56c05 Oct 29, 2018

@fmatthew5876

This comment has been minimized.

Copy link
Contributor Author

fmatthew5876 commented Oct 29, 2018

It looks like all generator chunks (even arrays) dissapear when defaulted. So that fixed a few 2k3 games. Also I found some of the 2k games have generator chunks their maps. Since all generator chunks disapear when defaulted, I've disabled is2k3 on them. Now any map from any version will just passthrough whatever generator chunks it may have.

The following games have 2 extra bytes after the Map data ends. I don't know if this is corruption or custom patch hackery. These bytes are placed after the terminating 0 and so liblcf does not read them.

./2k/Aliminator -VV 1.02 - Kelven - 2k- 13.08.2003 - 5C10
./2k/DBL2 - 5C10
./2k/Der Feind (Demo 2) - 1B34
./2k/Yoshis Maze Mania (v1.10) - 010C

@fdelapena fdelapena added the Savegames label Oct 30, 2018

@fmatthew5876

This comment has been minimized.

Copy link
Contributor Author

fmatthew5876 commented Nov 1, 2018

While testing Violated Heroine for the other PR, I found it's LDB doesn't copy cleanly. There is something really strange.

This LDB has a 7 character string "Unknown" for Skill chunk 0x7 failure_message. Failure message is supposed to be an integer reffering to one of the skill failure message types.

liblcf tries to interpret it as a 7 byte integer and throws up an error.

Has anyone seen this before? Did some version or patch of rm2k store a string in failure_message?

@CherryDT

This comment has been minimized.

Copy link

CherryDT commented Nov 1, 2018

This LDB has a 7 character string "Unknown" for Skill chunk 0x7 failure_message. Failure message is supposed to be an integer reffering to one of the skill failure message types.

It is amazing what kind of oddities you keep digging up. 😲
No I never heard of anything like this and it makes no real sense to me. I tried looking for the game online because I wanted to see if RPG_RT is patched somehow but I wasn't able to find a working download.

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Nov 1, 2018

Only thing I can say is that VH1 doesn't use the battle system at all.

try searching for "VH1", the downloads are usually hosted on mega iirc and the password is "nanako" (or "Nanako")

@fmatthew5876

This comment has been minimized.

Copy link
Contributor Author

fmatthew5876 commented Nov 1, 2018

So for my testing this all looks good.

@Ghabry could you run your script one more time to just sanity check? If it passes then I think we're done here.

The VH1 version I have a is translated one. Maybe the translator did this. Would need to check the chunks in the original Japanese.

fmatthew5876 added some commits Oct 18, 2018

Add Comparable? field to structs.csv
Generates operator== and operator!= for
structs with Comparable? ==1
Refactor Music and Sound
- Support operator==()
- name defaults to (OFF)
Generate operator== for Flags and use presentifdefault
Fixes SavePicture::Flags default behavior
Improve handling of Map generator chunks
- Works correctly on 2k3, passes through
  any already existing chunks in 2k.

@fmatthew5876 fmatthew5876 force-pushed the fmatthew5876:ldb_copy_lsd branch from bb11c0a to d8f1688 Nov 1, 2018

This was referenced Nov 2, 2018

@fmatthew5876 fmatthew5876 force-pushed the fmatthew5876:ldb_copy_lsd branch from 2f523a8 to d8f1688 Nov 3, 2018

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Nov 4, 2018

The only mismatches I get are the 4 LMUs you mentioned before (Extra bytes) and the "Dark Sword" LSD which uses DynRPG custom data.

Are you fine with these results? If yes I'm merging this.

@fmatthew5876

This comment has been minimized.

Copy link
Contributor Author

fmatthew5876 commented Nov 4, 2018

Let's merge it :+1

@Ghabry Ghabry merged commit 3c46af7 into EasyRPG:master Nov 4, 2018

4 checks passed

GNU/Linux Build finished.
Details
OSX Build finished.
Details
Windows Build finished.
Details
web Build finished.
Details
@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Nov 4, 2018

Another corner case is probably games that use a bogus engine translation that translated (OFF) in Music & Sound with something else, but doesn't seem to affect any of the games here.

@fmatthew5876

This comment has been minimized.

Copy link
Contributor Author

fmatthew5876 commented Nov 4, 2018

Would that not break RPG_RT when it tries to find some bogus filename?

Either way once we have automated testing setup with Jenkins it would be good to add more games to the test set.

@CherryDT

This comment has been minimized.

Copy link

CherryDT commented Nov 4, 2018

Bogus sounds are ignored. Only bogus sprites crash. It's just that the special effect of turning off other sounds wouldn't work.

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Nov 4, 2018

The Player has an heuristic to detect other forms of "(OFF)" btw., so in player this is no issue.

@fmatthew5876 fmatthew5876 deleted the fmatthew5876:ldb_copy_lsd branch Nov 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.