Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using modifications #483

Closed
ghost opened this issue Jul 7, 2023 · 23 comments
Closed

Using modifications #483

ghost opened this issue Jul 7, 2023 · 23 comments

Comments

@ghost
Copy link

ghost commented Jul 7, 2023

Hi

First of all: the project is amazing and I am at awe how smooth and nice it works. I salute the author and much respect for them for creating the project.

I would love to play Gothic 1/2 again, but it would be even better to play with my favorite modifications. Unfortunately I had very little success in launching the mods.

I use Linux and the vanilla game launches without any issues. I copied all necessary modification files to Gothic 1 Data/ModVDF and tried using -game: argument, pointing to the mod *.ini file. Main menu is changed as expected (so I assume that the mod *.ini file has been recognized and loaded), but the "New Game" does not launch.

This is the specific command I used and game engine response:

$ ./Gothic2Notr.sh -g1 -g /Games/gothic/drive_c/GOG\ Games/Gothic/ -game:/Games/gothic/drive_c/GOG\ Games/Gothic/System/EdycjaRozszerzona+.ini
OpenGothic v1.0 dev
no *.ini file in path - using default settings
GPU = NVIDIA GeForce GTX 1660 SUPER
Depth format = Depth32F Shadow format = Depth16
loading error: unable to open file

I wonder what does it mean that the engine was "unable to open file"? Is it mod-specific problem or I do something wrong?

I will be thankful for any suggestions.

@Nindaleth
Copy link
Contributor

Hi @CriticalGoose! Can you list the specific version of your mod, ideally provide a download link?

@Try
Copy link
Owner

Try commented Jul 8, 2023

Hi, @CriticalGoose , thanks for reporting!

@Nindaleth
I think this issue is related to 8fa4c41. If mod had no custom *.dat file, this will result in gameDatDef = ".DAT", and then rest falls apart

@Nindaleth
Copy link
Contributor

@Try good catch, that was silly of me. Sent a PR that hopefully fixes it.

@CriticalGoose can you check if patch from #484 helps?

@ghost
Copy link
Author

ghost commented Jul 8, 2023

Hey,

Thanks for the response, @Try @Nindaleth .

This is the mod I try to play: https://bractwospolszczenia.pl/showthread.php?tid=1285 (Edycja Rozszerzona 1.0.4)
This is the exact download link: https://drive.google.com/file/d/1MfSqE7B7gNOpvyRou3Rl9MASCtp91r1L/view

I tried patching with #484 like this:
git pull #484
make -C build -j $(nproc)

Then I launched the game with same command as before:
$ ./Gothic2Notr.sh -g1 -g /Games/gothic/drive_c/GOG\ Games/Gothic/ -game:/Games/gothic/drive_c/GOG\ Games/Gothic/System/EdycjaRozszerzona+.ini

Unfortunatelly the issues persists.

I'm not 100% sure I do it the correct way. Please let me know if I made a mistake!

@ghost
Copy link
Author

ghost commented Jul 8, 2023

I apologize, after checking again I found out that the response is different now:

OpenGothic v1.0 dev
no *.ini file in path - using default settings
GPU = NVIDIA GeForce GTX 1660 SUPER
Depth format = Depth32F Shadow format = Depth16
loading error: buffer underflow at byte 7718676 while reading 482164992 additional bytes [context: slicing]

Try added a commit that referenced this issue Jul 8, 2023
@Try
Copy link
Owner

Try commented Jul 8, 2023

Got up until loading screen with help of ce4b78d

Then script-loading throws exception, due to .text section is smaller, than it should be.
quick workaround to phoenix, is enough to reach loading screen, yet then VM screams with error, and crashes shortly after.

[phoenix] vm: internal exception: pushv: no symbol found for index
[phoenix] vm: internal exception: popping instance from empty stack
[phoenix] vm: internal exception: vm: division by zero
[phoenix] vm: internal exception: tried to pop_instance but frame does not contain am instance.
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: illegal access of type 7 on symbol LOOP_CONTINUE which is another type (2)
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: illegal access of type 2 on symbol SELF which is another type (7)
[phoenix] vm: internal exception: tried to pop_reference but frame does not contain a reference.
[phoenix] vm: internal exception: illegal access of type 2 on symbol $11133 which is another type (3)
[phoenix] vm: internal exception: vm: division by zero
[phoenix] vm: internal exception: illegal access of type 2 on symbol BOSSOWIE which is another type (3)
[phoenix] vm: internal exception: tried to pop_instance but frame does not contain am instance.
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: vm: division by zero
not implemented call [NPC_HASREADIEDRANGEDWEAPON]
[phoenix] vm: internal exception: vm: division by zero
[phoenix] vm: internal exception: tried to pop_instance but frame does not contain am instance.
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: Popping instance of wrong type: N7phoenix7c_spellE, expected N7phoenix5c_npcE
[phoenix] vm: internal exception: tried to pop_reference but frame does not contain a reference.
 (  crash with infinite recursion here )

CC @lmichaelis
mod download link: https://drive.google.com/file/d/1MfSqE7B7gNOpvyRou3Rl9MASCtp91r1L/view?usp=sharing

./Gothic2Notr.sh -g1 -g /Games/gothic/drive_c/GOG\ Games/Gothic/ -game:/Games/gothic/drive_c/GOG\ Games/Gothic/System/EdycjaRozszerzona+.ini

UPD:

DMA mod detected: Ikarus
DMA mod detected: LeGo

@lmichaelis
Copy link
Contributor

lmichaelis commented Jul 9, 2023

Hm I'm not sure what's happening here. There seems to be lots of broken instructions everywhere and the VM is just trying its best. For example, SPELL_LIGHT:

The definition looks fine ...

00004bc7 <SPELL_LIGHT>
	Name: SPELL_LIGHT
	Address: b9ec2
	Size: 0
	Type: instance
	Generated: No
	Flags:
		Const: Yes
		Return: No
		Member: No
		Extern: No
		Merged: No
	File Index: 458
	Line Start: 289
	Line Count: 9
	Char Start: 8734
	Char Count: 225
	Parent: C_SPELL_PROTO (54b)
	Definition:
		instance SPELL_LIGHT(C_SPELL_PROTO)

but the actual bytecode is very broken:

000b9ec2 <SPELL_LIGHT>:
   b9ec2:	3d 01 ce 0b 00     BL 000bce01 ; <B_STOPMAGICSLEEP2>
   b9ec7:	43 ee 04 00 00     PUSHVI 000004ee ; <SELF>
   b9ecc:	40 00 00 80 3f     PUSHI 1065353216
   b9ed1:	3e 1c 01 00 00     BE 0000011c ; <AI_WAIT>
   b9ed6:	3c                 RSR

I would expect this to (1) call C_SPELL_PROTO, (2) initialize the fields of C_SPELL and (3) return. Calling those external AI-functions in here is not the correct behavior since they aren't even available at this time because the engine is still initializing.

This prompted me to look at the Ikarus code and it too looks like it's broken:

luis:_COMPILED/ $ zscript -f GOTHIC.DAT -d -s MEM_INITALL | head -n 30
00007ecd <MEM_INITALL>:
    7ecd:	09                 MOVI
    7ece:	41 98 21 00 00     PUSHV 00002198 ; <_BIN_STREAMLEN.LEN>
    7ed3:	41 97 21 00 00     PUSHV 00002197 ; <_BIN_STREAMLEN.NLEN>
    7ed8:	12                 GTE
    7ed9:	4c 3d b0 00 00     BZ 0000b03d
    7ede:	40 01 00 00 00     PUSHI 1
    7ee3:	41 98 21 00 00     PUSHV 00002198 ; <_BIN_STREAMLEN.LEN>
    7ee8:	0d                 LSL
    7ee9:	41 98 21 00 00     PUSHV 00002198 ; <_BIN_STREAMLEN.LEN>
    7eee:	09                 MOVI
    7eef:	50 d8 1a 00 00     GMOVI 00001ad8 ; <MEM_STACKPOS>
    7ef4:	41 d7 1a 00 00     PUSHV 00001ad7 ; <MEMINT_STACKPOS.POSITION>
    7ef9:	41 99 21 00 00     PUSHV 00002199 ; <_BIN_STREAMLEN.POS>
    7efe:	09                 MOVI
    7eff:	41 8d 21 00 00     PUSHV 0000218d ; <_BIN_CCNT> = 0
    7f04:	41 8e 21 00 00     PUSHV 0000218e ; <_BIN_CLEN> = 0
    7f09:	41 98 21 00 00     PUSHV 00002198 ; <_BIN_STREAMLEN.LEN>
    7f0e:	3d f0 1f 00 00     BL 00001ff0 ; <MEM_REALLOC>
    7f13:	41 8d 21 00 00     PUSHV 0000218d ; <_BIN_CCNT> = 0
    7f18:	09                 MOVI
    7f19:	41 98 21 00 00     PUSHV 00002198 ; <_BIN_STREAMLEN.LEN>
    7f1e:	41 8e 21 00 00     PUSHV 0000218e ; <_BIN_CLEN> = 0
    7f23:	09                 MOVI
    7f24:	3c                 RSR
    7f25:	41 9b 21 00 00     PUSHV 0000219b ; <_BIN_EOF.LEN>
    7f2a:	09                 MOVI
    7f2b:	41 8e 21 00 00     PUSHV 0000218e ; <_BIN_CLEN> = 0
    7f30:	41 9b 21 00 00     PUSHV 0000219b ; <_BIN_EOF.LEN>
    7f35:	41 8b 21 00 00     PUSHV 0000218b ; <_BIN_CRSR>
luis:_COMPILED/ $ zscript -f GOTHIC.DAT -d -s MEM_INITALL | wc -l
33340

As you can see, the first weird thing that stands out is the BZ 0000b03d. This makes no sense since we're at 7ecd and the code are between 7f24 and b03d can never be executed anyways because of the RSR at 7f24. b03d just contains:

    b03d:       40 00 00 00 00     PUSHI 0
    b042:       3c                 RSR

You can also see my disassembler getting confused because it outputs way too many lines.

Also this thing with the text size seems weird. Its not something like an off-by-one or rounding error since it's declared as >400MB too large. Also, the symbol table is broken shows broken entries after index 19bd7 which points to an invalid symbol count being stored in the binary.

Maybe this is some sort of obfuscation? Maybe the compression algorithm did something weird? Maybe it points to even more Ikarus/LeGo trickery? I am not sure. All I can say is: The bytecode looks very broken everywhere I look.

EDIT:

I did some more investigating and I found that indeed, the symbol count is partially incorrect. The symbol count is set as 0x19c13 which is followed by a sort table of that size. The actual symbol table after that, however, contains only 0x19bd8 symbols, missing 59 symbols. I can't event consult the sort table for cross-reference by taking the maximum value stored in it since that too is 0x19c12 (index into the symbol table).

Are we even sure it works with the original game @CriticalGoose ?

@ghost
Copy link
Author

ghost commented Jul 9, 2023

Hello @lmichaelis
The original game launches and I managed to play it for 2-3 hours. The issue happens only while launching the modification.

lmichaelis added a commit to GothicKit/ZenKit that referenced this issue Jul 10, 2023
This fixes an issue with some modifications of Gothic which insert multiple newlines at the end of strings in scripts.

Related: Try/OpenGothic#483
@lmichaelis
Copy link
Contributor

I found the issue: A single string in the binary was terminated using multiple newline-chars which was not supported correctly. Could you try again with GothicKit/phoenix#b8d27fd @CriticalGoose ?

@Try
Copy link
Owner

Try commented Jul 10, 2023

@lmichaelis
Testing locally: game hangs at loading. Context:

// model_script.cc : make_event_tag
case mds::event_tag_type::window: {  // evt.type == mds::event_tag_type::dam_multiply
	auto frames = a.value_or(""); // "1.0"
	std::istringstream stream {frames};

	int32_t fr = 0;
       	while (!stream.eof()) { // fails to parse string, before eof
		stream >> fr;
		evt.frames.push_back(fr);
	}

So 2 issues here:

  1. From what I know dam_multiply should not have frames, - just single value
  2. Input sanitation - probably should look forward to move away from stringstream

Try added a commit that referenced this issue Jul 10, 2023
@Try
Copy link
Owner

Try commented Jul 10, 2023

After adding .fail() check to stream, loading fails in model_script_dsl.cc with multiple different syntax_error exceptions.

@Try
Copy link
Owner

Try commented Jul 10, 2023

With 8d1b9dc and quick-fixes for dam_multiply/stringstream/model_script_dsl, manage to get it running locally
(except million errors caused by ikarus)
изображение

Nice looking mod btw :) Curious: is there English translation ?

@Nindaleth
Copy link
Contributor

Nindaleth commented Jul 11, 2023

Curious: is there English translation ?

I can see something here: https://www.moddb.com/mods/gothic-edycja-rozszerzona-104-english-version

@ghost
Copy link
Author

ghost commented Jul 11, 2023

Nice looking mod btw :) Curious: is there English translation ?

@Try This specific modification is the unofficial extension to "Edycja Rozszerzona" and it doesn't have the support of the original author. As far as I know, there is no English translation of the original, but you can try English version of "Edycja Rozszerzona+" provided by Nindaleth. I cannot recommend it to you, though, as I didn't play it before and I've read mixed opinions on this one. I hoped to try it with OpenGothic and judge it myself.

I'm happy to see that there is some progress on the technical side of things. I regret not being able to help, since my experience with programming is very limited. I will be happy to help any way I can and I'm glad I've learned a few things in the process here (:

I tried launching Gothic 2 with this engine without any mods and it runs even better than the first one, but I wonder why you cannot make more than one step back (blocking) at a time? Is this a conscious decision or a bug?

lmichaelis added a commit to GothicKit/ZenKit that referenced this issue Jul 11, 2023
This commit has to be reverted since just skipping whitespace can cause issues with other scripts in which the symbol parent addresses get corrupted.

Related: b8d27fd#r121311819
Related: Try/OpenGothic#483
lmichaelis added a commit to GothicKit/ZenKit that referenced this issue Jul 11, 2023
These workarounds are required by some mods which won't function correctly otherwise.

Related: Try/OpenGothic#483
Closes: #72
@Try
Copy link
Owner

Try commented Jul 11, 2023

I tried launching Gothic 2 with this engine without any mods and it runs even better than the first one, but I wonder why you cannot make more than one step back (blocking) at a time? Is this a conscious decision or a bug?

no sure, I need to check it ( currently my local copy in semi-broken state :D )

revert "read string symbols with skipped whitespace"

In Gothic2 problematic symbol is TXT_GUILDS array:

const string TXT_GUILDS[66] =
{
	"Guildless",
	"Paladin",
	"Militia",
	"Citizen",
	"Magician",
	"Novice",
	"Dragon Hunter",
	"Mercenary",
	"Farmer",
	"Bandit",
	"Convict",
	"Seeker",
	"Land Dweller",
	"Pirate",
	"Water Mage",
	"D",
	"",
	"Meatbug",
...

"D" followed by "", represented as "D\n\n" in file, so skipping white-space symbols after string is not suppose to work.

In case of mod it's also TXT_GUILDS; buffer appears to be:
"Agresor\n\nJaszczur\n". - so possibly white-space skip only hides issue.

@lmichaelis
Copy link
Contributor

so possibly white-space skip only hides issue.

Yes and no. I did implement it incorrectly but in the mod script there is a string symbol which has a count of 1, meaning it's not an array, and a value of <some text here>\n\n where phoenix leaves the last \n untouched which causes all subsequent symbols to be read incorrectly.

lmichaelis added a commit to GothicKit/ZenKit that referenced this issue Jul 12, 2023
@lmichaelis
Copy link
Contributor

I've tested this latest patch with G1, G2 and G1 modded. The mod now loads up correctly and only Ikarus related need fixing. Those syntax errors should be considered as warnings only.

Try added a commit that referenced this issue Jul 12, 2023
@Try
Copy link
Owner

Try commented Jul 12, 2023

I tried launching Gothic 2 with this engine without any mods and it runs even better than the first one, but I wonder why you cannot make more than one step back (blocking) at a time? Is this a conscious decision or a bug?

Fixed in 6fa3f94

@ghost
Copy link
Author

ghost commented Jul 13, 2023

I've tested this latest patch with G1, G2 and G1 modded. The mod now loads up correctly and only Ikarus related need fixing. Those syntax errors should be considered as warnings only.

That's amazing, @lmichaelis! For some reason it is not the case for me.

I pulled the updates:

git pull --recurse-submodules
make -C build -j $(nproc)

Then I launched the game to test the engine:

./Gothic2Notr.sh -g ~/Games/gothic

It was a success, so I tried with a mod:

./Gothic2Notr.sh -g ~/Games/gothic -game:~/Games/gothic/System/EdycjaRozszerzona+.ini

After choosing the "New Game" entry, I received the following error:

OpenGothic v1.0 dev
no *.ini file in path - using default settings
no *.ini file in path - using default settings
no *.ini file in path - using default settings
GPU = NVIDIA GeForce GTX 1660 SUPER
Depth format = Depth32F Shadow format = Depth16
loading error: buffer underflow at byte 7718676 while reading 482164992 additional bytes [context: slicing]

I thought that maybe I didn't pull one of the updates for some reason, so I made a backup, downloaded and compiled the engine as instructed on the main git page, installed Gothic 1 again, copied mod files and nothing's changed...

Am I missing something?

@lmichaelis
Copy link
Contributor

Try cd-ing into lib/phoenix and running git checkout main and then git pull to get the latest patches of phoenix if Try hasn't updated the dependency in OpenGothic yet. Then rebuild.

@ghost
Copy link
Author

ghost commented Jul 13, 2023

It worked - big, big thanks!

image

It was nice to learn a thing or two in the process and I'm glad that the project is still going on. It deserves recognition, that's for sure. You should consider setting up a way to "buy you a coffee" or support you in some other way - I would be happy to say thank you.

Should I close the thread now?

@Nindaleth
Copy link
Contributor

I suggest renaming the issue to contain the mod name, for better discoverability and also just in case someone wanted to pile up another mod issues in here. Not sure if @Try wants to handle the Ikarus issues in this one too?

Try added a commit that referenced this issue Jul 14, 2023
@Try
Copy link
Owner

Try commented Jul 14, 2023

Not sure if @Try wants to handle the Ikarus issues in this one too?

We have ticket for Ikarus already: #231

@Try Try closed this as completed Jul 14, 2023
lmichaelis added a commit to GothicKit/ZenKit that referenced this issue Aug 20, 2023
These workarounds are required by some mods which won't function correctly otherwise.

Related: Try/OpenGothic#483
Closes: #72
(cherry picked from commit 3fe0f7b)
lmichaelis added a commit to GothicKit/ZenKit that referenced this issue Aug 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants