Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IWD2: crash when starting a new game #2082

Closed
2 of 7 tasks
MarcelHB opened this issue May 14, 2024 · 19 comments · Fixed by #2085
Closed
2 of 7 tasks

IWD2: crash when starting a new game #2082

MarcelHB opened this issue May 14, 2024 · 19 comments · Fixed by #2085

Comments

@MarcelHB
Copy link
Collaborator

MarcelHB commented May 14, 2024

Bug description

In IWD2, when starting a new game, by group or manual setup,

  1. we apparently load garbage (\r­¡║\r­¡║\r) as item name as seen in the stack below,
  2. the garbage name causes an error log line due to iconv errors, ➔ shouldn't print the erroneous string, as some lines below already
  3. and that log line itself is thus invalid and will cause libfmt to abort

Steps to reproduce

  1. Start IWD2
  2. New Game
  3. Pick a predefined group (1st one should do)
  4. Proceed until crash

Expected behavior

Can start a game.

Screenshots

Thread 1 hit Breakpoint 2, GemRB::StringFromEncodedData (inptr=0x18148530, length=12, encoding="UTF-8") at D:/Sources/gemrb/gemrb/core/Strings/StringConversion.cpp:58
58                      Log(ERROR, "String", "iconv failed to convert string {} from {} to UTF-16 with error: {}", in, encoding, strerror(errno));
(gdb) bt
#0  GemRB::StringFromEncodedData (inptr=0x18148530, length=12, encoding="UTF-8") at D:/Sources/gemrb/gemrb/core/Strings/StringConversion.cpp:58
#1  0x00007ffd2d4350a1 in GemRB::StringFromEncodedView[abi:cxx11](GemRB::StringViewImp<char const> const&, GemRB::EncodingStruct const&) (view=..., encoding=...) at D:/Sources/gemrb/gemrb/core/Strings/StringConversion.cpp:77
#2  0x00007ffd2d43533e in GemRB::StringFromUtf8[abi:cxx11](GemRB::StringViewImp<char const> const&) (utf8view=...) at D:/Sources/gemrb/gemrb/core/Strings/StringConversion.cpp:102
#3  0x00007ffd2d4353ab in GemRB::StringFromUtf8[abi:cxx11](char const*) (data=0x18148530 "Cache2\\\r­¡║\r­¡║\r.itm") at D:/Sources/gemrb/gemrb/core/Strings/StringConversion.cpp:107
#4  0x00007ffd2d436084 in GemRB::FileExists (path="Cache2\\\r­¡║\r­¡║\r.itm") at D:/Sources/gemrb/gemrb/core/System/VFS.cpp:218
#5  0x00007ffd2d433720 in GemRB::FileStream::Open (this=0x18148580, fname="Cache2\\\r­¡║\r­¡║\r.itm") at D:/Sources/gemrb/gemrb/core/Streams/FileStream.cpp:68
#6  0x00007ffd2d433d62 in GemRB::FileStream::OpenFile (filename="Cache2\\\r­¡║\r­¡║\r.itm") at D:/Sources/gemrb/gemrb/core/Streams/FileStream.cpp:201
#7  0x00007ffd730d1524 in SearchIn (path="Cache2", resRef=..., type="itm") at D:/Sources/gemrb/gemrb/plugins/DirectoryImporter/DirectoryImporter.cpp:48
#8  0x00007ffd730d16b9 in GemRB::DirectoryImporter::GetResource (this=0xd20f20, resname=..., type=1005) at D:/Sources/gemrb/gemrb/plugins/DirectoryImporter/DirectoryImporter.cpp:63
#9  0x00007ffd2d3e2c42 in GemRB::ResourceManager::GetResourceStream (this=0xeb6a00, ResRef=..., type=1005, silent=true) at D:/Sources/gemrb/gemrb/core/ResourceManager.cpp:165
#10 0x00007ffd2d350802 in GemRB::GameData::GetItem (this=0xeb6a00, resname=..., silent=true) at D:/Sources/gemrb/gemrb/core/GameData.cpp:171
#11 0x00007ffd2d3ac1c0 in GemRB::Interface::ResolveRandomItem (this=0x5ff2c0, itm=0x18148490) at D:/Sources/gemrb/gemrb/core/Interface.cpp:3249
#12 0x00007ffd2d3aba1d in GemRB::Interface::ReadItem (this=0x5ff2c0, str=0x1814cf00, itm=0x18148490) at D:/Sources/gemrb/gemrb/core/Interface.cpp:3097
#13 0x00007ffd2d3ab91a in GemRB::Interface::ReadItem (this=0x5ff2c0, str=0x1814cf00) at D:/Sources/gemrb/gemrb/core/Interface.cpp:3084
#14 0x00007ffd790d6451 in GemRB::CREImporter::ReadInventory (this=0x17fa0b40, act=0x18144540, slotCount=50) at D:/Sources/gemrb/gemrb/plugins/CREImporter/CREImporter.cpp:1121
#15 0x00007ffd790d4d3e in GemRB::CREImporter::GetActor (this=0x17fa0b40, is_in_party=0 '\000') at D:/Sources/gemrb/gemrb/plugins/CREImporter/CREImporter.cpp:924
#16 0x00007ffd7a076217 in GemRB::AREImporter::GetActor (this=0x148aed90, str=0x148aec40, actorMgr=std::shared_ptr<GemRB::ActorMgr> (use count 2, weak count 0) = {...}, map=0x17e93510)
    at D:/Sources/gemrb/gemrb/plugins/AREImporter/AREImporter.cpp:1119
#17 0x00007ffd7a0788bd in GemRB::AREImporter::GetMap (this=0x148aed90, resRef=..., day_or_night=true) at D:/Sources/gemrb/gemrb/plugins/AREImporter/AREImporter.cpp:1542
#18 0x00007ffd2d348ac1 in GemRB::Game::LoadMap (this=0x13391cd0, resRef=..., loadscreen=true) at D:/Sources/gemrb/gemrb/core/Game.cpp:842
#19 0x00007ffd2d347eba in GemRB::Game::GetMap (this=0x13391cd0, areaName=..., change=true) at D:/Sources/gemrb/gemrb/core/Game.cpp:671
#20 0x00007ffd2d3245b0 in GemRB::GameControl::ChangeMap (this=0x13183820, pc=0x144f3a40, forced=true) at D:/Sources/gemrb/gemrb/core/GUI/GameControl.cpp:2561
#21 0x00007ffd2d39fcf8 in GemRB::Interface::HandleFlags (this=0x5ff2c0) at D:/Sources/gemrb/gemrb/core/Interface.cpp:800
#22 0x00007ffd2d3a0a46 in GemRB::Interface::Main (this=0x5ff2c0) at D:/Sources/gemrb/gemrb/core/Interface.cpp:967
#23 0x00007ff694961667 in main (argc=1, argv=0xeb4ba0) at D:/Sources/gemrb/platforms/windows/GemRB.cpp:63

GemRB version (check as many as you know apply)

  • master as of this issue
  • 0.9.2
  • 0.9.1
  • 0.9.0

Video Driver (check as many as you know apply)

  • SDL1.2
  • SDL2 built with OPENGL_BACKEND enabled
  • SDL2 without OPENGL_BACKEND enabled
@lynxlynxlynx
Copy link
Member

Any group or a specific one?

@lynxlynxlynx lynxlynxlynx added this to the 0.9.3 - TBN milestone May 14, 2024
@MarcelHB
Copy link
Collaborator Author

I clicked the first one from the top, not sure what it is named in English.

@lynxlynxlynx
Copy link
Member

Lady's Lament. Works here in English. Try breaking in Interface::ResolveRandomItem to see how we resolve to that name. I doubt it is bad data in the CRE file, the way translations are handled, but the same argument could be used for the random treasure table.

@MarcelHB
Copy link
Collaborator Author

Must be way up the stack, will have a look:

(gdb) f 11
#11 0x00007ffd2d3ac1c0 in GemRB::Interface::ResolveRandomItem (this=0x5ff2c0, itm=0x17facb70) at D:/Sources/gemrb/gemrb/core/Interface.cpp:3249
3249                            const Item* item = gamedata->GetItem(itm->ItemResRef, true);
(gdb) p *itm
$1 = {ItemResRef = {str = "\r­¡║\r­¡║\r", static Size = 8 '\b', static npos = 255 ' '}, Expired = 0, Usages = {_M_elems = {1, 0, 0}}, Flags = 0, Weight = -1, MaxStackAmount = 0}

@MarcelHB
Copy link
Collaborator Author

(gdb) f 14
#14 0x00007ffd79546451 in GemRB::CREImporter::ReadInventory (this=0x12f24f70, act=0x17fa4ea0, slotCount=50) at D:/Sources/gemrb/gemrb/plugins/CREImporter/CREImporter.cpp:1121
1121                            CREItem *item = core->ReadItem(str);
(gdb) p *str
$3 = {_vptr.DataStream = 0x7ffd2d6b5a00 <vtable for GemRB::MemoryStream+16>, filename = {str = "10gob.cre\000\000\000\000\000\000\000"}, originalfile = "D:\\Games\\IWD2\\data\\CRE.bif", static InvalidPos = 18446744073709551615,
  static Error = -1, Pos = 2242, size = 2346, Encrypted = false, IsDataBigEndian = false}

that gets the data from here:

gemrb/gemrb/core/Interface.cpp

Lines 3081 to 3091 in 60dafac

CREItem *Interface::ReadItem(DataStream *str) const
{
CREItem *itm = new CREItem();
if (ReadItem(str, itm)) return itm;
delete itm;
return NULL;
}
CREItem *Interface::ReadItem(DataStream *str, CREItem *itm) const
{
str->ReadResRef( itm->ItemResRef );

Wrong stream position? Will investigate tomorrow.

@MarcelHB
Copy link
Collaborator Author

MarcelHB commented May 14, 2024

It's not exactly related to the group choice, for me it's any new game. I was just too lazy or ran into other trouble when trying to do that manually.

@MarcelHB MarcelHB changed the title IWD2: crash when starting a new game with a predefined group IWD2: crash when starting a new game May 14, 2024
@lynxlynxlynx
Copy link
Member

Yeah, makes sense, since you hit it with the basic goblin in the area. I see 10gob.cre does have a clean inventory, but one of the items is indeed randomized: 00rtgob1 will trigger a substitution. Is the resref bad for you already when entering Interface::ResolveRandomItem or does that happen in the loop?

@MarcelHB
Copy link
Collaborator Author

Yeah, makes sense, since you hit it with the basic goblin in the area. I see 10gob.cre does have a clean inventory, but one of the items is indeed randomized: 00rtgob1 will trigger a substitution. Is the resref bad for you already when entering Interface::ResolveRandomItem or does that happen in the loop?

Bad input from the outside:

(gdb) f 12
#12 0x00007ff908ceba1d in GemRB::Interface::ReadItem (this=0x5ff2c0, str=0x18097ef0,
    itm=0x18091bf0) at D:/Sources/gemrb/gemrb/core/Interface.cpp:3097
3097            if (ResolveRandomItem(itm)) {
(gdb) p *itm
$1 = {ItemResRef = {str = "\r­¡║\r­¡║\r", static Size = 8 '\b', static npos = 255 ' '},
  Expired = 0, Usages = {_M_elems = {1, 0, 0}}, Flags = 0, Weight = -1, MaxStackAmount = 0}

@lynxlynxlynx
Copy link
Member

It modifies itm, so make sure to check before ResolveRandomItem runs.

@MarcelHB
Copy link
Collaborator Author

MarcelHB commented May 15, 2024

Right, pickedItem sometimes is a 0-ResRef.

@MarcelHB
Copy link
Collaborator Author

00RTGob1 has 20 values in RNDTRES.2DA, but when debugging, there are 21 entries with \0\0... being the last one. Then the roll sometimes ends up there, and explode over that name later gives use empty parts with OOB access.

@lynxlynxlynx
Copy link
Member

So a problem in 2daImporter, I guess we still don't trim properly despite all the tests.

@MarcelHB
Copy link
Collaborator Author

Eh, RT_NORM.2DA it is. Anyway, yes, reproducible under test scenario. I'll take a closer look.

@MarcelHB
Copy link
Collaborator Author

MarcelHB commented May 15, 2024

As far as I can see, here you wanted Explode work like trailing space as empty string: 5e3192a

I was looking through all the SKILLBRD.2DAs and I found some in PST and BG1 that match your commit description, but then again, they only have a single column? For the 2DAImporter, that doesn't matter anyway since it cuts the trailing occurrences over the number of columns, as they are seen from the header.

In contrast this case here has 40 columns but 20 values + trailing space making the 21st, so that cut off is meaningless and we'll carry that around.

So a fix is probably violating that test case, but if it's only relevant for some edgy 2DA file, I guess we are fine anyway. There is also a test for that 2DA case.

@lynxlynxlynx
Copy link
Member

Yeah, that behaviour matches the other odd explode cases. The importer got fixed in 8a77bde, since this could only happen in the first line IIRC.

Since there's no way a 2da should be able to return an empty string except via an empty default value (just judging from our implementation, I doubt it's intentional), we could always just return the default value if the cell is empty. Which is something we can do at construction time and not repeat at each lookup.

@MarcelHB
Copy link
Collaborator Author

MarcelHB commented May 15, 2024

Shouldn't we clean up something here? I see three points:

  1. Explode shouldn't treat trailing space as the last element if space is set as delimiter. From what I understood, this SKILLBRD.2DA file was the only known case? 2DA importer generally ignores superfluous values by now, so everything is good.
  2. There is something else as well I'm not aware of r/n, or discriminating between a,,, on comma delimiter and a on space delimiter with potentially bogus trailing space in Explode makes the code even harder to get, we shoud make the Query* method return defVal if we actually have an empty value. Or just have 2DAImporter check the last cell for being empty and discard it.
  3. Now this ResolveRandomItem rolls over the actual amount of cells as seen in GetColumnCount, and looking at the file it has all kinds of lengths between nothing and 40, with a default value of *. Or we leave GetColumnCount as it is and have some last empty default-value cell, unless discarded in 2.
    1. Anyway, if we start to roll over additional * cells over some consistent default value logic, now we should check if we skew the outcome in some way? Maybe this way never that accurate in the first place, I don't know.
    2. As if it wasn't annoying, these values may actually look like BULL03*3 or * by default or * by text value somewhere in between.
    3. By the logic of Explode(str, '*') over a value * we get two parts of empty strings, or BULL03 + 3 over a full value. The current code does not look like it's suited to properly handle that case (line 3273).

@lynxlynxlynx
Copy link
Member

  1. You're right, I didn't consider the case when the delimiter is a space itself. As for the cases, I doubt I did a systematic search, this table in combination with the changes in the importer was just the first place I noticed it, since it broke level up. So there may be more.
  2. Like I said, I prefer discarding it immediately, so the query functions don't have to check all the time.
  3. Yuck, some lines have over 70 cells. Looks like the original had the same approach, leaving the limited amount of named columns and using direct grid lookups. So I think we should leave GetColumnCount alone; it's already parametrized anyway. And the way RtRows is filled looks ok to me — the fix for the previous two items should make it all dandy again.

@MarcelHB
Copy link
Collaborator Author

That looks like the least intrusive way to fix the 2DAImporter without breaking anything else:

diff --git a/gemrb/plugins/2DAImporter/2DAImporter.cpp b/gemrb/plugins/2DAImporter/2DAImporter.cpp
index 22ee5e3ee..ed93a9fe7 100644
--- a/gemrb/plugins/2DAImporter/2DAImporter.cpp
+++ b/gemrb/plugins/2DAImporter/2DAImporter.cpp
@@ -83,6 +83,11 @@ bool p2DAImporter::Open(std::unique_ptr<DataStream> str)
 		
 		auto sv = StringView(&line[pos + 1], line.length() - pos - 1);
 		rows.emplace_back(Explode<StringView, cell_t>(sv, ' ', std::max<size_t>(1, colNames.size() - 1)));
+		auto& row = rows.back();
+		// Explode may have returned trailing space as the last but empty item
+		if (!row.empty() && row.back().length() == 0) {
+			row.pop_back();
+		}
 	}
 
 	assert(rows.size() < std::numeric_limits<index_t>::max());

Changing Explode for the space-with-space-case should not be necessary for this one here.

@lynxlynxlynx
Copy link
Member

Looks fine. 👍🏾

MarcelHB added a commit to MarcelHB/gemrb that referenced this issue May 16, 2024
MarcelHB added a commit to MarcelHB/gemrb that referenced this issue May 16, 2024
Is potentially invalid UTF-8 and will then crash libfmt.

cf. gemrb#2082
@lynxlynxlynx lynxlynxlynx linked a pull request May 16, 2024 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants