Strange items.other/items.bad misalignment (0.47.04 linux64)

[lethosor]

Since #1558 was merged, various fields of world have been misaligned, at least on linux64. To paraphrase my (and @BenLubar's) findings on IRC:

• world.items.other.BAD and world.items.bad were actually the same vector, they just overlapped before (items_other_id was one larger than the size of the <static-array>). Now that the static-array is one larger, things after items are misaligned. Notably, immediately after items are artifacts and jobs, and the latter (due to having a vtable) causes crashes on world load in buildings_onUpdate().
• Commenting out world.items.bad causes DF to segfault on startup because cur_savegame is misaligned. But strangely, the position of jobs is correct compared to 0.47.04-r1,, and jobs is well before cur_savegame. The size of items also matches 0.47.04-r1 after this change. I'm not really sure how this is possible.
• It's a bit harder to remove BAD from the items_other_id enum (the only way I see to shrink the <static-array> at this point) because there's a job_item_vector_id::BAD item that depends on it. I don't know if job_item_vector_id::BAD is really necessary, though.

Basically, I have no idea what was causing the weird alignment issues I was seeing later on in world after fixing the size of items. Any help would be appreciated.

[PatrikLundell]

Win64:
I haven't had any issue until trying to look at this issue, and it could be a matter of barking up the wrong tree (or me missing something). I've been using the latest df-structures, with my PR on top, plus the associated DFHack PR. I made sure the structures were up to date, recompiled, started DF and loaded a save, and didn't see anything usual. I then pulled the latest DFHack, and DF crashed when trying to load a save (obviously after recompiling) (I've tried it twice, with the same save I typically use).

This is what the pull produced:
PS C:\dfhack> git pull origin develop
remote: Enumerating objects: 57, done.
remote: Counting objects: 100% (57/57), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 57 (delta 41), reused 53 (delta 37), pack-reused 0
Unpacking objects: 100% (57/57), done.
From https://github.com/DFHack/dfhack

• branch develop -> FETCH_HEAD
  d0c030c..f204465 develop -> origin/develop
  Removing depends/tthread/fast_mutex.h
  Merge made by the 'recursive' strategy.
  CMakeLists.txt | 4 +
  depends/tthread/CMakeLists.txt | 2 +-
  depends/tthread/fast_mutex.h | 249 --------------------------------
  depends/tthread/tinythread.h | 1 -
  docs/Compile.rst | 181 ++++++++++++++---------
  docs/Introduction.rst | 3 +-
  docs/changelog.txt | 7 +
  library/LuaTypes.cpp | 22 +++
  library/include/BitArray.h | 12 ++
  library/include/DataDefs.h | 19 +++
  library/xml | 2 +-
  plugins/labormanager/joblabormapper.cpp | 4 +
  plugins/labormanager/labormanager.cpp | 3 +
  plugins/rendermax/renderer_light.cpp | 4 +-
  plugins/rendermax/renderer_light.hpp | 24 +--
  plugins/rendermax/renderer_opengl.hpp | 22 ++-
  plugins/rendermax/rendermax.cpp | 24 ++-
  plugins/ruby/codegen.pl | 8 +
  plugins/ruby/ruby-autogen-defs.rb | 16 ++
  plugins/stonesense | 2 +-
  20 files changed, 250 insertions(+), 359 deletions(-)

Thus, something in that set changed DF from (apparently?) working to crashing. My primary (uneducated) guess would point to library/include/DataDefs.h, followed by BitArray.h and LuaTypes.cpp.

[lethosor]

It's actually something in library/xml - the size of the items.other array changed (changes in submodules generally don't show up in diff stats in the main dfhack repo)

[lethosor]

I switched world.items.other back to a static-array and now I'm getting a segfault on accessing world.machines (which also has a vtable). (Also when accessing world.cur_savegame, but that's later on.) I'm not sure what's going on here, since buildings.other was definitely the right size, but I'm inclined to roll back the df-other-vectors additions so we have a working develop build in the meantime.

[lethosor]

I rolled back some of the changes for now. Now that I think about it, one likely cause could be items being one vector too large and buildings (shortly after items, but before cur_savegame) being one vector too small, so fixing the size of items broke everything after buildings. I have some other PRs to deal with first, though.

[lethosor]

buildings.other is fixed now. items.other is trickier because of the world.items.other.BAD and world.items.bad overlap, since there are existing tools that use both names (and at least one cross-reference to items_other_id.BAD from somewhere else in the XML files that would need to be removed).

[quietust]

I question whether items_other_id.BAD actually exists, or if it's an instance of two very similar enums being shoehorned together (or perhaps a misunderstanding from some code that wasn't analyzed properly).

[lethosor]

Yeah, I'm probably going to get rid of items_other_id.BAD along with world.items.other.BAD. It's possible that some DF code accesses items.bad as one item past the end of items.other, but I'd be surprised if it deliberately does that instead of accessing items.bad directly.