Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codechange: Don't save unused NewGRF override mappings. #9202

Merged
merged 1 commit into from May 8, 2021

Conversation

@PeterN
Copy link
Member

@PeterN PeterN commented May 7, 2021

Motivation / Problem

Saveload code for NewGRF entity override manager saves all entries for a table, regardless of whether they are mapped or not.

This is not a huge issue but is wasteful for saving objects as 64000 empty entries as saved.

Description

This is resolved by testing if the override is valid, and only saving if so. The Saveload array already copes with sparse loading/saving so no saveload changes are needed.

Limitations

Checklist for review

Some things are not automated, and forgotten often. This list is a reminder for the reviewers.

  • The bug fix is important enough to be backported? (label: 'backport requested')
  • This PR affects the save game format? (label 'savegame upgrade')
  • This PR affects the GS/AI API? (label 'needs review: Script API')
    • ai_changelog.hpp, gs_changelog.hpp need updating.
    • The compatibility wrappers (compat_*.nut) need updating.
  • This PR affects the NewGRF API? (label 'needs review: NewGRF')
src/saveload/newgrf_sl.cpp Outdated Show resolved Hide resolved
@PeterN PeterN force-pushed the PeterN:sparse-overrides branch from 626fe49 to 9141018 May 7, 2021
@michicc
michicc approved these changes May 7, 2021
@LordAro LordAro merged commit 69e5da0 into OpenTTD:master May 8, 2021
12 checks passed
12 checks passed
@github-actions
Emscripten
Details
@github-actions
Commit checker
Details
@github-actions
Check preview needs update Check preview needs update
Details
@github-actions
Linux (clang, clang++, libsdl2-dev)
Details
@github-actions
Linux (gcc, g++, libsdl2-dev)
Details
@github-actions
Linux (gcc, g++, libsdl1.2-dev)
Details
@github-actions
Linux (gcc, g++, -DOPTION_DEDICATED=ON)
Details
@github-actions
Mac OS (x64, x86_64)
Details
@github-actions
Windows (windows-latest, x86)
Details
@github-actions
Windows (windows-latest, x64)
Details
@github-actions
Windows (windows-2016, x86)
Details
@github-actions
Windows (windows-2016, x64)
Details
@TrueBrain
Copy link
Member

@TrueBrain TrueBrain commented May 9, 2021

Seems valgrind doesn't fully agree with this PR:

==31684== Invalid read of size 2
==31684==    at 0x8B06F8: OverrideManagerBase::IsValidID(unsigned short) const (newgrf_commons.h:221)
==31684==    by 0x8B00F4: Save_NewGRFMapping(OverrideManagerBase const&) (newgrf_sl.cpp:33)
==31684==    by 0x8A9AB9: Save_IIDS() (industry_sl.cpp:93)
==31684==    by 0x8C21D3: SlSaveChunk(ChunkHandler const*) (saveload.cpp:1776)
==31684==    by 0x8C225C: SlSaveChunks() (saveload.cpp:1792)
==31684==    by 0x8C2C02: DoSave(SaveFilter*, bool) (saveload.cpp:2564)
==31684==    by 0x8C2CEA: SaveWithFilter(SaveFilter*, bool) (saveload.cpp:2590)
==31684==    by 0x8369CC: ServerNetworkGameSocketHandler::SendMap() (network_server.cpp:629)
==31684==    by 0x837C37: ServerNetworkGameSocketHandler::Receive_CLIENT_GETMAP(Packet*) (network_server.cpp:1015)
==31684==    by 0x7E695D: NetworkGameSocketHandler::HandlePacket(Packet*) (tcp_game.cpp:84)
==31684==    by 0x7E6E82: NetworkGameSocketHandler::ReceivePackets() (tcp_game.cpp:139)
==31684==    by 0x7F18D3: TCPListenHandler<ServerNetworkGameSocketHandler, (unsigned char)0, (unsigned char)1>::Receive() (tcp_listen.h:133)
==31684==  Address 0x80cab4a is 0 bytes after a block of size 74 alloc'd
==31684==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==31684==    by 0xAB7FA7: unsigned short* MallocT<unsigned short>(unsigned long) (alloc_func.hpp:69)
==31684==    by 0xAB5DE5: OverrideManagerBase::OverrideManagerBase(unsigned short, unsigned short, unsigned short) (newgrf_commons.cpp:48)
==31684==    by 0xAEA373: IndustryOverrideManager::IndustryOverrideManager(unsigned short, unsigned short, unsigned short) (newgrf_commons.h:239)
==31684==    by 0xAEA2D0: __static_initialization_and_destruction_0(int, int) (newgrf_industries.cpp:30)
==31684==    by 0xAEA339: _GLOBAL__sub_I__industry_mngr (newgrf_industries.cpp:692)
==31684==    by 0xCFAE2C: __libc_csu_init (in OpenTTD/build4openttd)
==31684==    by 0x54F203F: (below main) (libc-start.c:264)

When joining a multiplayer games. There are more similar errors following.

Maybe more useful what Thread sanitizer returns:

WARNING: ThreadSanitizer: heap-use-after-free (pid=1138)
  Read of size 2 at 0x7b1400002758 by main thread (mutexes: write M626):
    #0 OverrideManagerBase::IsValidID(unsigned short) const OpenTTD/src/saveload/../newgrf_commons.h:221:57 (openttd+0xfcf140)
    #1 Save_NewGRFMapping(OverrideManagerBase const&) OpenTTD/src/saveload/newgrf_sl.cpp:33:16 (openttd+0xfce092)
    #2 Save_IIDS() OpenTTD/src/saveload/industry_sl.cpp:93:2 (openttd+0xfbf28a)
    #3 SlSaveChunk(ChunkHandler const*) OpenTTD/src/saveload/saveload.cpp:1776:4 (openttd+0xff878e)
    #4 SlSaveChunks() OpenTTD/src/saveload/saveload.cpp:1792:3 (openttd+0xff7fe1)
    #5 DoSave(SaveFilter*, bool) OpenTTD/src/saveload/saveload.cpp:2564:2 (openttd+0xff5bd9)
    #6 SaveWithFilter(SaveFilter*, bool) OpenTTD/src/saveload/saveload.cpp:2590:10 (openttd+0xff5a12)
    #7 ServerNetworkGameSocketHandler::SendMap() OpenTTD/src/network/network_server.cpp:629:7 (openttd+0xed3ac3)
    #8 ServerNetworkGameSocketHandler::Receive_CLIENT_GETMAP(Packet*) OpenTTD/src/network/network_server.cpp:1015:15 (openttd+0xed8b9a)
    #9 NetworkGameSocketHandler::HandlePacket(Packet*) OpenTTD/src/network/core/tcp_game.cpp:84:58 (openttd+0xe44a91)
    #10 NetworkGameSocketHandler::ReceivePackets() OpenTTD/src/network/core/tcp_game.cpp:139:27 (openttd+0xe45708)
    #11 TCPListenHandler<ServerNetworkGameSocketHandler, (unsigned char)0, (unsigned char)1>::Receive() OpenTTD/src/network/core/tcp_listen.h:133:9 (openttd+0xe59ae8)
    #12 NetworkReceive() OpenTTD/src/network/network.cpp:975:10 (openttd+0xe52eec)
    #13 NetworkGameLoop() OpenTTD/src/network/network.cpp:1013:7 (openttd+0xe52a77)
    #14 GameLoop() OpenTTD/src/openttd.cpp:1470:3 (openttd+0x14224b6)
    #15 VideoDriver::GameLoop() OpenTTD/src/video/video_driver.cpp:37:3 (openttd+0x1066e00)
    #16 VideoDriver::Tick() OpenTTD/src/video/video_driver.cpp:103:9 (openttd+0x1067349)
    #17 VideoDriver_Dedicated::MainLoop() OpenTTD/src/video/dedicated_v.cpp:275:9 (openttd+0x10654b8)
    #18 openttd_main(int, char**) OpenTTD/src/openttd.cpp:800:30 (openttd+0x141d26d)
    #19 main OpenTTD/src/os/unix/unix.cpp:262:12 (openttd+0xee4ad2)

  Previous write of size 8 at 0x7b1400002758 by main thread:
    [failed to restore the stack]

  As if synchronized via sleep:
    #0 nanosleep <null> (openttd+0xa4990d)
    #1 void std::this_thread::sleep_for<long, std::ratio<1l, 1000000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000000l> > const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/thread:378:9 (openttd+0x1067eed)
    #2 VideoDriver::SleepTillNextTick() OpenTTD/src/video/video_driver.cpp:172:3 (openttd+0x10679f3)
    #3 VideoDriver_Dedicated::MainLoop() OpenTTD/src/video/dedicated_v.cpp:276:9 (openttd+0x10654c6)
    #4 openttd_main(int, char**) OpenTTD/src/openttd.cpp:800:30 (openttd+0x141d26d)
    #5 main OpenTTD/src/os/unix/unix.cpp:262:12 (openttd+0xee4ad2)

  Location is heap block of size 69 at 0x7b1400002710 allocated by main thread:
    #0 operator new(unsigned long) <null> (openttd+0xad8fcb)
    #1 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) <null> (libstdc++.so.6+0x142e7d)
    #2 DeterminePaths(char const*) OpenTTD/src/fileio.cpp:1049:52 (openttd+0x119ab54)
    #3 openttd_main(int, char**) OpenTTD/src/openttd.cpp:666:2 (openttd+0x141c251)
    #4 main OpenTTD/src/os/unix/unix.cpp:262:12 (openttd+0xee4ad2)

  Mutex M626 (0x7b34000002a8) created at:
    #0 pthread_mutex_lock <null> (openttd+0xa69486)
    #1 __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:749:12 (openttd+0xdc6a36)
    #2 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_mutex.h:100:17 (openttd+0xdcf268)
    #3 std::lock_guard<std::mutex>::lock_guard(std::mutex&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_mutex.h:159:19 (openttd+0xdce7c2)
    #4 VideoDriver::GameLoop() OpenTTD/src/video/video_driver.cpp:35:31 (openttd+0x1066df6)
    #5 VideoDriver::Tick() OpenTTD/src/video/video_driver.cpp:103:9 (openttd+0x1067349)
    #6 VideoDriver_Dedicated::MainLoop() OpenTTD/src/video/dedicated_v.cpp:275:9 (openttd+0x10654b8)
    #7 openttd_main(int, char**) OpenTTD/src/openttd.cpp:800:30 (openttd+0x141d26d)
    #8 main OpenTTD/src/os/unix/unix.cpp:262:12 (openttd+0xee4ad2)
PeterN added a commit to PeterN/OpenTTD that referenced this pull request May 9, 2021
@PeterN PeterN deleted the PeterN:sparse-overrides branch May 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants