Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AI can allocate more memory than the system has, crashing the game #6322

Closed
DorpsGek opened this issue Jun 15, 2015 · 12 comments
Closed

AI can allocate more memory than the system has, crashing the game #6322

DorpsGek opened this issue Jun 15, 2015 · 12 comments

Comments

@DorpsGek
Copy link

@DorpsGek DorpsGek commented Jun 15, 2015

fanioz opened the ticket and wrote:

I was just playing the game before sleep. Let it along night, and found it crash upon wake up with the following message " Out of memory. Cannot reallocate 201326592 bytes ". So I can't tell you exactly what happened before crash.
Attached the crash.* file

Attachments

Reported version: 1.5.0
Operating system: Windows


This issue was imported from FlySpray: https://bugs.openttd.org/task/6322
@DorpsGek
Copy link
Author

@DorpsGek DorpsGek commented Jun 16, 2015

frosch wrote:

Information from crash.dmp:
0012f434 006a6dae 00000016 0bb9a8ab 01000000 openttd!CustomAbort+0x10 [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\os\windows\crashlog_win.cpp @ 544]
0012f480 006a60a3 00000016 0043e62c 2074754f openttd!raise+0x17a [f:\dd\vctools\crt_bld\self_x86\crt\src\winsig.c @ 586]
0012f488 0043e62c 2074754f 6d20666f 726f6d65 openttd!abort+0x10 [f:\dd\vctools\crt_bld\self_x86\crt\src\abort.c @ 74]
0012f690 00469d46 00ee2dc0 0c000000 0053de3b openttd!error+0x3e [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\openttd.cpp @ 124]
0012f69c 0053de3b 0c000000 42b9dd8c 00530f35 openttd!ReallocError+0xe [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\core\alloc_func.cpp @ 31]
0012f6a8 00530f35 49d20020 0c000000 00000000 openttd!sq_vm_realloc+0x40 [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\3rdparty\squirrel\squirrel\sqmem.cpp @ 14]
0012f6c0 0052dff8 01000000 00000000 ffffffff openttd!sqvector::_realloc+0x34 [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\3rdparty\squirrel\squirrel\squtils.h @ 106]
0012f6f0 0052f205 08000000 36f7c1e8 0012f708 openttd!SQArray::Append+0x43 [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\3rdparty\squirrel\squirrel\sqarray.h @ 60]
0012f700 00531c13 0012f7a0 0054a684 08000000 openttd!sq_arrayappend+0x60 [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\3rdparty\squirrel\squirrel\sqapi.cpp @ 287]
0012f708 0054a684 08000000 2ffe9ce0 286dc128 openttd!array_append+0xd [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\3rdparty\squirrel\squirrel\sqbaselib.cpp @ 434]
0012f7a0 0054832e 286dc128 26580430 00000006 openttd!SQVM::CallNative+0x413 [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\3rdparty\squirrel\squirrel\sqvm.cpp @ 1189]
0012f940 00530608 286dc128 00f3d6f4 00000027 openttd!SQVM::Execute+0x9ca [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\3rdparty\squirrel\squirrel\sqvm.cpp @ 798]
0012f99c 0052d167 00001388 00001388 0052bab7 openttd!sq_resumecatch+0x6d [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\3rdparty\squirrel\squirrel\sqapi.cpp @ 1022]
0012f9a8 0052bab7 00000000 00000005 00000010 openttd!Squirrel::Resume+0x23 [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\script\squirrel.cpp @ 201]
0012f9f4 0054c0a6 26e0a2a8 00000000 13b6af78 openttd!ScriptInstance::GameLoop+0x236 [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\script\script_instance.cpp @ 239]
0012fa30 004407c4 13b6af78 00000000 00f6d90c openttd!AI::GameLoop+0xad [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\ai\ai_core.cpp @ 80]
0012fb6c 004409b3 76b3a256 00000000 005d3e9d openttd!StateGameLoop+0x142 [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\openttd.cpp @ 1389]
0012fb78 005d3e9d 00000000 00000000 0012fde0 openttd!GameLoop+0xea [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\openttd.cpp @ 1484]
0012fbc0 0043f90f 02443718 0012fdf8 7ffd4000 openttd!VideoDriver_Win32::MainLoop+0x2f8 [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\video\win32_v.cpp @ 1278]
0012fde0 005d61d4 00000001 0012fdf8 00000000 openttd!openttd_main+0xa71 [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\openttd.cpp @ 865]
0012fef8 006a84bc 00400000 00000000 00292937 openttd!WinMain+0x7b [c:\bamboo-agent-home\xml-data\build-dir\ottd-rls-w32bit\src\os\windows\win32.cpp @ 457]
0012ff88 764fee1c 7ffd4000 0012ffd4 7740399b openttd!__tmainCRTStartup+0x11a [f:\dd\vctools\crt_bld\self_x86\crt\src\crt0.c @ 275]


This comment was imported from FlySpray: https://bugs.openttd.org/task/6322#comment13973

@DorpsGek
Copy link
Author

@DorpsGek DorpsGek commented Jun 16, 2015

frosch wrote:

I.e. the crash was caused by an AI, allocating a huge array.


This comment was imported from FlySpray: https://bugs.openttd.org/task/6322#comment13974

@DorpsGek
Copy link
Author

@DorpsGek DorpsGek commented Oct 7, 2017

james1101 wrote:

I've also seen it happen with WmDot, in version 1.7.1:

  1. Select WmDot as an AI for the game. (settings to change to which values: Debug Level = 8; Max Atlas Size = 150; Build Attempts = 15)
  2. Start new game (large map size, 1M tiles or more or 1k by 1k or larger)
  3. Execute console command: "startai" until WmDot starts.
  4. Open AI/Game Script Debug window.
  5. Wait until it crashes.

This comment was imported from FlySpray: https://bugs.openttd.org/task/6322#comment14779

@TrueBrain
Copy link
Member

@TrueBrain TrueBrain commented Apr 12, 2018

Random idea: sandbox the AI so it is limited in how much memory it can allocate, and kill the AI before the OS kills the game if it overspends.

@TrueBrain TrueBrain changed the title Crash: Out of memory due to AI/GS (in particular when using 32bit build and 32bpp blitter) AI can allocate more memory than the system has, crashing the game Apr 12, 2018
@nielsmh
Copy link
Contributor

@nielsmh nielsmh commented Nov 1, 2018

Squirrel is a bit annoying in how it handles allocation. The ideal fix would be changing everything around so it uses an allocator object (which can then be an arena allocator or whatever you want) configured per SQVM, or rather per SQSharedState. I'm not entirely sure how much work it will be to make such a patch.

@stale
Copy link

@stale stale bot commented Jan 24, 2019

This issue has been automatically marked as stale because it has not had any activity in the last two months.
If you believe the issue is still relevant, please test on the latest nightly and report back.
It will be closed if no further activity occurs within 7 days.
Thank you for your contributions.

@stale stale bot added the stale label Jan 24, 2019
@LordAro LordAro added pinned and removed stale labels Jan 24, 2019
@nielsmh
Copy link
Contributor

@nielsmh nielsmh commented Jan 25, 2019

Followup on my previous comment, I attempted to track memory use in Squirrel, and it turned out it would likely require a lot of intrusive changes to have the VM link every allocation to a separate pool. Making a good fix for this will require someone willing to deep-dive into Squirrel internals.

@James103
Copy link
Contributor

@James103 James103 commented May 21, 2019

Despite @nielsmh's attempts to measure and limit memory allocations in Squirrel, the game can still crash with "Out of memory. Cannot reallocate 1200000000 bytes" when an AI initializes a huge array with for example, 100 million elements at once (local x = array(100000000);). This crash may happen even when the max memory allowed to AI/GS is 8 MiB.

@nielsmh
Copy link
Contributor

@nielsmh nielsmh commented May 21, 2019

It's not possible to safely fail an allocation inside Squirrel's allocator. I tried throwing an exception from the allocator as soon as the allocation would exceed the allotted, and it was not possible to reasonably correctly catch it everywhere it could occur. So allocation limit is only checked after each call into Squirrel.

It may be possible to have some kind of "may fail" flag to the allocation functions, that could be set when they're called in the context of allocating user data structures so those operations can fail in a safer manner, but it would be a much more intrusive change and require a bunch of extra care.

@glx22
Copy link
Contributor

@glx22 glx22 commented May 21, 2019

Can't allocation be checked in allocator malloc and realloc before the real allocation ?

@nielsmh
Copy link
Contributor

@nielsmh nielsmh commented May 21, 2019

How would you signal allocation failure in a safe way? If you just make malloc/realloc able to return a nullptr when the pool is exhausted you'd have to go through all of Squirrel and make sure all allocation results are checked and cause an appropriate failure.

If you instead throw an exception from malloc/realloc you have to at least go over all call sites into Squirrel (including all setup/registration code) and make sure to catch that exception, and make sure to take down the VM properly.

For either of those approaches you'd need some special mode in the allocator that makes it work despite the pseudo-OOM condition since otherwise Squirrel may not be able to unwind and shut down itself properly. At this point you may as well just add an "allowed to fail" flag to malloc/realloc and only set that in some selected call sites where OOM can be handled sensibly.

@andythenorth
Copy link
Contributor

@andythenorth andythenorth commented Oct 22, 2019

See also #7513, which is not the same issue, but similar area.

rubidium42 added a commit to rubidium42/OpenTTD that referenced this issue Apr 17, 2021
…an be allocated instead of crashing the whole game
rubidium42 added a commit to rubidium42/OpenTTD that referenced this issue Apr 17, 2021
…an be allocated instead of crashing the whole game
@michicc michicc closed this in e5fedcd Apr 17, 2021
LordAro added a commit to LordAro/OpenTTD that referenced this issue Apr 18, 2021
…an be allocated instead of crashing the whole game
LordAro added a commit that referenced this issue Apr 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants