Lua stability

[ilteroi]

i'm getting intermittent crashes from Lua during autoplay, typically in CallTestAll() ... in different hooks. this is annoying and seems impossible to fix, at least for me. does this happen to others as well?

anyway, which of these lua hooks are actually used? i know modders might want to use them, but i would like to disable them by default if we don't need them in base vp. opinions?

[LoneGazebo]

I get a crash here and there that stacks out to the LUA51 interface...

Have you tried luajit or whatever that 'lean' lua manager was called?

[LoneGazebo]

https://www.reddit.com/r/civ/comments/6a0j6l/civ_v_luajit_newer_version/

[ilteroi]

i think i tried that before and there were issues but i'll try again

[ilteroi]

still the same problem - but the crash is always an access violation in LuaSupport::CallTestAll(), so apparently it's an invalid pointer somewhere.

does anyone have an idea why that might be happening?

[LoneGazebo]

do you have a repeatable crash? you could try to see if the crash is detectable while connected to the DLL

[ilteroi]

nah, it's not repeatable. but annoying ...

[ryanmusante]

https://forums.civfanatics.com/threads/luajit-for-civilization-v-faster-turns.399635/

[ilteroi]

i patched out the lua callbacks (for my personal branch) and haven't seen a single crash since. works fine!

[LoneGazebo]

Yeah I think the additional lua callbacks added by whoward in his API were overtaxing stuff.

[LoneGazebo]

civ's lua interface isn't exactly...robust.

[LoneGazebo]

@ilteroi are you gonna push the lua callback opt out? We could lock it behind a define that modders can enable.

G

[Serpens66]

@ilteroi Hi there :)
where can I see which callbacks you mean? I also recently started my first steps on dll modding, but because of far too many bugs in multiplayer, I'm using whowards dll as base, instead of community patch (although I dont know if multiplayer bugs are dll or lua related).

Anyway which callbacks did you removed or where can I find "your branch"? My first goal is currently a bug free game, so if removing some callbacks might help...

Btw. I'm also struggling to build a dll, maybe you have a tip:
https://forums.civfanatics.com/threads/building-a-dll-based-mod-in-2019.645887/#post-15726745

[ilteroi]

well for building you should follow the guide in VPs readme.

the lua switch is in the VP master branch, but callbacks are enabled by default in CP ... search for CORE_DISABLE_LUA_HOOKS in CustomModOptions.xml

[Serpens66]

thank you very much, also for your and Gazebos work :)
Is there a thread (eg at civfanatics) where you write about dll stuff or I could ask questions, or simply open a new thread in community patch area?

And where can I find the documentation you mentioned? I readme I see a "how to build this" but it is only general information, no step by step nor trouble shooting. I tried it 2 times now, but I can not pass the "fatal error C1083: Cannot open include file: 'unordered_set': No such file or directory" error message from visual studio when trying to build the dll.

[ilteroi]

well, i don't really know your setup, but i can tell you that unordered set is part of the visual c++ stl implementation, so you should have it.

on my machine it's located at C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\include\unordered_set (for VC++ 2008) but each VS release has its own instance.

which version are you using? independent of the IDE version, you need to use the legacy 2008 toolset to compile/link the DLL.

also, unordered set is only used to store some non-essential advisor messages, you can just as well rip it out and see how far you get without it.

[Serpens66]

edit2:
solved the problem with building dll, I was missing the service pack1 for 2008, all those downloads are not easy to find and I refused to use a link in the civfanatics thread, but it was the solution.

edit1:
On topic of this github issue:
Can you tell me how you found out that a crash is related to a lua hook? I just started using whowards dll v94 (previously v88) and I already got 2 crashes within 2 hour of testing... Is there a step by step guide how to debug? (I know in readme is also sth written about that, but again not very detailed)

[ilteroi]

good point about the service pack ... anyway, did you manage to attach the visual studio debugger to a running civ game with a custom gamecore dll?

anyway if you experience a crash while running in the debugger it shows you where the exception happened. it may be that there is no source available for the crash location, but look at the callstack, if lua51.dll is involved that is a strong hint ...

[Serpens66]

found this guide on how to debug https://forums.civfanatics.com/threads/a-guide-to-debugging-crashes-with-a-custom-dll.506328 , I'm able to live debug now, at least break if there is a crash. But I can't set breakpoints myself, nor can I read minidump files (explained in that thread).

Anyway, is there no professional way to get a lua traceback, if the crash is caused by lua?
Eg. I get frequent crashes from the newst EUI, without any other mod installed, and would like to track it down. I will now try live debugging and hipe there is something crashing in the dll. But if it is a crash in lua, I wont be able to find the reason? (usually there is nothing about it in lua log)

Btw:
The 2 crashes I mentioned above (wth whowards dll) were caused when founding a pantheon without capital. But this is already fixed in CP, although it is part of "MOD_BALANCE_CORE"... So there is no easy way for me to find out all the vanilla bugfixes CP has, but whoward does not have?

edit:
yes, got a crash from EUI now and the live debugger only shows lua51. And in the lua.log is nothing... Is there really no C**lua debug library/code that can help here?!

[ilteroi]

• there is no list of bugfixes ... at this point major parts of the vanilla code have been replaced, so you cannot really compare them anymore
• lua has its own callstack and there even is a function DumpCallStack() but it's not used at the moment. maybe it's possible to store the lua_state* in GetInstance() to a global variable and call DumpCallStack() in the c++ exception handler ... check CreateMiniDump()

[Serpens66]

thanks. sounds good. unfortunately I'm no programmer and and dont know C++ (I'm currently learning by reading civ5 code). Is this something the CP-team will implement? It will also help to track down alot of crashes with CP-mods I guess.

edit: what does the stackwalker? Is this already sth for lua?

edit2: can you tell me how to read a minidump? I put the minidump in same folder like the dll and the pdb file and opened it with VS. There I see a "dump summary" and a list of modules. But nothing about the crash reason or any code. Where do I see the crash reason in the minidump? (I live debugged, so I know the reason for the crash, but I would like to find it in the dump file too, to learn how to read them).
When I hit "debug with native only", I get an error popup telling me that debug information for civilization5.exe is not there. Then I hit continue and... oh.. now I received the "do you want to break" message and can see code. But it is not the code from the dll, it is from a file called "xmemory"... does this mean the minidump itself crashed while trying to read it?!

[ilteroi]

i added lua stack logging ... good idea :)

[Serpens66]

github is really a strange place, wasnt able to find this topic, neither in closed issues nor in my profile history. Found it via email link.

ilteroi:
Could you explain how to use the lua stacking?
I built CP dll with the new code and put the dll into the new beta build from 18.April. I found an easy way to crash the game with a lua command (by accident):
pPlayer:AddNotification(NotificationTypes.NOTIFICATION_BARBARIAN, descr, title, -1, -1)
And when VS attached I saw, that there was a crash in CvPlot in getImprovementType. At first I had no clue how this was related to my code. Will your lua logging/callstack now help in any way to find out it was caused by this notification?

I still have trouble to open minidumps. When I open it, it shows nearly nothing. When hitting "debug with natives" it throws an error about civ5.exe and when I continue I got a break and indeed finally code from that getimprovementtype code. But that did not work for whowards dll, so I'm not sure if this is the supposed way.
Anyway, I see nothing about lua...

This specific crash shows C++ code, while other lua crashes only show LUA51.
So maybe your new lua callstack feature only works for the latter?