Lua scripts support #127

smbas · 2016-08-25T11:42:33Z

A basic support for the Lua scripts used in The Witcher. There is the ability to bind C++ to Lua and call Lua functions from C++.
No actual bindings are done, so.

The code is a bit raw, I think. So needs you feedback.

smbas · 2016-08-25T14:07:02Z

It doesn't build the toluapp directory for some reason. Did I miss something?
The builds on my machines are successful (on both Ubuntu 14.04 and Linux Mint 18).

DrMcCoy · 2016-08-25T14:18:33Z

It seems like you forgot to add the tolua directory to the SUBDIRS variables in the top-level Makefile.am: https://github.com/smbas/xoreos/blob/feature-witcher-lua-scripts/Makefile.am#L92

If you're building with CMake, that's not necessary, but the autotools build system needs it.

DrMcCoy · 2016-08-25T16:36:23Z

src/aurora/lua/scriptman.cpp

+	const int execResult = lua_dobuffer(_luaState, data, dataSize, path.c_str());
+	if (execResult != 0) {
+		warning("Failed to execute Lua file: %s", TypeMan.setFileType(path, kFileTypeLUC).c_str());
+	}


Hmm, you're leaking both the original stream as well as the memStream here.

DrMcCoy · 2016-08-25T16:42:19Z

Nice job in getting it to compile now. :)

Could you split up the last commit (the "BUILD: Proper Lua includes") and rebase the fixes into the original commits? I.e. the Makefile.am changes should into the first commit, the one that adds the toluapp directory in the first place, and the include fixes into the commits that add those files.

That involves an interactive git rebase. If you don't know how to do that, just say so and I'll take care of it once the PR is ready to be merged.

DrMcCoy · 2016-08-25T16:47:23Z

src/aurora/lua/scriptman.h

+	/** Was the script subsystem successfully initialized? */
+	bool ready() const;
+
+	/** Execute a sctipt file. */


Typo: s/sctipt/script/

Ah, you fixed the typo in a908f9b. If you could instead move the fix directly here (by using an interactive git rebase), i.e. have it look like you never made the typo at all, that would be better.

DrMcCoy · 2016-08-25T17:57:00Z

Apart from these issues here, your code is pretty clean and nice. :)

@clone2727 will add a few comments about the Lua parts later.

One thing, though, that's kinda missing in my eyes: actually using any of this in the Witcher engine. I.e. running any of the Lua scripts from within the game, even if it just results in a lot of warnings about missing functions, classes, what-have-you. As is, I can't really evaluate if any of this actually works.

I'm not sure yet how the Lua scripts are even called from within the engine, to be honest. I guess there are hooks for specific events? There's also the RunClientLua() NWScript engine function, which, I guess, would call ScriptManager::executeString()? These are mostly simple single-liners, though, and obviously not the brunt of the Lua functionality in the game. They also reference things that must have been set-up previously, either from full-fledged script or from the engine code. I for one have no clue. :)

smbas · 2016-08-25T18:57:01Z

Done. Hope I haven't missed anything.
I definitely need to build it using the autotools before pushing :)

Firstly, I don't know at the moment what scripts are called from the game. What I know is that there's the script "startup.luc", which call the others. And yes, RunClientLua() executes a string of Lua code.

To be able to execute Lua scripts we need to bind a lot of classes and functions. Without the bindings (or stubs at least) the execution will be interrupted immediately. Maybe there is a better way other than write stubs for every class and functions to make the scripts run, just skipping the actual logic. I haven't find any such way yet.

DrMcCoy · 2016-08-25T19:21:39Z

Done. Hope I haven't missed anything.

Looks good. :)

I don't know at the moment what scripts are called from the game

Ah, okay.

Maybe there is a better way other than write stubs for every class and functions to make the scripts run

Is there a way to add a generic "this is missing, just print its name" binding? Or I don't suppose you have lists of what is missing, so stubs could be added via a huge table, like what the NWScript system is doing?

I.e. basically what I would like to see, if possible, is kinda a mirror to the "TODO" warnings for unimplemented engine functions that the NWScript system does.

smbas · 2016-08-25T19:41:22Z

I'll work on stubs this weekend. It's a boring task, I said. Currently I just execute the startup script and see what's missing. Than write an appropriate stub. The problem here is that most stubs need to return a valid value in order to continue the execution of a script.

Also there are situations with an access to a nested variable or function. In this case Lua just throw the error "Can't access field ?". What field, who knows. You have just a question mark.
Happily there is a decompiler of Lua scripts. It makes such process a bit easier.

Currently, I would make something like this to bind some class to Lua:

class Control {
public:
    static void registerLuaBindings();

private:
    static void luaSomeMethod(lua_State *state);
};

But what about stubs? I think about creating a huge class LuaBindings, which have a lot of private classes that look like the above. Or better to create just plain functions with such signature, for example, addAbility_CDefs(). A name of a function + A name of a class. What do you think?

DrMcCoy · 2016-08-25T20:15:29Z

EDIT: Grml, why does GitHub disable Markdown for email replies...

I'll work on stubs this weekend.

Thanks. :)

Currently I start the execution of startup script and see what's missing. Than write an appropriate stub.

Yeah, that's what I feared. For NWScript, I was glad that the nwscript.nss (or nwscriptdefn.nss in The Witcher) files existed, so I could automate that.

Problem here is that most stubs need to return a valid value in order to continue the execution of a script.

Well, the NWScript engine functions that need to return something return an "empty" value, which evaluates to 0 for int, "" for string, etc. If the scripts actually need a real value, that won't work, yeah.

I think about creating a huge class LuaBindings, which have a lot of private classes that look like the above.

Hmm, yes, for stubbing, that's probably okay.

But I guess in the future, we probably want to implement them as real C++ classes with methods? Is that possible?

Otherwise, if we're creating them as plain functions, I'd rather have them in one C++ class with non-static methods even, so that when they're getting implemented, they can easily access game data. Similar to the Functions class in src/engines/witcher/script/functions.h. That works by using boost::function and boost::bind() to create a functor that binds the object to the function pointer as its this.

(Btw, I guess Witcher's script directory should be renamed to nwscript, then. And another directory, lua, would then contain the Lua side of things.)

smbas · 2016-08-26T07:33:39Z

But I guess in the future, we probably want to implement them as real C++ classes with methods? Is that possible?

Currently to do this you must create proxies to the actual methods of an object. As Lua only accepts functions with the signature: int func(lua_State *). So now we can create bindings only as static methods or free functions.

Maybe there more robust solution using C++ templates. Don't know about binding member functions, but I tried to bind a function with a custom signature that way. It looked like: Variables func(const Variables &).

smbas · 2016-09-01T17:52:04Z

Add stubs to allow execution of the global and startup scripts without errors. Not so a big win, because currently only the code from the global scope is executed. There are a lot of function that must be called from the engine.

Also I have implemented RunClientLua. But currently it's just throws exceptions that the global variable g_GuiInGame not found. It seems that the original engine has CGuiInGame::OnInitialize method, that call the appropriate code to initialize the GUI.

clone2727 · 2016-09-05T04:31:47Z

lua/llimits.h

@@ -128,7 +128,7 @@ typedef LUA_UACNUMBER l_uacNumber;
 ** type for virtual-machine instructions
 ** must be an unsigned with (at least) 4 bytes (see details in lopcodes.h)
 */
-typedef unsigned long Instruction;
+typedef unsigned int Instruction;


If this is a change to make it a fixed-width integer, it should just be uint32_t.

Agreed, uint32_t would make more sense.

DrMcCoy · 2016-09-06T18:01:25Z

Even though it doesn't functionally do much yet, it's nice to see how the code is supposed to be called and supposed to work. :)

It seems that the original engine has CGuiInGame::OnInitialize method, that call the appropriate code to initialize the GUI

Okay, interesting. So there's a lot of things that still need more research.

As for this PR, it looks really quite good. If you change these two things @clone2727 flagged, I'd say it's fit to be merged. :)

Two questions though:

How do you wish to be credited in the AUTHORS file?
Do you plan to extend upon this? Research more Witcher Lua stuff, implement more for xoreos, etc.? It's fine if you don't want to or can't for any reason, but I certainly would appreciate it.

tolua++ was used to bind C++ code to Lua scripts in The Witcher.

Lua scripts in The Witcher were compiled using a 32-bit compiler.

This methods are used to link/unlink C++ objects with their Lua instances in The Witcher. First of all, this is needed for calling the proper functions on inherited Lua objects from C++.

Using this inherited Lua objects can update the members of their C++ counterparts.

Far from a complete list, but it's enough to execute global and startup scripts without errors. Also currently only the code within the global scope are executed.

smbas · 2016-09-10T12:46:33Z

Done.

How do you wish to be credited in the AUTHORS file?

As you do for all contributors: name, username and email.

Do you plan to extend upon this? Research more Witcher Lua stuff, implement more for xoreos, etc.? It's fine if you don't want to or can't for any reason, but I certainly would appreciate it.

Yes, but currently I don't know how often I can do this. Next I want to research more about GUI system of The Witcher. The most part of it is implemented using Lua.

DrMcCoy · 2016-09-10T16:53:03Z

Okay, merged, thanks! :D

Yes, but currently I don't know how often I can do this.

Sure, no pressure.

Next I want to research more about GUI system of The Witcher.

Neat! Looking forward to this, then. :)

DrMcCoy · 2016-09-14T14:50:53Z

Hmm, there's a lot of memory issues valgrind finds in the Lua code, though: https://gist.github.com/DrMcCoy/4f2c93770c2f1a290965fa83c35e8b38 (a 64-bit build on amd64). Looks like it's all use after free?

smbas · 2016-09-14T19:35:03Z

Quick look at the Valgrind log tells that the problem seems to be the following. The Lua state is closed in the ScriptManager::deinit(). Then its destructor destroys the TableRefs that are stored in _luaObjectInstances. This includes the "unrefing" from the already freed Lua state.

Normally, it cannot happen. On the exit all Lua instances must be freed and the map _luaObjectInstances must be empty. You can find a warning about that in the log: https://github.com/xoreos/xoreos/blob/master/src/aurora/lua/scriptman.cpp#L57

Two possible quick solutions:

Temporary disable (comment) the RegisterSubst/UnRegisterSubst bindings.
Clear the _objectLuaInstances before closing the Lua state.

The right way of fixing it will be to provide the delete() method for objects that calls ``ScriptManager::unsetLuaInstanceForObject()`. But when I ran the game Lua wasn't complain about the absence of this method. It's not unusual so, because the most part of the Lua code isn't executed yet (only the code that are located in the global scope). Moreover there are probably some cleanup methods that must be called manually from the engine itself.

Also I think about storing some sort of a weak pointer to a Lua state in the TableRef. But I have no idea how to do it at the moment.

So that instances left unfreed do not try to access an already destroyed Lua state. See also #127 (comment)

DrMcCoy · 2016-09-14T22:12:47Z

Ah, okay, that makes sense. :)

Clear the _objectLuaInstances before closing the Lua state.

Yeah, that's what I added now. And with that , the valgrind issues are gone.

DrMcCoy reviewed Aug 25, 2016
View reviewed changes

clone2727 reviewed Sep 5, 2016
View reviewed changes

smbas added 9 commits September 10, 2016 14:33

ALL: Add tolua++ 1.0.92

e6d135c

tolua++ was used to bind C++ code to Lua scripts in The Witcher.

TOLUAPP: Silence few compiler warnings

f5b7c50

LUA: Use 32-bits LUAC file format on all platforms

2524411

Lua scripts in The Witcher were compiled using a 32-bit compiler.

AURORA: Add Lua script manager

ca43e12

WITCHER: Initialize Lua within the engine

fe5d0d2

AURORA: Add Lua stack wrapper

d937b00

AURORA: Bind the script manager to Lua

3b641a3

AURORA: Add the ability to ignore files by the script manager

6c965e6

AURORA: Add more getters to the Lua stack wrapper

ad189b2

smbas added 17 commits September 10, 2016 15:03

AURORA: Add Lua variable class

fa75e5d

AURORA: Implement stack operations for a list of variables

95b5948

AURORA: Implement calling Lua functions

dd81ad8

AURORA: Allow getting a Lua usertype value without a type check

3811279

AURORA: Add StackGuard class

a26a332

AURORA: Implement Lua::TableRef class

fae253e

AURORA: Implement Lua::FunctionRef class

5bb74eb

AURORA: Rewrite Lua callFunction() using TableRef & FunctionRef classes

8b47263

AURORA: Add some stub Lua bindings for the script manager

43d92db

AURORA: Allow access to Lua table elements using metamethods

405f903

AURORA: Implement RegisterSubst/UnregisterSubst Lua bindings

069438f

This methods are used to link/unlink C++ objects with their Lua instances in The Witcher. First of all, this is needed for calling the proper functions on inherited Lua objects from C++.

AURORA: Add utility functions to get C++ object from Lua

d0d88bb

AURORA: Add injection of the new index meta event into Lua tables

8f01764

Using this inherited Lua objects can update the members of their C++ counterparts.

AURORA: Add the ability to transfer object ownership to Lua

2f24580

WITCHER: Add Lua script stubs

c711a27

Far from a complete list, but it's enough to execute global and startup scripts without errors. Also currently only the code within the global scope are executed.

WITCHER: Implement RunClientLua script function

471490b

WITCHER: Rename 'script' directory to 'nwscript'

325f1ee

DrMcCoy merged commit 325f1ee into xoreos:master Sep 10, 2016

DrMcCoy added a commit that referenced this pull request Sep 14, 2016

AURORA: Force Lua instance deallocation before closing Lua state

abdf79b

So that instances left unfreed do not try to access an already destroyed Lua state. See also #127 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lua scripts support #127

Lua scripts support #127

smbas commented Aug 25, 2016

smbas commented Aug 25, 2016

DrMcCoy commented Aug 25, 2016

DrMcCoy Aug 25, 2016

DrMcCoy commented Aug 25, 2016

DrMcCoy Aug 25, 2016

DrMcCoy Aug 25, 2016

DrMcCoy commented Aug 25, 2016

smbas commented Aug 25, 2016 •

edited

Loading

DrMcCoy commented Aug 25, 2016

smbas commented Aug 25, 2016 •

edited

Loading

DrMcCoy commented Aug 25, 2016 •

edited

Loading

smbas commented Aug 26, 2016 •

edited

Loading

smbas commented Sep 1, 2016

clone2727 Sep 5, 2016

DrMcCoy Sep 6, 2016

DrMcCoy commented Sep 6, 2016

smbas commented Sep 10, 2016

DrMcCoy commented Sep 10, 2016

DrMcCoy commented Sep 14, 2016

smbas commented Sep 14, 2016

DrMcCoy commented Sep 14, 2016

Lua scripts support #127

Lua scripts support #127

Conversation

smbas commented Aug 25, 2016

smbas commented Aug 25, 2016

DrMcCoy commented Aug 25, 2016

DrMcCoy Aug 25, 2016

Choose a reason for hiding this comment

DrMcCoy commented Aug 25, 2016

DrMcCoy Aug 25, 2016

Choose a reason for hiding this comment

DrMcCoy Aug 25, 2016

Choose a reason for hiding this comment

DrMcCoy commented Aug 25, 2016

smbas commented Aug 25, 2016 • edited Loading

DrMcCoy commented Aug 25, 2016

smbas commented Aug 25, 2016 • edited Loading

DrMcCoy commented Aug 25, 2016 • edited Loading

smbas commented Aug 26, 2016 • edited Loading

smbas commented Sep 1, 2016

clone2727 Sep 5, 2016

Choose a reason for hiding this comment

DrMcCoy Sep 6, 2016

Choose a reason for hiding this comment

DrMcCoy commented Sep 6, 2016

smbas commented Sep 10, 2016

DrMcCoy commented Sep 10, 2016

DrMcCoy commented Sep 14, 2016

smbas commented Sep 14, 2016

DrMcCoy commented Sep 14, 2016

smbas commented Aug 25, 2016 •

edited

Loading

smbas commented Aug 25, 2016 •

edited

Loading

DrMcCoy commented Aug 25, 2016 •

edited

Loading

smbas commented Aug 26, 2016 •

edited

Loading