Lua Embedding #13
Comments
|
Current state of Lua work:
|
ghost
assigned
|
Next work will be to re-work Button and ObjectButton to perhaps just be scripted entities. Or maybe Button can remain but be a default Button configured Entity for extension by the user. |
|
Thinking about creating a statemanager lookup system, that will enable entites to perform function calls on any part of the active state or statemanager. Would be possible to call a few general StateManager functions that would influence the active state, such as setting exit, or attempt to call a Lua function. The Lua function call should allow for some guidance of where it should look, ie. State or Entites. Then will descend into the appropriate area, if Entity, find the entity by some name and try to call the function name passed on its Lua script. The difficulties in setting this up is that it's not possible to just load up the lua script file we want and call the function because it won't be in the context of the active lua state we're thinking of, but rather the script will be loaded into the lua state doing the loading, so it won't have the desired effect. Leads me to the StateManager based lookup system, as the StateManager is static and knows which State is running, which in turn knows its EntityManager and so forth, which enables us to perform a call into the lua state we want (on that's already running) through something that we can easily access. At least this is the idea. Implementation is yet to be done. This is more a note so I remember the idea. |
|
My commit from this evening:
Will be working on a lookup system next to dispatch function calls to the active state or to entities within the state. |
|
Idea for lookup system based on taking a peek at unity:
However this still doesn't solve the issue of passing varying parameters if the call goes through C++. Is there a way to bypass C++ entirely can call a function directly on the Entity's Lua state? Or is there a way to do it through C++? edit: What if i pushed the desired Entity function onto the lua stack to be called? |
|
Some notes to myself: but will this resolve the issue of handling varying/multiple arguments that are potentially of type that I've defined instead of a default type? How are my types represented in lua so that I know how to push and get them? Do i need to worry about this if i use lua_xmove to push data back and forth? Call could be something like callfunc(nargs, nreturns, args) |
|
Can now lookup LuaScripts by name, get their LuaScript instance pointer and call functions on them. These functions should take no parameters and return nothing, will be trying params/returns now. Unsure of whether or not lua_xmove will actually be the right function to transfer things as it mentions they should be members of the same global lua_state. These changes are in the experiments branch, in the LuaCrossStateCallTest project |
|
It seems the issue is that data I want isn't pushed onto the stack. Performing a stack dump on scriptB at the time of calling script:CallFunction(..) from scriptB: require "LuaScript"
require "Vector2"
print "Script B!"
script = LPC.LuaScript.GetScript("scriptA")
--vect = LPC.Vector2i(5, 4)
test = "Some Test string"
print (test)
script:CallFunction("Test", 2, 5)gives so how can i get the variable test? It seems luabind doesn't accept the "..." param, so i can't use a var args list, so I need someway to push an arbitrary amount of data to the stack so that it can be transferred over. |
|
Other idea: define my own generic call function based on the link above. How will I get the LuaScript class context if I write a version of lua's call_va? |
|
Super slow version is made! However due to some issues with lua_touserdata mangling my data the script has to be looked up by name each call, which is really really bad. Besides that, it appears to work to pass simple params, unsure on userdata. Need to look into return values as well. Will check out passing userdata and return values and then look into why lua_touserdata mangles my LuaScript* |
|
Return values work, userdata seems to be a bit problematic to pass side note to self: luacfunctions return # should be the number of return values to expect back from the function, so all the modules register functions should return 0. |
|
It seems that the userdata issue was related to the way the luabind modules defined. I made my own class and registered using the Lua C API and was able to pass it between states as lightuserdata. Only issue: Because each instance of the class is registered with the metatable that tracks whether or not it's a piece of that userdata an issue arises when passing userdata between states as the metatable context is lost. This leaves two options:
I'm unsure if this means that luabind will have to be abandoned and I hope not since i do really like it. Perhaps there's a way to get this working with luabind, although my previous attempts where pretty unsuccessful, even when using unsafe methods for casting userdata back to its expected type. Maybe if we dump luabind we can use LuaJIT? It's gonna hurt though to migrate everything over. Overloaded C++ functions? Maybe check size of stack and compare to num params each is expecting? Added a note about proving that we're calling into the existing state, although the usage of global someNum doesn't really prove anything. The fact that Script A! only prints once, proves that we're calling into the existing state. But i also wanted to try using the state's data as well. |
|
Interesting developments in the Lua C API work. I've decided to attempt to re-make the LuaScript class with the Lua C API as LuaCScript (see experiments branch) LuaCScript currently deals with double pointers because the TScriptMap is a map<string, LuaCScript*> so when you want to retrieve one to perhaps call a function on or get data, you get a LuaCScript** so that you can point to the pointer. Haven't yet put together the LuaCScript version of GenericCall but will work on that next. I also need to figure out usage of constructors and destructors for userdata that will be GC'd, such as the LuaRect. Non-GC'd object will be used as double-pointers in Lua and memory managed via C++ while GC'd objects will be used as single-pointers. Once I learn how to do everything that luabind currently hides I'll drop it and begin using the Lua C API, and look into making use of LuaJIT to boost speed. |
|
I've decided to go with option 3 for now:
I've written the generic call function for LuaCScript, next I need to write a parser for the udata signature that will step through the stack and at each udata perform the appropriate metatable registration. It's assumed that the script being called into has the lib associated with the various udata params open, or else it wouldn't be able to use them anyways, and the metatable registration would fail. |
|
Should also do a check to determine if a udata type list was given, ie. after popping the script udata, function name, # params and # results off the stack, if a udata type string was given the stack size should be # params + 1. If no udata type list given, check through the stack for udata. If any udata is found, print a warning about the speed impact of trying to resolve its type and register it appropriately and then attempt to resolve the udata type. Is there a way to find the metatable some udata is in just from the udata itself? |
|
Have decided to leave luabind in favor of the Lua C API.
|
|
Working on property accessors for set/get type behavior through --Get the value of x from r
num = r.x
--Set the value of x
r.x = 5I've figured a workaround for __newindex where instead of registering a table of functions i register a lookup function which will determine the appropriate setter and call it. Contemplating doing the same for getting where if only the index val was passed it would return the value at that point, however I think I could push a table of functions to __newindex and use those, but I'm not sure. I'll check back at Lua IRC later but will fully implement the crummy method for now to test it. Wrote an accessor lookup table which is registered in __newindex and works for sets but not for gets, ie. with print(r.x) for ex. it instead prints a function address of some sort using it as :accessor("x") works, but i think it should be in __index to be a getter, but I need the other functions in there as well. Need to work more on this. What I really need is to push tables to __index and __newindex but I'm unsure how to do this. Back to IRC tomorrow. Will work on udata parsing for now |
|
Addendum to udata signature idea: This may not be necessary, it looks like I can snag the metatable of objects on the stack, so i can define a function that all udata will have in its metatable like mytype or something, grab the metatable from the udata, call that function and check the result and use the result to select the appropriate metatable for the data to be registered with in the target state. See this SO post This worked well. Now udata's metatable has a type function which will return the C++ classname as a string, so in my call function i can check udata for this field and resolve the type and thus determine the correct metatable to register the type with in the recieving state. |
|
Have made a primitive version of the system which resolves the udata type and will add it to the appropriate table. Currently only for LuaRect, but generic version coming soon. udata signature string is not needed, so is removed. Generic version will step through the stack looking for udata, when it finds it it will perform readType call on the udata to get the typename, and then use this typename to get the appropriate addX function to add the correct metatable in the receiving state so that the data can be passed seamlessly. (User defined types??? i hope not.. maybe there's a way, but look at much much later) |
|
Current Progress:
|
|
Progress update:
The migration of master over to the Lua C API is going to take a long time, and longer to fully test and tweak. In addition I have to put in the Lua -> C++ call dispatcher, ie. calling into State functions. However this shouldn't be used for a lot I think, as the majority of user defined stuff should be written in Lua scripts running on Entity's which can be snagged and called Lua -> Lua. Anyways, gonna take some time to get this in. As an implementation note: The majority of Lua functions will be private besides those that are absolutely necessary to be in C++, and will be defined (in header & implementation) after all C++ members, and will have names beginning with lowercase characters. The hope is to make all Lua specific functions very obvious to try and keep the massive amount of them from getting in the way as much as possible. They should also be easily identifiable by the fact that they all take a lua_State pointer as a param. Additional note about implementation in the main engine: I may bump the Lua C API code into its own class or namespace, so it's separate entirely from the C++ class to avoid clutter. |
|
Decided on this method for the implementation: Files will be luacentity.h/.cpp and put into a folder titled luac located in the top directory. |
|
Lua good practice note: The stack should be back to its initiali configuration after the function call has finished, not the way i do it now, where i knock everything off the stack. |
|
Note to self: Use std::function for the function pointers, see http://en.cppreference.com/w/cpp/utility/functional/function cool new C++11 goodies Additional note: I don't think Lua will accept a std::function, will have to use pointers there. But other internal stuff can use it. Also note: None of the toString or concat operator functions will be written for the libs until everything else is written, because I need to write std::string conversion operators for all the classes that have Lua Libs. This std::string operator can also serve as a debug dump sort of thing, pushing all relevant information about the object into a string. There are various classes which should/shouldn't be managed by Lua. Classes that should be managed by C++ (use double pointers in the libs)
Classes that should be managed by Lua (single ptr)
Classes that will be very different, will just consist of a single class function table, no object metatables
Method for implementing the very different classes can be tested with the Debug class, as it has no dependencies. |
|
Entity class changes question: Should Update and Move be rolled into one function and should we call Physics::Move(deltaT) before returning from Entity::Update(deltaT) |
|
Should i print debug log error for invalid accessor indices? |
|
Work update Classes that should be managed by C++ (use double pointers in the libs)
Classes that should be managed by Lua (single ptr)
Classes that will be very different, will just consist of a single class function table, no object metatables
|
|
Question: for lua_xmove is the object metatable moved as well? I assumed no because i got "attempt to index userdata" errors, but maybe it just didn't know what to do with the table? I should check for certain. |
|
On the C++ side, some overloaded operators shouldn't be member functions? |
|
regarding my recent commit, what if i got a pointer to the entitymanager instead of operating on it through the state? It crashes when trying to call the entitymanager functions, so what would happen if i got it directly? Decided to get EntityManager shared_ptr directly, now also got State and Entity libs working along with cross lua-state calls, which does still need to be tested with user data params and return values. |
|
I think to do the Image, AnimatedImage and Text Lua libraries I should setup a Resource Manager of some sort as described in the issue about it and then use that to manage the memory and just give Lua a shared_ptr. Also, it should be possible to give Lua a shared_ptr since it doesn't use it internally at all, and only uses it when calling into the C library |
|
After looking into using shared_ptrs for lua userdata, it seems promising, however it could lead to interesting memory leaks for a few reasons. Lua's garbage collection is lazy, it seems that when something goes out of scope it isn't immediately deleted, Lua instead waits until there's a good amount of garbage to kick in the collector. This could lead to issues where you want an entity to remove the entity ptr from the state but Lua won't do it automatically. When an lua Entity shared_ptr is deleted, should it also be removed from the manager? Or should it stay and let the Entity take care of itself? I think a specific entity:delete function would be best option here and for above. This would remove the entity memory from Lua and remove it from the manager. If an entity is holding its own shared_ptr (as they will later), what will happen upon exiting the state? A lone shared_ptr will still exist in the Lua state, keeping the entity alive. A specific function should exist to delete the entity instead of relying on the constructor, as the entity may be kept alive by stray shared_ptrs. So when the entitymanager exits all the entity's and their scripts should be forced to quit. This may provide the proper memory freeing to enable usage of shared_ptrs without keeping alive dead objects. Will have to test it out, although I may be unable to until next weekend after midterms. |
|
Have realized a big issue with the cross-state call data transferring. lua_xmove is not a suitable solution for transferring parameters since it removes them from the caller stack, making the data unusable in the caller state after making the call. The data should be copied, not moved. Edit: The above is wrong, the data is still there but it looks like the metatable is removed. So the metatable needs to be restored to it somehow? It seems to copy primitive types correctly, but userdata gets messed up. So i think it's just losing the table. From making use of the new Lua accessible stack dump call, it's clear that the metatable is being lost on the caller's userdata. Why is this occurring? Is it an effect of lua_xmove? Once the data has been moved, i no longer have it to restore metatables so perhaps the solution is to make my own copier function after all? |
|
Unsure of how to restore the metatables after they're stripped by lua_xmove on the caller state as the userdata is no longer on the stack, so I'll be writing a copier function. Initial plan is to do it like so:
Self note: Need to make sure I'm following the naming convention i decided on. Functions for usage in C++ should have capital names, LuaCFunctions should have lowercase. It appears I've broken this convention, so I should rename some stuff. namely, checkX addX lua_openX and so on. |
|
Have written a set of copier functions in the LuaScriptLib class to facilitate transfer of some number of unknown data types between Lua states and preserve the stack ordering. I've tested it on primitives and the Vector2f, I need to write the remaining Allocate, Copy and Push functions for the other libraries. I don't need the table adders map anymore right? Since now the userdata metatable restoration is handled by the userdata copiers table |
|
I think having entity be a shared_ptr is causing some issues when the State script closes after the EntityManager has freed and then tries to garbage collect any entity shared_ptrs it may be holding which have been invalidated by the EntityManager closing. weak_ptr I believe should resolve this issue, providing the necessary safety check and not implying ownership, since the script shouldn't really "own" the entity shared_ptr, it's just using it on occasion. |
|
Discovered an issue with using weak/shared ptr for Entity in lua. I forgot to change the lua entity allocation in the StateLib when getting an entity via State.getEntity to use shared_ptr when I changed EntityLib so it was using Entity **luaE = (Entity**)lua_newuserdata(l, sizeof(Entity*));
luaE = entity.get();(entity is a std::shared_ptr) So it gave the illusion of working with shared ptr but really wasn't at all. This may explain why sometimes the program crashes when exiting. Anyways after migrating it to use the right method it crashes anytime I try to set the Lua shared_ptr to an existing shared_ptr in C++. It appears to be performing a swap operation? Not good at all. Same issue with weak_ptrs. So it looks like it's back to raw pointers? I'm sure someone said this was possible in the Lua IRC maybe I'll check back. I suppose for now I'll make it use raw pointers so I can continue testing my cross-state call stuff. Rest of program is ok, just don't use the EntityLib for a bit. |
|
Thanks to the Lua IRC channel turning me on to placement syntax, shared/weak ptrs are ok! Fantastic. Should I be doing this for creation of all my objects? hmm Decided yes. Now that things are back on track, I'll be moving the EntityLib shared_ptr over to a weak_ptr, finish setting up the other C++ managed data libs that aren't Image/AnimatedImage/Text and then create the resource manager and set up those final three libraries. |
|
With the addition of the C/P/A functions and the placement new stuff here's an update on what needs to be done. Classes that should be managed by C++ (use double pointers in the libs)
Classes that should be managed by Lua (single ptr)
Classes that will be very different, will just consist of a single class function table, no object metatables
|
|
For functions where only one type is valid at each param, it's not needed to use LuaScriptLib::readType to do type checking, as XLib::checkX will throw an error which will get logged to Debug Log if the type is wrong, and is verbose enough to find the issue, so no extra work on our part is needed. For overloaded functions we'll still need to do the check ourselves. Also noted that some libraries I forgot to update to use AllocateX instead of lua_newuserdata (besides in the Allocate function) and need to update those. |
|
Have written and tested all the Lua libraries. Remaining work to be done:
|
|
For some reason the exact same version of the template call function for LuaScript doesn't compile in the main engine but does work in the experiments LuaCrossStateCallTest project. Why this is i have no idea, everything is identical but it refuses to figure out the trailing return type i think. I'm not sure what the reason for this is since the exact same code works in the other branch. |
|
I've noticed that self is a parameter used in lua when a function like function Blah:fcnName()
endis called the metatable (object?) calling the function via myObj:fcnName()is passed in as self (myObj). This clashes with the name I've chosen for the "this" entity pointer being pushed into the Lua states, which is named self as a global. I should change this name. Related to this I'm curious about providing some simple object methods for the Lua scripts and then instead of free functions in the scripts it'd be something like: --player.lua
player = Object(parent) --parent is an optional parameter specifying the class to inherit from
function player:init()
--init the player
endand so forth like this. Then the C++ side would get the global object with the same name as the script and call functions on it. This may be neat in that users could write their own parent classes and inherit from them in their script classes and such, giving them the benefits of reusing code and whatnot. I'll check this out in the experiments branch soon. |
|
I'd like to make the camera scriptable, it'd also be nice to change the map some from scripts, ie. change tiles or such |
|
For implementing a class system, i'd like to try it out with a stripped down version of the entity system and such and see how it works. I think it'll be a good idea, since then we can use inheritance and such but i'd like to sort it out in the experiments branch first. |
|
Should probably change the assert that the states are equal to throw/log a debug error |
|
I think LuaRefs should be held by the LuaScript they exist in, some issues could come up if they become too separated. I'd also like to have the ability to just pass regular data in as params like a standard function call, and then the script would create the appropriate LuaParams, some kind of flag would be needed for passing references though. Since the script will hold them in some kind of structure (map perhaps?) maybe pass in a string with a flag of some sort? Hmm. To do this would it be best to hide the distinction between udata and raw data? Ie. define UDataLib even for primitive types, then LuaUdataParam and LuaPrimitiveParam become the same class. Then when figuring out the LuaParam type: template<class T, class... Args>
void callFcn(str name, T param, Args... args){
//...
LuaParam<T> luaParam = LuaParam<T>(param);
//...
} |
|
Things become more complicated: my primitive types and ptr types have some conflicts with how they need to be treated in order to be able to manipulate the actual object. Ie, the primitive types have to be used via a pointer, but the other types are already a pointer so they're fine. Something more clever needs to be thought of to resolve this + the conflicts with the regular primitives. I'll have to be more clever hah After some more thinking, they only differ in how they're handled when i want to get a value from Lua, everything is pushed on by value. However for primitives we get them by value, for my pointer types I can also get by value since they're already pointers but for my primitive types (Vector2f, Rectf) I need to get a reference/pointer to the one in Lua so I can work with the object itself instead of a copy that'll be discarded. So pushing on could be done generically sort of i think, but handling return values will require more thought. Perhaps I should return by value? And only the libraries themselves will operate directly on the reference? Perhaps a GetCopy function can be added to UdataLib that returns a copy of the object. |
|
I'd like to make it possible to attach scripts to the camera and map as well. The goal being that we should be able to create the editor in itself, ie. the tilebar is a scripted entity, the map is scripted, etc, etc. |
Implement Lua for scripting objects behavior
Need a better way for Lua scripts to load the modules they need through something like require or such
How should the State class be handled?
Should states be run via script? Hmm. What to do with states..
How to save data from script?
Helpful Reading Material
Programming in Lua: The C API
Luabind Documentation
The text was updated successfully, but these errors were encountered: