Efficiently binding a LOT of things to sol

[Sixmorphugus]

Hi,
As you might know I have been writing a lua binder for the Unreal Engine using this library.
Currently, I am binding everything by creating simple usertypes (create_simple_usertype), adding functions/members one by one via set, and adding that to the lua state directly. This reduces compile time a lot and fixes all the errors caused by the compiler running out of space.

However, set still generally goes through more than 40 levels of template code. This means that one engine module (for example, the very large core module with 1,456 classes) will take a very long time to compile and debug. I am aware that sol is made to simplify binding things to lua, but would at this point be very interested in knowing if any lower level methods of binding a user type are available.

I have already done several things to mitigate the long compilation, such as duplicating the default UHT behavior of splitting generated source files into multiple compilation units. However, this doesn't solve the problem of set being very complex. With the information we already use to create the bindings (information that comes from UHT before our generator runs) we can probably provide much of the information that is currently being found automatically.

Our autogenerated binding functions currently look like this:

#pragma once
#include "D:\Repos\BrickSpaceStuff\BrickSpaceEngine\Engine\Source\Runtime\Engine\Classes\Camera\PlayerCameraManager.h" // Engine
#pragma warning( disable : 4503 ) // disable irrelevant "name truncated" warning

PRAGMA_DISABLE_DEPRECATION_WARNINGS // disable unreal's depreciation warnings
PRAGMA_DISABLE_OPTIMIZATION // disable code optimization (shortens compile time)
void FEngineSolBind::PlayerCameraManager(sol::state& LuaState) {
	// Sol code to bind class "APlayerCameraManager"
	auto NewType = LuaState.create_simple_usertype<APlayerCameraManager>();

	// base classes
	NewType.set(sol::base_classes, sol::bases<AActor, UObject>());

	// constructor lambdas
	//NewType.set("new", sol::overload([](void) {return NewObject<APlayerCameraManager>();}, [](FString Name) {return NewObject<APlayerCameraManager>((UObject*)GetTransientPackage(), FName(*Name));}));
	//NewType.set("newadv", sol::overload(&NewObject<APlayerCameraManager>(UObject*), &NewObject<APlayerCameraManager>(UObject*, FName, EObjectFlags, UObject*, bool, FObjectInstancingGraph*), &NewObject<APlayerCameraManager>(UObject*, UClass*, FName, EObjectFlags, UObject*, bool, FObjectInstancingGraph*)));

	// members
	NewType.set("TransformComponent", &APlayerCameraManager::TransformComponent);
	NewType.set("DefaultFOV", &APlayerCameraManager::DefaultFOV);
	NewType.set("DefaultOrthoWidth", &APlayerCameraManager::DefaultOrthoWidth);
	NewType.set("DefaultAspectRatio", &APlayerCameraManager::DefaultAspectRatio);
	NewType.set("DefaultModifiers", &APlayerCameraManager::DefaultModifiers);
	NewType.set("FreeCamDistance", &APlayerCameraManager::FreeCamDistance);
	NewType.set("FreeCamOffset", &APlayerCameraManager::FreeCamOffset);
	NewType.set("ViewTargetOffset", &APlayerCameraManager::ViewTargetOffset);
	NewType.set("bIsOrthographic", sol::property([](APlayerCameraManager& Obj) { return Obj.bIsOrthographic; }, [](APlayerCameraManager& Obj, uint8 Val) { Obj.bIsOrthographic = Val; }));
	NewType.set("bDefaultConstrainAspectRatio", sol::property([](APlayerCameraManager& Obj) { return Obj.bDefaultConstrainAspectRatio; }, [](APlayerCameraManager& Obj, uint8 Val) { Obj.bDefaultConstrainAspectRatio = Val; }));
	NewType.set("bUseClientSideCameraUpdates", sol::property([](APlayerCameraManager& Obj) { return Obj.bUseClientSideCameraUpdates; }));
	NewType.set("ViewPitchMin", &APlayerCameraManager::ViewPitchMin);
	NewType.set("ViewPitchMax", &APlayerCameraManager::ViewPitchMax);
	NewType.set("ViewYawMin", &APlayerCameraManager::ViewYawMin);
	NewType.set("ViewYawMax", &APlayerCameraManager::ViewYawMax);
	NewType.set("ViewRollMin", &APlayerCameraManager::ViewRollMin);
	NewType.set("ViewRollMax", &APlayerCameraManager::ViewRollMax);

	// methods
	NewType.set("StopAllCameraAnims", &APlayerCameraManager::StopAllCameraAnims);
	NewType.set("StopCameraAnimInst", &APlayerCameraManager::StopCameraAnimInst);
	NewType.set("StopAllInstancesOfCameraAnim", &APlayerCameraManager::StopAllInstancesOfCameraAnim);
	NewType.set("SetManualCameraFade", &APlayerCameraManager::SetManualCameraFade);
	NewType.set("StopCameraFade", &APlayerCameraManager::StopCameraFade);
	NewType.set("StartCameraFade", &APlayerCameraManager::StartCameraFade);
	NewType.set("StopAllCameraShakes", &APlayerCameraManager::StopAllCameraShakes);
	NewType.set("StopAllInstancesOfCameraShake", &APlayerCameraManager::StopAllInstancesOfCameraShake);
	NewType.set("StopCameraShake", &APlayerCameraManager::StopCameraShake);
	NewType.set("PlayCameraShake", &APlayerCameraManager::PlayCameraShake);
	NewType.set("ClearCameraLensEffects", &APlayerCameraManager::ClearCameraLensEffects);
	NewType.set("RemoveCameraLensEffect", &APlayerCameraManager::RemoveCameraLensEffect);
	NewType.set("AddCameraLensEffect", &APlayerCameraManager::AddCameraLensEffect);
	NewType.set("GetCameraLocation", &APlayerCameraManager::GetCameraLocation);
	NewType.set("GetCameraRotation", &APlayerCameraManager::GetCameraRotation);
	NewType.set("GetFOVAngle", &APlayerCameraManager::GetFOVAngle);
	NewType.set("RemoveCameraModifier", &APlayerCameraManager::RemoveCameraModifier);
	//NewType.set("AddNewCameraModifier", &APlayerCameraManager::AddNewCameraModifier);
	NewType.set("GetOwningPlayerController", &APlayerCameraManager::GetOwningPlayerController);
	NewType.set("OnPhotographyMultiPartCaptureEnd", &APlayerCameraManager::OnPhotographyMultiPartCaptureEnd);
	NewType.set("OnPhotographyMultiPartCaptureStart", &APlayerCameraManager::OnPhotographyMultiPartCaptureStart);
	NewType.set("OnPhotographySessionEnd", &APlayerCameraManager::OnPhotographySessionEnd);
	NewType.set("OnPhotographySessionStart", &APlayerCameraManager::OnPhotographySessionStart);
	NewType.set("PhotographyCameraModify", &APlayerCameraManager::PhotographyCameraModify);

	LuaState.set_usertype("PlayerCameraManager", NewType);
}
PRAGMA_ENABLE_OPTIMIZATION
PRAGMA_ENABLE_DEPRECATION_WARNINGS

[ThePhD]

I'm afraid we don't supply a lower-level binding as -- surprisingly -- we are only doing the barest minimum to actually bind things with simple usertype's set. The code will only get more efficient if we use C++17's Fold Expressions and Constexpr-If, but Visual Studio 2017 supports neither of these and likely won't until 2019 or 2020 (e.g., far too late for it to matter).

As a side note, I see a "#pragma once" in there. If this is in a header, you should move it to single compilation unit immediately, or to multiple compilation unit if you're not using the 64-bit Toolset to enable having greater than 4 GB of compiler space available and your bindings are truly that massive (assuming Visual Studio: if you're using more than Visual Studio, you'll be fine: gcc/clang will just attempt to consume all available RAM by default). This will mean that this only gets compiled once in a while, whenever the bindings are updated, rather than with every build.

Finally, the only thing you can do that's "lower level" is... well, setting up your own kind of usertype. Since you're automatically generating the bindings, you can use pieces and parts of sol2 (like tables, function binding support, sol::metatable_key_t, etc.) to glue all of your stuff into a metatable and figure out all the fun things I had to figure out while learning what it takes to support all the fun things C++ has (it looks like you're using mostly functions, so that'll be easy on you here, but other things might ba- I LIED, you're using properties too, you'll be in for a fun time!

Still, templates ultimately cause the compile-time churn because it's "generating" things at compile-time. If you want to hammer it down, then you can essentially write chunks of the wrapper yourself and stamp them out ahead-of-time. Unfortunately, "export" templates were dropped from C++ as a feature, so I can't even use that for this niche use case...

[Sixmorphugus]

Alright, thanks. It's included in a single compilation unit at the moment, the #pragma once is just there for legacy reasons. I guess I can look at the set source to see how you did different things...

[ThePhD]

As a side note: you know a LOT of what you're doing before-hand. The distinct thing about all the templates you're working with is that they're generating new function types for each distinct signature (of which there are a LOT of distinct ones).

You can cut down on this by "hand-crafting" your own metatable. You can get one started in sol2 by doing the usual

LuaState.new_usertype<ClassName>("ClassNameInLua");

Now, here's where things get tricky. You use sol::propertys. These require some management and therefore I suggest that you add them using the set function or by passing them straight to new_usertype. HOWEVER, the majority of what you're working with are functions. Since your generator is already doing a lot of the heavy lifting, you should be able to inject a multitude of classes using your own lua_CFunction wrapper. lua_CFunction is the raw backbone by which the Lua API allows you to call pretty much everything else. You can hard-generate what sol2's doing for you at compile-time with your generator. For example, for a class of ClassName and a method named XXXX, you can stamp out the code manually like this:

inline int ClassName_XXXX_f_g (lua_State* L) {
     // Get the class here
     ClassName* c = sol::stack::get<ClassName*>(c, 1);
     // add safety checks here
     // Use the generator to yank out arguments
     decltype(auto) result = (c->XXXX)( sol::stack::get<Arg1>(L, 2), sol::stack::get<Arg2>(L, 3), sol::stack::get<Arg3>(L, 4), ... );

     // push the result
     // this generally uses copy-semantics behavior: to get
     return stack::push(L, std::forward<decltype(result)>(result));
}

This is essentially how member functions are handled. There are special cases I'm not talking about here because there's a LOT of those but this is probably what you want. Then, you can add it to the usertype metatable at runtime using sol::table's simple syntax:

sol::table TypeTable = LuaState["ClassNameInLua"];
TypeTable["XXXX"] = &ClassName_XXXX_f_g;

You can get even more performance and save Lua serialization space by using sol::c_call, which lifts the named function up into a compile-time wrapper and thusly avoids serialization costs (but still generates some templates. Not as much, but still does):

sol::table TypeTable = LuaState["ClassNameInLua"];
TypeTable["XXXX"] = &sol::c_call<decltype(&ClassName_XXXX_f_g), &ClassName_XXXX_f_g>;

Remember lua_CFunctions are raw and we push them directly with 0 management on our end: that means sol2 is generating less compiler stuff and ultimately just shoves that right in. You forsake a great degree of checking, exception inspection/safety-netting, and compiler-safety that sol2 has built up over the years, but if your only goal is increasing compilation speed you can use sol2 at this level. None of what I've shown you here is undocumented black magic: the Stack API is there, sol::table is a common, documented type, and the fact that you can add at runtime to a sol usertype table is there. The use of sol::c_call lua C functions is also documented here.

The reason sol2 can't dump out this code for you to save in a serializable format is because we do this at compile-time, on-demand, when you hit F7/Ctrl+B (whatever your build button is). This means it keeps itself updated and doesn't require you to re-serialize crap.

But since you're running a generator before-hand, you can write your own generator on top of sol2: do feel free to use it as you wish! And let me know how this goes: if this becomes the case, I might make this issue part of the documentation if there's any success in reducing compiler overhead.

Also, @OrfeasZ @eliasdaler maybe you two would want to take a look at this too, since it might be of interest to you. It's about as close to "internals documentation" as you can get, really...!

Good luck.

[ThePhD]

Added a link to here from the docs, since it might as well be there in case somebody wants to try their hand at using sol2 on a more "manual" level: http://sol2.readthedocs.io/en/latest/compilation.html#compile-speed-improvemements

[Sixmorphugus]

Just out of curiousity, what makes sol::property so complicated?

[ThePhD]

sol::property is only accomplish-able when you work on something's metatable and hook what's known as its __index and __newindex functions. When you do this, the conventional method of binding methods no longer works properly because of complicated stuff regarding what Table/Userdata Lua hands you when it "fails a lookup" when searching for the name of a function given obj.thing or obj.thing = blah.

This creates a terribly "fun" problem when trying to keep both performance and have the damn thing work. You can't just dump a bunch of functions into Lua's metatable. You have to bind __index and __newindex, store all the functions (and variables) you saved somewhere, and then look up which one is supposed to be which.

That is, of course, just the beginning.

Did you want to support base classes? How about calling a variable like a function? How about using functions as variables (sol::property)? Then, users want to have read only variable. Write only variables (wat?). And they don't just want to bind member functions.

They want to bind function objects, variables, things that may or may not have annnnything to do with the class they're being bound too, but provide a value in a consistent and easy way that looks like it comes "from the instance". Oh, and don't forget people want to add stuff at runtime, because the system wasn't complicated enough already: which, by the way, how do you know that someone's adding a variable versus a function at runtime?

Finally, now that you've created a system that can handle all of that, correctly, and the bug reports stop...!

Is it fast ?

[Sixmorphugus]

I understand your pain. Sol2 is great. Thanks for the fast response!

Also, it supports write-only? Interesting.

[Sixmorphugus]

Oh, one more question. It's to do with the above code you provided as an example of a lua_CFunction that makes use of sol.

What would be the best thing to do if required arguments are either of the wrong type or missing? I.e. what would the safety checks you omitted look like? I know about luaL's checknumber, checkstring etc. I'm wondering if there are other things to do. For now, I'm just doing return 0; when this happens, but getting sol to throw the normal error would be interesting.

[ThePhD]

It does support write-only.

sol::stack::get defaults to using a checked version if you have SOL_CHECK_ARGUMENTS on. If it's not on, you need to manually check each other.

SOL_SAFE_USERTYPE goes through and for each "member-function"-alike it calls, it gets the Object Type T first as a pointer, checks if its null or if it's not of the right type, and throws a specific error if it is. You can simulate this by doing T* pself = sol::stack::get<T*>();, and then doing a if (pself == nullptr) return lua_error("error: self argument is nil or not the correct type. Please make sure to call your instance-functions with obj:func( ... ) or obj.func(obj, ... )");

There's also some other checks we do for when people do new_usertype and bind constructors and the like. If you need overloading, that's REALLY hard to do, and I would recommend just going with sol::overload and binding that particular function through sol2's high-level mechanisms, as duplicating that behavior is error-prone and hard to get right.

[Sixmorphugus]

Alright, I'll use the return sol::lua_error syntax then.

The generated functions are nice. Currently, they look like this (though there are some mistakes such as the &&s and not checking c):

inline int AActor_IsActorBeingDestroyed_luabind(lua_State* L) {
	AActor* c = sol::stack::get<AActor*>(L, 1);

	// Call the function
	bool Ret = c->IsActorBeingDestroyed();

	sol::stack::push_reference_multi(L, Ret);	return 1;
}

inline int AActor_MakeNoise_luabind(lua_State* L) {
	AActor* c = sol::stack::get<AActor*>(L, 1);

	// Safely get params
	sol::optional<float> Arg0 = sol::stack::check_get<float>(L, 2);
	sol::optional<APawn*> Arg1 = sol::stack::check_get<APawn*>(L, 3);
	sol::optional<FVector> Arg2 = sol::stack::check_get<FVector>(L, 4);
	sol::optional<float> Arg3 = sol::stack::check_get<float>(L, 5);
	sol::optional<FName> Arg4 = sol::stack::check_get<FName>(L, 6);

	if (!Arg0 && !Arg1 && !Arg2 && !Arg3 && !Arg4)
	{
		return 0;
	}

	// Call the function
	c->MakeNoise(Arg0.value(), Arg1.value(), Arg2.value(), Arg3.value(), Arg4.value());

	return 0;
}

[ThePhD]

That was pseudo code, but lua_error isn't part of sol2: it's a Lua C API function: https://www.lua.org/manual/5.3/manual.html#lua_error

[Sixmorphugus]

Oh. Alright.

[guijun]

compile speed is nightmare.
I have no tool yet for retrieve functions and vars automaticly..

my binding files has about 4000 lines and most of them are coded manually ....

[ThePhD]

@guijun Place all your bindings in either a single .cpp file or multiple .cpp files. Only update the files when you need to make a change to the binding. This will keep your compile speeds lower and only invoke the pain when you actually change the bindings.

I am working on new ways to bind things since compile times has become more or less a pain for most users. I just haven't been able to write one that has the same performance as new_usertype just yet...

[guijun]

Hi, Thanks.

I have to keep it onhold and continue when your new release the speed boosted build~