Finalizing our API...

[UltraEngine]

I want to make sure my design is optimal before I finalize our game engine design here:
https://github.com/UltraEngine/Documentation/blob/master/CPP/Scripting.md#exposing-c-classes-to-lua

Please let me know if any of the design factors below can be improved.

Shared pointers and nil

This is what I do to handle functions that accept a shared pointer that may be NULL:

L->set_function("CreateBox",
	[](World* w) { if (w == NULL) return CreateBox(NULL); else return CreateBox(w->As<World>()); }
);

Casting Shared Pointers

For type casting of shared pointers, I am declaring my own function for each class like this:

L->set_function("Monster", [](Object* m) { if (m == NULL) return NULL; else return m->As<Monster>(); } );

Object::As is a method in the base object class that uses shared_from_this to cast shared pointers:

shared_ptr<T> As() { return std::dynamic_pointer_cast<T>(shared_from_this()); }

String Classes

In our engine we use two string classes for strings:

class String : public std::string
{};

class WString : public std::wstring
{};

Is there any way to make Lua automatically assign strings to this class, or should I use lua versions of each function that receives and returns a string?:

void Print(const String& s);

L->set_function("Print", [](std::string s) { Print(String(s)); });

Please let me know if you have any suggestions. Thank you for this great library, I am excited to be able to finally bring this to market using sol!

[OrfeasZ]

For shared pointers I think sol will throw an error if you try to accept a shared_ptr<T> and pass in nil from Lua. You could use a pointer like you do here, or alternatively you could provide an explicit overload for the nil case:

L->set_function("CreateBox",
	sol::overload(
		[](std::shared_ptr<World> w) { return CreateBox(w); },
		[](std::nullptr_t) { return CreateBox(nullptr); }
	)		
);

For your string classes, you could define custom sol pushers and getters so it can be automatically converted for you:

String sol_lua_get(sol::types<String>, lua_State* L, int index, sol::stack::record& tracking)
{
	int absindex = lua_absindex(L, index);
	auto str = sol::stack::get<std::string>(L, absindex);
	tracking.use(1);
	return String(str);
}

int sol_lua_push(sol::types<String>, lua_State* L, String& str)
{
	return sol::stack::push(L, str.c_str());
}

[OrfeasZ]

Oops, you'll also need a sol_lua_check for the String:

template <typename Handler>
bool sol_lua_check(sol::types<String>, lua_State* L, int index, Handler&& handler, sol::stack::record& tracking)
{
	int absindex = lua_absindex(L, index);
	auto success = sol::stack::check<std::string>(L, index, handler);
	tracking.use(1);
	return success;
}

[UltraEngine]

If I try to expose the string classes to sol it won't compile with the public inheritance of std::string and std::wstring, but it does compile if I make those private, so there is probably some room for improvement there. I will also look into pushers/getters.

I like your solution for null shared pointers, it seems a little bit simpler.

[UltraEngine]

I was actually able to get wide strings working. They can concentate with Lua strings, display in the debugger, and my own or Lua's print function:

#include "UltraEngine.h"

using namespace UltraEngine;

class MyWString;

class MyString : private String
{
public:
    MyString(std::string s) { assign(s); };
    friend int main(int argc, const char* argv[]);
    MyString operator+(const MyString& s) { return (std::string(*this)) + std::string(s); };
    MyString operator+(const std::string& s) { return (std::string(*this)) + String(s); };
    friend MyWString;
};

class MyWString : private WString
{
public:
    MyWString(std::string s) { assign(WString(s)); };
    MyWString(std::wstring s) { assign(s); };
    friend int main(int argc, const char* argv[]);
    MyWString operator+(const MyWString& s) { return (std::wstring(*this)) + std::wstring(s); };
    MyWString operator+(const std::string& s) { return (std::wstring(*this)) + WString(s); };
};

MyString GetString()
{
    return MyString(std::string("How are you? "));
}

MyWString GetWString()
{
    return MyWString(std::wstring(L"Сколько вам лет? "));
}

int main(int argc, const char* argv[])
{
    //Get command-line options
    auto cl = ParseCommandLine(argc, argv);

    //Enable script debugging if the -debug switch is specified
    if (cl["debug"].is_boolean() and cl["debug"] == true)
    {
        RunScript("Scripts/Modules/Debugger.lua");
    }

    //Create a timer
    auto timer = CreateTimer(490);

    //Poll the debugger every timer tick
    ListenEvent(EVENT_TIMERTICK, timer, std::bind(&PollDebugger, 500));

    auto L = Core::LuaState();
    L->set_function("MyPrint", sol::overload(
        [](MyString& s) { Print(s); },
        [](MyWString& s) { Print(s); }
    ));

    L->set_function("GetString", GetString);
    L->set_function("GetWString", GetWString);

    L->new_usertype<MyString>(
        "STRINGCLASS",
        sol::meta_function::concatenation, sol::overload(
            [](MyString& s1, MyString& s2) { return s1 + s1; },
            [](MyString& s1, MyWString& s2) { return MyWString(s1) + s2; },
            [](MyString& s1, std::string& s2) { return s1 + MyString(s2); },
            [](std::string& s1, MyString& s2) { return MyString(s1) + s2; }
        ),
        sol::meta_function::equal_to, sol::overload(
            [](MyString& s1, MyString s2) {return s1 == s2; },
            [](MyString& s1, std::string s2) {return s1 == s2; },
            [](MyString& s1, MyWString s2) {return MyWString(s1) == s2; }
        ),
        sol::meta_function::to_string, [](MyString& s) { return std::string(s); }
    );

    L->new_usertype<MyWString>(
        "WSTRINGCLASS",
        sol::meta_function::concatenation, sol::overload(
            [](MyWString& s1, MyWString& s2) { return s1 + s2; },
            [](MyWString& s1, MyString& s2) { return s1 + MyWString(s2); },
            [](MyWString& s1, std::string& s2) { return s1 + MyWString(s2); },
            [](std::string& s1, MyWString& s2) { return MyWString(s1) + s2; }
        ),
        sol::meta_function::equal_to, sol::overload(
            [](MyWString& s1, MyWString s2) {return s1 == s2; },
            [](MyWString& s1, std::string s2) {return s1 == s2; },
            [](MyWString& s1, MyString s2) {return s1 == s2; }
        ),
        sol::meta_function::to_string, [](MyWString& s) { return std::string(s.ToUtf8String()); }
    );
    
    //Run the main script
    RunScript("Scripts/Main.lua");

    return 0;
}

Main.lua:

local a = GetWString();
local b = GetString();

local r = a == b

local c = a..b
local d = "Test: "
local e = d..c
MyPrint(e)
print(e)
print("ok")

[UltraEngine]

Okay, for strings what works best is a wide string wrapper class:

struct WStringWrapper
{
    WString s;
    WStringWrapper(const std::string& s);
    WStringWrapper(const std::wstring& s);
};

L->new_usertype<WStringWrapper>(
    "WStringWrapperClass",
    sol::meta_function::concatenation, sol::overload(
        [](WStringWrapper& s1, WStringWrapper& s2) { return WStringWrapper(s1.s + s2.s); },
        [](WStringWrapper& s1, std::string s2) { return WStringWrapper(s1.s + s2); },
        [](std::string s1, WStringWrapper& s2) { return WStringWrapper(s1 + s2.s); }
    ),
    sol::meta_function::equal_to, sol::overload(
        [](WStringWrapper& s1, WStringWrapper s2) { return s1.s == s2.s; },
        [](WStringWrapper& s1, std::string s2) { return s1.s == s2; },
        [](std::string s1, WStringWrapper s2) { return s1 == s2.s; }
    ),
    sol::meta_function::less_than, sol::overload(
        [](WStringWrapper& s1, WStringWrapper s2) {return s1.s < s2.s; },
        [](WStringWrapper& s1, std::string s2) {return s1.s < WString(s2); },
        [](std::string s1, WStringWrapper s2) {return WString(s1) < s2.s; }
    ),
    sol::meta_function::to_string, [](WStringWrapper& s) { return std::string(s.s.ToUtf8String()); }
);

A typical function definition now looks like this:

L->set_function("ExtractExt", sol::overload(
	[](std::string s) { return ExtractExt(s); },// Returns a narrow string
	[](Core::WStringWrapper& s) { return Core::WStringWrapper(ExtractExt(s.s)); }// Returns a wide string
));

I am not going to try to make strings in Lua a class with methods like in C++ and C# because the dynamic typing of Lua would make it difficult to tell if a variable is a Lua string or a wide string.

My last wish is to be able to overload properties so I can set a string value with either raw Lua strings or my wide string class, but this seems to not work:

"name", sol::property(
	[](Entity& entity) { return Core::WStringWrapper(entity.name); },
	sol::overload(
		[](Entity& entity, const std::string s) { entity.name = s; },
		[](Entity& entity, const Core::WStringWrapper& s) { entity.name = s.s; }
	)
)

[UltraEngine]

I tried using sol_lua_get and sol_lua_check from the code above, but when I add these it breaks the concatenation overloads in the class definition. I think it's probably not possible to make it work all at once.

Maybe the string wrapper can be improved in the future, but it's good enough for now.

[UltraEngine]

It doesn't appear my equal_to function is ever being called when comparing a WStringWrapper with a Lua string:

            sol::meta_function::equal_to, sol::overload(
                [](WStringWrapper& s1, WStringWrapper& s2)
                {
                    return s1.s == s2.s;
                },
                [](WStringWrapper& s1, std::string s2) {
                    return s1.s == WString(s2);
                },
                [](std::string s1, WStringWrapper& s2) {
                    return WString(s1) == s2.s;
                }
            ),

Here is how it is being used. Event.text is a WStringWrapper, so ExtractExt returns another WStringWrapper, and then on the next line it is being compared to a Lua string:

        local ext = ExtractExt(event.text)
        if ext == "wav" then

However, none of the equal_to functions above get triggered. I'm using Lua 5.4.4.

[UltraEngine]

Maybe this is the cause...?
#619 (comment)

[UltraEngine]

Ugh, this is awful. I'm going to roll all this string stuff back. Lua should just use wide strings and drop all this nonsense.

[UltraEngine]

I think this is what you actually want to do.

When sending strings to Lua always convert to UTF-8:

L->set_function("CurrentDir", []() { return std::string(CurrentDir().ToUtf8String()); });

When getting strings back from Lua, convert from UTF8 to wide strings and proceed:

L->set_function("Notify", [](std::string s) { Notify(WString(s)); } );

[OrfeasZ]

The sol_lua_x functions will not be useful if your type is also defined as a usertype. I would suggest not defining them as usertypes and instead defining sol_lua_x functions for them (like above) that can normalize your strings into UTF-8 for lua, and back into whatever encoding you need in C++.

[UltraEngine]

Okay, what I have now is perfect. Thank you everyone for your input.