Overloaded function calling wrong version, in release mode only

[JoshKlint]

I have a problem with this overload:

L->set_function("CreateWidget",
	sol::overload(
		sol::resolve<shared_ptr<Widget>(shared_ptr<GUI>, const std::string&, const int, const int, const int, const int)>(&CreateWidget),
		sol::resolve<shared_ptr<Widget>(shared_ptr<Widget>, const std::string&, const int, const int, const int, const int)>(&CreateWidget)
	)
);

In debug mode it works as expected. In release mode, the first function is always called, even though the parameter calls for the second overload. This was quite difficult to track down. What is going on here?

Both the GUI and Widget class are derived from a base Object class.

[ThePhD]

Sounds like the safeties are getting turned off in Release mode and you're getting any-old-matching, because strict checking is disabled.

[JoshKlint]

How do I change that? The function this is being called from within is being executed using lua_pcall().

[JoshKlint]

It looks like this code has an example of the sol way to replace lua_pcall, and from what I gather, the sol calling methods add the safety checks in(?):
#865

Do I need to not use lua_pcall? Is there another replacement for lua_pcall?

[JoshKlint]

I replaced the call to lua_pcall with this code but the problem persists!

bool Entity::CallFunction(const std::string& name)
{
	if (VirtualMachine::L == nullptr) VirtualMachine::Reset();

	auto L = VirtualMachine::L;
	bool success = false;
	auto self = dynamic_pointer_cast<Entity>(Self());
		 
	int stacksize = lua_gettop(L);
	sol::stack::push(L, self);
	sol::stack::get_field(L, name);
	if (lua_isfunction(L, -1))
	{
		auto f = sol::protected_function(L, -1);
		if (f.valid())
		{
			auto result = f(self);
			if (result.status() == sol::call_status::ok) success = true;
		}
	}

	lua_settop(L, stacksize);
	return success;
}

[JoshKlint]

I defined SOL_ALL_SAFETIES_ON = 1 and an error is occurring when the above code is run, but I don't know how to catch the error and print it out.

auto f = sol::protected_function(L);	
if (f.valid())
{
	sol::protected_function_result result;
	try
	{
		result = f(self);
	}
	catch (const sol::error & e)
	{
		Print("Lua error: " + std::string(e.what()));
	}
	if (result.status() == sol::call_status::ok)
	{					
		success = true;
	}
	else
	{
		Print("Some error occurred, I don't know what!");
	}
}

[JoshKlint]

Okay, I have error handling working now. It says that no overload of the function was found with the given arguments, but I know it is there.

auto L = VirtualMachine::L;
bool success = false;
auto self = dynamic_pointer_cast<Entity>(Self());
		
sol::function errorfunc = {};
lua_getglobal(L, "LuaErrorHandler");
if (lua_isfunction(L, -1)) errorfunc = sol::function(L);

int stacksize = lua_gettop(L);
sol::stack::push(L, self);
sol::stack::get_field(L, name);
if (lua_isfunction(L, -1))
{
	auto f = sol::protected_function(L);	
	if (f.valid())
	{
		if (errorfunc.valid()) f.set_default_handler(errorfunc);
		sol::protected_function_result result;
		try
		{
			result = f(self);
		}
		catch (const sol::error & e)
		{
			Print("Lua error: " + std::string(e.what()));
		}
		if (result.status() == sol::call_status::ok)
		{					
			success = true;
		}
		else
		{
			Print("Some error occurred, I don't know what lol!");
		}
	}
}

[JoshKlint]

I found one way to write the function that works correctly in release mode:

L->set_function("CreateWidget",
	sol::overload
	(
		[](Widget* wid, std::string s, int x, int y, int w, int h) { return CreateWidget(dynamic_pointer_cast<Widget>(wid->Self()), s, x, y, w, h); },
		//sol::resolve<shared_ptr<Widget>(shared_ptr<Widget>, const std::string&, const int, const int, const int, const int)>(&CreateWidget),
		sol::resolve<shared_ptr<Widget>(shared_ptr<GUI>, const std::string&, const int, const int, const int, const int)>(&CreateWidget)
	)
);

I am using strict checks for everything except floating points / integers:

#define SOL_NO_CHECK_NUMBER_PRECISION 1
#define SOL_ALL_SAFETIES_ON 1
#include <sol/sol.hpp>

It is pretty dangerous to have these types of errors occurring. Any ideas how this can be detected better?

[ThePhD]

After deep investigation, this seems like a SERIOUS bug in Visual C++. In release builds, it is selecting the wrong template function and this is extremely concerning to me that it is doing this despite there not being an ODR violation that I can find.

I will need to spend more time looking into this and possibly producing a repro for the VC++ team, or finding out if there is some deeper bug in my code. I can't guarantee I will get back to you quickly, as I have spent a lot of my time just to confirm Visual C++ might be behaving poorly...

[stonedreamforest]

@ThePhD
I have encountered this problem before ,try disable comdat folding, this is a vc++ bug, Because it can't be reproduced in other compilers.

[ThePhD]

Well, the docs there make it easy to see why RelWithDebInfo builds don't show the problem and only strictly release builds...

Goddamn I spent days tracking this. Thanks for the tip @stonedreamforest

[ThePhD]

Specifying /OPT:NOREF,NOICF,NOLBR to the linker makes this program work properly.

This is a Visual C++ bug. Nothing I can do to help you here, @JoshKlint ; either write the function different or turn it off.

#include <iostream>

#define SOL_ALL_SAFETIES_ON 1
#include <sol/sol.hpp>

struct Object {
	int value;
};

struct Widget : Object {
	double wid_value;

	Widget(int v, double wv) {
		this->value = v;
		this->wid_value = wv;
	}
};

struct Wadget : Object {
	bool wad_value;

	Wadget(int v, bool wv) {
		this->value = v;
		this->wad_value = wv;
	}
};

std::shared_ptr<Object> f (std::shared_ptr<Widget>, int a) {
	std::cout << "Widget" << std::endl;
	return std::make_shared<Widget>(a, 4.0);
}

std::shared_ptr<Object> g (std::shared_ptr<Wadget>, int a) {
	std::cout << "Wadget" << std::endl;
	return std::make_shared<Wadget>(a, false);
}

int main() {
	sol::state lua;
	lua.open_libraries(sol::lib::base);

	//lua.new_usertype<Widget>("Widget", sol::base_classes, sol::bases<Object>());
	//lua.new_usertype<Wadget>("Wadget", sol::base_classes, sol::bases<Object>());

	std::shared_ptr<Widget> wid = std::make_shared<Widget>(0, 2.0);
	std::shared_ptr<Wadget> wad = std::make_shared<Wadget>(1, true);

	lua["create"] = sol::overload(&f, &g);

	sol::protected_function create = lua["create"];
	std::cout << "create(wid, 2)" << std::endl;
	create(wid, 2);
	std::cout << "create(wad, 4)" << std::endl;
	create(wad, 4);

	return 0;
}

[JoshKlint]

Wow, that is interesting. I also lost a couple days on this trying to figure out what was going on. My program works now with the original commands, with the fix above. Thank you!

[romanholidaypancakes]

Specifying /OPT:NOREF,NOICF,NOLBR to the linker makes this program work properly.

This is a Visual C++ bug. Nothing I can do to help you here, @JoshKlint ; either write the function different or turn it off.

#include <iostream>

#define SOL_ALL_SAFETIES_ON 1
#include <sol/sol.hpp>

struct Object {
	int value;
};

struct Widget : Object {
	double wid_value;

	Widget(int v, double wv) {
		this->value = v;
		this->wid_value = wv;
	}
};

struct Wadget : Object {
	bool wad_value;

	Wadget(int v, bool wv) {
		this->value = v;
		this->wad_value = wv;
	}
};

std::shared_ptr<Object> f (std::shared_ptr<Widget>, int a) {
	std::cout << "Widget" << std::endl;
	return std::make_shared<Widget>(a, 4.0);
}

std::shared_ptr<Object> g (std::shared_ptr<Wadget>, int a) {
	std::cout << "Wadget" << std::endl;
	return std::make_shared<Wadget>(a, false);
}

int main() {
	sol::state lua;
	lua.open_libraries(sol::lib::base);

	//lua.new_usertype<Widget>("Widget", sol::base_classes, sol::bases<Object>());
	//lua.new_usertype<Wadget>("Wadget", sol::base_classes, sol::bases<Object>());

	std::shared_ptr<Widget> wid = std::make_shared<Widget>(0, 2.0);
	std::shared_ptr<Wadget> wad = std::make_shared<Wadget>(1, true);

	lua["create"] = sol::overload(&f, &g);

	sol::protected_function create = lua["create"];
	std::cout << "create(wid, 2)" << std::endl;
	create(wid, 2);
	std::cout << "create(wad, 4)" << std::endl;
	create(wad, 4);

	return 0;
}

hi @ThePhD
Is this still a problem in vs2022? I don't want to close it because closing makes the file bigger.