Returning nullptr from function doesn't give nil

[okonk]

Hi, I'm not sure what the expected behaviour is, but in an older version of sol2 this was giving nil as I expect, so maybe there's a bug here?

I've simplified it to the following test case:

#include <sol.hpp>
#include <memory>

class Test {

};

std::shared_ptr<Test> ReturnsNull() {
    return nullptr;
}

int main() {
    sol::state lua;
    lua.open_libraries(sol::lib::base);
    lua.new_usertype<Test>("Test");
    lua.set_function("returns_null", ReturnsNull);
    lua.script("print(returns_null() == nil)");
}

Lua is printing false when I expect it should be true.

[ThePhD]

This behavior was recently changed. When someone pushes a std::shared_ptr<T> that is nullptr, and then tries to get a std::shared_ptr out in a C++ function, it would crash and/or fail because we pushed nil and not a userdata that was nullptr.

This scenario, of course, now creates the problem you're seeing here.

I might be able to massage this to behave better by using a metatables == operator, but I need to test if it even kicks in for this case.

[ThePhD]

An __eq metamethod cannot be used for this case:

Behavior similar to the addition operation, except that Lua will try a metamethod only when the values being compared are either both tables or both full userdata and they are not primitively equal. The result of the call is always converted to a boolean.

So we're a little screwed at this point. The best thing I can suggest is writing a function is_nil. That, or we would have to let returning a nullptr result in pushing nil into Lua, but ultimately disallow void func (std::shared_ptr<T>&/std::unique_ptr<T>&/my_special_ptr<T>&) from being a thing.

[ThePhD]

That, or we demand people to do

void func( sol::optional<std::shared_ptr<T>&> maybe_ptr ) {
     ....
}

That way, if someone passes nil to that function, it won't crash? There's a lot of design choices here and I'm honestly not sure which is the right one.

[ThePhD]

@okonk Pull from the latest. It exhibits the behavior you expect and want.

Have fun!

[okonk]

Nice, thanks. :)