__len metamethod result is nil in Lua despite returning a number

[CodeSmile-0000011110110111]

Trying to return a plain number from the __len metamethod results in the value being nil:

var enumTable = new LuaTable();
var lenFunc = new LuaFunction("enum.__len", (context, token) =>
{
    Console.WriteLine("enum.__len called ...");
    return new ValueTask<Int32>(context.Return(3));

});
enumTable.Metatable = new LuaTable();
enumTable.Metatable[Metamethods.Len] = lenFunc;
state.Environment["testEnum"] = enumTable;

Test script:

print(tostring(testEnum))
print("len = " .. tostring(#testEnum))

Output:

table: 27252167
enum.__len called ...
len = nil

If I do the setup entirely in Lua, it returns the expected result:

local t = {}
local mt = { __len = function(t) return 123 end }
setmetatable(t, mt)
print(tostring(#t))

Output: 123

Using 0.5 dev branch, pulled today. Confirmed by typing this in the solution's ConsoleApp1 sandbox project.

In case it helps, this is the entire output:

Source Code --------------------------------------------------
print(tostring(testEnum))
print("len = " .. tostring(#testEnum))


local t = {}
local mt = { __len = function(t) return 123 end }
setmetatable(t, mt)
print(tostring(#t))

Chunk[0]==================================================
Parameters:0
Code --------------------------------------------------
[0]     1               GETTABUP 0 0 -1
[1]     1               GETTABUP 1 0 -2
[2]     1               GETTABUP 2 0 -3
[3]     1               CALL 1 2 0
[4]     1               CALL 0 0 1
[5]     2               GETTABUP 0 0 -1
[6]     2               LOADK 1 3
[7]     2               GETTABUP 2 0 -2
[8]     2               GETTABUP 3 0 -3
[9]     2               LEN 3 3
[10]    2               CALL 2 2 2
[11]    2               CONCAT 1 1 2
[12]    2               CALL 0 2 1
[13]    5               NEWTABLE 0 0 0
[14]    6               NEWTABLE 1 0 1
[15]    6               CLOSURE 2 0
[16]    6               SETTABLE 1 -5 2
[17]    7               GETTABUP 2 0 -6
[18]    7               MOVE 3 0
[19]    7               MOVE 4 1
[20]    7               CALL 2 3 1
[21]    8               GETTABUP 2 0 -1
[22]    8               GETTABUP 3 0 -2
[23]    8               LEN 4 0
[24]    8               CALL 3 2 0
[25]    8               CALL 2 0 1
[26]    8               RETURN 0 1
LocalVariables --------------------------------------------------
[0]     t       14      27
[1]     mt      17      27
Constants --------------------------------------------------
[0]     print
[1]     tostring
[2]     testEnum
[3]     len\ =\
[4]     __len
[5]     setmetatable
UpValues --------------------------------------------------
[0]     _ENV    1       0

Chunk[0]==================================================
Parameters:1
Code --------------------------------------------------
[0]     6               LOADK 1 0
[1]     6               RETURN 1 2
[2]     6               RETURN 0 1
LocalVariables --------------------------------------------------
[0]     t       0       3
Constants --------------------------------------------------
[0]     123
UpValues --------------------------------------------------

Output --------------------------------------------------
table: 27252167
enum.__len called ...
len = nil
123
Result --------------------------------------------------
End --------------------------------------------------

Process finished with exit code 0.

[CodeSmile-0000011110110111]

Curious, I added an __index func too:

var indexFunc = new LuaFunction("enum.__index", (context, token) =>
{
    Console.WriteLine("enum.__index called ...");
    return new ValueTask<Int32>(context.Return(1234567890));

});
enumTable.Metatable[Metamethods.Index] = indexFunc;

Usage:
print("index = " .. tostring(testEnum.DoesntExist))

Output:

enum.__index called ...
index = 1234567890

So this works, meaning the issue is with the __len metamethod respectively the # operator.

[CodeSmile-0000011110110111]

LuaVirtualMachine:2033 (in ExecuteUnaryOperationMetaMethod)

var task = func.Func(new() { Access = access, ArgumentCount = argCount, 
    ReturnFrameBase = newFrame.ReturnBase }, context.CancellationToken);

// omitted ..

var RA = context.Instruction.A + context.FrameBase;
var results = stack.GetBuffer()[newFrame.Base..];
stack.Get(RA) = results.Length == 0 ? default : results[0];

The last line causes the value on the stack to change from "3" to "nil" because results contains just nil values. The newFrame.Base is +1 higher than the return value index on the stack.

[CodeSmile-0000011110110111]

I just pulled to see if it is fixed. Actually, I made a proper fork this time, pulled that, added the few changes I had made (unrelated to this), and verified that __len C# metamethod works in ConsoleApp1.

Strangely after building to DLL and importing in Unity, I still get the return value nil'ed. I also restarted Unity and verified the DLL timestamps in the project. It must be the correct build since I can step through the LuaVirtualMachine code from within Unity, which matches the current v0.5 dev branch.

For context, I stepped through here (LuaVirtualMachine, lines 2128+):

            var result = stack.Count > newFrame.ReturnBase
                ? stack.Get(newFrame.ReturnBase)
                : default;
            stack.Get(context.Instruction.A + context.FrameBase) = result;
            stack.PopUntil(newBase - variableArgumentCount);
            context.Thread.PopCallStackFrame();
            return true;

The stack has 6 elements, here's the values I see (in parentheses the values I see in ConsoleApp1):
stack.Count = 6 (5)
newFrame.ReturnBase = 5 (4)
result = 3 (3)
context.Instruction.A = 4 (3)
context.FrameBase = 1 (0)
newBase = 5 (4)
variableArgumentsCount = 0 (0)

After PopUntil, the number "3" on stack index 5 changes to nil.

Stack array after PopUntil, highlighted line is the one that got cleared:

For reference:
[3] = "print" function
[4] = "tostring" function

My len metamethod is this and I verified that it returns the number 3:

private static readonly LuaFunction __lenList = new(Metamethods.Len, (context, _) =>
{
	var self = context.GetArgument<LuaList<T>>(0);
	var value = self.ManagedList?.Count ?? 0;
	return new ValueTask<Int32>(context.Return(value));
});

[CodeSmile-0000011110110111]

I was able to find a reproducable test case that works in ConsoleApp1!

The important difference is assigning to a local variable. In that case __len returns nil.

test.lua

print(#existinglist)    -- "3"
local existing = existinglist
print(#existing)        -- "nil"

print(#makelist())      -- "3"
local list = makelist()
print(#list)            -- "nil"

values set in C#:

state.Environment["existinglist"] = new LuaValue(new LuaList<string>());
state.Environment["makelist"] = new LuaFunction("makelist", (context, _) =>
{
    var list = new LuaValue(new LuaList<string>());
    return new ValueTask<int>(context.Return(list));
});

Simplified version of LuaList:

public sealed class LuaList<T> : ILuaUserData
{
    static readonly LuaFunction __lenList = new(Metamethods.Len, (context, _) =>
    {
        return new ValueTask<int>(context.Return(3));
    });

    static LuaTable s_Metatable;
    public LuaTable Metatable { get => s_Metatable; set => throw new NotImplementedException(); }

    public LuaList()
    {
        s_Metatable = new LuaTable();
        s_Metatable[Metamethods.Len] = __lenList;
    }
}

Chunk dump:

Chunk[0]==================================================
Parameters:0
Code --------------------------------------------------
[0]     1               GETTABUP 0 0 -1
[1]     1               GETTABUP 1 0 -2
[2]     1               LEN 1 1
[3]     1               CALL 0 2 1
[4]     2               GETTABUP 0 0 -2
[5]     3               GETTABUP 1 0 -1
[6]     3               LEN 2 0
[7]     3               CALL 1 2 1
[8]     5               GETTABUP 1 0 -1
[9]     5               GETTABUP 2 0 -3
[10]    5               CALL 2 1 2
[11]    5               LEN 2 2
[12]    5               CALL 1 2 1
[13]    6               GETTABUP 1 0 -3
[14]    6               CALL 1 1 2
[15]    7               GETTABUP 2 0 -1
[16]    7               LEN 3 1
[17]    7               CALL 2 2 1
[18]    7               RETURN 0 1
LocalVariables --------------------------------------------------
[0]     existing        5       19
[1]     list    15      19
Constants --------------------------------------------------
[0]     print
[1]     existinglist
[2]     makelist
UpValues --------------------------------------------------
[0]     _ENV    1       0

Output --------------------------------------------------
3
nil
3
nil
Result --------------------------------------------------
End --------------------------------------------------

I checked with luac.nl (Lua 5.2.4) - bytecode dumps match.

[CodeSmile-0000011110110111]

Note: I can't reproduce it with a table in any way.