Two-arity callbacks receive raw `Lua.VM.State` (not `Lua.t`) when entered via `Lua.encode!/2` or nested in a `set!` value

[smn]

Again, thanks for the work on this package :)

We're testing this against:

• Lua 1.0.0-rc.3
• Elixir 1.20.0 / Erlang/OTP 28

I'm not sure if this is the intended behaviour but a two-arity Elixir callback (fn args, state -> {results, state} end) receives a different state argument — and must return a different state shape — depending on how it entered the VM:

How the callback enters the VM	state it receives
Lua.set!(lua, [:f], fun) — function directly at the path	Lua.t
deflua + Lua.load_api/3	Lua.t
Lua.set!(lua, [:t], %{"f" => fun}) — function inside a value	raw Lua.VM.State
Lua.encode!(lua, fun) — closure handed to Lua at runtime	raw Lua.VM.State

The Lua.set!/3 docs teach the Lua.t convention (treat state as a Lua.t, use the public API on it, return it as received). That holds for a function set directly at a path and for deflua. But a closure that reaches the VM as an encoded value — either via Lua.encode!/2, or nested inside a value passed to set! — is invoked with the raw Lua.VM.State instead.

The raw Lua.VM.State is unusable with the public API: Lua.decode!/2, Lua.encode!/2, Lua.get_private!/2, etc. all expect Lua.t and raise no function clause matching. And the two return conventions reject each other:

• a raw-state callback returning {results, %Lua{}} fails with native function returned invalid result ..., expected {results, state}
• a set!-at-path callback returning {results, %Lua.VM.State{}} fails with deflua functions must return encoded data

So the same closure cannot work in both positions. This bites any factory-style API — an Elixir function that builds a closure and hands it to Lua via Lua.encode!/2 (e.g. a translator bound to a locale, or a store handle bound to a config). Written against the documented set! convention it works at a path and crashes as an encoded value:

** (Lua.RuntimeException) Lua runtime error: no function clause matching in Lua.get_private!/2

We'd expect a two-arity callback to receive Lua.t (and accept {results, %Lua{}} back) regardless of how it entered the VM, matching the documented Lua.set!/3 convention. The workaround today is to hand-wrap the raw state as %Lua{state: raw} to reach the public API — which reaches into the struct's internals.

Here's a standalone test that pins the documented convention where it already holds (first block, passing) and asserts the same convention for the two encoded-value entry points (second block, failing as of 1.0.0-rc.3):

defmodule Lua.CallbackStateAsymmetryTest do
  @moduledoc """
  A two-arity Elixir callback (`fn args, state -> {results, state} end`)
  receives a different `state` argument — and must return a different state
  shape — depending on how it entered the VM:

  | How the callback enters the VM                                  | `state` it receives |
  | --------------------------------------------------------------- | ------------------- |
  | `Lua.set!(lua, [:f], fun)` — function directly at the path       | `t:Lua.t/0`         |
  | `deflua` + `Lua.load_api/3`                                      | `t:Lua.t/0`         |
  | `Lua.set!(lua, [:t], %{"f" => fun})` — function inside a value   | raw `Lua.VM.State`  |
  | `Lua.encode!(lua, fun)` — closure handed to Lua at runtime       | raw `Lua.VM.State`  |

  The raw `Lua.VM.State` is unusable with the public API — `Lua.decode!/2`,
  `Lua.encode!/2`, `Lua.get_private/2`, etc. all expect `t:Lua.t/0` and raise
  "no function clause matching" — and the two return conventions reject each
  other:

    * a raw-state callback returning `{results, %Lua{}}` fails with
      `"native function returned invalid result ..., expected {results, state}"`
    * a `set!`-at-path callback returning `{results, %Lua.VM.State{}}` fails
      with `"deflua functions must return encoded data"`

  So the same closure cannot work in both positions. The `Lua.set!/3` docs
  teach the `t:Lua.t/0` convention, but any factory-style API — an Elixir
  function that builds a closure and returns it to Lua via `Lua.encode!/2`
  (e.g. a translator bound to a locale, a store handle bound to a config) —
  gets the raw state and must hand-wrap it in `%Lua{state: raw}` to use the
  public API, reaching into the struct's internals.

  The first describe block pins the documented `t:Lua.t/0` convention where it
  already holds (passing). The second asserts the same convention for the
  other two entry points and fails as of 1.0.0-rc.3.
  """
  use ExUnit.Case, async: true

  defmodule DefluaProbe do
    @moduledoc false
    use Lua.API, scope: "probe"

    deflua whoami(), state do
      {[inspect(state.__struct__)], state}
    end
  end

  # Reports the struct name of the state the VM handed the callback.
  defp probe_callback do
    fn _args, state -> {[inspect(state.__struct__)], state} end
  end

  # A callback written exactly the way the `Lua.set!/3` docs teach: treat
  # `state` as a `t:Lua.t/0`, use the public API on it, return it as received.
  defp documented_convention_callback do
    fn _args, state -> {[Lua.get_private!(state, :secret)], state} end
  end

  describe "callbacks that receive the public Lua.t (documented convention)" do
    test "a function set! directly at a path receives Lua.t" do
      lua = Lua.set!(Lua.new(), [:probe], probe_callback())

      assert {["Lua"], _} = Lua.eval!(lua, "return probe()")
    end

    test "a deflua function loaded via load_api receives Lua.t" do
      lua = Lua.load_api(Lua.new(), DefluaProbe)

      assert {["Lua"], _} = Lua.eval!(lua, "return probe.whoami()")
    end

    test "a set!-at-path callback can use the public Lua API on its state" do
      lua =
        Lua.new()
        |> Lua.put_private(:secret, "from-private")
        |> Lua.set!([:fetch], documented_convention_callback())

      assert {["from-private"], _} = Lua.eval!(lua, "return fetch()")
    end
  end

  describe "callbacks that enter the VM as encoded values (asymmetric)" do
    # FAILS: returns ["Lua.VM.State"]. Lua.encode!/2 stores the closure bare,
    # so the VM invokes it with its internal state rather than wrapping it the
    # way Lua.set!/3 does for a function set directly at a path.
    test "a closure embedded via encode! receives the same Lua.t" do
      {fun, lua} = Lua.encode!(Lua.new(), probe_callback())
      lua = Lua.set!(lua, [:probe], fun)

      assert {["Lua"], _} = Lua.eval!(lua, "return probe()")
    end

    # FAILS: returns ["Lua.VM.State"]. set! only wraps a function that is
    # itself the value at the path — a function reachable *inside* the value
    # (here, a table field) is encoded bare, like Lua.encode!/2 does.
    test "a function nested inside a table passed to set! receives the same Lua.t" do
      lua = Lua.set!(Lua.new(), [:api], %{"probe" => probe_callback()})

      assert {["Lua"], _} = Lua.eval!(lua, "return api.probe()")
    end

    # FAILS: the practical consequence. The identical callback — written
    # against the documented set! convention — works when registered at a path
    # and crashes when handed to Lua as an encoded value:
    #
    #   ** (Lua.RuntimeException) Lua runtime error:
    #      no function clause matching in Lua.get_private!/2
    #
    # And side-stepping the public API doesn't rescue it: returning the %Lua{}
    # shape from the encoded position trips "native function returned invalid
    # result", while returning the raw shape from the set! position trips
    # "deflua functions must return encoded data". No single closure satisfies
    # both entry points.
    test "the same callback works identically via set! and via encode!" do
      lua = Lua.put_private(Lua.new(), :secret, "from-private")

      lua = Lua.set!(lua, [:registered], documented_convention_callback())
      {encoded, lua} = Lua.encode!(lua, documented_convention_callback())
      lua = Lua.set!(lua, [:embedded], encoded)

      assert {["from-private", "from-private"], _} =
               Lua.eval!(lua, "return registered(), embedded()")
    end
  end
end

[davydog187]

This is not intended behavior. Working on a fix