Allow use within busted unit tests

[dainkaplan]

Not sure on the technical hurdles here, but being able to run a debugger within your unit tests would be awesome.

Busted allows calling the runner from within a regular lua file, so the launcher should be possible:

require 'busted.runner'()

describe("a test", function()
  -- tests to here
end)

(see documentation here: https://olivinelabs.com/busted/#usage)

And running it:

lua test.lua

However, when trying to combine the debugger and the above, the debugger gets halted before the first test is actually run. Could be due a limitation of lua51, not sure.

[tomblind]

I've attempted to reproduce this, but it seems to work fine for me. 😄

What happens when the debugger 'halts'? Is there an error? Can you set "verbose": true in the launch.json and post the log from the debug console?

[dainkaplan]

I’ll check the latest and see what whats what! Will report back. Thanks for looking into it, @tomblind

[astrochili]

@tomblind I can provide the case. My lunch.json:

{
    "configurations": [     
        {
            "name": "Test",
            "type": "lua-local",
            "request": "launch",
            "program": {
                "command": "busted"
            },
            "args": [
                "${workspaceFolder}/test/start.lua"
            ],
            "verbose": true
        }
    ]
}

My test/start.lua:

if os.getenv("LOCAL_LUA_DEBUGGER_VSCODE") == "1" then
    require("lldebugger").start()
end

describe("a test", function()
  -- tests to here
end)

Busted falls on describe() or it() call. This is verbose output:

[request] launchRequest
[request] setBreakPointsRequest
[request] configurationDoneRequest
[info] launching `"busted" /Users/.../test/start.lua` from "/Users/..."
[info] process launched
[request] threadsRequest
[message] {"tag":"$luaDebug","message":"step","type":"debugBreak","threadId":1,"breakType":"step"}
[command] break clear
[message] {"tag":"$luaDebug","type":"breakpoints","breakpoints":[]}
[command] break set /usr/local/share/lua/5.3/busted/init.lua:65
[message] {"tag":"$luaDebug","type":"breakpoints","breakpoints":[{"file":"/usr/local/share/lua/5.3/busted/init.lua","enabled":true,"line":65}]}
[command] cont
[message] {"tag":"$luaDebug","message":"\nstack traceback:\n\t/usr/local/share/lua/5.3/busted/core.lua:268: in global 'describe'\n\t/Users/.../test/start.lua:5: in main chunk\n\t[C]: in function 'xpcall'\n\t/usr/local/share/lua/5.3/busted/core.lua:178: in field 'safe'\n\t/usr/local/share/lua/5.3/busted/block.lua:146: in field 'execute'\n\t/usr/local/share/lua/5.3/busted/init.lua:7: in upvalue 'executor'\n\t/usr/local/share/lua/5.3/busted/core.lua:312: in function </usr/local/share/lua/5.3/busted/core.lua:312>\n\t[C]: in function 'xpcall'\n\t/usr/local/share/lua/5.3/busted/core.lua:178: in field 'safe'\n\t/usr/local/share/lua/5.3/busted/core.lua:312: in field 'execute'\n\t/usr/local/share/lua/5.3/busted/execute.lua:58: in local 'execute'\n\t/usr/local/share/lua/5.3/busted/runner.lua:197: in function 'busted.runner'\n\t/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted:3: in main chunk\n\t[C]: in ?","type":"debugBreak","threadId":1,"breakType":"error"}
[error] 
stack traceback:
	/usr/local/share/lua/5.3/busted/core.lua:268: in global 'describe'
	/Users/.../test/start.lua:5: in main chunk
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.3/busted/core.lua:178: in field 'safe'
	/usr/local/share/lua/5.3/busted/block.lua:146: in field 'execute'
	/usr/local/share/lua/5.3/busted/init.lua:7: in upvalue 'executor'
	/usr/local/share/lua/5.3/busted/core.lua:312: in function </usr/local/share/lua/5.3/busted/core.lua:312>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.3/busted/core.lua:178: in field 'safe'
	/usr/local/share/lua/5.3/busted/core.lua:312: in field 'execute'
	/usr/local/share/lua/5.3/busted/execute.lua:58: in local 'execute'
	/usr/local/share/lua/5.3/busted/runner.lua:197: in function 'busted.runner'
	/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted:3: in main chunk
	[C]: in ?
[request] threadsRequest
[command] threads
[message] {"tag":"$luaDebug","threads":[{"active":true,"id":1,"name":"main thread"}],"type":"threads"}
[request] stackTraceRequest 0/20 (thread 1)
[command] thread 1
[message] {"tag":"$luaDebug","frames":[{"line":268,"active":true,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"describe"},{"source":"/Users/.../test/start.lua","line":5},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":146,"source":"/usr/local/share/lua/5.3/busted/block.lua","func":"execute"},{"line":7,"source":"/usr/local/share/lua/5.3/busted/init.lua","func":"executor"},{"source":"/usr/local/share/lua/5.3/busted/core.lua","line":312},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":312,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"execute"},{"line":58,"source":"/usr/local/share/lua/5.3/busted/execute.lua","func":"execute"},{"source":"/usr/local/share/lua/5.3/busted/runner.lua","line":197},{"source":"/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted","line":3},{"source":"[C]","line":-1}],"type":"stack"}
[request] evaluateRequest items
[command] thread 1
[message] {"tag":"$luaDebug","frames":[{"line":268,"active":true,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"describe"},{"source":"/Users/.../test/start.lua","line":5},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":146,"source":"/usr/local/share/lua/5.3/busted/block.lua","func":"execute"},{"line":7,"source":"/usr/local/share/lua/5.3/busted/init.lua","func":"executor"},{"source":"/usr/local/share/lua/5.3/busted/core.lua","line":312},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":312,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"execute"},{"line":58,"source":"/usr/local/share/lua/5.3/busted/execute.lua","func":"execute"},{"source":"/usr/local/share/lua/5.3/busted/runner.lua","line":197},{"source":"/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted","line":3},{"source":"[C]","line":-1}],"type":"stack"}
[command] frame 1
[message] {"tag":"$luaDebug","frames":[{"line":268,"active":true,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"describe"},{"source":"/Users/./test/start.lua","line":4..4},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":146,"source":"/usr/local/share/lua/5.3/busted/block.lua","func":"execute"},{"line":7,"source":"/usr/local/share/lua/5.3/busted/init.lua","func":"executor"},{"source":"/usr/local/share/lua/5.3/busted/core.lua","line":312},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":312,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"execute"},{"line":58,"source":"/usr/local/share/lua/5.3/busted/execute.lua","func":"execute"},{"source":"/usr/local/share/lua/5.3/busted/runner.lua","line":197},{"source":"/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted","line":3},{"source":"[C]","line":-1}],"type":"stack"}
[command] eval items
[message] {"tag":"$luaDebug","type":"result","result":{"type":"nil","value":"nil"}}
[request] scopesRequest
[command] thread 1
[message] {"tag":"$luaDebug","frames":[{"line":268,"active":true,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"describe"},{"source":"/Users/.../test/start.lua","line":5},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":146,"source":"/usr/local/share/lua/5.3/busted/block.lua","func":"execute"},{"line":7,"source":"/usr/local/share/lua/5.3/busted/init.lua","func":"executor"},{"source":"/usr/local/share/lua/5.3/busted/core.lua","line":312},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":312,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"execute"},{"line":58,"source":"/usr/local/share/lua/5.3/busted/execute.lua","func":"execute"},{"source":"/usr/local/share/lua/5.3/busted/runner.lua","line":197},{"source":"/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted","line":3},{"source":"[C]","line":-1}],"type":"stack"}
[command] frame 1
[message] {"tag":"$luaDebug","frames":[{"line":268,"active":true,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"describe"},{"source":"/Users/.../test/start.lua","line":5},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":146,"source":"/usr/local/share/lua/5.3/busted/block.lua","func":"execute"},{"line":7,"source":"/usr/local/share/lua/5.3/busted/init.lua","func":"executor"},{"source":"/usr/local/share/lua/5.3/busted/core.lua","line":312},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":312,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"execute"},{"line":58,"source":"/usr/local/share/lua/5.3/busted/execute.lua","func":"execute"},{"source":"/usr/local/share/lua/5.3/busted/runner.lua","line":197},{"source":"/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted","line":3},{"source":"[C]","line":-1}],"type":"stack"}
[request] variablesRequest locals
[command] locals
[message] {"tag":"$luaDebug","variables":[{"type":"nil","value":"nil","name":"trace"},{"type":"table","length":0,"value":"[table: 0x7f8612651570]","name":"ctx"},{"type":"function","value":"[function: 0x7f861245b1e0]","name":"fn"},{"type":"string","value":"\"Test case\"","name":"name"}],"type":"variables"}

By the way, without this code

if os.getenv("LOCAL_LUA_DEBUGGER_VSCODE") == "1" then
    require("lldebugger").start()
end

busted runs fine from VSCode, but without debugger of course.

[tomblind]

Ok, I do get the error using the busted cli. It turns out it's not an error at all: busted is using lua's debug library to get a callstack and this falsely triggers a debug break, because that's one way the debugger detects errors when running code outside of an xpcall. If you hit F5 and continue, the test script will still run and can be debugged.

The easiest workaround for this right now is to run busted using a lua interpreter instead of the busted cli, since this uses xpcall for error trapping. So, in your, launch.json:

"program": {
    "lua": "lua",
    "file": "${workspaceFolder}/test/start.lua"
}

and then in start.lua:

require("busted.runner")()

It looks like this is how @dainkaplan was doing it originally though, so I'm not sure why the error occurred for him.

I'll have to think on this more, since it would be good to support running from the custom cli.

[dainkaplan]

I haven’t been able to retest yet but hope to get to this finally this weekend! As @tomblind said that’s exactly how I was trying to run it. You’re using lua51?

[tomblind]

Yup - lua51

Make sure you don't use

if os.getenv("LOCAL_LUA_DEBUGGER_VSCODE") == "1" then
    require("lldebugger").start()
end

when running this way. It's not necessary when using the lua interpreter and could cause this issue to still arise.

[astrochili]

@tomblind @dainkaplan Guys, I tried your suggestion to use lua lunching instead CLI. This is what happens:
https://drive.google.com/file/d/1H_tJAbdvA9iXIyEoWWY_fQL6JxMCyWIl/view?usp=sharing

I use Lua 5.3 and can't test it with Lua 5.1 at this moment. Any ideas?

[tomblind]

I can't access the doc - can you post it here?

[astrochili]

I can't access the doc - can you post it here?

That's a video, I updated the sharing settings, try again please :)

[tomblind]

Oops. The environment isn't being set properly. I have a fix for that but haven't pushed it yet. Sorry 😄

[tomblind]

Actually I was wrong - the environment issue should not a problem when running in lua-interpreter mode.

Looking more closely at your video - there's something strange going on in the call stack. It looks like the debugger is rehooking a second time, which is probably messing up the environment. I have no idea off the top of my head why that would happen, though. It could be a lua5.3 thing - I'll have to figure out how to get that working and see.

[astrochili]

Actually I was wrong - the environment issue should not a problem when running in lua-interpreter mode.

Looking more closely at your video - there's something strange going on in the call stack. It looks like the debugger is rehooking a second time, which is probably messing up the environment. I have no idea off the top of my head why that would happen, though. It could be a lua5.3 thing - I'll have to figure out how to get that working and see.

I'd like to help you with testing. I tried to install busted via luarocks for lua 5.1 but without success.

[tomblind]

I managed to get busted working with 5.3 and reproduce your issue. Unfortunately, I don't think debugging using the interpreter in this way will work. When busted.runner is required, it re-evaluates any code passed to lua's command line, which runs a second debugger hook. Unfortunately, this second hook doesn't get access to the busted environment, hence the error.

However, I have fixed the original issue that occurred when running via the busted cli (6ac5b86) and it will be in the next release (which I hope to do today or tomorrow). With this fix, you can also run using a lua interpreter, but you have to set it up as a custom environment (use "command": "lua" in launch.json instead of "lua": "lua").