neotest-dotnet Adapter

[Issafalcon]

Hi @rcarriga.

Firstly, I'd like to say, you're doing an amazing job crafting the testing ecosystem in neovim! The extensible unified framework approach is great!

I'd like to get involved. I'm working on making dotnet development better in neovim, and I'm creating a plugin to extend the omnisharp lsp. I got up to writing some stuff for the testing side (much like how omnisharp-vim has a built in testing solution), but then I saw nvim-neotest, and thought it would be a much better idea to create an adapter for the testing functionality for dotnet.

I was hoping you, or other contributors might be able to help me translate what I have into an adapter. Here is what I have so far:

local omnisharp_client = require("neosharper.omnisharp.client")
local project_actions = require("neosharper.omnisharp.actions.project")
local code_actions = require("neosharper.omnisharp.actions.code")
local constants = require("neosharper.constants")
local request_params = require("neosharper.omnisharp.requests.basic_request_params")
local dap = require("neosharper.dap")

local M = {}

local debugStarted = false
local waitingForDebugger = false
local dotnet_test_pid

local function find_tests_in_file(code_elements, test_list)
  test_list = test_list or {}

  for _, element in ipairs(code_elements) do
    -- Check the elements for tests
    if element.Properties and element.Properties.testMethodName and element.Properties.testFramework then
      table.insert(test_list, element)
    end

    if element.Children then
      find_tests_in_file(element.Children, test_list)
    end
  end

  return test_list
end

local function find_test(tests)
  for _, test in ipairs(tests) do
    local test_ranges = test.Ranges.full
    if vim.fn.line(".") >= test_ranges.Start.Line and vim.fn.line(".") <= test_ranges.End.Line then
      return test
    end
  end
end

local function attach_debugger_to_debugged_test(test_output, debugger_code_window)
  for _, output in ipairs(test_output) do
    print(output)
    dotnet_test_pid = dotnet_test_pid or string.match(output, "Process Id%p%s(%d+)")
    if string.find(output, "Waiting for debugger attach...") or string.find(output, "Please attach debugger") then
      waitingForDebugger = true
    end
  end
  if dotnet_test_pid ~= nil and waitingForDebugger and not debugStarted then
    debugStarted = true
    vim.schedule(function()
      vim.fn.win_gotoid(debugger_code_window)
      dap.debug_attach(dotnet_test_pid)
    end)
  end
end

local function run_dotnet_test(cmd, debug_mode)
  local initial_win_id = vim.fn.win_getid()
  vim.api.nvim_create_buf(false, true)
  vim.cmd("botright new")
  vim.cmd("resize " .. 5)

  vim.fn.termopen(cmd, {
    env = (debug_mode and { ["VSTEST_HOST_DEBUG"] = "1", ["DOTNET_CLI_HOME"] = "/tmp" }) or nil,
    on_stdout = function(_, return_val)
      if debug_mode then
        attach_debugger_to_debugged_test(return_val, initial_win_id)
      end
    end,
    on_stderr = function(error, return_val)
      print(return_val)
      print(error)
    end,
    on_exit = function()
      waitingForDebugger = false
      dotnet_test_pid = nil
      debugStarted = false
    end,
  })
end

function M.discover_tests()
  local params = request_params.make_basic_request_params()

  omnisharp_client.make_request_async(constants.omnisharpEndpoints.discoverTests, params, function(err, result)
    if err then
      error("Error fetching test data from Omnisharp-lsp: " .. err)
    else
      put(result)
    end
  end)

  return "Finished the discover tests"
end

function M.debug_test()
  local code_structure = code_actions.get_code_structure()

  local tests = find_tests_in_file(code_structure.Elements)
  local current_test = find_test(tests)

  if current_test == nil then
    vim.api.nvim_echo({ { "No test found", "WarningMsg" } }, true, {})
  end

  local projectInfo = project_actions.get_project_details()
  local dotnet_test_cmd = "dotnet test "
    .. projectInfo.MsBuildProject.TargetPath
    .. " --filter "
    .. current_test.Properties.testMethodName

  run_dotnet_test(dotnet_test_cmd, true)
end

return M

For completion, here is the dap wrapper in my plugin:

local dap = require("dap")
local debug_options = require("neosharper.options").get_options().debugging

local M = {}

local function get_dap_adapter_config()
  return {
    type = 'executable',
    command = debug_options.netcoredbg_path,
    args = {'--interpreter=vscode'}
  }
end

local function get_dap_adapter_for_attach(host, port)
  return {
    port = tonumber(port),
    host = host,
    type = 'executable',
    command = debug_options.netcoredbg_path,
    args = {'--interpreter=vscode'},
  }
end

function M.terminate_dap_session()
  if dap.session() then
    dap.terminate()
    dap.close()
  end
end

function M.debug_launch(launchSettings, programPath)
  dap.set_log_level("TRACE")
  dap.launch(get_dap_adapter_config(), {
      type = "netcoredbg",
      name = "launch - netcoredbg",
      request = "launch",
      program = programPath,
      -- TODO: True for now. Make configurable for user
      stopAtEntry = debug_options.stop_at_entry,
      cwd = launchSettings.filePath,
      args = { '--launch-profile="PatientDemandApp.WebApi"'}
  })
end

function M.debug_attach_remote(port, host, cwd)
  dap.set_log_level("TRACE")
  dap.attach(get_dap_adapter_for_attach(host, tonumber(port)), {
    type = "netcoredbg",
    name = "remote attach - netcoredbg",
    request = "attach",
    connect = {
      port = tonumber(port),
      host = host
    },
    -- mode = "remote",
    cwd = cwd
  })
end

function M.debug_attach(pid)
  dap.launch(get_dap_adapter_config(),
    {
      type = "netcoredbg",
      name = "attach - netcoredbg",
      request = "attach",
      processId = pid
    }
  )
end

return M

To summarise, I am leveraging the omnisharp server in LSP mode to find the appropriate test from cursor position (or all tests in a file). Running the test(s) is straightforward, but debugging requires that dotnet test gets run with specific environment variables set (VSTEST_HOST_DEBUG) so that the test pauses and outputs a PID, which can then be used to attach the debugger.

I'm not entirely sure what this might look like in a neotest adapter (reading through the existing ones, I have some idea, but not sure if it would work in the same way as I have it in the code above).

So my questions are:

1. Would it be reasonable to assume users of neotest-dotnet would have omnisharp lsp installed / attached to the buffer? An option would be to check for the omnisharp client, but fall back to using treesitter queries like the rest of neotest. The omnisharp lsp requests just provide some convenient ways to get test names and other test metadata.
2. How would the run (with debug env variables set) -> pause -> extract PID -> attach debugger process look like in the adapter code?

[rcarriga]

Hey delighted to hear you want to work on this!

Would it be reasonable to assume users of neotest-dotnet would have omnisharp lsp installed / attached to the buffer?

I don't think that'd be unreasonable especially if it's a popular LSP for dotnet (I'm not at all familiar to I've no idea 😅).

An option would be to check for the omnisharp client, but fall back to using treesitter queries like the rest of neotest. The omnisharp lsp requests just provide some convenient ways to get test names and other test metadata.

If you're going to implement the treesitter anyway, does it still make sense to use the LSP as well? Is there some metadata that LSP provides that you can't get with treesitter? Of course there's no reason not to use LSP other than the potential issues of maintaining two solutions with one perhaps being neglected.

How would the run (with debug env variables set) -> pause -> extract PID -> attach debugger process look like in the adapter code?

That's an interesting one, you could just start the process before returning the arguments to provide to nvim-dap. So your plugin would completely control the process and then return the nvim-dap args once it is ready to go. All of the adapter functions can be async so you can wait, poll etc without blocking NeoVim.

[Issafalcon]

Thanks for the reply @rcarriga !

So the benefit of using the LSP is that all the various unit testing frameworks (xUnit, Nunit, MSTest, Specflow etc) are accounted for during test discovery. I haven't used treesitter queries too extensively, but I assume I would also need to account for all of the various attributes and quirks of each framework when writing my queries to gain the same sort of functinality. Not only that, the omnisharp-lsp would (I assume) also work for other languages that work with the dotnet runtime and their various testing frameworks. Also, there are other metadata benefits from using the LSP, like automatically being able to find the project reference that the test is part of, and the framework that is being used, which is required in some cases when using CLI commands to run tests.

I think supporting omnisharp-lsp use would be quite easy, but getting equivalent functionality from treesitter alone would require a lot more work. I think in the first instance, I would use the LSP. If enough people would like to see the ability to use the adapter without having to have omnisharp-lsp client attached (maybe they use the alternative csharp-lsp) then I could add the treesitter support as required. The omnisharp LSP solution wouldn't require much ongoing maintenance, but the treesitter work could be added to to provide very similar functionality.

I'll make a start on this soon. Any other general tips for adapter writing?

[rcarriga]

OK that all makes a lot of sense 👍 Sounds good!

In terms of advice, the current documentation should be a good start but there is definitely some things lacking. Also have a look at the existing issues/repos for other adapters #9 #6 #5, you might find them useful. The main issue people seem to have is result collection which makes sense so feel free to ask for advice with specific problems. Also if you're using sumneko_lua LSP I'd recommend enabling type checking for neotest as many of the functions the neotest.lib are type annotated and it'll help for discovery. Please not if you're using the lib functions, the only public interface is neotest.lib, don't import lower level modules

[Issafalcon]

Nice one! Thanks again for the info!

[iabdelkareem]

Hey @Issafalcon I'm a developer working with .NET & Neovim. I found your post while searching if there anyone mentioned a neotest adapter for .NET.

I'd love to join forces with you and contribute in developing & testing this adapter and to build a better .NET development experience in Neovim in general ^^

[Issafalcon]

Cool! Sorry for the late reply, I just started a new job so busy getting up to speed with everything! Hopefully I'll be able to start planning out the adapter and maybe starting out on some code this weekend. Always happy to collaborate!

[Issafalcon]

Hi @rcarriga - I've made a start on the adapter https://github.com/Issafalcon/neotest-dotnet. It would be good to get your opinion on the following approach. Essentially it boils down to the 'using LSP vs using treesitter' discussion above. The LSP methods offer all the metadata I think I need (class names, namespace names, method names, ranges), but not in the format treesitter might return. The other caveat, is that the project needs to have been built using the dotnet build command, in order for tests to be discovered this way.

1. run is_test_file method using omnisharp_lsp requests to determine if each file ending in .cs or .fs contains any tests.

• Unfortunately there is no file naming convention that can reliably be predicted for test files. Anything with those extensions is fair game, which could be hundreds (or dare-i-say thousands) of files.
• For each file_name argument passed in, the adapter file would send a 'code_structure' request to the omnisharp lsp and then run some processing to determine if it contained any tests. Only then could the file be considered a 'test file'
• I'm not sure if it makes sense to do this upfront before running a treesitter query on these files, or if simply saying every file ending in .cs or .fs is a 'test file' and then relying on the treesitter queries in the discover_positions interface method to pick out the tests. Bearing in mind, there could be hundreds of files with this extension in a single project.

2. Use treesitter queries to pick out the tests, as per the other adapters (it seems to me like neotest is geared towards handling and parsing treesitter nodes, so my original thought of using the lsp to extract the tests might end up being more complex)

• Alternatively, if you can suggest a way of converting the test elements returned from the lsp code_structure request to a format that neotest would expect from the discover_positions method, then it might be better to do this (I could then cache all the test elements discovered in the is_test_file method and simply parse the metadata to fit the expected return model)
• This is what I get from my custom test_discovery method using the LSP (note this is a set of the extracted test elements, but the parent would be something with `Kind = "class", having similar properties without specifying testFramework or testMethodName on it, so where treesitter parser extracts the class or namespace, we would be able to obtain this as well)

{
    DisplayName = "IsPrime_InputIs1_ReturnFalse()",
    Kind = "method",
    Name = "IsPrime_InputIs1_ReturnFalse",
    Properties = {
      accessibility = "public",
      static = false,
      testFramework = "mstest",
      testMethodName = "Prime.UnitTests.Services.PrimeService_IsPrimeShould.IsPrime_InputIs1_ReturnFalse"
    },
    Ranges = {
      attributes = {
        End = {
          Column = 20,
          Line = 15
        },
        Start = {
          Column = 8,
          Line = 15
        }
      },
      full = {
        End = {
          Column = 9,
          Line = 21
        },
        Start = {
          Column = 8,
          Line = 15
        }
      },
      name = {
        End = {
          Column = 48,
          Line = 16
        },
        Start = {
          Column = 20,
          Line = 16
        }
      }
    }
  }

FYI - @iabdelkareem - If you're keen to help on this, I'll create some initial issue items against the repo once we determine the best approach.

[Issafalcon]

Just looking through the parser function in neotest-vim-test. Starting to think it might be easier to write the treesitter queries for all the different test case possibilities 😄

[rcarriga]

There's nothing in Neotest that couples it with treesitter so you should have no issue with using the LSP information as long as you can determine the nesting of the tests/namespaces. Once you can get them into a nested list structure (see https://github.com/nvim-neotest/neotest/blob/master/tests/unit/lib/positions/init_spec.lua for examples of what's expected), the Tree class can parse the list. Does the LSP provide the structure or represent the relationships somehow? If not you could always use the ranges to infer them.

I do think that checking for tests in is_test_file would be a good idea, otherwise the summary tree will show all files in the project which wouldn't be the best experience IMO. I'd say avoid caching for now, and see if the performance really requires it after implementing. If you cache, you'll have to handle invalidating it 😅

[Issafalcon]

Thanks for the reply! That's exactly the kind of example structure I was looking for! The LSP does provide the structure, and nests it appropriately, so it might be easier than I expected in that case. I'll carry on with the LSP approach.

[rcarriga]

Just in case you hadn't seen it as well, the tree Position elements are defined here https://github.com/nvim-neotest/neotest/blob/master/lua/neotest/types/init.lua. Some of the examples from above are incomplete so just wanted to be sure you had the right info.

[iabdelkareem]

@Issafalcon That's great ^^ I'm currently on vacation. I'll be able to help out after 3 weeks

[Issafalcon]

Hi @rcarriga. By way of an update, I'm making some good (albeit slow) progress with the dotnet adapter. I'm using a luarocks module xml2lua so that I can easily parse the .trx file that microsoft produce. Consuming this for plugin users might not always be so straightforward I imagine (depending on their plugin manager - packer has builtin luarocks install option).

I did notice the new #70 stream option, but the test output streamed directly from the runner is nowhere near as well structured as the trx.

Anyway, if using luarocks isn't a deal-breaker for an adapter, I'll carry on with the current implementation, and it shouldn't be too long before it's usable.

[rcarriga]

Great to hear there's progress!

Neotest won't impose what can/can't be used within an adapter, since they're downstream so as long as users are happy to install, then there's no issue.

However for this case, I've actually already embedded xml2lua in the neotest lib 😅 You can use require("neotest.lib").xml.parse.

For streaming you don't need to use the streamed output, for example neotest-python doesn't. If you can get the xml streamed somehow then you could use that!

[Issafalcon]

Ah, nice! Well that makes things easier 😄 Not sure how to get the streamed xml, so I'll stick to just parsing the output file. Another question I had was about using some of the async stuff in the util lib. I was going to switch to making async calls instead of synch calls to the LSP during the adapter methods, because I'm finding for large files, the lsp can take a while to get the code structure. These requests will be given callbacks that will receive error / response data. What's the best way to handle this in the adapter methods I'm writing. Would the callback need to forward on the response to some other method of neotest base?

[rcarriga]

So the async lib can convert a callback function to an async function using async.wrap. This is already built-in for vim.lsp.buf_request_all (see https://github.com/nvim-lua/plenary.nvim/blob/master/lua/plenary/async/lsp.lua) which can be used as async.lsp.buf_request_all.

For more complex patterns, for example calls that require multiple callbacks, you can use the async.control module (see https://github.com/nvim-lua/plenary.nvim/blob/master/lua/plenary/async/control.lua) to coordinate the flow. The control module is used several times in neotest if you would like some examples

[Issafalcon]

Hi @rcarriga - So, after a lot of back and forth on a few approaches, I think I've finally got something that can be widely used. https://github.com/Issafalcon/neotest-dotnet. There are many more features that can be added (debugging, itemized test output files with more verbose detail etc.) but I hope it's good enough to be incorporated into the ecosystem as a start!

[rcarriga]

Brilliant to hear! I've added it to the README 😄

[Issafalcon]

Hi @rcarriga . So I'm trying to add support for parameterized tests in the dotnet adapter. The tests are parameterized by adding an attribute per additional test above the main test method. My idea is to customize the build_position function in the treesitter parser and then add separate test nodes (each requiring a custom test name so dotnet runs correctly) for each of the attributes.

My question is, how does neotest determine what is a nested test? I'd like each of the parameterized tests to nest underneath the main test method in the summary list. Is it determined by the treesitter ranges alone, or do the test nodes need to be parsed in a particular order as well?

[Issafalcon]

I've implemented the best approach I could think of (code now in main). It seems to be working nicely now:

[rcarriga]

Good to hear you've got it working at least!

I think I see what you mean with the capturing issues but I'm definitely a bit unclear on it just because I've absolutely experience with C#. You don't necessarily have to do what you need with captures, they're just the easy declarative way of doing it. You can use the node methods to explore the tree (see :h tsnode) which I think will allow you to do the more complex traversal that you need. If you want to provide a sample test file I can give it a shot as well!

[Issafalcon]

Oh man, I didn't even know about that treesitter api! That does change things! As I have it working now, I'll probably put something on the backlog to update it to use the more efficient way of doing it with tsnode. However, if you wanted to have a crack at it, that would be great.

The sample files I usually use are in https://github.com/dotnet/samples/tree/main/core/getting-started. That repo has various dotnet sample projects maintained by microsoft. You'll need to install the dotnet core 3.1 SDK, and run dotnet build from the directory containing the .sln file of those individual projects.

[rcarriga]

Actually managed to do it without needing to do traversal, you can do it with just a second query on the test definition node.
I think you would be able to have one query for parameterized and non-parameterized tests with this approach but not sure.

local async = require("neotest.async")
local lib = require("neotest.lib")
local ts = lib.treesitter

async.run(function()
  local param_query = vim.treesitter.parse_query(
    "c_sharp",
    [[
      ;;query
      (attribute_list
        (attribute
          name: (identifier) @attribute_name (#any-of? @attribute_name "InlineData")
          ((attribute_argument_list) @parameters)
        )
      )
    ]]
  )
  local parse_queries = [[
    ;; query
    (method_declaration
      name: (identifier) @test.name
      parameters: (parameter_list
        (parameter
          name: (identifier)
        )*
      ) @parameter_list
    ) @test.definition

    ]]
  local tree = ts.parse_positions("test.cs", parse_queries, {
    nested_tests = true,
    build_position = function(file_path, source, captured_nodes)
      local match_type = "test"
      local name = vim.treesitter.get_node_text(captured_nodes[match_type .. ".name"], source)
      local definition = captured_nodes[match_type .. ".definition"]

      local node = {
        type = match_type,
        path = file_path,
        name = name,
        range = { definition:range() },
      }
      local nodes = { node }

      local capture_indices = {}
      for i, capture in ipairs(param_query.captures) do
        capture_indices[capture] = i
      end
      local params_index = capture_indices["parameters"]

      for _, match in param_query:iter_matches(captured_nodes[match_type .. ".definition"], source) do
        local params_node = match[params_index]
        local params_text = vim.treesitter.get_node_text(params_node, source)
        nodes[#nodes + 1] = vim.tbl_extend(
          "force",
          node,
          { name = node.name .. params_text, range = { params_node:range() } }
        )
      end

      return nodes
    end,
  })
  print(vim.inspect(tree:to_list()))
end)

[Issafalcon]

Excellent! Thanks for your help on this. I'll add another task in to see if this can work in place of what I have already.

[Issafalcon]

Hi @rcarriga - I have another question for you, as I've hit a hurdle I can't think how to overcome. Maybe it's just not possible yet.

Having implemented debugging support in neotest-dotnet (entry point https://github.com/Issafalcon/neotest-dotnet/blob/4e71023855e4de9053c43e0829b2be439b3d33ec/lua/neotest-dotnet/init.lua#L129). I've started the debug process outside the workings of neotest core, so that I can wait for the subprocess ID of the dotnet runner to be returned from the output in order to attach the debugger. It's working well, and I'm outputting the results from the dotnet test command in a new buffer. However, the issue is that neotest picks up the output from the netcoredbg process, which doesn't contain the output from the actual test.

I've added a screencast to the adapter github page https://github.com/Issafalcon/neotest-dotnet so you can see what I mean.

I think it's because neotest might be creating a different results file when using debug strategy, which isn't a trx file (so when I try and parse it, it fails and looks like the test run failed).

Is there a simple way to fix this? At the moment I just inform people that debugging will always produce a failed output, but at least the additional stream output is there in the other buffer so they can see the actual test results.

[Issafalcon]

Thanks @rcarriga! I'll add in the output file from stdout of the debugged process as you suggest. The second part is something I thought saw happening via the log files (2 different trx files were being created, one of which either has nothing in it, or bogus output that can't be parsed as expected). I'll be honest, I haven't tried to identify that issue recently, so it could have just been a red herring. If I see it again after I add in the output fix, I'll grab the logs and some more information. For the time being, I'm just adding support for the remaining frameworks, and doing some bug squashing 😃

[Issafalcon]

Finally got round to getting this in. Works nicely now! I don't suppose there is an easier way of switching to use a different strategy internally to the adapter is there?

I've taken the standard dap strategy and modified it for the adapter, and I couldn't figure out an easy way of switching it over for the user. So instead of using the dap strategy, I always want to use my custom one if the user is running with the dap strategy. It would save them having to have different keybinds or commands when using the dotnet adapter.

I'm currently requiring the users to set up their own commands for it:
lua require("neotest").run.run({strategy = require("neotest-dotnet.strategies.netcoredbg"), is_custom_dotnet_debug = true})

[rcarriga]

We could allow the build spec strategy field to be a strategy function so that the adapter could check if the strategy is dap and return its custom version to run

[Issafalcon]

That would be ideal!

[rcarriga]

I've added that to core, so just set strategy to your custom function in the build spec when users select the dap strategy, and it should work 😄

[Issafalcon]

Hi @rcarriga. I'm looking at implementing a new feature, to discover parameterized tests that are not inline. This requires actually compiling the C# code. See Issafalcon/neotest-dotnet#55 for details.

My question for you is, what would be the best way of running the dotnet test -t process in the background, as few times as possible? I am envisioning running it once in the 'root' directory to list all the tests during the initial file discovery process, and then caching this result in a tmp file somewhere. Then, the discover_tests function can check it if the treesitter queries determines that we have parameterized tests. The tricky bit would be figuring out when to bust this cache and regenerate the list of tests using the command. It can be pretty slow to run, so I only want to do it when it's necessary.

[rcarriga]

A very simple approach could be to have a simple loop running that will refresh tests at most every x ms using a nio.control.Event and nio.sleep. I do something similar in some of the consumers (

neotest/lua/neotest/consumers/state/init.lua

Lines 13 to 25 in 92badcc

	local updated_event = nio.control.event()
	tracker = StateTracker:new(client)
	local function update_positions()
	while true do
	updated_event.wait()
	updated_event.clear()
	for _, adapter_id in ipairs(tracker.adapter_ids) do
	tracker:update_positions(adapter_id)
	tracker:update_counts(adapter_id)
	end
	nio.sleep(50)
	end
	end

) but you could do the sleep before running so that it "debounces" the user writing several times in quick succession

[Issafalcon]

Ah, good idea! I think this will work well enough. I'll give it a go!

Thanks!