Nvim-dap, Pytest, Debugpy and Docker

[olimorris]

Firstly, this isn't an issue with the plugin at all, more of a general question on being able to use vim-ultest to debug remotely into a Docker container.

With my current config (for reference rather than to understand), I use a pretty hefty command to pass to the command line which triggers Docker to wait for any feedback from nvim-dap:

docker-compose -f "./docker-compose.yml" exec -T -w /usr/src/app debug python -m debugpy --listen ' ..
                debug_host .. ':' .. debug_port .. ' --wait-for-client -m pytest ' .. test_method[1]

Where test_method is the test name for the nearest test (something I pinched from vim-test). Using the instructions in the vim-ultest docs, combined with my previous setup, I form the following:

require("ultest").setup({
        builders = {
            ['python#pytest'] = function(cmd)
                local debug_host = '0.0.0.0'
                local debug_port = 5678

                local test_method = fn['test#python#pytest#build_position']('nearest', {
                    file = fn['expand']('%'),
                    line = fn['line']('.'),
                    col = fn['col']('.')
                })

                local args = 'docker-compose -f "./docker-compose.yml" exec -T -w /usr/src/app debug python -m debugpy --listen ' .. debug_host .. ':' .. debug_port .. ' --wait-for-client -m pytest ' .. test_method[1]
                return {
                    dap = {
                        type = "python",
                        request = "attach",
                        connect = {
                            host = debug_host,
                            port = debug_port
                        },
                        args = args,
                        mode = "remote",
                        name = "Remote Attached Debugger",
                        cwd = fn.getcwd(),
                        pathMappings = {{
                            localRoot = fn.getcwd(), -- Wherever your Python code lives locally.
                            remoteRoot = "/usr/src/app" -- Wherever your Python code lives in the container.
                        }}
                    }
                }
            end
        }
    })

When I run this with UltestDebugNearest I get Invalid adapter: nil which is an nvim-dap error. Which seems odd as I use it with nvim-dap in my previous setup.

Is there anything obvious I may be overlooking? Granted I need to actually make use of the vim-ultest cmd function to pass to nvim-dap.

[rcarriga]

Haha wow that is a chunky config alright.

My first question is to ask if there is a reason for not using the supplied cmd argument to your builder? It should just be as simple as

           function(cmd)
                local debug_host = '0.0.0.0'
                local debug_port = 5678

                local args = 'docker-compose -f "./docker-compose.yml" exec -T -w /usr/src/app debug python -m debugpy --listen ' .. debug_host .. ':' .. debug_port .. ' --wait-for-client -m ' .. table.concat(cmd, " ")
                return {....

Though I don't think that'd causing the issue you're having.

That error comes from almost immediately after calling nvim-dap's run function so shouldn't be hard to debug.
In your local installation of nvim-dap can you add just

print(vim.inspect(M.adapters), vim.inspect(config))

just after here https://github.com/mfussenegger/nvim-dap/blob/master/lua/dap.lua#L226

That should give a good hint as to what's going wrong

[olimorris]

So I'm now not getting an error...but I'm not getting anything either:

g['test#python#pytest#executable'] = 'pytest'

    require("ultest").setup({
        builders = {
            ['python#pytest'] = function(cmd)
                local debug_host = '0.0.0.0'
                local debug_port = 5678
                
                local args = 'docker-compose -f "./docker-compose.yml" exec -T -w /usr/src/app debug python -m debugpy --listen ' .. debug_host .. ':' .. debug_port .. ' --wait-for-client -m pytest ' .. table.concat(cmd, " ")
                return {
                    dap = {
                        type = "python",
                        request = "attach",
                        connect = {
                            host = debug_host,
                            port = debug_port
                        },
                        args = args,
                        mode = "remote",
                        name = "Remote Attached Debugger",
                        cwd = fn.getcwd(),
                        pathMappings = {{
                            localRoot = fn.getcwd(), -- Wherever your Python code lives locally.
                            remoteRoot = "/usr/src/app" -- Wherever your Python code lives in the container.
                        }}
                    }
                }
            end
        }
    })

The output from dap.lua is:

{
  python = {
    host = "0.0.0.0",
    port = 5678,
    request = "attach",
    type = "server"
  }
} {
  args = 'docker-compose -f "./docker-compose.yml" exec -T -w /usr/src/app debug python -m debugpy --listen 0.0.0.0:5678 --wait-for-client -m pytest tests/test_basic.py::test_that_this_adds_up',
  connect = {
    host = "0.0.0.0",
    port = 5678
  },
  cwd = "/Users/Oli/Code/Python/UAPI/Decoder",
  mode = "remote",
  name = "Remote Attached Debugger",
  pathMappings = { {
      localRoot = "/Users/Oli/Code/Python/UAPI/Decoder",
      remoteRoot = "/usr/src/app"
    } },
  request = "attach",
  type = "python"
}

Which looks like it should work just fine. Feel this must be some weirdness in the config itself

[rcarriga]

Yep that looks fine to me and if it's reaching nvim-dap like that then it's likely a config problem. Try setting the nvim-dap log level to DEBUG and seeing if there are any errors.

If you supply the same config to nvim-dap directly do you get the same behaviour?

Also I'm guessing you just mistyped but in your config you have -m pytest ' .. table.concat(cmd, " ") where it should be -m ' .. table.concat(cmd, " "). The printed config looks right though.

[olimorris]

So had another go at this tonight. Went back to basics and ensured I got everything working for debugging locally, which I have. I then subbed in the config for nvim-dap Docker debugging:

(Anything commented out is config related to local debugging and not Docker)

cmd 'packadd nvim-dap'

local dap = require('dap')
dap.set_log_level('DEBUG')

-- dap.adapters.python = {
--     type = 'executable';
--     command = '/Users/Oli/.asdf/shims/python3';
--     args = { '-m', 'debugpy.adapter' };
-- }
-- dap.configurations.python = {
--     type = 'python'; -- the type here established the link to the adapter definition: `dap.adapters.python`
--     request = 'launch';
--     name = "Launch file";
--     pythonPath = '~/.asdf/shims/python3'
-- }

dap.adapters.python = {
    type = "server",
    host = '0.0.0.0',
    port = 5678,
}
dap.configurations.python = {
    type = "python",
    request = "attach",
    connect = {
        port = 5678,
        host = '0.0.0.0'
    };
    mode = "remote",
    name = "Remote Attached Debugger",
    cwd = vim.fn.getcwd(),
    pathMappings = {
        {
            localRoot = vim.fn.getcwd(), -- Wherever your Python code lives locally.
            remoteRoot = "/usr/src/app", -- Wherever your Python code lives in the container.
        };
    };
}

g['test#python#pytest#executable'] = 'pytest'

local docker_cmd = 'docker-compose -f "./docker-compose.yml" exec -T -w /usr/src/app debug python -m debugpy --listen 0.0.0.0:5678 --wait-for-client -m pytest '

require("ultest").setup({
    builders = {
        ['python#pytest'] = function(cmd)

            return {
                dap = {
                    type = 'python',
                    request = 'attach',
                     -- module = cmd[1],
                    module = docker_cmd,
                    args = {cmd[2]}
                }
            }
        end
    }
})

This yields the following as debug information and just hangs:

[ DEBUG ] 2021-04-22T23:25:41Z+0100 ] ...e/nvim/site/pack/packer/opt/nvim-dap/lua/dap/session.lua:711 ]	"request"	{
  arguments = {
    adapterID = "nvim-dap",
    clientId = "neovim",
    clientname = "neovim",
    columnsStartAt1 = true,
    linesStartAt1 = true,
    locale = "en_GB@currency=GBP.UTF-8",
    pathFormat = "path",
    supportsRunInTerminalRequest = true,
    supportsVariableType = true
  },
  command = "initialize",
  seq = 0,
  type = "request"
}

Let me know if you see anything obvious that I'm missing.

Shall continue to play around with it this weekend.

[rcarriga]

So I'm not too familiar with the docker side of nvim-dap but to me it looks like config you're returning to vim-ultest is the wrong one.

From what I can tell you should be running your docker command as a separate job (using vim.loop.spawn, jobstart or whatever) and then return the pythonAttachConfig from the example in the docs. So it'd be something like

require("ultest").setup(
  {
    builders = {
      ["python#pytest"] = function(cmd)
        local docker_cmd =
          'docker-compose -f "./docker-compose.yml" exec -T -w /usr/src/app debug python -m debugpy --listen 0.0.0.0:5678 --wait-for-client -m ' ..
          table.concat(cmd, " ")

        -- You can attach output handlers but anything needed by vim-ultest should be caught by the "attach" adapter
        vim.fn.jobstart(docker_cmd)

        return {
          dap = {
            type = "python",
            request = "attach",
            connect = {
              port = 5678, -- Need to open bind this port to your container
              host = "0.0.0.0"
            },
            mode = "remote",
            name = "Remote Attached Debugger",
            cwd = vim.fn.getcwd(),
            pathMappings = {
              {
                localRoot = vim.fn.getcwd(),
                remoteRoot = "/usr/src/app" -- Likely need to change this
              }
            }
          }
        }
      end
    }
  }
)

[olimorris]

Thank you. Some solid advice.

So I have it working nicely. The only downside is the inclusion of a wait function. Doesn't seem to be any other way to know if you're connected to the Docker container or not.

Let me know if you'd like me to write this up in a Wiki btw. Whilst us Docker users are the minority it's still an interesting use case and I love that we can use vim-ultest as the driver for using nvim-dap.

Final code is:

local dap = require('dap')
dap.set_log_level('DEBUG')

dap.adapters.python = {
    type = "server",
    host = '0.0.0.0',
    port = 5678,
}

g['test#python#pytest#executable'] = 'pytest'

require("ultest").setup({
    builders = {
        ['python#pytest'] = function(cmd)

            local docker_cmd =
                'docker-compose -f "./docker-compose.yml" exec -T -w /usr/src/app debug python -m debugpy --listen 0.0.0.0:5678 --wait-for-client -m ' ..
                table.concat(cmd, " ")

            g.debug_job_id = fn.jobstart(docker_cmd)
            utils.wait(2)

            return {
                dap = {
                    type = "python",
                    request = "attach",
                    connect = {
                        port = 5678,
                        host = '0.0.0.0'
                    };
                    mode = "remote",
                    name = "Remote Attached Debugger",
                    cwd = fn.getcwd(),
                    pathMappings = {
                        {
                            localRoot = fn.getcwd(), -- Wherever your Python code lives locally.
                            remoteRoot = "/usr/src/app", -- Wherever your Python code lives in the container.
                        };
                    };
                }
            }
        end
    }
})

where my wait function is defined as:

function wait(seconds)
    local start = os.time()
    repeat
    until os.time() > start + seconds
end

GIF:

[rcarriga]

That's great to hear! 😁 As an alternative to the wait, could you add the -d flag to the exec command to detach? If that works OK (provided debugpy doesn't require an interactive session for some reason) then it should be as simple as swapping

            g.debug_job_id = fn.jobstart(docker_cmd)

with

            g.debug_job_id = fn.system(docker_cmd)

which will block until it returns.

I'd definitely like to add this to the wiki btw!

[olimorris]

I will write this up in the next week!