CLI flag to launch Pulumi program under a debugger

[swgillespie]

Since we get to use ✨ real code ✨ in Pulumi programs, sometimes you just need to look at your code in a debugger. It would be cool if we could do this from the pulumi CLI.

Language hosts could optionally respond to an RPC request to launch a given program under a debugger where the debugger is listening to a given port.

[ellismg]

Moving this out. We'll probably do this around the same time that we try to support doing more Pulumi stuff from within the language host itself (e.g. when we support managing the lifecycle of a stack in the source program itself, so you can write tests using your favorite framework).

[mikhailshilkov]

With .NET support, this should get up in the queue IMO, and .NET folks love their F5-debugging.

[rienafairefr]

Adding my 2cts, not sure why that works but it seems to, by using the node-terminal type in launch.json

{

    "version": "0.2.0",
    "configurations": [
         {
            "name": "Deploy run",
           "command": "pulumi up", 
           "type": "node-terminal"
         }
    ]
}

typescript program, pulumi 3.10.3, node 12, vscode 1.59.1
I can put a breakpoint in the index.ts, hit the debug, et voilà.

[aureq]

For people using Python and VSCode, here is an alternative.

1. Install debugpy as part of your environment.
2. In your VSCode project add a new launch configuration and select Python and then Remote Attach. Keep the default (localhost and 5678)
3. Add the code below to your code

import debugpy
debugpy.listen(("localhost", 5678))
max_wait_s = 30
while max_wait_s >= 0:
    if debugpy.is_client_connected():
        break

    time.sleep(1)
    max_wait_s -= 1

To start debugging, simply run pulumi up or pulumi preview from the command line and after a few seconds, hit F5 in VSCode. VSCode should pause the execution at the first encountered breakpoint.

If no debugger is attached after 30 seconds, execution will continue.

[mritd]

For people using GoLand and Go (This idea may apply to other languages):

• 1. First of all, Pulumi directly executes the binary file/script when it starts, and Pulumi sets specific environment variables when executing the binary file (I guess other languages may be the same).
• 2. Secondly, Pulumi allows you to set the location of the executable file.
• 3. I think any modern language should support "remote debugging"; for GoLand, it uses dlv for remote debugging;

After understanding the execution method, we only need to create a simple script, then execute the dlv debugging command in the script, and finally turn on the remote Debug function of GoLand, we can happily Coding.

Below is the detailed tutorial:

This is my project structure:

~/o/p/xxxx ❯❯❯ tree
.
├── Pulumi.yaml
├── common.go
├── debug.sh
├── deployment.go
├── go.mod
├── go.sum
├── goapp
├── main.go
└── service.go

0 directories, 9 files

Compile Code to bin file

For GoLand's remote Debug, we need to compile our code using the following command:

# this will generate an executable file with debug symbols (goapp)
go build -o goapp -gcflags "all=-N -l" .

Create debug script

We create a bash script to execute the dlv command:

⚠️ Note: Please replace ./goapp with a real absolute path (I'm not sure if you don't do this will cause problems, but using an absolute path must be fine)

#!/bin/bash
dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec ./goapp

Save the above code as debug.sh, and execute chmod +x debug.sh to increase its executable permissions.

Update Pulumi.yaml

We need to update Pulumi.yaml to let Pulumi execute our debug.sh script:

⚠️ The following binary configuration should also use the absolute path of debug.sh

name: test
description: Debug example
runtime:
  name: go
  options:
    binary: /Users/bleem/ops/pulumi/xxxx/debug.sh

Create GoLand Remote Debug configuration

Start debugging

Execute the pulumi up command in the terminal, which will block pulumi and display a line of log:

Type                 Name       Plan     Info
pulumi:pulumi:Stack  xxxxx-dev           API server listening at: [::]:2345

Finally, click the Debug button on GoLand to let us "find bugs, fix bugs, create bugs".

[zivyatziv]

For people using Python and VSCode, here is an alternative.

1. Install debugpy as part of your environment.
2. In your VSCode project add a new launch configuration and select Python and then Remote Attach. Keep the default (localhost and 5678)
3. Add the code below to your code

import debugpy
debugpy.listen(("localhost", 5678))
max_wait_s = 30
while max_wait_s >= 0:
    if debugpy.is_client_connected():
        break

    time.sleep(1)
    max_wait_s -= 1

To start debugging, simply run pulumi up or pulumi preview from the command line and after a few seconds, hit F5 in VSCode. VSCode should pause the execution at the first encountered breakpoint.

If no debugger is attached after 30 seconds, execution will continue.

I would like to suggest a better approach which pauses the code and waits for a client to attach:

import debugpy

debugpy.listen(("0.0.0.0", 5678))
print("debugpy is listening, attach by pressing F5 or ►")

debugpy.wait_for_client()
print("Attached to debugpy!")

Since I am working with venv in pulumi, I would often get this error:

Unable to open '__main__.py': Unable to read file '/pulumi/__main__.py' (Error: Unable to resolve non-existing file '/pulumi/__main__.py').

You can solve this issue by changing the remote path in the launch.json like so:

"pathMappings": [
    {
        "localRoot": "${workspaceFolder}/",
        "remoteRoot": "${workspaceFolder}/"     # <-- default is empty which results in the root project directory 
    }
]

[harryttd]

I'm using Typescript and this works for me:

❯ code --version
1.63.0
7db1a2b88f7557e0a43fec75b6ba7e50b3e9f77e
x64

package.json:

Notice the "debug" prompt above the scripts section. In between lines 5 and 6. You can click that and vscode will open up a debug console. Works perfectly.
https://stackoverflow.com/a/70661424/6379084

[secustor]

I can confirm simply starting it in a debugger as @harryttd described, works in Webstorm too.

[ghostsquad]

For those using Go - there are a couple additional steps needed.

First, you will need a time.Sleep in your code to give you a chance to attach before the code you want to debug is run. Unfortunately Go does not have a "wait for attach" API, which means this is a little clunkier than in other languages.

Second, since go run strips debugging symbols, you will need to manually go build and then put a section like this in Pulumi.yaml:

runtime:
    name: go
    options:
        binary: goapp

Adjust the binary: configuration to point at the binary you have built locally, and then pulumi up will deploy using that binary and you can attach to it with your debugger.

Is it possible to use any of the --config* flags to dynamically set/merge these debugging properties? It's not clear how these flags should be used.

Do you have any additional steps for getting this to work in Jetbrains Goland?

Edit: I missed this comment regarding delve and remote debugging in goland (#1372 (comment))

[Frassle]

Note that nodejs programs can be quite easily debugged using vscodes auto-attach system (https://code.visualstudio.com/docs/nodejs/nodejs-debugging#_auto-attach)

Turn on auto attach and then run pulumi preview/up in the vscode terminal and vscode will see the new node process and attach to it.

[zivyatziv]

Note that nodejs programs can be quite easily debugged using vscodes auto-attach system (https://code.visualstudio.com/docs/nodejs/nodejs-debugging#_auto-attach)

Turn on auto attach and then run pulumi preview/up in the vscode terminal and vscode will see the new node process and attach to it.

Or you can create a new terminal with the type JavaScript debug terminal And that's it :)

[cristichiru]

I have found a simpler way to debug pulumi go project in VSCode.

There is no need to build the project upfront nor add options.binary.

1. As @lukehoban suggested, add to the beginning of your pulumi.Run func: time.Sleep(20 * time.Second)

2. Set a breakpoint somewhere in your code

3. Run pulumi up or pulumi preview. Wait a couple of seconds to allow go build (invoked by pulumi) to finish and start a process pulumi-go.NNNNNNNNNN

4. In VSCode, assuming you have an "Attach to Process" configuration, start debugging (F5)

   A list of processes should be displayed, filter to pulumi-go
   

[Pithikos]

Anyone got this working with IPython?

E.g.

import pulumi
from pulumi_gcp import storage

# Create a GCP resource (Storage Bucket)
bucket = storage.Bucket('my-bucket', location="eu")

import IPython; IPython.embed(using=False)

The debug point is then totally skipped when running Pulumi

Diagnostics:
  pulumi:pulumi:Stack (trade-engine-staging):
    Python 3.9.16 (main, Dec  7 2022, 01:11:51)
    Type 'copyright', 'credits' or 'license' for more information
    IPython 8.10.0 -- An enhanced Interactive Python. Type '?' for help.
    In [1]: Do you really want to exit ([y]/n)?

Resources:
    + 1 created
    1 unchanged

[Kos-UlpiaTech]

Below is an enhanced single click solution which worked for me to debug my Python Pulumi project on a Mac:

1. Create a tasks.json file in .vscode with content

{
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "2.0.0",
    "tasks": [
      {
        "label": "pulumi up",
        "type": "shell",
        "isBackground": true,
        "command": "pulumi up",
        "presentation": {
          "reveal": "silent"
        },
        // This task is run before some debug tasks.
        // Problem is, it's a watch script, and since it never exits, VSCode
        // complains. All this is needed so VSCode just lets it run.
        "problemMatcher": [
            {
            "pattern": [
                {
                "regexp": ".",
                "file": 1,
                "location": 2,
                "message": 3
                }
            ],
            "background": {
                "activeOnStart": true,
                "beginsPattern": ".",
                "endsPattern": ".",
            }
            }
        ]
      }, 
      {
        "label": "Wait 30 seconds",
        "type": "shell",
        "command": "sleep 3",
        "presentation": {
          "reveal": "silent"
        }
      },
      {
        "label": "Run pulumi up and wait to start",
        "dependsOn": ["pulumi up", "Wait 30 seconds"]
      }
    ]
  }

2. Create tasks launch.json file in .vscode with contents

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Remote Attach",
            "type": "python",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "preLaunchTask": "Run pulumi up and wait to start",
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "."
                }
            ],
            "justMyCode": true
        }
    ]
}

3. Add the following line to the settings.json file in VS Code
   "terminal.integrated.profiles.osx": { "zsh": { "path": "/bin/zsh", "args": ["-l", "-i"] } }
4. Add the below lines of code to your main.py file

import debugpy

debugpy.listen(("localhost", 5678))
debugpy.wait_for_client()

Run the debugger with the "Python: Remote Attach" configuration

Note: If the solution throws an exception and you receive errors that the port is already taken when you attempt to run the debugger you may want to check for a previously run and not shut down debugpy process and shut it down with the following commands:

ps aux | grep debugpy
kill PID of the debugpy process

Thank you to all people in this thread for providing their solutions which I based mine on.

[weijiany]

There is a way for intelliJ.

1. Using this command to start inspector NODE_OPTIONS='--inspect-brk' pulumi preview -s [your stack name]. The default server address is localhost:9229, you could find it in this document.
2. Opening your project via intelliJ. Following the config:
   
3. Adding your breakpoint and then ⌘ + D start your debug mode. 🎉

[sanmai-NL]

@swgillespie This issue/discussion would better be refined into issues/topics for specific languages.

[ami-descope]

There is a way for intelliJ.

1. Using this command to start inspector NODE_OPTIONS='--inspect-brk' pulumi preview -s [your stack name]. The default server address is localhost:9229, you could find it in this document.
2. Opening your project via intelliJ. Following the config:
   
3. Adding your breakpoint and then ⌘ + D start your debug mode. 🎉

@weijiany thank you so much!!!! this is life changing!

[tazik561]

1. NODE_OPTIONS='--inspect-brk' pulumi preview -s

How did you use ? In the first step when I write this command on terminal and tap to enter the pulumi starts to work! since there is problem in my pulumi codes after starting I will get error. I want to debug it but this guild does not work for me.
I am using dot net

[Frassle]

I am using dot net

NODE_OPTIONS is a nodejs specific option. The simplest thing I've seen for dotnet is to add a call to System.Diagnostics.Debugger.Launch() at the start of your program.

[tazik561]

I am using dot net

NODE_OPTIONS is a nodejs specific option. The simplest thing I've seen for dotnet is to add a call to System.Diagnostics.Debugger.Launch() at the start of your program.

Thanks for response , If I run it without pulumi I got this error :

Unhandled exception. System.InvalidOperationException: Program run without the Pulumi engine available; re-run using the `pulumi` CLI
   at Pulumi.Deployment..ctor(RunnerOptions runnerOptions)
   at Pulumi.Deployment.<>c__119`1.<RunAsync>b__119_0()
   at Pulumi.Deployment.CreateRunnerAndRunAsync(Func`1 deploymentFactory, Func`2 runAsync)
   at RR.ClientPortalApi.Infra.Program.<Main>()

How can I use with pulumi preview ? I mean If I run pulumi preview by terminal , How can I debug it ?

[Frassle]

Add a call to System.Diagnostics.Debugger.Launch() at the start of your program and then run pulumi preview.

pulumi will run the program with the environment needed for the sdk to work, and the Launch() method should trigger a debugger attach dialog when it's hit.

N.B.
You might need to use Break on linux instead of Launch.

[tazik561]

I am using mac, I was using both but not working :

    public class SharedStack : Stack
    {
#if DEBUG
        private const string BuildConfiguration = "Debug";
#else
        private const string BuildConfiguration = "Release";
#endif

        public SharedStack() => CreateResources();

    

        private void CreateResources()
        {
            System.Diagnostics.Debugger.Break();

[EronWright]

Summarizing how to attach across the SDKs, such that the program waits for the client to attach.

1. nodejs - use --inspect-brk flag when spawning node, attach by pid.
2. golang - use dlv exec command to run the program, attach by port. Build using debug flags.
3. python - use debugpy in entrypoint script to listen and wait, attach by port. Must install debugpy into venv.
4. csharp - poll System.Diagnostics.Debugger.IsAttached in entrypoint script to wait, attach by pid.
5. java - use jdwp JVM args, attach by port.