How to debug cmdline scripts with options used in a sequence of piped cmds with VS Code

[udgithub]

Hello all!

Perhaps you can help me cause I don't knwo how I can debug something like
"cat myfile | python-script-1.py --option1=x | python-script-2.py --option2=y "?
I tried to use debugpy too. But I have to call both scripts like so
"python -m debugpy --listen 5678 --wait-for-client python-script-1.py --option1=x"
"python -m debugpy --listen 5679 --wait-for-client python-script-2.py --option2=y"
and I don't know what to do if I want to debug both scripts at once. Perhaps there is a way to bring vs code into listen mode and connect to it from the cmdline scripts? This is the way I was used to work with when remote debugging PHP with xdebug.

Thanks a lot in advance.

Uwe

[int19h]

There is indeed a way to have VSCode listen for connections and for the clients to connect - use "listen" in your launch.json and debugpy --connect when launching your app (or debugpy.connect() if you prefer to do it from code). But I don't think that'll help here because each time you do launch/F5, it's still a single debug session, so as soon as there's an incoming connection, the session is considered running and further connections are rejected. There's no way to just keep it running and accept more connections and spin up new debug sessions for each.

However, you should be able to debug both scripts at once using your original setup if you use two separate debug configurations (one for each port) and attach to both processes sequentially. That is, you'd need to select your first debug config, do F5 on that (thus attaching to the first process), then switch debug config to the other one, and do F5 again.

Another possibility would be to try to utilize subprocess debugging support. The problem there is that it needs to somehow know that you're starting a Python subprocess in order to intercept the command line and insert debugpy there. It does that for all common ways of launching processes, including low-level APIs like calling CreateProcess via ctypes. But it can't do that for os.system that you'd have to use here to make piping work, because all the syntax there is inherently shell-specific. So it would require you to write a Python script that does the equivalent of your pipeline, but using the Python stdlib (e.g. process.Popen) - then it should be able to do everything automatically.

[udgithub]

Hello Pavel,

thanks a lot for this fast and helpful response. Great !!

I was already thinking about this way to use several debug configs (one per port).
But at the end I decided to write better tests per script so that it should be good enough to debug one functionality after the other.
But I will try the listen/connect way. I played around with it but it didn't work, cause I did not call the debugpy.connect() from within my script.

An hour later: I just tried it with this launch.json config:

        {
            "name": "Python: Listen 5678",
            "type": "python",
            "request": "attach",
            "listen": {
                "host": "localhost",
                "port": 5678
            },
        },

This works for doing something like this.
"cat myfile | python -m debugpy --connect 5678 ./python-script-1.py --option1=x | ./python-script-2.py --option2=y "
or
"cat myfile | ./python-script-1.py --option1=x | python -m debugpy --connect 5678 python-script-2.py --option2=y "

But on my first try I could not debug both like this:
"cat myfile | python -m debugpy --connect 5678 ./python-script-1.py --option1=x | python -m debugpy --connect 5678 python-script-2.py --option2=y "

But now I got it.
I use two port 5678 and 5679
"cat myfile | python -m debugpy --connect 5678 ./python-script-1.py --option1=x | python -m debugpy --connect 5679 python-script-2.py --option2=y "
and I added a second launch.json config "listen 5679" like the one above but for port 5679.

And I could manage to start two debug sessions in one running VS code instance like this:

1. select "listen 5678" in the debug pull down menu (the one with the green triangle)
2. Press F5 to start the first debug session
3. select "listen 5679" in the debug pull down menu.
4. Pressing F5 twice did not work for me but when I tried the "Start debugging" from the "Run" menu this worked and I got a request from VS code if I want to start a second debug session.

I wasn't aware that this is possible in VS code.
So, thanks again for showing up this solution and have a nice day.

Ah, there is one open question to me:
Why is it not possible to use port 5678 twice for debugging?
I opened a second debug session within VS code using port 5678.
But the first breakpoint did only work for the second script in the sequence of calls in the pipe.

Best regards, Uwe

[int19h]

The problem is that at the moment, when you press F5, this always corresponds to a single debug session with possible child subprocesses. So if your main script spins up another Python instance, that will actually connect to the same port for debugging, and things will work - but it works because it reports itself as a subprocess of the main process. If a new connection comes in on the same port but is not a related process, the debug adapter rejects it. Fundamentally this is because the current implementation of adapter assumes that there's only one root process in the session, and some of its logic would break if that is not the case.

But in principle, we could implement multi-rooted process trees, and then this would "just work", so please file an enhancement request.

[udgithub]

Thanks - the feature request has been made: #1501