Debug ports should be configurable

[cpvandehey]

Expected behavior

I am using python and the MS ptvsd package. It works well with skaffold and VSCode, but it does not scale well in a platform setting with regards to debugging multiple services at once. skaffold is smart enough to dynamically find different ports when a developer launches more than once project, but the port selected for debugging should be configurable to keep consistency.

Here is the code that selects a default port 5678 if no ptvsd command already exists. The skaffold configuration should be setup to specify ports on a per project basis that remain CONSISTENT. In the end, a developer should be able to set up a debug configuration once within a project and never have to change it based on how many other services are running in parallel.

• Skaffold version: version: latest
• Operating system: mac
• Contents of skaffold.yaml: n/a

[briandealwis]

Thanks for the feedback @cpvandehey! How were you thinking this configuration would look like? What should happen if the debug port is unavailable? Should debug abort, or select a random port?

[cpvandehey]

How were you thinking this configuration would look like?

In the skaffold.yaml docs there is a section called portForward. I assume that a section right below this called debugPortForward would be intuitive and easy.

What should happen if the debug port is unavailable? Should debug abort, or select a random port?

Yeah, I have some opinions here. Most apps that allow specific port selection error out when another process is hanging onto the port. I suggest following that pattern.

BTW, I can certainly help contribute, but I am not sure if I am "allowed" to yet by my employer and the google container tools crew. You all made slick tools and I'd love to help out/write some go for once.

[briandealwis]

@cpvandehey Have you tried adding an explicit portForward for these debug ports? I can't think of any reason why this wouldn't work. Skaffold uses kubectl port-forward under the hood and the actual connection is established on-demand.

[cpvandehey]

This does work, but skaffold debug needs to pass these ports in the start command as well!

This is what happens in the python command:
"python". "-m", "ptvsd", "--listen", "0.0.0.0:SOMEDEBUGPORT"

[briandealwis]

debug tries to use the same debug ports as is normally used by the language runtime support:

• go/dlv: port 56268
• jvm: port 5005
• nodejs: port 9229
• python/ptvsd: 5678

A different port will only be used if there is another container in the same pod using the port, and debug uses a stable algorithm to find the next port.

[cpvandehey]

I agree with your analysis. This is what I believe should be configurable. The defaults are fine when you work with one project and one project only. It does not scale well when you'd like each project to have an independent and specific debug port that you can configure VScode to use within each repo (potentially even committing the .vscode/ directory to make it even more standard for teammates).

[cpvandehey]

Please confirm if the issue makes sense. I can certainly provide more specific examples

[briandealwis]

I'm reluctant to introduce a debug-specific configuration option when there is a mechanism that already works.

What would make more sense to me is to add support for portForward to target a container-port by name or port number. The debug ports are all named by the underlying debug protocol (jdwp, dab, devtools, dlv).

BTW: are you using Cloud Code for VSCode? It uses Skaffold under the hood and leverages Skaffold's APIs to automatically connect debug sessions.

[cpvandehey]

Perhaps I'm not making a clear enough point or I am misunderstanding how this works. Let me be a tad more verbose. Thanks for your patience and willingness to understand this!

when there is a mechanism that already works

You're 100% right that it works, but the DX is not quite there. It works well with one project running in debug mode. You can always expect it to claim a port on the running pod, forward it to local host on port 5678 (or find a nearby port that is open) and you can configure a launch.json setting in the project to attach a remote debugger on the forwarded port. After the appropriate networking is configured, Skaffold beautifully injects some flags and configuration for ptvsd to the python launch script on your pod (specifically "-m", "ptvsd", "--listen", "0.0.0.0:5678").

Again, I am happy to know that this reliably works and I can launch more projects that have dynamic port allocation, but this is a bad DX when I have many projects that should have consistent configuration.

Suppose I have 10 microservices to run/debug locally that are going to use skaffold. Instead of having specifically defined ports that each use on every launch regardless of the order, I have to accept the reality that the debug ports can change based on the order in which I launch the projects (which eventually leads to more manual configuration at the time of debugging).

For ex:

• I launch project A in debug mode. Project A forwards a debug port to 5678 on my localhost and I manually configure my launch.json to attach to 5678.
• I launch project B in debug mode. Project B forwards a debug port to 5679 on my localhost and I manually configure my launch.json to attach to 5679.
• I tear down all projects and launch in the reverse order.
• I have to manually map the debug ports in both projects's launch.json files again.

With regards to your suggestions on using portForward to define specific debug ports:
This gets developers about halfway through the steps necessary to set up a debugger. The most crucially forgotten step is the configuration and flag injection to set up ptvsd (specifically "-m", "ptvsd", "--listen", "0.0.0.0:PORT"). While I could manually modify the start up command whenever I want to debug a service, it would make far more sense for skaffold to simply inject which port I want to debug with in the ptvsd startup command (the same way it does so with a dynamically allocated port).

--

I can try out Cloud Code to see if that abstracts away all of this. It looks really useful (especially because its supported by all popular IDEs for a super flexible DX).

[briandealwis]

Just to be clear, my suggestion is this:

1. Assuming the general practice for structuring Kubernetes applications, you have your containers in separate pods (more on multiple containers in a pod below, however). Since each pod has an independent port-space, debug will allocate a port, which is usually port 5678 for python.
2. For each pod, you add portForward: definition in your skaffold.yaml that maps port 5678 to your desired local port for that pod/container.

For example, with 3 containers in podA, podB, podC, your skaffold.yaml will have:

    portForward:
    - resourceType: pod
      resourceName: podA
      port: 5678
      localPort: 7000
    - resourceType: pod
      resourceName: podB
      port: 5678
      localPort: 7001
    - resourceType: pod
      resourceName: podC
      port: 5678
      localPort: 7002

You then set up your launch.json to use those local ports (7000 – 7002). Note that the remote port for each pod is fixed 5678. Since you control the pod names (podA, podB, podC) then you choose how to map projects to local ports, and so you maintain the correlation to your launch.json. If you have multiple skaffold.yamls, allocate entirely separate port ranges. Then you can launch them independently and in different orders.

Two notes to the above:

1. debug chooses a default port for each language runtime providing that port is not already allocated elsewhere in the pod. But debug also attempts to parse the container command-line to determine if a port was already configured. I should teach debug to also attempt to search for an existing containerPort on the container definition.
2. A pod with several debuggable containers should have a stable allocation as the list of containers is an array and each container is considered in turn. So container[0] will be allocated port 5678, container[1] port 5679, etc.

So if this doesn't work for you, I'd appreciate a sketch of what your application looks like in terms of k8s resources and your containers.

I assume that a section right below this called debugPortForward would be intuitive and easy.

To the contrary, it would require significant effort and change within Skaffold. Our focus is instead to surface debugging information to be used by tools like Cloud Code to spare humans from managing this port mapping toil. And note that when a container is redeployed (e.g., you use a deployment, and the original pod is terminated), the old container continues to occupy the original local debug port until it is terminated, and so the new container will not be mapped to the original local port.

[cpvandehey]

Ah. This example makes sense. Thank you for writing that out and for your patience!

--

I just finished watching a cloud code demo at NEXT. I'm leaning towards just setting it up now.

[cpvandehey]

We can close this as there are multiple workarounds. Thanks again