Skip to content

Multi core debugging

Haneef Mohammed edited this page Oct 13, 2022 · 35 revisions

Multi-core debugging

This feature is now available as of version 1.4.X

Cortex-Debug allows multi-core debugging with multiple VSCode debug sessions in one instance of VSCode. Multiple sessions can be independent of multi-core but from a look and feel they feel similar. Multiple sessions can also be started using a Launch Group or Compound Launch and they can work. There are a couple of issues with debugging the type of targets we deal with

  • All launches are done near simultaneously. Usually, there is a dependency but no dependency can be enforced (not easily anyway)
  • Information may need to be passed between instances about TCP ports to connect to, etc.
  • Synchronized Restart/Reset/Stop cannot be done

We addressed these issues using Chained Configurations. The theory is as follows. You will need a primary configuration (launch or attach type) that will start the GDB Server (OpenOCD, STLink-GdbServer, pyocd, JLinkGDBServer, etc.). The first instance in most cases (except JLink) creates TCP ports for multiple debuggers (gdb) to connect to; one for each processor core. In the case of JLink, one server is launched for each processor core. The primary configuration can optionally launch additional configurations. In launch.json, it can look something like

    // Following two are needed for cases where gdb-server is shared. Like OpenOCD and PyOCD.
    // Not for JLink and STLink where the sharing happens at the LINK level and not at the server level
    "numberOfProcessors": 3,
    "targetProcessor": 1,
    "chainedConfigurations": {
        "enabled": true,
        "waitOnEvent": "postInit",
        "detached": false,                         // Set this to true for JLink/STLink, false for most other type servers
        "lifecycleManagedByParent": true,
        "launches": [                              // Array of launches. Order is respected
            {
                "name": "Attach Network Core",     // Name of another configuration
                "folder": "${workspaceFolder}/../net"
            }
        ]
    }

Most of the properties are available at the top level and for each launch. When specified at the top level, they are inherited by all the launches and each launch item can provide overrides. There is nothing that stops a child configuration from having its own chained configurations. Hence the term chained, although that is not recommended or tested.

One of the design goals is to have each launch configuration be independent of each other as much as possible. You should be able to use any chained configuration in a standalone scenario. As such, certain items may collide when present in a chained situation. Multiple SWO/RTT configurations SVD file specification is another instance. Typically, there should be only one configuration taking this responsibility. We may have a facility to handle this in later revisions See Issue#593

Configurations are organized as a tree with the primary configuration at the root (more info below). They are typically related in some way. The child launches do not even have to be cortex-debug debug configurations. Thay can be any valid configuration specified in launch.json

If the top-level session is invoked without debugging, so will all the children.

Configuration properties

Please use InelliSense that is built into VSCode as an aid when typing. Following are the properties relevant to chained configurations.

Name Type Description
enabled boolean At the top level, disables all chained configurations. You can enable at the top level but disable it for a chained launch individually
waitOnEvent enum ["postInit" (default), "postStart"]. "postStart" means launch the configuration after the gdb-server has started and is ready for gdb connections. "postInit" means wait until all initialization commands have been completed -- generally, this also includes programming the device for one or more cores. "postInit" is a better option in most cases. With "postInit", (normally) the firmware is at the reset-handler and before even a single instruction is executed unless something is done in the "pre...", "override...", "post..." properties by your configuration. Note that "postInit" happens before "postStartSession" commands. "waitOnEvent" can be different for launches.
name string Exact name of the launch configuration. See folder below.
folder string Default: "". If not specified or an empty string, it means the current folder but it can be another folder in the workspace. You can either use the full-path name or relative path name of the folder as it appears in the .code-workspace file. Case matters. And, it has to be a known VSCode workspace folder and not just any ordinary folder
delayMs number Number of milliseconds to wait to launch this configuration after the waitOnEvent event. There is naturally a 5-millisecond delay between launches but you can add to that. The delays are additive in the chain.
detached boolean If 'true' means that the chained launch configuration will have its own server and not shared with the parent. This is typical for 'JLink' gdb-servers. For servers like OpenOCD, pyOCD, STLink-GdbServer, this should be false especially for multi-core devices. Unless specified, this is automatically done for you.
lifecycleManagedByParent boolean See Life cycle management section below.
launches array Array of launch objects where each launch object describes a chained launch. Applies to top-level only.

The properties folder and name are critical to launching another session. The folder should point to a directory to a directory where there is a <folder>/.vscode/launch.json file which contains a configuration name. Leaving folder empty means using the current launch.json file. There is no error checking possible until run-time.

Life cycle management

This affects actions like Restart, Reset, Stop, and Disconnect.

All chained launch configurations are maintained as a tree that represents a grouping and parent-child relationship.

Note: Each node in the tree can have its own setting for lifecycleManagedByParent. Even when certain nodes are detached, there is still only one root internally which is the primary launch configuration. Hopefully, what we got works for your situation. Visually, in the CALL stack, you can see lifecycle dependency with managed children indented a bit whereas non-managed items appear as items at the top-level -- even if all of them were launched by the same chained launches and some sessions may or may not share a server.

"lifecycleManagedByParent": false

When lifecycleManagedByParent is false, this item is independently controlled. There are still things that can affect that session. When you Stop or Disconnect the (primary) session that launched the server, the server is killed (by us). So, if there are any children still alive, they will die as well (eventually) as they will lose their connection to the HW. Normally, it is okay to Stop or Disconnect a configuration and the others will keep running until their server exits. The life-cycle dependency can be seen visually in the CALL STACK window as dependent ones are indented a bit.

"lifecycleManagedByParent": true

This is mostly managed by VSCode itself. Note that VSCode controls what happens when VSCode native buttons, commands, and keyboard shortcuts are invoked. When this is true, such global commands are directed to the root of the tree by VSCode automatically and then disseminated to each of the participating children in a top-down fashion. For Reset and Restart you must have proper customizations in your launch.json to make them do the right/expected thing. The defaults may work.

Life cycle management is independent of the detached property. The detached property simply says if the session is going to share the parent's server or start its own server. In the case of JLink, all items may be detached from each other but you can still choose how their lice cycles are managed.

So, internally there are two trees of sessions. One is organized by life-cycle management (this is what is visible in the CALL STACK window) and one is organized by who launched what. You will never see the latter in the UI.

Terminating a session

There are two terms of importance:

  • Stop: For native programs, this means killing the program. For us, it means to terminate the debug session, leave the target in an arbitrary state (depends on your gdb-server), generally halted.
  • Disconnect: It means to leave the program running. Not every gdb-server does this right. They may do the same thing as Stop or worse. Note that in the gdb world, this is called Detach. It is the opposite of attach.

Usually, there is not a problem with terminating a child (leaf) session where the child either disconnects or stops it is all good. But when a parent is asked to end a session, the parent (root) has to send a request to all its lifecycle-dependent children to terminate. Children are independent of each other running in different processes and VSCode coordinates some of that. The only API we have is to also ask the child to terminate but the API cannot specify which method (Stop or Disconnect). This may change in the future as we will make that request with Microsoft. Our current plan is to send the same terminate request-type to all the children using a side-channel API. Not sure of side-effects

Pause/Resume/Breakpoints/Watchpoints

These are NOT considered global events/actions. You can use the Cross Trigger Interface/Matrix (CTI/CTM) to configure a synchronized halt.

Windows management

  • Variables, Watch, and Stack windows should work normally and as expected. In the Stack Window though there will be one top-level item per configuration. The child configurations will be nested from its parent. We are still experimenting with this and are not sure what is possible with VSCode yet. This is something we have minimal control over.

  • Debug Console: There will be one "Debug Console" for each session. In the "Debug Console", there is a drop-down list of sessions.

  • gdb-server: There will be one tab per gdb-server launched in the chain. This is also true for multi-session cases where each session that launched a gdb-server has a console. We will not have one tab per child when the server is shared. This affects things like semi-hosting which can be confusing in a multi-core environment. Additional gdb-servers will be named gdb-server-# where # is a number. We try to preserve the output from old sessions and re-use the tabs from the old sessions.

  • Registers: All the sessions are presented in the Registers window in a Tree view. You can only see the registers from the last breakpoint or pause for a given session.

  • Peripherals: Each launch configuration can have an (optional) SVD file provided and the will all show up in the Peripheral Panel. Normally, in multi-core debug scenarios, you should have just one SVD file for all the sessions but this design allows even debugging different processors (from different vendors) all at the same time. We do not check to see if the SVD files you have provided are duplicates. Note that no updates occur if the Peripheral Tree is collapsed, but it does slow down the startup because SVD files are generally large. You may not see a visible slowdown as SVD files are read in a way that does not affect significantly the main debugging process.

    There are instances where some peripherals may only be accessible from a given core. There are also instances where the same block name is in actuality a different instance and may even have the same base address. This is common with CoreSight (Debug) components, System Control Blocks, etc. We cannot determine which component is redundant and which is not. Note that you can only look at peripherals where the respective core is in a halted state.

  • RTT & SWO: RTT and SWO are session aware. i.e., each session can have such a thing or not. Normally, there is one instance of RTT and SWO for the entire processor. But having multiple processors with their own RTT/SWO can also work. On a single device, please make sure that SWO/RTT are handled just by one session in the group...there is nothing in the extension that prevents it if your gdb-server and device can support that.

ST-Link notes

STMicroelectronics has a couple of dual-core micocontrollers in its portfolio. Below is an example of using Cortex-Debug to work on an STM32WL55 which has a Cortex-M4 and a Cortex-M0+. Following instructions should be valid for STM32WB series and maybe other dual core MCUs from STMicro.

The systems starts with the startup of the Cortex-M4. The firmware on the Cortex-M4 then enables some flags to wake up the Cortex-M0+.

To use Cortex-Debug, three launch configs can be used : one for the Cortex-M4, one for the Cortex-M0+, and one that uses chainedConfigurations to enable debugging both cores at the same time.

Debugging the Cortex-M0+ can only be done in attach mode because of the way the core is woken up by the Cortex-M4.

When debugging the Cortex-M4, it is recommended to also flash the firmware of the Cortex-M0+. This is done below using loadFiles.

Here is an example of launch.json :

        {
            "name": "Debug CM4 - ST-Link",
            "cwd": "${workspaceFolder}",
            "type": "cortex-debug",
            "executable": "${workspaceFolder}/build/project_CM4",
            "loadFiles": [
                "${workspaceFolder}/build/project_CM4",
                "${workspaceFolder}/build/project_CM0PLUS",
            ],
            "request": "launch",        
            "servertype": "stlink",
            "device": "STM32WL55JC",
            "interface": "swd",
            "runToEntryPoint": "main",
            "svdFile": "${workspaceFolder}/STM32WL5x_CM4.svd",
            "v1": false,
            "showDevDebugOutput": "both",
            "serverArgs": [
                "-l", "1",
                "-m", "0",
                "-k",
                "-t","-s"
            ],
        },
        {
            /* In this launch config, the CM4 debug is started,
            then we wait 10s for the M0 to boot. After than,
            a launch config to attach to the CM0PLUS is started.
            */
            "name": "Debug CM4+CM0 - ST-Link",
            "cwd": "${workspaceFolder}",
            "type": "cortex-debug",
            "executable": "${workspaceFolder}/build/project_CM4",
            "loadFiles": [
                "${workspaceFolder}/build/project_CM4",
                "${workspaceFolder}/build/project_CM0PLUS",
            ],
            "request": "launch",
            "servertype": "stlink",
            "device": "STM32WL55JC",
            "interface": "swd",
            "serialNumber": "",
            // "runToEntryPoint": "main",       // No run to main to let the CM4 reach the start point of the CM0PLUS
            "svdFile": "${workspaceFolder}/STM32WL5x_CM4.svd",
            "v1": false,
            "showDevDebugOutput": "both",
            "serverArgs": [
                "-l", "1",
                "-m", "0",
                "-k",
                "-t","-s"
            ],

            "chainedConfigurations": {
                "enabled": true,
                "waitOnEvent": "postInit",
                "detached": true,
                "delayMs":5000,    // Wait 5s, could be optimized
                "lifecycleManagedByParent": true,   
                "launches": [ 
                    {
                        "name": "Attach CM0 - ST-Link",
                        "folder": "${workspaceFolder}"
                    }
                ]
            }
        },
        {
            "name": "Attach CM0 - ST-Link",
            "cwd": "${workspaceFolder}",    
            "type": "cortex-debug",
            "executable": "${workspaceFolder}/build/project_CM0PLUS",
            "request": "attach", 
            "servertype": "stlink",
            "device": "STM32WL55JC",
            "interface": "swd",
            "serialNumber": "",
            "svdFile": "${workspaceFolder}/STM32WL5x_CM0P.svd",
            "v1": false,
            "showDevDebugOutput": "both",
            "serverArgs": [
                "-l", "1",
                "-m", "1",
                "-t", "-s",
            ],
        }

Several things are important to take from this launch.json example:

  • -m option is used to indicate which Core the debugger will be connected
  • -t indicate that the ST-LINK probe is shared
  • The delay for the chained configuration. You can also start the M4 launch, click continue, and then start the M0 launch. You can try optimizing the delay. If you get Target unknown error 32 on the launch of the second gdbserver, it means the Cortex-M4 did not have time to start the Cortex-M0+.
  • Only the Cortex-M4 uses the option -k (connect under reset) because of the startup process of the Cortex-M0+
  • Connecting to the Cortex-M0+ is always done in attach mode

Other notes

While some gdb-servers are shared and others are separate, we have chosen a design to use separate GDB instances for each session. While it may be possible to use one GDB instance to debug multiple targets, it is confusing and hard to manage. This design aspect is unlikely to change.