How to Debug when we have multiple cores (OpenOCD or PyOCD)

[haneefdm]

We have a chip with two or more cores. The first core is a CM0 and the second one is a CM4. This causes a big problem as 99% of the time, we have to debug the CM4. Problem is with tcp port selection. Both PyOCD and OpenOCD, will allocate the specified tcp port (or default) to the first processor and then start incrementing the port number. With other debuggers, we ask the server to start at 'x' and ask the debugger to connect to 'x+1'. Since Cortex-Debug does the port selection (find free ports, which is nice), it is hardcoded to use that port.

Is there a workaround? My current workaround is to edit the extension :-)

Potential Fixes?

We need to know the number of processors and a processor offset to debug. Also, where we need portastic.find() to find the required number of ports. Then we have to use the ports in the way, the gdbserver allocates, in this case (PyOCD and OpenOCD), that is simple.

Another way perhaps is to run the gdbserver externally and have Cortex-Debug use a port that is given rather than hard coded. This has the advantage of running one server and attaching two Cortex-Debug sessions, to debug both processors at the same time. This could fix Issue #36 as well.

Both ways?

Please see the output of OpenOCD below. I highlighted below lines that show that our extension is connected to the wrong processor.

Open On-Chip Debugger 0.10.0+dev-2.2.0.22 (2019-03-22-11:25)
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
adapter speed: 1500 kHz
adapter speed: 1000 kHz
** Auto-acquire enabled, use "set ENABLE_ACQUIRE 0" to disable
cortex_m reset_config sysresetreq
cortex_m reset_config vectreset
adapter_nsrst_delay: 200
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Started by VSCode
Info : CMSIS-DAP: SWD Supported
Info : CMSIS-DAP: FW Version = 1.2.0
Info : CMSIS-DAP: Interface Initialised (SWD)
Info : SWCLK/TCK = 1 SWDIO/TMS = 1 TDI = 0 TDO = 0 nTRST = 0 nRESET = 1
Info : CMSIS-DAP: Interface ready
Info : VTarget = 3.293 V
Info : kitprog3: acquiring PSoC device...
Info : clock speed 1000 kHz
Info : SWD DPIDR 0x6ba02477
Info : psoc6.cpu.cm0: hardware has 4 breakpoints, 2 watchpoints
Info : psoc6.cpu.cm0: external reset detected

---

** Silicon: 0xE206, Family: 0x000, Rev.: 0x00 (**)
** Detected Device: CY8C6247BZI-D54
** Detected Main Flash size, kb: 1024
** Flash Boot version 1.01
** Chip Protection: VIRGIN

---

Info : psoc6.cpu.cm4: hardware has 6 breakpoints, 4 watchpoints
Info : psoc6.cpu.cm4: external reset detected
Info : Listening on port 50000 for gdb connections
Info : Listening on port 50001 for gdb connections
Info : accepting 'gdb' connection on tcp/50000
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0x00001f2c msp: 0x080477a8
Info : New GDB Connection: 1, Target psoc6.cpu.cm0, state: halted
Warn : Only resetting the Cortex-M core, use a reset-init event handler to reset any peripherals or configure hardware srst support.
Info : kitprog3: acquiring PSoC device...
target halted due to debug-request, current mode: Thread

[haneefdm]

This whole port selection is complicated and makes it even worse by having multiple CPUs in the same chip.

I found a workaround for OpenOCD, but it is not pretty. In the chain, if you have multiple cpus (in my case two, but could be more). you can do the following (I am going to use my example) in your config tcl file. You can have more than one config file, btw.

# Redirect cpus that you are not interested in to some other port. In this case, I have two
# and I want to debug the psoc6.cpu.cm4 which is not the first in the chain. Assign them far
# away from the cpy that you are interested
psoc6.cpu.cm0 configure -gdb-port 60000
#add more if needed XXX configure -gdb-port 60000

the cpu names are assigned by the vendor they don't have to look like xxx.cpu.yyy. They can be anything. You can find out the cpu (target) names by doing the following. or look at the output of OpenOCD carefully

foreach t [target names] {
    puts "Target: $t"
}

A bit more fancy/generic stuff, still not perfect though

set junk_port 60000
set my_cpu "psoc6.cpu.cm4";  # This is what I want to debug
foreach t [target names] {
    if {[string compare $t $my_cpu] != 0} {
        $t configure -gdb-port $junk_port
        puts "My Msg: CPU $t should now be using gdb port $junk_port"
        incr junk_port
    }
}

Now, OpenOCD will just use what ideal port Cortex-Debug deems is good for the cpu you are debugging.

Can this fail, yes. If whatever junk_port we chose happens to used by something else. Try another junk_port.

Until @Marus or someone else figures out a better fix, these are hints for you to use.

[Marus]

@haneefdm - The v0.2.7 has finally been released - my client issue ended up taking a lot longer than I expected to fully deal with because of things outside my control so didn't get back to that as quickly as I would have liked.

This adds a new server type external - you need to specify the gdbTarget launch.json option; this should allow you to at least work around this by manually running your OpenOCD session and then specifying with GDB port to connect to; not intended for most cases, but could help for some strange cases like this, at least until we have proper multiple target support.

You may also need to play with the overrideAttachCommands, overrideLaunchCommands and overrideRestartCommands; although I do think that the defaults there will work for OpenOCD in most cases (people trying to use different external GDB servers will need those more).

[haneefdm]

@Marus, Yes, the external server type works, but it is tedious. I am working on a fix for this. From launch.json, I need to know how many processors and which processor in the jtag chain, they want to debug. This will allocate the right number of TCP ports and will make gdb use the correct tcp port. I can test with pyocd and openocd. With openocd, I have been using tcl workarounds but I don't see a workaround for pyocd. The same goes for swoPort

If the user wants to debug more than one processor simultaneously, then the external is the only viable method at least for all but one processor.

Thoughts?

[haneefdm]

For what it is worth. You can do the following to disable a gdb port for a processor.

psoc6.cpu.cm0 configure -gdb-port disabled

[Marus]

Yeah, I’m still trying to come up with a good solution here; ideally you’d actually be able to have multiple simultaneous debug sessions that you’d launch together (sort of like the debug adapter + extension launch configuration we have when developing the extension). The issue is the two debug adapters would run individually and not have any knowledge about what the other is doing in terms of the GDB server and flashing the device and what-not.

My vague idea - but I haven’t really thought it through enough to have a clear plan yet - is to move the responsibilities for launching the GDB server into the extension, while the debug adapter remains responsible for the GDB session only. That way you’d have some way to tie the two configurations together and the extension could determine if there is a pre-existing GDB server to use (and if so what ports and stuff to use) or if it needs to launch a new server. But then you of course need to have some way to tie them together in the configurations and be able to determine when you should close the server and other things.

This would come with a lot of coordination, in particular could be problematic when it comes to the flashing and what is responsible for that. That could get really tricky in the multi-core scenario - since they have shared memory and peripherals you need to make sure that the second to launch doesn’t flash the memory disrupting the other core (I think you pretty much always have to have all but one of them strictly be an attach type vs launch type - but even that could be difficult as you may not be able to guarantee that the “launch” type configuration launches first if trying to debug both cores simultaneously.)

I think what I really need to do is get something setup with one of the multi-core devices that you had Alan send me so I can get a better feel for using it and the process.

[haneefdm]

I had similar thoughts and reservations. But I finally have a simple enough implementation that works for a more common use case. First, the use case. Most of the time you are only debugging a program in one core. But, this may not be the first core in the chain. Let us call this the primary core.

• In the launch.json for the primary core where you tell us how many cores you got and which core you want to debug. This is important because then we can allocate the proper number of gdb/swo/tcl TCP ports and launch the gdb-server correctly. GDB, however, is asked to connect to the right target core.
• If you want to debug another core, you use the external server method. You will not be able to use the SWO port this way but we can add that later. You can do this without ever having to start a server manually. But, when the primary debug session ends, so will the others.

I have this working now. Required very few changes, all localized. One issue I came up with is that we may want to allocate ports that we do not even need. I ran into this with pyocd. We only know about the gdb-port. I think we also need to allocate the telnet-port. We do this for JLInk; we never use the 'console-port' but we allocate anyways. I will make this change.

Here is an example launch.json for mbedOS where I am debugging the second processor in the chain.

        {
            "cwd": "${workspaceRoot}",
            "executable": "${workspaceRoot}/BUILD/${workspaceRootFolderName}.elf",
            "name": "Cortex-Debug pyocd",
            "request": "launch",
            "type": "cortex-debug",
            "servertype": "pyocd",
            "runToMain": true,
            "debugServer": 4711,
            "numberOfProcessors": 2,
            "currentProcessor": 1,
        },

[Marus]

Yeah, I think this is a good interim solution for sure. that would cover at least a good portions of the use cases; and certainly a lot easier to implement than trying to move the GDBserver management up to the extension level.

[Marus]

Although I’m not sure I like the property name “currentProcessor” - I’m thinking maybe “targetProcessor” is a better term.

[haneefdm]

Yup. targetProcessor is much better

[haneefdm]

I found that with pyocd the existing targetId option in launch.json, you can specify which target you want to debug. Hard to figure out what the value should be. But, it takes an eternity (almost two minutes), before it will succeed and a debug session starts. Setting the explicit debug chain options takes 1-2 seconds (not including programming time) to get started.

I have not found a similar option with openocd. So, I am thinking, getting specific about the debug chain is still a good (not great) idea.

[haneefdm]

With the latest beta release, this is addressed. See changelog

https://github.com/Marus/cortex-debug/releases/tag/v0.3.5-beta1