Document an approach to debugging FastCS IOCs in cluster.

[gilesknap]

Document an approach to setting breakpoints and running a python debugger against a fast CS IOC deployed into a cluster.

• should not require re-deploying but enable debugging in place production IOC
• may require changes to helm charts, container images etc.

[gilesknap]

First up, this works really really well:

# setup a debugger ephemeral container against a traditional IOC
module load pollux
kubectl debug -n p47-beamline bl47p-mo-sim-01-0 --profile general -c debugger --image=ghcr.io/epics-containers/epics-base-developer:7.0.9ec4 --target bl47p-mo-sim-01 -- bash -c "sleep infinity"

# get a shell in the debugger
kubectl exec -tin p47-beamline bl47p-mo-sim-01-0 -c debugger -- bash

Now we are in a neworkHost=true pod but running as root with all of the epics tools, compilers and uv. Our processes are running as root but the original IOC container is running as p47-detector. We are sharing the process namespace with the IOC container:

root ➜ / 
$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
37887        1  0.0  0.0   4444  1804 pts/0    Ss+  12:06   0:00 bash /epics/ioc/start.sh
37887        7  0.0  0.1 129228 26104 pts/0    Sl+  12:06   0:00 /venv/bin/python /venv/bin/stdio-expose /epics/ioc/start.sh
37887       10  0.0  0.0   2788   516 pts/0    S+   12:06   0:00 /bin/sh -c /epics/ioc/start.sh
37887       12  0.2  0.0 1911308 9108 pts/0    Sl+  12:06   0:01 /epics/ioc/bin/linux-x86_64/ioc /epics/runtime/st.cmd
root        71  0.0  0.0   2684   496 ?        Ss   12:06   0:00 sleep infinity
root       333  0.0  0.0   5764  3684 pts/0    Ss   12:13   0:00 bash
root       650  0.0  0.0   7876  2412 pts/0    R+   12:18   0:00 ps aux

Adding extra tools on the fly is also easy:

$ uv pip install copier
Using Python 3.13.5 environment at: /venv
Audited 1 package in 3ms
root ➜ / 
root ➜ / 
$ apt update &> /dev/null; apt install vim
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
...
The following NEW packages will be installed:
  libgpm2 libsodium23 vim vim-common vim-runtime xxd
0 upgraded, 6 newly installed, 0 to remove and 10 not upgraded.
Need to get 9786 kB of archives.
After this operation, 44.1 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
...

[gilesknap]

I'm coming around to the idea of installing gdb and debugpy into production containers. Then we can exec debugpy in the runtime container as needed.

For now I'm going to pause this and move on to dev in cluster to see if that shakes out any improvements to this approach.