-
Notifications
You must be signed in to change notification settings - Fork 101
Debugging CUBE Containerized Services
CUBE is a complex software infrastructure, comprising the main core backend, as well as a constellation of ancillary services. Usually, if things break, the first place to start debugging is to make sure the ancillary services are all OK. This page documents a workflow that can be useful in debugging these CUBE services pfcon, pfioh, and pman within their respective containers.
Due to the distributed and containerized nature of CUBE, debugging the ancillary services pfcon, pfioh, and pman can be particularly difficult. An initial strategy is to first debug these services outside of ChRIS in a non-containerized setup. If all works "outside" of ChRIS, then a next step is running the services in containers and talking to them from another container, and finally, calling the services via CUBE.
The workflow is to start the whole CUBE system containerized. Then, once all the services are up, to pause and take down each service in turn and restart in an interactive mode. This allows for much more responsive console response, and, if a source file mapping is effected into the container (see below), allows for interactive debugging.
In order to fully debug, the source code of the particular service needs to be volume mapped into its respective container in the correct place. This necessitates small transitory changes to the docker-compose.yml.
Let's assume we want to debug the pman source code in a running active pman container. Further, let's assume we have checked out the pman repository at the same tree level as the ChRIS_ultron_backend repo. Now, in the docker-compose.yml add the following line to the volumes: section of the pman_service:
- ../pman/pman/pman.py:/usr/local/lib/python3.6/dist-packages/pman/pman.pyRepeat for other services if/as necessary
- ../pfioh/pfioh/pfioh.py:/usr/lib/python3.6/site-packages/pfioh/pfioh.py - ../pfcon/pfcon/pfcon.py:/usr/local/lib/python3.6/dist-packages/pfcon/pfcon.pyNote that though these service comprise many files (as witnessed in their respective repos), for the overwhelming majority of debug cases you will be interested in only one file per service as shown above.
It is important to meaningfully pause/stop the flow of setup execution in CUBE. This should happen after the initial containerized services have all been launched but before real processing has started. In the current CUBE, this is best effected by starting the management script with a -p flag,
*make* -p
and when execution pauses, to remove and restart the relevant services (see below).
Additionally. add a breakpoint to charm.py with a pudb.set_trace() to a relevant line of code. At time of writing (May 2019), a good place to "break" is in the app_service_call() method of charm.py.
If tests run fine, but subsequent use cases, fail, then simply wait until CUBE has been fully instantiated.
From the CUBE repo dir, run (add a -d for debugging verbosity)
*make*or, if you have local docker images:
*make* localThe simplest way to debug is to open four terminals, say in a grid. In each terminal cd to the CUBE source repo.
Each terminal will be used to run a specific service.
| Terminal 1 | Terminal 2 |
|---|---|
cd <CUBErepo> |
cd <CUBErepo> |
| Terminal 2 | Terminal 3 |
|---|---|
cd <CUBErepo> |
cd <CUBErepo> |
Now, do
| Terminal 1 | Terminal 2 |
|---|---|
*destroy* ; |
|
sudo rm -fr FS ; |
|
rm -fr FS ; |
|
*make* -d -p local |
(nothing) |
| Terminal 2 | Terminal 3 |
|---|---|
| (nothing) | (nothing) |
for ease of copy/paste, the command in terminal 1 is
*destroy* ; sudo rm -fr FS ; rm -fr FS ; *make* -d -p localOnce execution pauses at the breakpoint, stop the target services and restart in an interactive manner. This keeps the terminal output much more current.
There is stop/restart order dependency. First pfcon, then pfioh and pman in any order (or even together).
The docker make file, docker-make-chris_dev.sh provides a restart feature for a given service. For services, see below. Also, the local in the below is only for cases where you have actually built local docker images (see the Dockerfile) of each repo.
In many cases, esp with a source file mapping, the local is not needed.
Now, do
export HOST_IP=$(ip route | grep -v docker | awk '{if(NF==11) print $9}')The idea behind this debugging approach is to first start services with docker-compose and then restart then in interactive mode. Doing this, however, breaks the management of docker-compose and requires some tweaking. One of these tweaks is to add an entry in the /etc/hosts of the chris_dev_backend container for a host pfcon_service. For example,
10.23.130.106 pfcon_servicewhere the IP must be the host IP of the machine you are running on. If you are using the standard FNNDSC bash environment with appropriate aliases, you could do (use appropriate IP for your host):
dkee chris_dev "echo '${HOST_IP} pfcon_service' >> /etc/hosts"
dkee chris_dev "cat /etc/hosts"where
alias dkl='docker ps -a'
function dkee {
NAME=$1
CMD="$2"
ID=$(dkl | grep $NAME | head -n 1 | awk '{print $1}')
docker exec $ID bash -c "$CMD"
}and dkee is a function that "executes" and "exits" some command within a container regex.
| Terminal 1 | Terminal 2 |
|---|---|
# First do this |
|
dkee chris_dev "echo '10.23.130.106 pfcon_service' >> /etc/hosts" |
|
| (wait) | *make* -r pfcon local |
| Terminal 2 | Terminal 3 |
|---|---|
# Then do this |
# And finally |
*make* -r pfioh local |
*make* -r pman local |
Assuming satisfied preconditions, let's say hello to pfcon. It will in turn ask each of pfioh and pman hello and return the response.
./pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd --httpResponseBodyParse --jsonwrapper 'payload' --msg \
'{ "action": "hello",
"meta": {
"askAbout": "sysinfo",
"echoBack": "Hi there!",
"service": "host"
}
}' --quiet --jsonpprintindent 4In this call, be sure that the HOST_IP env variable is set correctly.
./pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd \
--httpResponseBodyParse --jsonwrapper 'payload' \
--msg '
{ "action": "coordinate",
"threadAction": true,
"meta-store": {
"meta": "meta-compute",
"key": "jid"
},
"meta-data": {
"remote": {
"key": "%meta-store"
},
"localSource": {
"path": "/etc"
},
"localTarget": {
"path": "/usr/users/test/foo/feed_90/simplefsapp_90/data",
"createDir": true
},
"specialHandling": {
"op": "plugin",
"cleanup": true
},
"transport": {
"mechanism": "compress",
"compress": {
"encoding": "none",
"archive": "zip",
"unpack": true,
"cleanup": true
}
},
"service": "host"
},
"meta-compute": {
"cmd": "$execshell $selfpath/$selfexec /share/outgoing --saveinputmeta --saveoutputmeta --dir ./",
"auid": "rudolphpienaar",
"jid": "89",
"threaded": true,
"container": {
"target": {
"image": "fnndsc/pl-simplefsapp",
"cmdParse": true
},
"manager": {
"image": "fnndsc/swarm",
"app": "swarm.py",
"env": {
"meta-store": "key",
"serviceType": "docker",
"shareDir": "%shareDir",
"serviceName": "89"
}
}
},
"service": "host"
}
}
'In this call, be sure that the HOST_IP env variable is set correctly.
pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd \
--httpResponseBodyParse --jsonwrapper 'payload' \
--msg '
{ "action": "coordinate",
"threadAction": true,
"meta-store": {
"meta": "meta-compute",
"key": "jid"
},
"meta-data": {
"remote": {
"key": "%meta-store"
},
"localSource": {
"path": "/neuro/users/rudolphpienaar/Pictures"
},
"localTarget": {
"path": "/home/rudolph/tmp/Pictures",
"createDir": true
},
"specialHandling": {
"op": "plugin",
"cleanup": true
},
"transport": {
"mechanism": "compress",
"compress": {
"encoding": "none",
"archive": "zip",
"unpack": true,
"cleanup": true
}
},
"service": "host"
},
"meta-compute": {
"cmd": "$execshell $selfpath/$selfexec --prefix test- --sleepLength 0 /share/incoming /share/outgoing",
"auid": "rudolphpienaar",
"jid": "89",
"threaded": true,
"container": {
"target": {
"image": "fnndsc/pl-simpledsapp",
"cmdParse": true
},
"manager": {
"image": "fnndsc/swarm",
"app": "swarm.py",
"env": {
"meta-store": "key",
"serviceType": "docker",
"shareDir": "%shareDir",
"serviceName": "89"
}
}
},
"service": "host"
}
}
'Running all the services containerized can result in a lag in debugging, mostly because log files sometimes need to be fully flushed. At times, it is better to run the services non-containerized.
In such instances, add a breakpoint using pudb.set_trace() typically in charm.py. Then start the CUBE dev environment in a containerized fashion:
sudo rm -fr /hostFS/storeBase/* ; sudo rm -fr /usr/users/* ; *make*When execution stops at the breakpoint, kill all the ancillary containers
dkrm pfcon
dkrm pfioh
dkrm pmanwhere dkrm is actually a function I have in .bashrc
dkrm ()
{
NAME=$1;
ID=$(dkl | grep $NAME | awk '{print $1}');
docker stop $ID && docker rm -vf $ID
}and then restart these services directly as per instructions above.