-
Notifications
You must be signed in to change notification settings - Fork 101
Debugging CUBE Containerized Services
CUBE is a complex software infrastructure, comprising the main core backend, as well as a constellation of ancillary services. Usually, if things break, the first place to start debugging is to make sure the ancillary services are all OK. This page documents a workflow that can be useful in debugging these CUBE services pfcon, pfioh, and pfcon within their respective containers.
Due to the distributed and containerized nature of CUBE, debugging the ancillary services, pfcon, pfioh, and pman can be particularly difficult. An initial strategy is to first debug these services outside of ChRIS in a non-containerized setup. If all works "outside" of ChRIS, then a next step is running the services in containers and talking to them from another container, and finally, calling the services via CUBE.
The workflow is to start the whole CUBE system containerized. Then, once all the services are up, to pause and take down each service to debug and restart in an interactive mode.
Also, in order to fully debug, the source code of the particular service needs to be volume mapped into its respective container in the correct place. This necessitates small transitory changes to the docker-compose.yml.
Let's assume we want to debug the pman source code in a running active pman container. Further, let's assume we have checked out the pman repository at the same tree level as the ChRIS_ultron_backend repo. Now, in the docker-compose.yml add the following line to the volumes: section of the pman_service:
- ../pman/pman/pman.py:/usr/local/lib/python3.5/dist-packages/pman/pman.pyRepeat for other services if/as necessary
- ../pfioh/pfioh/pfioh.py:/usr/lib/python3.5/site-packages/pfioh/pfioh.py - ../pfcon/pfcon/pfcon.py:/usr/local/lib/python3.5/dist-packages/pfcon/pfcon.pyIt is important to meaningfully pause/stop the flow of setup execution in CUBE. This should happen after the initial containerized services have all been launched but before meaningful processing has started. In the current CUBE, there is a logical break in "Step 10 -- Running the django unit tests" where the script stops to ask if it should delete a prior database. This of course assumes that CUBE has been at least once prior.
Alternatively. add a breakpoint to charm.py. with a pudb.set_trace() to a relevant line of code. At time of writing, I use a breakpoint in the app_service() method before the conditional call to self.app_service_fsplugin_setup() (around line 495).
If tests run fine, but subsequent use cases, fail, then simply wait until CUBE has been fully instantiated.
From the CUBE repo dir, run
*make* localThe simplest way to debug is to open four terminals. Each terminal will be used to run a specific service. Typically from left to right the terminals should be anchored in the following source repositories:
| Terminal 1 | Terminal 2 | Terminal 3 | Terminal 4 |
|---|---|---|---|
| pfurl | pfcon | pfioh | pman |
Once execution pauses at the breakpoint, stop the target services and restart in an interactive manner. This keeps the terminal output much more current.
There is stop/restart order dependency. First pfcon, then pfioh, and then pman.
The docker make file, docker-make-chris_dev.sh provides a restart feature for a given service. For services, see below.
*make* -r pfcon local*make* -r pfioh local*make* -r pman localAssuming satisfied preconditions, let's say hello to pfcon. It will in turn ask each of pfioh and pman hello and return the response.
./pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd --httpResponseBodyParse --jsonwrapper 'payload' --msg \
'{ "action": "hello",
"meta": {
"askAbout": "sysinfo",
"echoBack": "Hi there!",
"service": "host"
}
}' --quiet --jsonpprintindent 4In this call, be sure that the HOST_IP env variable is set correctly.
./pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd \
--httpResponseBodyParse --jsonwrapper 'payload' \
--msg '
{ "action": "coordinate",
"threadAction": true,
"meta-store": {
"meta": "meta-compute",
"key": "jid"
},
"meta-data": {
"remote": {
"key": "%meta-store"
},
"localSource": {
"path": "/etc"
},
"localTarget": {
"path": "/usr/users/test/foo/feed_90/simplefsapp_90/data",
"createDir": true
},
"specialHandling": {
"op": "plugin",
"cleanup": true
},
"transport": {
"mechanism": "compress",
"compress": {
"encoding": "none",
"archive": "zip",
"unpack": true,
"cleanup": true
}
},
"service": "host"
},
"meta-compute": {
"cmd": "$execshell $selfpath/$selfexec /share/outgoing --saveinputmeta --saveoutputmeta --dir ./",
"auid": "rudolphpienaar",
"jid": "89",
"threaded": true,
"container": {
"target": {
"image": "fnndsc/pl-simplefsapp",
"cmdParse": true
},
"manager": {
"image": "fnndsc/swarm",
"app": "swarm.py",
"env": {
"meta-store": "key",
"serviceType": "docker",
"shareDir": "%shareDir",
"serviceName": "89"
}
}
},
"service": "host"
}
}
'In this call, be sure that the HOST_IP env variable is set correctly.
pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd \
--httpResponseBodyParse --jsonwrapper 'payload' \
--msg '
{ "action": "coordinate",
"threadAction": true,
"meta-store": {
"meta": "meta-compute",
"key": "jid"
},
"meta-data": {
"remote": {
"key": "%meta-store"
},
"localSource": {
"path": "/neuro/users/rudolphpienaar/Pictures"
},
"localTarget": {
"path": "/home/rudolph/tmp/Pictures",
"createDir": true
},
"specialHandling": {
"op": "plugin",
"cleanup": true
},
"transport": {
"mechanism": "compress",
"compress": {
"encoding": "none",
"archive": "zip",
"unpack": true,
"cleanup": true
}
},
"service": "host"
},
"meta-compute": {
"cmd": "$execshell $selfpath/$selfexec --prefix test- --sleepLength 0 /share/incoming /share/outgoing",
"auid": "rudolphpienaar",
"jid": "89",
"threaded": true,
"container": {
"target": {
"image": "fnndsc/pl-simpledsapp",
"cmdParse": true
},
"manager": {
"image": "fnndsc/swarm",
"app": "swarm.py",
"env": {
"meta-store": "key",
"serviceType": "docker",
"shareDir": "%shareDir",
"serviceName": "89"
}
}
},
"service": "host"
}
}
'Running all the services containerized can result in a lag in debugging, mostly because log files sometimes need to be fully flushed. At times, it is better to run the services non-containerized.
In such instances, add a breakpoint using pudb.set_trace() typically in charm.py. Then start the CUBE dev environment in a containerized fashion:
sudo rm -fr /hostFS/storeBase/* ; sudo rm -fr /usr/users/* ; *make*When execution stops at the breakpoint, kill all the ancillary containers
dkrm pfcon
dkrm pfioh
dkrm pmanwhere dkrm is actually a function I have in .bashrc
dkrm ()
{
NAME=$1;
ID=$(dkl | grep $NAME | awk '{print $1}');
docker stop $ID && docker rm -vf $ID
}and then restart these services directly as per instructions above.