-
Notifications
You must be signed in to change notification settings - Fork 101
Debugging CUBE Containerized Services
CUBE is a complex software infrastructure, comprising the main core backend, as well as a constellation of ancillary services. Usually, if things break, the first place to start debugging is to make sure the ancillary services are all OK. This page documents a workflow that can be useful in debugging these CUBE services. Essentially, instructions on starting each core service directly (i.e. not in containers) is provided.
Due to the distributed and containerized nature of CUBE, debugging the ancillary services, pfcon, pfioh, and pman can be particularly difficult. First, try and debug these services outside of ChRIS in a non-containerized setup. Once that has been done, you can move to debugging the services while containerized.
The workflow is to start the whole CUBE system containerized. Then, once all the services are up, to pause and take down each service to debug and restart in an interactive mode.
Assuming we want to debug pman, in the docker-compose.yml add the following line to the volumes: section of the pman_service:
- ../pman:/usr/local/pmanThis assumes that the pman code is located one level up in the directory tree.
By default, the docker-entrypoint.py will assume the command to run is located at
/usr/local/bin/pmanand has some code to that effect
def pman_do(args, unknown):
str_otherArgs = ' '.join(unknown)
str_CMD = "/usr/local/bin/pman %s" % (str_otherArgs)
# str_CMD = "/usr/local/pman/bin/pman %s" % (str_otherArgs)
return str_CMDbut by volume mounting the source tree into the container, this entry point needs to be changed to
def pman_do(args, unknown):
str_otherArgs = ' '.join(unknown)
# str_CMD = "/usr/local/bin/pman %s" % (str_otherArgs)
str_CMD = "/usr/local/pman/bin/pman %s" % (str_otherArgs)
return str_CMDAdd a pudb.set_trace() to a relevant line of code in charm.py. At time of writing, I use a breakpoint on line 503.
From the CUBE repo dir, run
*make* localThe simplest way to debug is to open four terminals. Each terminal will be used to run a specific service. Typically from left to right the terminals should be anchored in the following source repositories:
| Terminal 1 | Terminal 2 | Terminal 3 | Terminal 4 |
|---|---|---|---|
| pfurl | pfcon | pfioh | pman |
Once execution pauses at the breakpoint, stop the target service, i.e. pman
docker-compose stop pman_service && docker-compose rm -f pman_serviceOnce stopped, and assuming breakpoints are added to the pman source, run
docker-compose run --service-ports pman_serviceIn this call, be sure that the HOST_IP env variable is set correctly.
./pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd --httpResponseBodyParse --jsonwrapper 'payload' --msg '
{ "action": "coordinate",
"threadAction": true,
"meta-store": {
"meta": "meta-compute",
"key": "jid"
},
"meta-data": {
"remote": {
"key": "%meta-store"
},
"localSource": {
"path": "/etc"
},
"localTarget": {
"path": "/usr/users/test/foo/feed_90/simplefsapp_90/data",
"createDir": true
},
"specialHandling": {
"op": "plugin",
"cleanup": true
},
"transport": {
"mechanism": "compress",
"compress": {
"encoding": "none",
"archive": "zip",
"unpack": true,
"cleanup": true
}
},
"service": "host"
},
"meta-compute": {
"cmd": "$execshell $selfpath/$selfexec /share/outgoing --saveopts /share/opts.json --dir ./",
"auid": "rudolphpienaar",
"jid": "89",
"threaded": true,
"container": {
"target": {
"image": "fnndsc/pl-simplefsapp",
"cmdParse": true
},
"manager": {
"image": "fnndsc/swarm",
"app": "swarm.py",
"env": {
"meta-store": "key",
"serviceType": "docker",
"shareDir": "%shareDir",
"serviceName": "89"
}
}
},
"service": "host"
}
}
'In this call, be sure that the HOST_IP env variable is set correctly.
pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd --httpResponseBodyParse --jsonwrapper 'payload' --msg '
{ "action": "coordinate",
"threadAction": true,
"meta-store": {
"meta": "meta-compute",
"key": "jid"
},
"meta-data": {
"remote": {
"key": "%meta-store"
},
"localSource": {
"path": "/neuro/users/rudolphpienaar/Pictures"
},
"localTarget": {
"path": "/home/rudolph/tmp/Pictures",
"createDir": true
},
"specialHandling": {
"op": "plugin",
"cleanup": true
},
"transport": {
"mechanism": "compress",
"compress": {
"encoding": "none",
"archive": "zip",
"unpack": true,
"cleanup": true
}
},
"service": "host"
},
"meta-compute": {
"cmd": "$execshell $selfpath/$selfexec --prefix test- --sleepLength 0 /share/incoming /share/outgoing",
"auid": "rudolphpienaar",
"jid": "89",
"threaded": true,
"container": {
"target": {
"image": "fnndsc/pl-simpledsapp",
"cmdParse": true
},
"manager": {
"image": "fnndsc/swarm",
"app": "swarm.py",
"env": {
"meta-store": "key",
"serviceType": "docker",
"shareDir": "%shareDir",
"serviceName": "89"
}
}
},
"service": "host"
}
}
'Running all the services containerized can result in a lag in debugging, mostly because log files sometimes need to be fully flushed. At times, it is better to run the services non-containerized.
In such instances, add a breakpoint using pudb.set_trace() typically in charm.py. Then start the CUBE dev environment in a containerized fashion:
sudo rm -fr /hostFS/storeBase/* ; sudo rm -fr /usr/users/* ; *make*When execution stops at the breakpoint, kill all the ancillary containers
dkrm pfcon
dkrm pfioh
dkrm pmanwhere dkrm is actually a function I have in .bashrc
dkrm ()
{
NAME=$1;
ID=$(dkl | grep $NAME | awk '{print $1}');
docker stop $ID && docker rm -vf $ID
}and then restart these services directly as per instructions above.