-
Notifications
You must be signed in to change notification settings - Fork 101
Debugging CUBE Containerized Services
CUBE is a complex software infrastructure, comprising the main core backend, as well as a constellation of ancillary services. Usually, if things break, the first place to start debugging is to make sure the ancillary services are all OK. This page documents a workflow that can be useful in debugging these CUBE services. Essentially, instructions on stopping and rerunning pman within a container is provided as an example.
Due to the distributed and containerized nature of CUBE, debugging the ancillary services, pfcon, pfioh, and pman can be particularly difficult. An initial strategy is to first debug these services outside of ChRIS in a non-containerized setup. If all works "outside" of ChRIS, then a next step is running the services in containers and talking to them from another container, and finally, calling the services via CUBE.
The workflow is to start the whole CUBE system containerized. Then, once all the services are up, to pause and take down each service to debug and restart in an interactive mode. Using a docker logs --follow is often not reactive enough, hence the recommendation is to stop and restart each service.
Assuming we want to debug the pman source code as well, in the docker-compose.yml add the following line to the volumes: section of the pman_service:
- ../pman:/usr/local/pmanThis assumes that the pman code is located one level up in the directory tree. Repeat of course for other services if necessary
- ../pfioh:/usr/local/pfioh - ../pfcon:/usr/local/pfconBy default, the docker-entrypoint.py will assume the command to run is located at
/usr/local/bin/pmanand has some code to that effect
def pman_do(args, unknown):
str_otherArgs = ' '.join(unknown)
str_CMD = "/usr/local/bin/pman %s" % (str_otherArgs)
# str_CMD = "/usr/local/pman/bin/pman %s" % (str_otherArgs)
return str_CMDbut by volume mounting the source tree into the container, this entry point needs to be changed to
def pman_do(args, unknown):
str_otherArgs = ' '.join(unknown)
# str_CMD = "/usr/local/bin/pman %s" % (str_otherArgs)
str_CMD = "/usr/local/pman/bin/pman %s" % (str_otherArgs)
return str_CMDNote -- you will need to rebuild the container after changing the docker entry point script.
If the system needs to be debugged during testing, Add a pudb.set_trace() to a relevant line of code in charm.py. At time of writing, I use a breakpoint on line 503.
If tests run fine, but subsequent use cases, fail, then simply wait until CUBE has been fully instantiated.
From the CUBE repo dir, run
*make* localThe simplest way to debug is to open four terminals. Each terminal will be used to run a specific service. Typically from left to right the terminals should be anchored in the following source repositories:
| Terminal 1 | Terminal 2 | Terminal 3 | Terminal 4 |
|---|---|---|---|
| pfurl | pfcon | pfioh | pman |
Once execution pauses at the breakpoint, stop the target services and restart in an interactive manner. This keeps the terminal output much more current.
There is stop/restart order dependency. First pfcon, then pfioh, and then pman.
The docker make file, docker-make-chris_dev.sh provides a restart feature for a given service. For services, see below.
*make* -r pfcon local*make* -r pfioh local*make* -r pman localAssuming satisfied preconditions, let's say hello to pfcon. It will in turn ask each of pfioh and pman hello and return the response.
./pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd --httpResponseBodyParse --jsonwrapper 'payload' --msg \
'{ "action": "hello",
"meta": {
"askAbout": "sysinfo",
"echoBack": "Hi there!",
"service": "host"
}
}' --quiet --jsonpprintindent 4In this call, be sure that the HOST_IP env variable is set correctly.
./pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd --httpResponseBodyParse --jsonwrapper 'payload' --msg '
{ "action": "coordinate",
"threadAction": true,
"meta-store": {
"meta": "meta-compute",
"key": "jid"
},
"meta-data": {
"remote": {
"key": "%meta-store"
},
"localSource": {
"path": "/etc"
},
"localTarget": {
"path": "/usr/users/test/foo/feed_90/simplefsapp_90/data",
"createDir": true
},
"specialHandling": {
"op": "plugin",
"cleanup": true
},
"transport": {
"mechanism": "compress",
"compress": {
"encoding": "none",
"archive": "zip",
"unpack": true,
"cleanup": true
}
},
"service": "host"
},
"meta-compute": {
"cmd": "$execshell $selfpath/$selfexec /share/outgoing --saveopts /share/opts.json --dir ./",
"auid": "rudolphpienaar",
"jid": "89",
"threaded": true,
"container": {
"target": {
"image": "fnndsc/pl-simplefsapp",
"cmdParse": true
},
"manager": {
"image": "fnndsc/swarm",
"app": "swarm.py",
"env": {
"meta-store": "key",
"serviceType": "docker",
"shareDir": "%shareDir",
"serviceName": "89"
}
}
},
"service": "host"
}
}
'In this call, be sure that the HOST_IP env variable is set correctly.
pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd --httpResponseBodyParse --jsonwrapper 'payload' --msg '
{ "action": "coordinate",
"threadAction": true,
"meta-store": {
"meta": "meta-compute",
"key": "jid"
},
"meta-data": {
"remote": {
"key": "%meta-store"
},
"localSource": {
"path": "/neuro/users/rudolphpienaar/Pictures"
},
"localTarget": {
"path": "/home/rudolph/tmp/Pictures",
"createDir": true
},
"specialHandling": {
"op": "plugin",
"cleanup": true
},
"transport": {
"mechanism": "compress",
"compress": {
"encoding": "none",
"archive": "zip",
"unpack": true,
"cleanup": true
}
},
"service": "host"
},
"meta-compute": {
"cmd": "$execshell $selfpath/$selfexec --prefix test- --sleepLength 0 /share/incoming /share/outgoing",
"auid": "rudolphpienaar",
"jid": "89",
"threaded": true,
"container": {
"target": {
"image": "fnndsc/pl-simpledsapp",
"cmdParse": true
},
"manager": {
"image": "fnndsc/swarm",
"app": "swarm.py",
"env": {
"meta-store": "key",
"serviceType": "docker",
"shareDir": "%shareDir",
"serviceName": "89"
}
}
},
"service": "host"
}
}
'Running all the services containerized can result in a lag in debugging, mostly because log files sometimes need to be fully flushed. At times, it is better to run the services non-containerized.
In such instances, add a breakpoint using pudb.set_trace() typically in charm.py. Then start the CUBE dev environment in a containerized fashion:
sudo rm -fr /hostFS/storeBase/* ; sudo rm -fr /usr/users/* ; *make*When execution stops at the breakpoint, kill all the ancillary containers
dkrm pfcon
dkrm pfioh
dkrm pmanwhere dkrm is actually a function I have in .bashrc
dkrm ()
{
NAME=$1;
ID=$(dkl | grep $NAME | awk '{print $1}');
docker stop $ID && docker rm -vf $ID
}and then restart these services directly as per instructions above.