Skip to content
Permalink
Browse files

[CI] Randomize RPCService's port and handle NoSuchProcess error prope…

…rly (#690)

* [CI] Randomize RPCService's port in test_container_rpc.sh

We should avoid the port collisions between test cases as much as possible.
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1934/ shows the port collision to us.

* Fix my mistake

* Handle psutil.NoSuchProcess error properly

Current Jenkins CI shows the following errors to us.
```
 [unittest_rpc_container] Traceback (most recent call last):
 [unittest_rpc_container] File ../python/rpc_test_container.py, line 1, in <module>
 [unittest_rpc_container] import rpc
 [unittest_rpc_container] File /clipper/containers/python/rpc.py, line 20, in <module>
 [unittest_rpc_container] import clipper_admin.metrics as metrics
 [unittest_rpc_container] File /clipper/clipper_admin/clipper_admin/metrics/__init__.py, line 6, in <module>
 [unittest_rpc_container] if not server.redis_daemon_exist():
 [unittest_rpc_container] File /clipper/clipper_admin/clipper_admin/metrics/server.py, line 134, in redis_daemon_exist
 [unittest_rpc_container] process_names = [psutil.Process(pid).name() for pid in pids]
 [unittest_rpc_container] File /usr/local/lib/python2.7/dist-packages/psutil/__init__.py, line 338, in __init__
 [unittest_rpc_container] self._init(pid)
 [unittest_rpc_container] File /usr/local/lib/python2.7/dist-packages/psutil/__init__.py, line 378, in _init
 [unittest_rpc_container] raise NoSuchProcess(pid, None, msg)
 [unittest_rpc_container] psutil._exceptions.NoSuchProcess: psutil.NoSuchProcess no process found with pid 162
```

* Fix location
  • Loading branch information...
withsmilo committed May 12, 2019
1 parent 2ed4976 commit 6466d7d2b26141c7f28a4bc447618044b6256341
Showing with 21 additions and 9 deletions.
  1. +9 −1 clipper_admin/clipper_admin/metrics/server.py
  2. +12 −8 containers/test/test_container_rpc.sh
@@ -131,7 +131,15 @@ def redis_daemon_exist():
# situation is that we are in a container without any other python2
# process.
pids = psutil.pids()
process_names = [psutil.Process(pid).name() for pid in pids]
process_names = []

for pid in pids:
try:
name = psutil.Process(pid).name()
except psutil.NoSuchProcess:
name = None
process_names.append(name)

return 'redis-server' in process_names


@@ -22,7 +22,10 @@ then
fi

success=false
rpc_service_port=17000 # for test only

PORT_RANGE_START=10000
PORT_RANGE_END=20000
RPC_SERVICE_PORT=`perl -e "print int(rand($PORT_RANGE_END-$PORT_RANGE_START)) + $PORT_RANGE_START"`

function clean_up {
# Perform program exit housekeeping
@@ -48,8 +51,8 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd $DIR

# Start python rpc test container
echo "Starting python RPC test container..."
python ../python/rpc_test_container.py --rpc_service_port $rpc_service_port &
echo "Starting python RPC test container... (port:$RPC_SERVICE_PORT)"
python ../python/rpc_test_container.py --rpc_service_port $RPC_SERVICE_PORT &

# Deprecate JVM containers
# cd ../jvm
@@ -68,20 +71,21 @@ cd $DIR/../../
cd container
make container_rpc_test
container_uptime_seconds=180
./container_rpc_test -t $container_uptime_seconds -p $rpc_service_port &
echo "Starting cpp RPC test container... (port:$RPC_SERVICE_PORT)"
./container_rpc_test -t $container_uptime_seconds -p $RPC_SERVICE_PORT &

sleep 10s

cd $DIR/../../debug/src/benchmarks
make rpctest
echo "Executing RPC test (first iteration)..."
REDIS_PORT=$1
./rpctest --num_containers=2 --timeout_seconds=30 --redis_port $REDIS_PORT --rpc_service_port $rpc_service_port
echo "Executing RPC test (first iteration)... (redis port:$REDIS_PORT, rpc_service_port:$RPC_SERVICE_PORT)"
./rpctest --num_containers=2 --timeout_seconds=30 --redis_port $REDIS_PORT --rpc_service_port $RPC_SERVICE_PORT
redis-cli -p $REDIS_PORT "flushall"
echo "Sleeping for 5 seconds..."
sleep 5s
echo "Executing RPC test (second iteration)..."
./rpctest --num_containers=2 --timeout_seconds=30 --redis_port $REDIS_PORT --rpc_service_port $rpc_service_port
echo "Executing RPC test (second iteration)... (redis port:$REDIS_PORT, rpc_service_port:$RPC_SERVICE_PORT)"
./rpctest --num_containers=2 --timeout_seconds=30 --redis_port $REDIS_PORT --rpc_service_port $RPC_SERVICE_PORT
redis-cli -p $REDIS_PORT "flushall"
echo "TEST PASSED!"
success=true

0 comments on commit 6466d7d

Please sign in to comment.
You can’t perform that action at this time.