Skip to content

Catch pipeline failures and return proper error status. #243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 15, 2016
Merged

Catch pipeline failures and return proper error status. #243

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fe15a81
Catch pipeline failures and return proper error status.
ndokos May 16, 2016
cf59a06
Check status of backgrounded commands in pbench-postprocess-tools.
ndokos Jul 12, 2016
f4f5618
Add unit tests.
ndokos Jul 11, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions agent/base
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
#!/bin/bash

export PBENCH_debug_mode=0
# pipeline status is set to the status of the last command that *failed*
# in the pipeline (or 0 if all succeed): this way "ssh foo | sed '...' "
# will catch any ssh failure
set -o pipefail

# very first thing to do is figure out which pbench
# we are
Expand Down
21 changes: 21 additions & 0 deletions agent/bench-scripts/test-bin/ssh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,24 @@ echo "$0 $*" >> $_testlog
if [[ "$4" == "netstat" && "$5" == "-tlpn" ]]; then
echo "tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN 5830/uperf"
fi

while true ;do
case $1 in
-o)
shift 2
;;
-n)
shift 1
;;
*)
break
;;
esac
done
remote=$1

if [[ "$remote" == "fubar" ]] ;then
exit 255
else
exit 0
fi
16 changes: 16 additions & 0 deletions agent/util-scripts/gold/pbench-clear-tools/test-12.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
+++ Running test-12 pbench-clear-tools
removing tools-default/mpstat
running . /opt/pbench-agent/profile; pbench-clear-tools --group default on fubar
Removing the remote remote@fubar
--- Finished test-12 pbench-clear-tools (status=1}
+++ pbench tree state
/var/tmp/pbench-test-utils/pbench
/var/tmp/pbench-test-utils/pbench/tmp
/var/tmp/pbench-test-utils/pbench/tools-default
--- pbench tree state
+++ pbench.log file contents
grep: /var/tmp/pbench-test-utils/pbench/pbench.log: No such file or directory
--- pbench.log file contents
+++ test-execution.log file contents
/var/tmp/pbench-test-utils/test-execution.log:/var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no -o ConnectTimeout=1 -n fubar . /opt/pbench-agent/profile; pbench-clear-tools --group default
--- test-execution.log file contents
25 changes: 25 additions & 0 deletions agent/util-scripts/gold/pbench-postprocess-tools/test-13.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
+++ Running test-13 pbench-postprocess-tools
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
--- Finished test-13 pbench-postprocess-tools (status=2}
+++ pbench tree state
/var/tmp/pbench-test-utils/pbench
/var/tmp/pbench-test-utils/pbench/pbench.log
/var/tmp/pbench-test-utils/pbench/tmp
/var/tmp/pbench-test-utils/pbench/tmp/tools-default
/var/tmp/pbench-test-utils/pbench/tools-default
/var/tmp/pbench-test-utils/pbench/tools-default/remote@fubar
/var/tmp/pbench-test-utils/pbench/tools-default/remote@fubar:
--- pbench tree state
+++ pbench.log file contents
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-postprocess-tools]started: --dir=/var/tmp/pbench-test-utils/pbench/tmp --group=default --iteration=1
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-postprocess-tools]running this tool on fubar: ssh -o StrictHostKeyChecking=no -n fubar pbench-postprocess-tools --iteration=1 --group=default --dir=/var/tmp/pbench-test-utils/pbench/tmp
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-postprocess-tools]started: copying tool data ( MB) from fubar
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-postprocess-tools]copying tool data failed for remote fubar
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-postprocess-tools]completed:
--- pbench.log file contents
+++ test-execution.log file contents
/var/tmp/pbench-test-utils/test-execution.log:/var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no -n fubar pbench-postprocess-tools --iteration=1 --group=default --dir=/var/tmp/pbench-test-utils/pbench/tmp
/var/tmp/pbench-test-utils/test-execution.log:/var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no -n fubar du -sm /var/tmp/pbench-test-utils/pbench/tmp/tools-default
/var/tmp/pbench-test-utils/test-execution.log:/var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no -n fubar cd /var/tmp/pbench-test-utils/pbench/tmp/tools-default && tar cf - *
--- test-execution.log file contents
26 changes: 26 additions & 0 deletions agent/util-scripts/gold/pbench-register-tool-set/test-11.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
+++ Running test-11 pbench-register-tool-set
--- Finished test-11 pbench-register-tool-set (status=4}
+++ pbench tree state
/var/tmp/pbench-test-utils/pbench
/var/tmp/pbench-test-utils/pbench/pbench.log
/var/tmp/pbench-test-utils/pbench/tmp
/var/tmp/pbench-test-utils/pbench/tools-default
/var/tmp/pbench-test-utils/pbench/tools-default/remote@fubar
/var/tmp/pbench-test-utils/pbench/tools-default/remote@fubar:
--- pbench tree state
+++ pbench.log file contents
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] tool_opts: --interval=3
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] tool_opts[0]="--interval=3"; pbench-register-tool --name=mpstat --group=default $label_opt -- "${tool_opts[@]}" 2>&1
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] tool_opts: --interval=3
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] tool_opts[0]="--interval=3"; pbench-register-tool --name=vmstat --group=default $label_opt -- "${tool_opts[@]}" 2>&1
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] tool_opts: --interval=3
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] tool_opts[0]="--interval=3"; pbench-register-tool --name=iostat --group=default $label_opt -- "${tool_opts[@]}" 2>&1
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] tool_opts: --record-opts=record -a --freq=100
/var/tmp/pbench-test-utils/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] tool_opts[0]="--record-opts=record -a --freq=100"; pbench-register-tool --name=perf --group=default $label_opt -- "${tool_opts[@]}" 2>&1
--- pbench.log file contents
+++ test-execution.log file contents
/var/tmp/pbench-test-utils/test-execution.log:/var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no fubar tool_opts[0]="--interval=3"; pbench-register-tool --name=mpstat --group=default -- "${tool_opts[@]}" 2>&1
/var/tmp/pbench-test-utils/test-execution.log:/var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no fubar tool_opts[0]="--interval=3"; pbench-register-tool --name=vmstat --group=default -- "${tool_opts[@]}" 2>&1
/var/tmp/pbench-test-utils/test-execution.log:/var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no fubar tool_opts[0]="--interval=3"; pbench-register-tool --name=iostat --group=default -- "${tool_opts[@]}" 2>&1
/var/tmp/pbench-test-utils/test-execution.log:/var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no fubar tool_opts[0]="--record-opts=record -a --freq=100"; pbench-register-tool --name=perf --group=default -- "${tool_opts[@]}" 2>&1
--- test-execution.log file contents
2 changes: 1 addition & 1 deletion agent/util-scripts/gold/pbench-stop-tools/test-05.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
+++ Running test-05 pbench-stop-tools
[warn][1900-01-01T00:00:00.000000] Too many pids for turbostat: 123463 123464 123465 123466 123467 -- maybe old tools running? Use pbench-kill-tools.
--- Finished test-05 pbench-stop-tools (status=0}
--- Finished test-05 pbench-stop-tools (status=1}
+++ pbench tree state
/var/tmp/pbench-test-utils/pbench
/var/tmp/pbench-test-utils/pbench/pbench.log
Expand Down
2 changes: 1 addition & 1 deletion agent/util-scripts/gold/pbench-stop-tools/test-06.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
+++ Running test-06 pbench-stop-tools
--- Finished test-06 pbench-stop-tools (status=0}
--- Finished test-06 pbench-stop-tools (status=1}
+++ pbench tree state
/var/tmp/pbench-test-utils/pbench
/var/tmp/pbench-test-utils/pbench/pbench.log
Expand Down
15 changes: 15 additions & 0 deletions agent/util-scripts/pbench-clear-tools
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#!/bin/bash
# -*- mode: shell-script; indent-tabs-mode: t; sh-basic-offset: 8; sh-indentation: 8; sh-indent-for-case-alt: + -*-
Copy link
Member

@portante portante Jul 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we specify real tabs instead of spaces for emacs?

Copy link
Member Author

@ndokos ndokos Jul 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought this is what indent-tabs-mode: t does. The doc says

Indentation can insert tabs if this is non-nil.


script_path=`dirname $0`
script_name=`basename $0`
Expand Down Expand Up @@ -58,6 +59,8 @@ pushd $pbench_run >/dev/null
# this tool group's directory which stores options, etc.
tool_group_dir="tools-$group"
if [ -d "$tool_group_dir" ]; then
typeset -i nerrs=0
ssh_opts="$ssh_opts -o ConnectTimeout=1"
for this_tool_file in `/bin/ls $tool_group_dir`; do
if [ "$this_tool_file" == "label" ]; then
continue;
Expand All @@ -72,6 +75,17 @@ if [ -d "$tool_group_dir" ]; then
remote_hostname=`echo $this_tool_file | awk -F@ '{print $2}'`
echo running ". ${pbench_install_dir}/profile; pbench-clear-tools $group_opt $name_opt" on $remote_hostname
ssh $ssh_opts -n $remote_hostname ". ${pbench_install_dir}/profile; pbench-clear-tools $group_opt $name_opt" | sed -e 's/\(.*\)/['$remote_hostname']\1/g'
rc=$?
if [ $rc != 0 ] ;then
nerrs=$nerrs+1
# we can't get to the remote but if the intent is to clear all tools
# then we remove the local entry for the remote
if [ -z "$name" ] ;then
echo "Removing the remote $this_tool_file"
/bin/rm -f "$tool_group_dir/$this_tool_file"
fi
continue
fi
remaining_remote_tools=`ssh $ssh_opts -n $remote_hostname ". ${pbench_install_dir}/profile; pbench-list-tools $group_opt"`
if [ -z "$remaining_remote_tools" ]; then
echo "The remote host $remote_hostname no longer has tools registered for $group group, so the "remote@$remote" entry in the local $tool_group_dir directory will be removed"
Expand All @@ -96,3 +110,4 @@ if [ -d "$tool_group_dir" ]; then
fi
fi
popd >/dev/null
exit $nerrs
40 changes: 37 additions & 3 deletions agent/util-scripts/pbench-postprocess-tools
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ if [ $? -ne 0 ]; then
printf -- "\t will store and process data\n"
printf "\n"
printf -- "\t-i str --iteration=num, num = a number representing the\n"
printf -- "\t iteration data was collected for"
printf -- "\t iteration data was collected for\n"
exit 1
fi
eval set -- "$opts";
Expand Down Expand Up @@ -74,6 +74,7 @@ while true; do
esac
done

typeset -i nerrs=0
iteration_num=`echo $iteration | awk -F- '{print $1}'`
# this tool group's directory which stores options, etc.
if [ -d "$pbench_run/tools-$group" ]; then
Expand All @@ -92,6 +93,13 @@ fi
tool_output_dir="$dir/tools-$group"
mkdir -p $tool_output_dir

# Try to prevent a cascade of tools running.
if [ "$action" == "start" ] ;then
# Kill any tools running from a previous incantation before
# starting this one.
pbench-kill-tools --group $group
Copy link
Member

@portante portante Jul 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a significant change to the PR. Can you update the summary line for the commit message to somehow include this and then call this behavior change out explicitly in the commit message?

fi

function move_tool_data {
local remote_host=$1
local remote_label=$2
Expand All @@ -100,6 +108,12 @@ function move_tool_data {
tool_data_size=`ssh $ssh_opts -n $remote_host du -sm $tool_output_dir | awk '{print $1}'`
debug_log "[$script_name]started: copying tool data ($tool_data_size MB) from $remote_host"
ssh $ssh_opts -n $remote_host "cd $tool_output_dir && tar cf - *" | tar mxf -
rc=$?
if [ $rc != 0 ] ;then
debug_log "[$script_name]copying tool data failed for remote $remote_host"
popd > /dev/null
return $rc
fi
# if the full hostname was used in pbench-register-tool --remote, make sure that is preserved in the directory name
if [ $remote_host != "$remote_shost" ]; then
if [ -e "$remote_label:$remote_shost" ]; then
Expand All @@ -112,11 +126,14 @@ function move_tool_data {
debug_log "[$script_name]completed: copying of tool data on $remote_host"
debug_log "[$script_name]started: deleting tool data on $remote_host"
ssh $ssh_opts -n $remote_host "cd $tool_output_dir && /bin/rm -rf *"
rc=$?
debug_log "[$script_name]completed: deleting tool data on $remote_host"
popd >/dev/null
return $rc
}

### phase 1: for each tool, call the tool script with --$action (start, stop, or postprocess)
pids=""
for this_tool_file in `/bin/ls $tool_group_dir`; do
if [ "$this_tool_file" == "label" ]; then
continue;
Expand All @@ -128,6 +145,7 @@ for this_tool_file in `/bin/ls $tool_group_dir`; do
# tool options are stored on the remote host's tool file, so no need to pass it here
debug_log "[$script_name]running this tool on $remote: ssh $ssh_opts -n $remote pbench-$action-tools --iteration=$iteration --group=$group --dir=$dir"
ssh $ssh_opts -n $remote pbench-$action-tools --iteration=$iteration --group=$group --dir=$dir &
pids="$pids $!"
else
# tool is local
# assemble the tool options in to an array
Expand Down Expand Up @@ -159,12 +177,19 @@ for this_tool_file in `/bin/ls $tool_group_dir`; do
fi
else
$pbench_bin/tool-scripts/$name --$action --iteration=$iteration --group=$group --dir=$dir "${tool_opts[@]}" &
pids="$pids $!"
fi
fi

fi
done
wait
for p in $pids ;do
wait $p
rc=$?
if [[ $rc -ne 0 ]] ;then
nerrs=$nerrs+1
fi
done
if [ "$action" == "postprocess" ]; then
# phase 2: now that the local results are ready, move them
# down to $tool_output_dir/[$label:]$hostname.
Expand All @@ -185,12 +210,21 @@ if [ "$action" == "postprocess" ]; then

### phase 3: copy over data from remote hosts
# for the remote tools, copy over the postprocess data
pids=""
for this_tool_file in `/bin/ls $tool_group_dir | grep "^remote"`; do
remote_hostname=`echo "$this_tool_file" | awk -F@ '{print $2}'`
label=`cat $tool_group_dir/$this_tool_file`
# copy over the data from postprocessing
move_tool_data $remote_hostname $label &
pids="$pids $!"
done
for p in $pids ;do
wait $p
rc=$?
if [[ $rc -ne 0 ]] ;then
nerrs=$nerrs+1
fi
done
wait
fi
debug_log "[$script_name]completed: $@"
exit $nerrs
1 change: 1 addition & 0 deletions agent/util-scripts/pbench-register-tool
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,3 +179,4 @@ else # register this tool on the remote host
echo "$label" >"$this_tool_file"
fi
fi
exit $rc
11 changes: 11 additions & 0 deletions agent/util-scripts/pbench-register-tool-set
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,13 +90,24 @@ if [ -z "$interval" ] ;then
interval=3
fi

typeset -i nerrs=0
default_tool_set=$(getconf.py -l default-tool-set pbench/tools)
case "$toolset" in
default)
for i in $default_tool_set; do
pbench-register-tool --name=$i $remote $label $group -- --interval="$interval"
rc=$?
if [ $rc != 0 ] ;then
nerrs=$nerrs+1
fi
done
# low overhead perf
pbench-register-tool --name=perf $remote $label $group -- --record-opts="record -a --freq=100"
rc=$?
if [ $rc != 0 ] ;then
nerrs=$nerrs+1
fi
;;
esac

exit $nerrs
2 changes: 2 additions & 0 deletions agent/util-scripts/samples/pbench-agent.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[pbench/tools]
default-tool-set = mpstat, vmstat, iostat
1 change: 1 addition & 0 deletions agent/util-scripts/samples/pbench-clear-tools/test-12/tools-default/mpstat
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
--interval: 34
Empty file added agent/util-scripts/samples/pbench-clear-tools/test-12/tools-default/remote@fubar
View file
Open in desktop
Empty file.
Empty file added agent/util-scripts/samples/pbench-postprocess-tools/test-13/tools-default/remote@fubar
View file
Open in desktop
Empty file.
Loading