Skip to content

Check for and install python-pandas #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 22, 2016
Merged

Check for and install python-pandas #383

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0192eda
Check for and install python-pandas
ndokos Sep 14, 2016
907af6b
Rebase and fix unittests
ndokos Sep 15, 2016
001ff23
Consistent use of = as equality operator in tests
ndokos Sep 21, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion agent/base
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,8 @@ if [[ -z "$_PBENCH_BENCH_TESTS" ]]; then

else
function check_install_rpm {
echo $@
echo $1
#$2 is the version which will vary
return 0
}
fi
Expand Down
6 changes: 4 additions & 2 deletions agent/bench-scripts/gold/pbench-fio/test-04.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
+++ Running test-04 pbench-fio
pbench-fio 2.14
pbench-fio
python-pandas
The following jobfile was created: /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/1-rw-4KiB/sample1/fio.job
[global]
bs=4k
Expand Down Expand Up @@ -1109,7 +1110,8 @@ Iteration 6-randrw-1024KiB complete (6 of 6), with 1 pass and 6 failures
/var/tmp/pbench-test-bench/pbench/tools-default/sar
--- pbench tree state
+++ pbench.log file contents
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-fio]pbench-fio is installed
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-fio]pbench-fio 2.14 is installed
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-fio]python-pandas is installed
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] fio_device_check() /tmp/fio
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] fio: Going to run [/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/bm --output-format=json /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/1-rw-4KiB/sample1/fio.job ]
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] post-processing fio result
Expand Down
10 changes: 6 additions & 4 deletions agent/bench-scripts/gold/pbench-fio/test-05.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
+++ Running test-05 pbench-fio
pbench-fio 2.14
pbench-fio
verifying clients have fio installed
python-pandas
The following jobfile was created: /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/1-throughput-4KiB/sample1/fio.job
[global]
bs=4k
Expand Down Expand Up @@ -565,8 +566,9 @@ Iteration 3-throughput-1024KiB complete (3 of 3), with 1 pass and 6 failures
/var/tmp/pbench-test-bench/pbench/tools-default/sar
--- pbench tree state
+++ pbench.log file contents
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-fio]pbench-fio is installed
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-fio]pbench-fio 2.14 is installed
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] verifying clients have fio installed
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-fio]python-pandas is installed
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] fio_device_check() /tmp/fio foo,bar
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] creating directories on the clients
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] opening port 8765 on firewall on the clients
Expand Down Expand Up @@ -914,7 +916,7 @@ Iteration 3-throughput-1024KiB complete (3 of 3), with 1 pass and 6 failures
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no bar mkdir -p /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/3-throughput-1024KiB/sample1
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no bar mkdir -p /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/3-throughput-1024KiB/sample1
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no bar mkdir -p /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/3-throughput-1024KiB/sample1
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no bar /opt/pbench-agent/bench-scripts/pbench-fio --install
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no bar /opt/pbench-agent/bench-scripts/pbench-fio --remote-only --install
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no bar pushd /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/1-throughput-4KiB/sample1 >/dev/null; screen -dmS fio-server bash -c ''/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/bm' --server 2>&1 >client-result.txt'
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no bar pushd /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/1-throughput-4KiB/sample1 >/dev/null; screen -dmS fio-server bash -c ''/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/bm' --server 2>&1 >client-result.txt'
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no bar pushd /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/1-throughput-4KiB/sample1 >/dev/null; screen -dmS fio-server bash -c ''/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/bm' --server 2>&1 >client-result.txt'
Expand Down Expand Up @@ -1005,7 +1007,7 @@ Iteration 3-throughput-1024KiB complete (3 of 3), with 1 pass and 6 failures
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no foo mkdir -p /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/3-throughput-1024KiB/sample1
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no foo mkdir -p /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/3-throughput-1024KiB/sample1
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no foo mkdir -p /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/3-throughput-1024KiB/sample1
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no foo /opt/pbench-agent/bench-scripts/pbench-fio --install
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no foo /opt/pbench-agent/bench-scripts/pbench-fio --remote-only --install
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no foo pushd /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/1-throughput-4KiB/sample1 >/dev/null; screen -dmS fio-server bash -c ''/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/bm' --server 2>&1 >client-result.txt'
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no foo pushd /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/1-throughput-4KiB/sample1 >/dev/null; screen -dmS fio-server bash -c ''/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/bm' --server 2>&1 >client-result.txt'
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/ssh -o StrictHostKeyChecking=no foo pushd /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/1-throughput-4KiB/sample1 >/dev/null; screen -dmS fio-server bash -c ''/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/bm' --server 2>&1 >client-result.txt'
Expand Down
6 changes: 4 additions & 2 deletions agent/bench-scripts/gold/pbench-fio/test-06.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
+++ Running test-06 pbench-fio
pbench-fio 2.14
pbench-fio
python-pandas
The following jobfile was created: /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/1-throughput-4KiB/sample1/fio.job
[global]
bs=4k
Expand Down Expand Up @@ -564,7 +565,8 @@ Iteration 3-throughput-1024KiB complete (3 of 3), with 1 pass and 6 failures
/var/tmp/pbench-test-bench/pbench/tools-default/sar
--- pbench tree state
+++ pbench.log file contents
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-fio]pbench-fio is installed
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-fio]pbench-fio 2.14 is installed
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] [pbench-fio]python-pandas is installed
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] fio_device_check() /tmp/fio
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] fio: Going to run [/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/bm --output-format=json /var/tmp/pbench-test-bench/pbench/fio__1900-01-01_00:00:00/1-throughput-4KiB/sample1/fio.job ]
/var/tmp/pbench-test-bench/pbench/pbench.log:[debug][1900-01-01T00:00:00.000000] post-processing fio result
Expand Down
2 changes: 1 addition & 1 deletion agent/bench-scripts/gold/pbench-uperf/test-00.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
+++ Running test-00 pbench-uperf
pbench-uperf 1.0.4
pbench-uperf
Starting iteration (1 of 2)
test sample 1 of 2
client[127.0.0.1]protocol[tcp]test[rr]instances[1]size[64] <-> server[127.0.0.1]
Expand Down
2 changes: 1 addition & 1 deletion agent/bench-scripts/gold/pbench-uperf/test-01.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
+++ Running test-01 pbench-uperf
pbench-uperf 1.0.4
pbench-uperf
Starting iteration (1 of 2)
test sample 1 of 2
client[c1]node[1]protocol[tcp]test[rr]instances[1]size[64] <-> server[s1]node[0]
Expand Down
60 changes: 44 additions & 16 deletions agent/bench-scripts/pbench-fio
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ maxstddevpct=5 # maximum allowable standard deviation in percent
max_failures=6 # after N failed attempts to hit below $maxstddevpct, move on to the nest test
supported_test_types="read,write,rw,randread,randwrite,randrw"
install_only="n"
remote_only="n"
config=""
rate_iops=""
test_types="read,randread" # default is -non- destructive
Expand Down Expand Up @@ -134,10 +135,13 @@ function fio_usage() {
printf -- "\t--install\n"
printf "\t\tinstall only\n"
printf "\t\tDefault is n\n"
printf -- "\t--remote-only\n"
printf "\trun this on the remotes only\n"
}

function fio_process_options() {
opts=$(getopt -q -o jic:t:b:s:d:r: --longoptions "help,max-stddev:,max-failures:,samples:,direct:,sync:,install,clients:,client-file:,iodepth:,ioengine:,config:,jobs-per-dev:,job-mode:,rate-iops:,ramptime:,runtime:,test-types:,block-sizes:,file-size:,targets:,tool-group:,postprocess-only:,run-dir:,directory:,numjobs:,job-file:,pre-iteration-script:," -n "getopt.sh" -- "$@");
opts=$(getopt -q -o jic:t:b:s:d:r: --longoptions "help,max-stddev:,max-failures:,samples:,direct:,sync:,install,remote-only,clients:,client-file:,iodepth:,ioengine:,config:,jobs-per-dev:,job-mode:,rate-iops:,ramptime:,runtime:,test-types:,block-sizes:,file-size:,targets:,tool-group:,postprocess-only:,run-dir:,directory:,numjobs:,job-file:" -n "getopt.sh" -- "$@");

if [ $? -ne 0 ]; then
printf "\t${benchmark}: you specified an invalid option\n\n"
fio_usage
Expand All @@ -154,6 +158,10 @@ function fio_process_options() {
shift;
install_only="y"
;;
--remote-only)
shift;
remote_only="y"
;;
--max-stddev)
shift;
if [ -n "$1" ]; then
Expand Down Expand Up @@ -359,7 +367,7 @@ function fio_process_options() {
;;
esac
done
if [ $postprocess_only == "n" ]; then
if [ "$postprocess_only" = "n" ]; then
benchmark_run_dir="$pbench_run/${benchmark}_${config}_$date"
else
if [ -z "$run_dir" ]; then
Expand All @@ -375,24 +383,36 @@ function fio_process_options() {

# Ensure the right version of the benchmark is installed
function fio_install() {
if [ "$postprocess_only" == "n" ]; then
if [ "$postprocess_only" = "n" ]; then
if check_install_rpm $benchmark_rpm $ver; then
debug_log "[$script_name]$benchmark_rpm is installed"
debug_log "[$script_name]$benchmark_rpm $ver is installed"
else
debug_log "[$script_name]$benchmark_rpm installation failed, exiting"
debug_log "[$script_name]$benchmark_rpm $ver installation failed, exiting"
exit 1
fi
fi

fi
if [ ! -z "$clients" ] ; then
debug_log "verifying clients have fio installed"
echo "verifying clients have fio installed"
for client in `echo $clients | sed -e s/,/" "/g`; do
ssh $ssh_opts $client ${pbench_install_dir}/bench-scripts/$script_name --install &
ssh $ssh_opts $client ${pbench_install_dir}/bench-scripts/$script_name --remote-only --install &
done
wait
fi
if [ "$install_only" == "y" ]; then
exit 0
}

# install python-pandas on the controller: fiologparser_hist.py needs it
function pandas_install() {
if [ "$postprocess_only" = "n" ]; then
if check_install_rpm python-pandas; then
debug_log "[$script_name]python-pandas is installed"
else
debug_log "[$script_name]python-pandas installation failed, exiting."
debug_log "[$script_name]On RHEL, python-pandas is in the EPEL repo."
debug_log "[$script_name]See https://fedoraproject.org/wiki/EPEL for details."
exit 1
fi
fi
}

Expand All @@ -401,7 +421,7 @@ function print_iteration {
printf "\n%28s" "$1" >>$benchmark_summary_txt_file
printf "\n%s" "$1" >>$benchmark_summary_csv_file
hists=`cat $iteration_dir/reference-result/fio.job 2>/dev/null | grep "^log_hist_msec"`
if [ $1 == "iteration" ]; then
if [ "$1" = "iteration" ]; then
# this is just a label, so no links here
if [ ! -z "$hist_interval" ]; then
printf "\n%28s %s %s %s" "iteration" "summary" "hist-results" "tools">>$benchmark_summary_html_file
Expand Down Expand Up @@ -442,7 +462,7 @@ function fio_device_check() {
local dev=""
local client=""
local rc=0
if [ "$postprocess_only" == "n" ]; then
if [ "$postprocess_only" = "n" ]; then
debug_log "fio_device_check() $devs $clients"
for dev in `echo $devs | sed -e s/,/" "/g`; do
if echo $dev | grep -q "^/dev/"; then
Expand Down Expand Up @@ -543,7 +563,7 @@ function fio_run_job() {
local client_opts=""

if [ ! -z "$client_file" ] ;then
typeset -i nclients=$(wc -l $client_file)
typeset -i nclients=$(wc -l $client_file | cut -d ' ' -f 1)
client_opts="--client=$client_file --max-jobs=$nclients"
elif [ ! -z "$clients" ]; then
local max_jobs=0
Expand Down Expand Up @@ -624,7 +644,7 @@ function fio_run_benchmark() {
echo "Restarting iteration $iteration ($count of $total_iterations)"
log "Restarting iteration $iteration ($count of $total_iterations)"
fi
if [ "$postprocess_only" == "n" ]; then
if [ "$postprocess_only" = "n" ]; then
mkdir -p $iteration_dir
else
if [ ! -e $iteration_dir ]; then
Expand Down Expand Up @@ -659,7 +679,7 @@ function fio_run_benchmark() {
fi
benchmark_results_dir="$iteration_dir/sample$sample"
benchmark_tools_dir="$benchmark_results_dir/tools-$tool_group"
if [ "$postprocess_only" == "n" ]; then
if [ "$postprocess_only" = "n" ]; then
mkdir -p $benchmark_results_dir
fio_job_file="$benchmark_results_dir/fio.job"
fio_create_jobfile "$test_type" "$ioengine" "$block_size" "$iodepth" "$direct" "$sync" "$runtime" "$ramptime" "$file_size" "$rate_iops" "$dev" "$fio_job_file"
Expand Down Expand Up @@ -757,11 +777,11 @@ function fio_run_benchmark() {
else
# delete the tool data [and respose time log for rr tests] from the other samples to save space
# this option is off by default
if [ "$keep_failed_tool_data" == "n" ]; then
if [ "$keep_failed_tool_data" = "n" ]; then
/bin/rm -rf $iteration_dir/$sample_dir/tools-* $iteration_dir/$sample_dir/response-times.txt
fi
# since non reference-result sample data is rarely referenced, tar it up to reduce the number of files used
if [ "$tar_nonref_data" == "y" ]; then
if [ "$tar_nonref_data" = "y" ]; then
pushd "$iteration_dir" >/dev/null; tar --create --xz --force-local --file=$sample_dir.tar.xz $sample_dir && /bin/rm -rf $sample_dir; popd >/dev/null
fi
fi
Expand Down Expand Up @@ -850,6 +870,14 @@ function fio_print_summary() {
fio_process_options "$@"
fio_install

# pandas installed only on controller
if [ "$remote_only" = "n" ] ;then
Copy link
Member

@portante portante Sep 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this supposed to only have one "=" for the comparison?

Copy link
Member Author

@ndokos ndokos Sep 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both = and == work the same way, at least when comparing strings in [ ... ] constructs. I use = mostly but as you noticed, I'm not consistent :-) I'll change the == to = throughout.

pandas_install
fi
if [ "$install_only" = "y" ]; then
exit 0
fi

mkdir -p $benchmark_run_dir
export benchmark config
pbench-collect-sysinfo --group=$tool_group --dir=$benchmark_run_dir beg
Expand Down
2 changes: 1 addition & 1 deletion agent/util-scripts/gold/pbench-register-tool/test-00.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
+++ Running test-00 pbench-register-tool
pbench-sysstat 11.2.0
pbench-sysstat
mpstat tool is now registered in group default
--- Finished test-00 pbench-register-tool (status=0}
+++ pbench tree state
Expand Down