Merge pull request cms-sw#205 from kmcdermo/minor_script_mods

minor fixups to scripts

README.md

@@ -226,15 +226,20 @@ The main script for collecting plots and sending them to LXPLUS can be called by
 ./web/move-benchmarks.sh ${outdir_name} ${suite} ${afs_or_eos}
 ```
 
-where again, ```${suite}``` defaults to ```forPR```. ```${outdir_name}``` will be the top-level directory where the output is collected and eventually shipped to LXPLUS. This will call ```./web/collectBenchmarks.sh ${outdir_name} ${suite}```, which will sort the files, and then call ```./web/tarAndSendToLXPLUS.sh ${outdir_name} ${suite} ${afs_or_eos}```, which packs up the top-level output dir and copies it to either an /afs or /eos userspace on LXPLUS. This will also run another script remotely on LXPLUS to copy ```web/index.php``` into each directory to have a nice web GUI for the plots. N.B. There are some assumptions on the remote directory structure, naming, and files present in order for ```web/tarAndSendToLXPLUS.sh``` to work. Please consult ```web/README_WEBPLOTS.md``` for setting this up properly!
+where again, ```${suite}``` defaults to ```forPR```. ```${outdir_name}``` will be the top-level directory where the output is collected and eventually shipped to LXPLUS. This script first calls ```./web/collectBenchmarks.sh ${outdir_name} ${suite}```, which will sort the files, and then calls the script ```./web/copyphp.sh```, which copies ```web/index.php``` into the ```${outdir_name}``` to have a nice GUI on the web, and finally calls ```./web/tarAndSendToLXPLUS.sh ${outdir_name} ${suite} ${afs_or_eos}```, which packs up the top-level output dir and copies it to either an /afs or /eos userspace on LXPLUS. 
 
-Lastly, the option ```${afs_or_eos}``` takes either of the following arguments: ```afs``` or ```eos```, and defaults to ```afs```. The mapping of the username to the remote directories is in ```web/tarAndSendToLXPLUS.sh```. If an incorrect string is passed, the script will exit. N.B. AFS is being phased out at CERN, so the preferred option is ```eos```.
+The option ```${afs_or_eos}``` takes either of the following arguments: ```afs``` or ```eos```, and defaults to ```eos```. The mapping of the username to the remote directories is in ```web/tarAndSendToLXPLUS.sh```. If an incorrect string is passed, the script will exit. 
+
+**IMPORTANT NOTES**
+1) AFS is being phased out at CERN, so the preferred option is ```eos```.
+
+2) There are some assumptions on the remote directory structure, naming, and files present in order for ```web/tarAndSendToLXPLUS.sh``` to work. Please consult ```web/README_WEBPLOTS.md``` for setting this up properly!
 
 **IMPORTANT DISCLAIMERS**
 
 1. There is a script: ```./xeon_scripts/trashSKL-SP.sh``` that is run at the very end of the ```./web/move-benchmarks.sh``` script that will delete log files, pngs, validation directories, root files, and the neat directory created to house all the validation plots.  This means that if the scp fails, the plots will still be deleted locally, and you will be forced to re-run the whole suite!!  You can of course comment this script out if this bothers you.
 
-2. ```web/tarAndSendToLXPLUS.sh``` has a number of pieces that execute code on LXPLUS, which includes executing scripts to copy ```web/index.php``` removing the copied over tarball of plots. If you are uncomfortable with this, you can comment it out.
+2. ```web/tarAndSendToLXPLUS.sh``` executes a script remotely on LXPLUS when using AFS, which makes the directory readable to outside world. If you are uncomfortable with this, you can comment it out. If your website is on EOS, then please ignore this disclaimer.
 
 ### Section 5.iv: Interpreting the results
 

val_scripts/validation-cmssw-benchmarks.sh

@@ -147,4 +147,8 @@ do echo ${!plot} | while read -r pN pO
     done
 done
 
+## Final cleanup
 make distclean ${mVal}
+
+## Final message
+echo "Finished physics validation!"

web/README_WEBPLOTS.md

@@ -10,7 +10,10 @@ Apologies on the ordered list style: GitHub Flavored Markdown ignores CSS/HTML s
    1) Intro
    2) Step-by-step instructions for setting up your website
    3) Setting permissions for your website
-   4) Integration with the scripts in this directory
+   4) Special Notes
+      1) Disclaimer on email addresses
+      2) Notes on ```${webdir}```
+      3) Passwordless scp on LXPLUS
 2) DEPRECATED: Setting up a website with an AFS userspace
 
 ## Section 1: Setting up a website with an EOS userspace
@@ -72,35 +75,25 @@ Upon trying to access your website now via a web browser (or another user's webs
 
 **Some discussion on the ```.htaccess``` file**: If you would like to use your personal website for more than just this project, it is recommended that you create a subdirectory ```${mictrkdir}``` under ```${webdir}```. In ```${mictrkdir}```, you then can create another ```.htaccess``` file which includes the restricted access for only members of the mic-trk e-group using the line from above: ```Require ADFS_GROUP mic-trk-rd```. This line would then need to be removed in your top-level ```${webdir}/.htaccess```, in case you want others to have access to other subdirectories related to physics analysis, RECO convener duties, etc. 
 
-**IMPORTANT DISCLAIMER**: It is imperative that you have your primary email address associated to your CERN account (go to CERN accounts to check this) be the same email used for sending+receiving emails from the mictrk e-group. Otherwise, the line ```Require ADFS_GROUP mic-trk-rd``` will lock you out of viewing your own website on a browser! Unless you have some special CERN account, your primary email for your CERN account is a ```@cern.ch``` Outlook address. 
-
-### Section 1.iv: Integration with the scripts in this directory
-
-In order to take advantage of the ```web/move-benchmarks.sh``` script, you will need to copy the script: ```copyphp.sh``` and php file: ```index.php``` to your top-level ```${webdir}``` on LXPLUS, i.e.:
-
-```
-scp copyphp.sh index.php ${USERNAME}@lxplus.cern.ch:/eos/user/${FIRST_LETTER}/${USERNAME}/${webdir}
-```
-
-where, again ```${webdir}``` == "www" if you followed the CERNBox instructions exactly. 
-
-When running ```./web/move-benchmarks ${plotdir}```, the ```copyphp.sh ${plotdir}``` script will run remotely, assuming this script can be found in your ```${webdir}```. ```copyphp.sh``` will copy ```index.php``` into ```${plotdir}``` as well as all subdirectories underneath it.
-
-**Note on ```${webdir}```**
+### Section 1.iv: Special Notes
+
+#### Section 1.iv.a: Disclaimer on email addresses
+
+It is imperative that you have your primary email address associated to your CERN account (go to CERN accounts to check this) be the same email used for sending+receiving emails from the mictrk e-group. Otherwise, the line ```Require ADFS_GROUP mic-trk-rd``` will lock you out of viewing your own website on a browser! Unless you have some special CERN account, your primary email for your CERN account is a ```@cern.ch``` Outlook address. 
 
+
+#### Section 1.iv.b: Notes on `${webdir}`
+
 - If ```${webdir} != "www"```, then you will have to modify the variable ```LXPLUS_OUTDIR``` in ```web/copyAndSendToLXPLUS.sh``` to match the name for ```${webdir}```. 
-- If you decided to make a subdirectory under ```${webdir}``` specifically for this project, then you will have to **either**:
-  1) Set ```LXPLUS_OUTDIR=${webdir}/${mictrkdir}```, and then move ```index.php``` and ```copyphp.sh``` to ```${mictrkdir}```.
-  2) **Or**, keep ```index.php``` and ```${copyphp.sh}``` in ```${webdir}```, and make the following changes to ```web/copyAndSendToLXPLUS.sh```
-     1) Make a new variable ```LXPLUS_WEBDIR=${webdir}```, and set ```LXPLUS_OUTDIR=${mictrkdir}```.
-     2) Modify the ```scp``` to be: ```scp -r ${tarball} ${LXPLUS_HOST}:${LXPLUS_WORKDIR}/${LXPLUS_WEBDIR}/${LXPLUS_OUTDIR}```
-     3) Modify the ```cd``` to be: ```cd ${LXPLUS_WORKDIR}/${LXPLUS_WEBDIR}/${LXPLUS_OUTDIR}```
-     4) Add this line under the untar (i.e. ```tar -zxvf```): ```cd ${LXPLUS_WORKDIR}/${LXPLUS_WEBDIR}```
-     4) Modify the call to ```copyphp.sh``` to be: ```./copyphp.sh ${LXPLUS_OUTDIR}/${dir}```
-
-While option ii) is significantly more modifications, it allows the user to keep the ```copyphp.sh``` script in the top-level ```${webdir}``` for future use with other projects.
-
-Lastly, make sure to read Section 10.ii.b in the main README.md on how to take advantage of passwordless scp for transferring plots to LXPLUS via ```./web/move-benchmarks.sh ${plotdir}```.
+- If you decided to make a subdirectory under ```${webdir}``` specifically for this project, then may wish to make the following modifications to: ```web/copyAndSendToLXPLUS.sh```
+  1) Make a new variable ```LXPLUS_WEBDIR=${webdir}```, and set ```LXPLUS_OUTDIR=${mictrkdir}```.
+  2) Modify the ```scp``` to be: ```scp -r ${tarball} ${LXPLUS_HOST}:${LXPLUS_WORKDIR}/${LXPLUS_WEBDIR}/${LXPLUS_OUTDIR}```
+  3) Modify the ```cd``` to be: ```cd ${LXPLUS_WORKDIR}/${LXPLUS_WEBDIR}/${LXPLUS_OUTDIR}```
+  4) Add this line under the untar (i.e. ```tar -zxvf```): ```cd ${LXPLUS_WORKDIR}/${LXPLUS_WEBDIR}```
+
+#### Section 1.iv.c: Passwordless scp to LXPLUS
+
+Make sure to read Section 10.ii.b in the main README.md on how to take advantage of passwordless scp for transferring plots to LXPLUS via ```./web/move-benchmarks.sh ${plotdir}```.
 
 ## Section 2: DEPRECATED: Setting up a website with an AFS userspace
 
@@ -119,7 +112,7 @@ Lastly, make sure to read Section 10.ii.b in the main README.md on how to take a
    4) ```touch .htaccess```
    5) open .htaccess in an editor and paste the following: ```Options +Indexes```
 
-3) Then copy in really the very useful ```index.php``` into ```${dir}```
+3) Then copy in really the very useful ```index.php``` into ```${dir}``` (optional: will simply make the top-level web GUI nice)
 
 4) Once set up and website is live, copy plots and directories into ```${dir}```
 5) ```cd ${dir}```

web/collectBenchmarks.sh

@@ -216,3 +216,6 @@ do
 	done
     done
 done
+
+# Final message
+echo "Finished collecting benchmark plots into ${dir}!"

web/copyphp.sh

@@ -3,4 +3,4 @@
 dir=${1}
 
 #cp index.php into all subdirectories
-find ${dir} -mindepth 0 -type d -exec cp index.php {} \;
+find ${dir} -mindepth 0 -type d -exec cp web/index.php {} \;

web/makereadable.sh

@@ -4,5 +4,3 @@ dir=${1}
 
 fs setacl ${dir} webserver:afs read
 afind ${dir} -t d -e "fs setacl -dir {} -acl webserver:afs read"
-
-./copyphp.sh ${dir}

web/move-benchmarks.sh

@@ -3,7 +3,7 @@
 # command line input
 dir=${1:-"benchmarks"} # Main output dir name
 suite=${2:-"forPR"} # which set of benchmarks to run: full, forPR, forConf
-afs_or_eos=${3:-"afs"} # which user space to use: afs or eos
+afs_or_eos=${3:-"eos"} # which user space to use: afs or eos
 
 # source global variables
 source xeon_scripts/common-variables.sh ${suite}
@@ -13,11 +13,18 @@ source xeon_scripts/init-env.sh
 echo "Moving plots and text files locally to ${dir}"
 ./web/collectBenchmarks.sh ${dir} ${suite}
 
-# Now copy to lxplus
+# Next copy index.php into ouput dir
+echo "Copying index.php into ${dir}"
+./web/copyphp.sh ${dir}
+
+# Then copy to lxplus
 echo "Moving plots and text files remotely to lxplus"
 ./web/tarAndSendToLXPLUS.sh ${dir} ${suite} ${afs_or_eos}
 
 # Final cleanup of directory
 echo "Removing local files"
 ./xeon_scripts/trashSKL-SP.sh 
 rm -rf ${dir}
+
+# Final message
+echo "Finished moving benchmark plots to LXPLUS!"

web/tarAndSendToLXPLUS.sh

@@ -3,7 +3,7 @@
 # command line input
 dir=${1:-"benchmarks"} # Main output dir name
 suite=${2:-"forPR"} # which set of benchmarks to run: full, forPR, forConf
-afs_or_eos=${3:-"afs"} # which user space to use: afs or eos
+afs_or_eos=${3:-"eos"} # which user space to use: afs or eos
 
 # in case this is run alone
 source xeon_scripts/common-variables.sh ${suite}
@@ -36,7 +36,7 @@ scp -r ${tarball} ${LXPLUS_HOST}:${LXPLUS_WORKDIR}/${LXPLUS_OUTDIR}
 # Make outdir nice and pretty
 if [[ "${afs_or_eos}" == "afs" ]]
 then
-    echo "Unpack tarball and execute remotely ./makereadable.sh ${dir}"
+    echo "Unpacking tarball and executing remotely: ./makereadable.sh ${dir}"
     SSHO ${LXPLUS_HOST} bash -c "'
     cd ${LXPLUS_WORKDIR}/${LXPLUS_OUTDIR}
     tar -zxvf ${tarball}
@@ -45,16 +45,18 @@ then
     exit
     '"
 else
-    echo "Unpack tarball and execute remotely ./copyphp.sh ${dir}"
+    echo "Unpacking tarball"
     SSHO ${LXPLUS_HOST} bash -c "'
     cd ${LXPLUS_WORKDIR}/${LXPLUS_OUTDIR}
     tar -zxvf ${tarball}
-    ./copyphp.sh ${dir}
     rm -rf ${tarball}
     exit
     '"
 fi
 
 # remove local tarball
-echo "Remove local tarball of plots"
+echo "Removing local tarball of plots"
 rm ${tarball}
+
+# Final message
+echo "Finished tarring and sending plots to LXPLUS!"

xeon_scripts/benchmark-cmssw-ttbar-fulldet-build.sh

@@ -141,3 +141,6 @@ done
 
 ## Final cleanup
 make distclean ${mOpt}
+
+## Final message
+echo "Finished compute benchmarks on ${ben_arch}!"

xeon_scripts/runBenchmark.sh

@@ -41,3 +41,6 @@ echo "Producing plots from text files"
 
 ##### Final cleanup #####
 make distclean
+
+##### Final message #####
+echo "Finished benchmarking and validation suite!"