Move the MET nightly build and Fortify runs from dakota to kiowa as the met_test user.

[JohnHalleyGotway]

Describe the Task

Currently, nightly builds for the develop and main_v9.1 branch are run on dakota as johnhg. In addition, fortify is run as johnhg, writing to the same output directory as the NB run. Then, some scripts are run as bullock on dakota to scrape the fortify pdf files and email if any counts have changed.

This task is to do the following:
(1) Reorganize the scripts directory into build, regression, and fortify sub-directories.
(2) Add a dev_met_env directory with files defining the required environment for building on dakota or kiowa. Add files for dev_met_env.dakota and dev_met_env.kiowa.
(3) Update the scripts to read those environment setting files.
(4) Move the cron jobs over from dakota to kiowa and run them as the met_test user.

Time Estimate

1 - 2 days.

Sub-Issues

Consider breaking the task down into sub-issues.
No sub-issues required.

Relevant Deadlines

None.

Funding Source

None.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required: John HG
• Select scientist(s) or no scientist required: No scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
• Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy
  None.

Task Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s), Project(s), Milestone, and Linked issues
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

Reorganized these scripts into directories and made a few improvements. Below is an email from Randy explaining how the Fortify scraping and comparison is done:

It's not a script, and it's not just one job, but it should be easy to turn into a single script. It all takes place in /home/bullock/fortify, if you need to browse there.

There are three separate cron jobs. The first one, at 6:00 am, copies the pdf summary produced by fortify into a local area. Here's the script:

   cp -u -v /d3/projects/MET/MET_regression/develop/NB*/fortify_sca/MET-develop/met/MET-develop_*.pdf   /home/bullock/fortify/pdf_new

  chmod 444 /home/bullock/fortify/pdf_new/*

The 2nd cron job, at 6:15 am, runs /usr/bin/ps2ascii on the pdf to convert it into an ascii file. The ascii output is truly awful to look at, but it let's us use things like "grep Low", "grep High", and "grep Critical" on it to grab the counts. The counts are then appended to a file /home/bullock/fortify/counts_new.txt. I ran "tail" on that file just now, and here's the output:

 9 27 2020 1024 351 0
 9 28 2020 1024 351 0
 9 29 2020 1024 351 0
 9 30 2020 1024 351 0
 10 1 2020 1024 351 0
 10 2 2020 1024 351 0
 10 3 2020 1024 351 0
 10 4 2020 1024 351 0
 10 5 2020 1024 351 0
 10 6 2020 1053 353 0

The columns are month, day, year, low count, high count, critical count. This tool, /usr/bin/pdf2ascii, is the the core tool that makes this whole thing work.

The last cron job, running at 6:30 am, reads the counts file, makes sure that the counts for both today and yesterday are there, does the subtraction and composes and sends the email. (Or doesn't send the email, if all the differences are zero.)

Shouldn't be too hard to make this into a single script. I never got around to it ... instead, I just adapted the code that produced the old png graphics (remember those?), and called it good enough.

[JohnHalleyGotway]

pdftohtml -f 2 -l 2 MET-develop_20201006_rev.pdf(base) met_test@kiowa:/d1/projects/MET/MET_regression/fortify/NB20201006/MET-develop/met$ grep -A 1 High MET-develop_20201006_revs.html
High

353

(base) met_test@kiowa:/d1/projects/MET/MET_regression/fortify/NB20201006/MET-develop/met$ grep -A 1 Medium MET-develop_20201006_revs.html
(base) met_test@kiowa:/d1/projects/MET/MET_regression/fortify/NB20201006/MET-develop/met$ grep -A 1 Low MET-develop_20201006_revs.html
Low

1053

[JohnHalleyGotway]

OK, this work is finished. Listed below is the new crontab entry for the met_test user on kiowa:

# Run nightly tests for MET
10 00 * * * /d1/projects/MET/MET_regression/MET-develop/scripts/regression/test_nightly.sh develop   > /d1/projects/MET/MET_regression/develop/NB`date +\%Y\%m\%d`.cronlog 2>&1
10 02 * * * /d1/projects/MET/MET_regression/MET-develop/scripts/regression/test_nightly.sh main_v9.1 > /d1/projects/MET/MET_regression/main_v9.1/NB`date +\%Y\%m\%d`.cronlog 2>&1
00 04 * * * /d1/projects/MET/MET_regression/MET-develop/scripts/fortify/run_nightly.sh     develop   > /d1/projects/MET/MET_regression/fortify/NB`date +\%Y\%m\%d`.cronlog 2>&1

These are running fine now and exercise:

• fortify/run_fortify_sca.sh and fortify/run_nightly.sh
• regression/test_nightly.sh, regression/test_regression.sh, and regression/test_unit.sh

Testing of 2 additional scripts is shown below:

• build/met_build.sh and build/met_checkout_and_build.sh

# creates met-9.1.20201007.tar.gz
/d1/projects/MET/MET_regression/MET-develop/scripts/build/met_checkout_and_build.sh tag met-9.1 
# creates met-develop.20201007.tar.gz
/d1/projects/MET/MET_regression/MET-develop/scripts/build/met_checkout_and_build.sh branch develop 

The result in a tarfile named "".

• regression/regression_runtimes.ksh on kiowa

cd /d1/projects/MET/MET_regression
MET-develop/scripts/regression/regression_runtimes.ksh develop/NB20201007

Output is:

develop-ref = 14.545  	develop = 14.704  	DIFF =	0.159   	TEST = ascii2nc_TRMM_3hr
develop-ref = 18.763  	develop = 18.982  	DIFF =	0.219   	TEST = ascii2nc_TRMM_12hr
develop-ref = 0.292   	develop = 0.289   	DIFF =	-0.003  	TEST = ascii2nc_GAGE_24hr
...