Jenkins: Performance data upload from Compy, Cori

[sarats]

This script facilitates automatic data upload from Compy's
performance archive into PACE.

It requires the Jenkins user to run this manually the first time
to use the Github credentials to create an authorized token for
later uploads automatically.

It stores the token information in $HOME/.pacecc.

It does not require cloning of E3SM sources or builds.

It moves certain files from the machine's PERF_ARCHIVE_DIR to the
older (OLD_PERF_ARCHIVE_DIR) after data is processed.

It deletes very large files (>50MB) that are not meant to be placed in the
performance archive by design.

Testing:
Successfully uploaded performance data from Compy by commenting out the boilerplate initialization.
#source $SCRIPTROOT/util/setup_common.sh
I didn't load any Python module, system Python was sufficient. In case of Python environment issues, just unload any custom Python envs.

[sarats]

As this tool doesn't need E3SM sources/builds, think if it needs all of the standard boilerplate initialization that's used for running E3SM tests.

As mentioned, it requires manual intervention the first time it's run by the Jenkins user to create a Github token and automates future uploads.

General info:
https://climatemodeling.science.energy.gov/technical-highlights/performance-analytics-computational-experiments-pace
Video - How to Upload Data
https://www.youtube.com/watch?v=Vd3q3mJPQFs&feature=youtu.be

[sarats]

How do I designate a file to be uploaded into dashboard from Jenkins? In this case, I want the pace log file to be easily accessible from there.

[sarats]

@worleyph @rljacob Just FYI.

[sarats]

Relevant excerpt from running standalone script:

+ SCRIPTROOT=/qfs/people/sree118/repos/E3SM_test_scripts
+ export CIME_MACHINE=compy
+ CIME_MACHINE=compy
+ PERF_ARCHIVE_DIR=/compyfs/performance_archive
+ OLD_PERF_ARCHIVE_DIR=/compyfs/OLD_PERF
+ cd /compyfs/performance_archive
+ '[' 0 -ne 0 ']'
+ wget -O pace-upload https://pace.ornl.gov/static/tools/pace-upload
--2020-05-28 15:05:18--  https://pace.ornl.gov/static/tools/pace-upload
Resolving pace.ornl.gov (pace.ornl.gov)... 128.219.184.19
Connecting to pace.ornl.gov (pace.ornl.gov)|128.219.184.19|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11154 (11K) [application/octet-stream]
Saving to: ‘pace-upload’

100%[==================================================================================================================>] 11,154      --.-K/s   in 0s

2020-05-28 15:05:18 (63.3 MB/s) - ‘pace-upload’ saved [11154/11154]

+ '[' 0 -ne 0 ']'
+ chmod +x pace-upload
+ echo 'Large file list:'
Large file list:
+ find . -size +50M -exec ls -lh '{}' ';'
+ find . -size +50M -exec rm '{}' ';'
+ '[' 0 -ne 0 ']'
++ date +%Y_%m_%d
+ curdate=2020_05_28
+ perl e3sm_perf_archive.perl
+ mv e3sm_perf_archive_compy_2020_05_28_out.txt performance_archive_compy_all_2020_05_28
+ ./pace-upload --perf-archive ./performance_archive_compy_all_2020_05_28
Authorizing sarats for PACE...

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
            Welcome to PACE sarats

Validating Experiments in ./performance_archive_compy_all_2020_05_28...
Validation Success

Compressing valid experiments...
Compression Success

Uploading...
Upload Success

Parsing experiments...
Parse Success

Downloading Report log...
Download Success
Report saved as: 'pace-sarats-2020-05-28-18:11:57.log'


+ '[' 0 -ne 0 ']'
+ mv pace-sarats-2020-05-28-18:11:57.log performance_archive_compy_all_2020_05_28
+ tar zcf performance_archive_compy_all_2020_05_28.tar.gz performance_archive_compy_all_2020_05_28
++ date +%Y-%m
+ curmonth=2020-05
+ mkdir -p /compyfs/OLD_PERF/2020-05
+ mv performance_archive_compy_all_2020_05_28 performance_archive_compy_all_2020_05_28.tar.gz /compyfs/OLD_PERF/2020-05

[wadeburgess]

You want jenkins to run your script, and this will upload a file to the E3SM CDash project? If this is correct, in which section of CDash will your file appear? None of the existing sections seem to be a good fit. The CDash is for testing results. Is there a better place for the file to land than CDash?

[sarats]

I think a new Cdash section for PACE might not be a bad idea. We will have an uploader for each major platform and I foresee nightly uploads. The machine POCs would be responsible for checking the logs, so would probably be better to add to Cdash rather than something else.

[rljacob]

Can we just make a new cdash dashboard? I want the E3SM one to focus just on E3SM tests.

[sarats]

@rljacob These are just nightly upload tasks, not really tests. It seems like an overkill to create an entire dashboard for a handful of tasks (compy, cori, anvil, alcf) but if you really want them separate, we can.

For now, let's get the Jenkins tasks there without the dashboard stuff. If this script design looks ok, I will template the other platform scripts based on this.

[jgfouca]

You can remove all the manual error checking.

[sarats]

I have refactored the common core to a utility script (pace_archive.sh) for reuse on different platforms beginning with Compy and Cori.

The first time this script is run, it stores following details in Jenkins user's $HOME/.pacecc:

$ cat ~/.pacecc 
[GITHUB]
username = <user>
token = <github_token>

[sarats]

cc @philipwjones

[jgfouca]

Looks good