A few scripts to automate approval / testing process
Python Shell C JavaScript Other
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
DMWM Don't check __init__.py and don't worry about print, futurize natural… Dec 28, 2016
batch-relvals Added a script to run relvals using the batch-system Mar 21, 2014
comparisons add deep CSV tags Feb 2, 2017
das-utils Update ib-eos-files.py Feb 9, 2017
es-kibana-dashboards renames es schema files Jul 26, 2016
es-templates es tempaltes updated Jun 10, 2016
jenkins more cleanup for keytab Feb 21, 2017
stitched simplify stitched port logic Aug 18, 2016
templates typp fixed for IB pages Bugdet to Budget Oct 19, 2016
tests new config flag NO_IB to not build IB Sep 5, 2016
.gitignore Initial version. Aug 21, 2013
.travis.yml Add test for config.map schema. Oct 15, 2014
FAQ.md Answer to @wddgit question in form of a FAQ. Apr 27, 2015
IBPageHead.txt added missing txt pages for build sets Oct 29, 2015
IBPageTail.txt added missing txt pages for build sets Oct 29, 2015
README.md updates README file Jun 6, 2016
RelValArgs.py use SIGSEGV for all CMSSW_[1-7] and SIGTERM for any thing above Nov 30, 2016
add-externals-gh-labels.py move api_rate_limits in github_utils and use it from various jobs Nov 17, 2016
auto-create-pull-requests increase default timeout Jul 18, 2016
auto-update-git-branches make use of local cmssw.git ref Jul 19, 2016
autoInstall.sh do not assume that cms-bot is always available in worksapce Sep 26, 2016
autoInstallLogs.sh follow old repo symlink Jun 19, 2016
backport-pr.py fix backport pr Dec 7, 2016
build-cmssw-ib-with-patch allow to build only cmssw-tool-conf Feb 7, 2017
build-cmssw-validation-with-patch add root.spec in the debug package list Jul 29, 2015
build-fwlite-ib remove cmssdtdev ref Sep 23, 2016
build-node-stats do nothing for now to avoid the email flood Aug 6, 2015
build-release force delete any cmsdist tag for release before creating a new one Oct 17, 2016
build-release-watchers.yaml Remove myself from all scripts. Jul 13, 2016
buildLogAnalyzer.py updated build logs parser to find all errors Jan 20, 2017
buildSetTest.py move ib validation scripts in here from int-build Oct 29, 2015
build_hosts.txt updates build hosts file Dec 7, 2016
categories.py add DQM/EcalMonitorDbModule under dqm Feb 9, 2017
category-watchers.yaml Update category-watchers.yaml Apr 25, 2016
checkDirSizes.py move ib validation scripts in here from int-build Oct 29, 2015
checkLibDeps.py move ib validation scripts in here from int-build Oct 29, 2015
checkLogFile.py move ib validation scripts in here from int-build Oct 29, 2015
checkPyConfigs.py move ib validation scripts in here from int-build Oct 29, 2015
checkTestLog.py fix for unit test log parsing Feb 13, 2016
cleanup-auto-build use userbase build directory for auto-build cleanup Sep 20, 2016
cleanup-cmssdt Update cleanup-cmssdt Dec 9, 2016
cleanup-tags keep only last 30 ib tags Jan 4, 2016
cms-externals-pr-test Distinguish between posting the results to cmsdist or the external re… Dec 10, 2014
cms-jenkins-api Add script to communicate with jenkins with a sso connection. Apr 17, 2014
cms_static.py use 13028 and 13029 issues as cms-bot commands Jan 22, 2016
cmsdist-comp-pr-process.py ping command to not override the any previous command Dec 15, 2016
cmsdist_merge_permissions.py if comment not valid then return false Dec 12, 2016
cmssw-pr-test-config 2017 WFs now in limited, not needed as extra tests Jun 28, 2016
cmssw_authors.py updates github committers list Nov 15, 2016
cmsutils.py try to run one more cmsRun process Nov 3, 2015
comment-gh-pr increase default timeout Jul 18, 2016
compare-material-budget use material-budget directory Oct 18, 2016
config-disabled.map use V00-30-XX pkgtools for every branch Dec 5, 2016
config.map added 83X build rules Feb 20, 2017
create-gh-pr.py removes extra lines Dec 15, 2016
create-gh-release.py fixed typo for creating GH release Sep 5, 2016
create-github-hooks increase default timeout Jul 18, 2016
cvmfsInstall.sh remove old week support Feb 6, 2017
deprecate-releases Work without whitelist. Dec 21, 2013
deprecate_releases.py Update deprecate_releases.py Dec 14, 2016
docker_launcher.sh more cleanup for keytab Feb 21, 2017
es-cleanup-indexes Cleanup more spurios logs. Mar 31, 2015
es-reindex An helper script to copy indexes around. Jan 19, 2015
es_cmsdoxygen_apache.py es monitor for doxygen added Sep 23, 2016
es_cmsrep_apache.py typo Sep 21, 2016
es_cmssdt_apache.py typo Sep 21, 2016
es_doxygen_apache.py make es_doxygen_apache.py executable Oct 21, 2016
es_hypernews.py add hn heart beat Mar 18, 2016
es_hypernews_log.py process mail log , input method changed , timestamp info updated Apr 1, 2016
es_ib_build_stats.py Update es_ib_build_stats.py May 15, 2016
es_ibs_log.py due to local ibeos files , now no need to track block/datasets Nov 14, 2016
es_iwyu_logs.py es tempaltes updated Jun 10, 2016
es_relval_log.py keep track of all requested data files Dec 8, 2016
es_utils.py ignores ssl verify Nov 9, 2016
forward-pull-requests increase default timeout Jul 18, 2016
forward_ports_map.py Add support for slc6_amd64_gcc700 Jan 20, 2017
gen-relval-jobs.py adding verbose info and error handling Apr 19, 2016
generate-categories-json Added mapping of people to categories to simpily visualization Oct 17, 2014
generate-json-performance-charts added yMin=0 to the url of the graphs May 8, 2014
generate-performance-summary Allow specifing non local server. Jun 6, 2014
get-git-tags use http header info to get rate limits to avoid extra api call Sep 6, 2016
get-pr-branch remove argparse deps and use directly argv Feb 21, 2017
get-production-arch fix production arch logic to use correct release queue Dec 15, 2016
get_repo_authors.py allow all cmssw authors with more than 10 commits to start tests Nov 15, 2016
github-rate-limits use http header info to get rate limits to avoid extra api call Sep 6, 2016
github_hooks_config.py enable push hook for cmsdist and cmssw Jan 4, 2016
github_utils.py get commit from tag/branch for release build issues Dec 15, 2016
github_webhook use jenkins alias instead of vm host name Sep 14, 2016
githublabels.py added requires-external label is cmssw PR depends on cmsdist changes Feb 17, 2017
gitmergesgraph.py IB Pages: use regular expression to identify automatic forward ports May 6, 2015
groups.yaml adding new BTV software conveners Jul 26, 2016
ib-cleanup-obsolete-week Make scripts executables. Mar 18, 2015
ib-pr-workflow-changed.py added das_client for IBs and PR testing Oct 26, 2016
ib-rotate-weeks use new week scheme to rotating ibs Jan 14, 2017
ib-weeks Added new ib-week nweek-02460 Feb 18, 2017
ignore-releases-for-tests Update ignore-releases-for-tests May 12, 2016
import-stitched.sh only create cmssw work area if git release is set Apr 26, 2016
init.sh helper bach script added Nov 9, 2015
is-weekly-ib-available.sh Update is-weekly-ib-available.sh Oct 6, 2016
jenkins-artifacts check if jenkins arts exists Oct 14, 2016
kill-build-release pass remote workspace to kill the release build Aug 27, 2015
logRootQA.py add my logroot qa to run-pr-comparisons Aug 6, 2015
logUpdater.py remove cmssdtdev ref Sep 23, 2016
logwatch.py es monitor for doxygen added Sep 23, 2016
merge-git-branch send git pull failed messages for auto merge of git branches Mar 1, 2016
merge-pull-request increase default timeout Jul 18, 2016
mirror-git-repositories updated git mirror for gitlab Jun 7, 2016
mirror-root git mirror script updated Jun 7, 2016
modify_comment.py increase default timeout Jul 18, 2016
new-release-cycle increase default timeout Jul 18, 2016
parse_iwyu_logs.py removes duplicate lines in the iwyu html report pages Jun 14, 2016
parse_jenkins_builds.py Update parse_jenkins_builds.py Feb 29, 2016
parse_workflow_time.py add python script for workflow time calculation Oct 19, 2015
port-pull-request.py increase default timeout Jul 18, 2016
pr-schedule-tests for master branch return the release queue Feb 20, 2017
prepare-repo-clone-for-port.sh do not push or create PR for dryrun Dec 7, 2015
process-build-release-request fix production arch logic to use correct release queue Dec 15, 2016
process-cmsdist-request increase default timeout Jul 18, 2016
process-error-reports increase default timeout Jul 18, 2016
process-partial-logs-relval.py bug fix: accidently forgot to parse the logs Oct 23, 2015
process-pull-request Update process-pull-request Nov 17, 2016
process_pr.py orp rejection should not close PR Feb 20, 2017
query-and-process-prs move api_rate_limits in github_utils and use it from various jobs Nov 17, 2016
query-new-pull-requests use the full repo name Aug 24, 2015
release-deploy-afs run cmspkg upgrade to get latest cmspkg Jul 23, 2016
release-notes increase default timeout Jul 18, 2016
releases.map Add CMSSW_7_1_25_patch5 for slc6_amd64_gcc481 (production arch) as Pr… Feb 14, 2017
releases.py Update releases.py Dec 28, 2016
report-build-release-status afs installation skip message added Jan 10, 2017
report-cmsdist-pull-request-results increase default timeout Jul 18, 2016
report-pull-request-results use correct directory path for alternative-comparisons Nov 14, 2016
report-summary-merged-prs do not show ib page for NO_IB Jan 12, 2017
report_size.py converted report_size.bash to report_size.py and report uniq volumes May 11, 2016
run-cmsdist-tests.sh Update run-cmsdist-tests.sh Feb 6, 2017
run-hlt-validation run voms-proxy-init Aug 9, 2016
run-ib-addon.py timeout addon tests after 1.5 hours Sep 27, 2016
run-ib-comparison-baseline use run-ib-testbase for run-ib-comparison-baseline Nov 11, 2016
run-ib-geometry pushd used so no need to have cd Mar 15, 2016
run-ib-igprof make use of run-ib-testbase.sh for igprof test Nov 11, 2016
run-ib-iwyu use send_jenkins_artifacts to send iwyu results to cmssdt Nov 11, 2016
run-ib-material-budget split material-budget in so that it can be run from pr tests too Oct 11, 2016
run-ib-relval.py reduce parallel workflows for threaded ibs Jan 11, 2017
run-ib-testbase.sh some debug info for ib test Dec 8, 2016
run-igprof-pr more cmsbuild.keytab cleanup Feb 21, 2017
run-material-budget compare-material-budget update to compare Pr tests results too Oct 11, 2016
run-pr-comparisons validateJR.sh is executable now Feb 2, 2017
run-pr-tests more cmsbuild.keytab cleanup Feb 21, 2017
run-tests-release typo: fix das client wrapper name Nov 15, 2016
runPyRelValThread.py Update runPyRelValThread.py Oct 6, 2016
runTests.py ignore ExpressionEvaluatorUnitTest for lxplus Mar 16, 2016
run_clang_static_analysis Update run_clang_static_analysis Oct 11, 2016
run_class_dumper.sh Remove -disable-check options from checker options May 24, 2016
schedule-additional-tests use local config.map file Dec 7, 2016
scram-package-monitor-sender Split scram package monitor in 2 scripts. Jun 2, 2015
scram-package-monitor-timestamps Split scram package monitor in 2 scripts. Jun 2, 2015
splitDepViolationLog.py move ib validation scripts in here from int-build Oct 29, 2015
splitUnitTestLog.py fix for unit test log parsing Feb 13, 2016
super-users.yaml Decommissioning @ktf. Sic transit gloria mundi. Apr 28, 2015
syncInstall.sh do not assume that cms-bot is always available in worksapce Sep 26, 2016
tag-and-schedule-ibs Update tag-and-schedule-ibs Feb 7, 2017
tag-ib increase default timeout Jul 18, 2016
update-github-hooks-ip create github hook ip file in /data/sdt Nov 30, 2015
update-release-map commit meesage for release to contain the arch type Mar 23, 2016
updateVOTags Add updateVOTags to cms-bot. Jan 22, 2014
upload-release disable debug option for release build/upload Oct 5, 2015
watchers.yaml Update watchers.yaml for makortel Feb 8, 2017

README.md

Introduction.

cms-bot started as a single script used to drive PR approval and grew to be the core of the whole release engineering process for CMSSW.

Setup

To have it working you'll need a ~/.github-token which can access the cms-sw organization.

Release integration

process-pull-request: this is the script which updates the status of a CMSSW PR. It parses all the messages associated to the specified PR and if it spots a transition (e.g. a L2 signature) it posts a message acknowledging what happended, updates the tags etc. The state of the PR is fully obtained by parsing all the comments, so that we do not have to maintain our own state tracking DB.

  • watchers.yaml: contains all the information required by process-pull-requests to notify developers when a PR touches the packages they watch.

Release building

  • process-build-release-request
  • release-build: script used to build a release which has been requested through a Github issue.
  • upload-release: script used to upload a release to the repository. When the job processing build requests spots a request to upload, it SSH to the build machine which has the release and executes this script.
  • config.map: semicolon separated key=value pairs formatted file with release queue related information. Each line represent a release queue. In particular CMSDIST_TAG is used to point to the CMSDIST tag to be used by the release building process.

Logging

Logging happens at many different level but we are trying to unify things using Elasticsearch for "live" data from which we dump precomputed views on a basis.