-
Notifications
You must be signed in to change notification settings - Fork 38
CRAB Testing
This page describes CRABServer and CRABClient tests executed on each PR in corresponding GitHub repositories. The way in which CRABServer and CRABClient test are executed is identical, therefore the general approach how testing works will be explained without focusing on one specific repository. All needed files which are used in CRAB testing can be found in CMSDOCKS Gitlab repository.
All test are executed as Docker containers in DMWM Jenkins. CRABServer tests configuration can be found at https://cmssdt.cern.ch/dmwm-jenkins/job/DMWM-CRABServer-PR-test/configure while CRABClient at https://cmssdt.cern.ch/dmwm-jenkins/job/DMWM-CRABClient-PR-test/configure. However, you will need to request access rights to view/edit it (you can do that by writing an email to Shahzad). All CRAB jobs logs are kept for 7 days, maximum 50 jobs logs in total.
After new PR is created, Jenkins generates a report that has 4 sections common to both CRABClient and CRABServer PR tests: 1) Pylint; 2) Python3 compatibility; 3) Default; 4) Unit tests. CRABClient has one additional section called Functional tests where functional tests are executed. Each of this section has an indication whether that test passed or failed. All tests feed their results back to DMWM-[CRABClient|CRABServer]-PR-test, which does a final analysis of the results and decides if PR passes or fails.
DMWM-[CRABClient|CRABServer]-PR-test is responsible for testing Pull requests and then triggering all other subprojects tests and then finally aggregating all those results and reporting back to the PR using a docker container and running this script.
All the jenkins jobs in https://cmssdt.cern.ch/dmwm-jenkins currently use Jenkins's github pull request builder. This is a jenkins plugin that interfaces github pull requests with a jenkins job.
We can trigger a jenkins job simply dropping a comment in the PR, but the set of accepted commands is more limited that what is available to CMSSW. In particular, we can write
- "test this please" for a one time test run. We often use this to trigger again
DMWM-[CRABClient|CRABServer]-PR-test
References:
- https://github.com/dmwm/CRABServer/issues/7156 This is the issue that triggered the creation of this section of the docs. It contains some more details about the integrations of github with jenkins.
Pylint configuration can be found in WMCore repository. So in case you want to run pylint test on your code before creating PR, use this file.
Things in the pylint report that are bold are causing PR tests to fail. Function which decides if test failed can be found here, while short summary would look like this:
- If pylint report has
warningsorerrors, then pylint test fails. However, there is a predefined list of warnings that do not cause pylint test to fail. This list is made of the following warnings:['0511', '0703', '0613']; - If pylint report score is below 8, then pylint test fails;
Pylint report in GitHub is divided into 3 sections:
- warnings and errors that must be fixed - these are warnings and errors in pylint report that are in bold and makes pylint test to fail;
- warnings - these are warnings and errors that that do not make pylint report to fail, i.e. following warnings:
['0511', '0703', '0613']; - comments to review - everything else what pylint report complains about are marked as comments.
As of now, unit tests are not executed for CRAB code, however, since same code to produce the final report is used for WMCore and they do have unit tests executed, we have this section in the report as well. One can notice that this type of test always pass the test.
As of now, functional tests are only executed on CRABClient repository. CRABClient is deployed directly from github and is used to executed various CRABClient commands to check if basic functionality (crab submit, crab status etc.) still works after changes made in PR.
Purpose: Test if crab submit works using different CMSSW releases.
Configuration:
- CMSSW releases: Test is being executed using multiple
CMSSWreleases. Releases can be managed/updated in Jenkins job configuration page underConfiguration Matrixsection. Identical tasks are being submitted to all provided releases and they are executed in parallel if there are enough free machines to run tests.
Implementation: Once CRAB task is submitted, crab status command is being executed maximum 3 times in configured time intervals. If after checking task's status it is not SUBMITTED, then status is checked again after N seconds.
Result: each submitted task is considered as one test, meaning that it has result passed or failed. Definition of terms if provided below:
1.Passed: both crab submit command and the submission to the grid succeeded, i.e. dbStatus variable is equal to SUBMITTED
2.Failed: dbStatus variable is not equal to SUBMITTED because one of the following cases: crab submit command succeed but after checking task's status for maximum N times, task's status is still in ['HOLDING', 'QUEUED', 'NEW']; crab submit command succeed but status of the task is not in ['HOLDING', 'QUEUED', 'NEW', 'SUBMITTED']; status of the tasks cannot be checked because crab status command cannot be executed on the task.
If there is at least one task having status failed, then entire test is considered as failed. If there are any tasks that failed, in the final report table with given task information together with the link to the log file of that submission is provided.