#339: Test prod CRABClient using test11 REST instance and CMSSW_12_5_2_patch1 CMSSW release

[cmsbuild]

Tests started for following configuration:

Configuration:

• CRABClient_version: prod
• REST_Instance: test11
• CMSSW_release: CMSSW_12_5_2_patch1
• SCRAM_ARCH: el8_amd64_gcc10

Tests started:

• Client_Validation
• Client_Configuration_Validation
• Task_Submission_Status_Tracking

Started at: 2022-11-21 19:08:48
Link to the job

[cmsbuild]

Task submission for Task_Submission_Status_Tracking successfully ended.

221121_181020:cmsbot_crab_20221121_191020
221121_181022:cmsbot_crab_20221121_191021
221121_181023:cmsbot_crab_20221121_191023
221121_181025:cmsbot_crab_20221121_191024
221121_181026:cmsbot_crab_20221121_191026
221121_181031:cmsbot_crab_20221121_191027

Task submission for Client_Configuration_Validation successfully ended.

221121_180924:cmsbot_crab_transferOutputs
221121_180926:cmsbot_crab_transferLogs
221121_180927:cmsbot_crab_activity
221121_180929:cmsbot_crab_inputFiles
221121_180930:cmsbot_crab_disableAutomaticOutputCollection
221121_180931:cmsbot_crab_outputFiles
221121_180933:cmsbot_crab_allowUndistributedCMSSW
221121_180934:cmsbot_crab_maxMemoryMB
221121_180936:cmsbot_crab_maxJobRuntimeMin
221121_180937:cmsbot_crab_numCores
221121_180939:cmsbot_crab_scriptExe
221121_180940:cmsbot_crab_scriptArgs
221121_180942:cmsbot_crab_sendPythonFolder
221121_180952:cmsbot_crab_sendExternalFolder
221121_180953:cmsbot_crab_inputDBS
221121_180954:cmsbot_crab_useParent
221121_180956:cmsbot_crab_secondaryInputDataset
221121_180957:cmsbot_crab_lumiMaskFile
221121_180959:cmsbot_crab_lumiMaskUrl
221121_181002:cmsbot_crab_outLFNDirBase
221121_181003:cmsbot_crab_runRange
221121_181005:cmsbot_crab_ignoreLocality
221121_181006:cmsbot_crab_userInputFiles
221121_181008:cmsbot_crab_whitelist
221121_181009:cmsbot_crab_blacklist
221121_181011:cmsbot_crab_ignoreGlobalBlacklist
221121_181012:cmsbot_crab_voRole
221121_181013:cmsbot_crab_voGroup
221121_181015:cmsbot_crab_scheddName
221121_181017:cmsbot_crab_collector
221121_181018:cmsbot_crab_extraJDL

Task submission for Client_Validation_Suite successfully ended.

221121_180922:cmsbot_crab_20221121_190921

Finished at: 2022-11-21 19:10:31
Find submission log here

[cmsbuild]

Test: Client validation
Result: SUCCEEDED
Finished at: 2022-11-21 19:31:52
Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_ClientValidation/310/console
All executed commands: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_ClientValidation/310//artifact/client-validation.log/*view*/
Message: All commands were executed successfully.

[cmsbuild]

Test: Client configuration validation
Result: FAILED
Attempt: 0 out of 4. Test failed. Investigate manually
Finished at: 2022-11-21 19:32:17
Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_ClientConfigurationValidation/510/console

RETRY TESTS:
 none

SUCCESSFUL TESTS:
 none

FAILED TESTS:
transferOutputs-check.sh 221121_180924:cmsbot_crab_transferOutputs - 1
transferLogs-check.sh 221121_180926:cmsbot_crab_transferLogs - 1
activity-check.sh 221121_180927:cmsbot_crab_activity - 1
inputFiles-check.sh 221121_180929:cmsbot_crab_inputFiles - 1
disableAutomaticOutputCollection-check.sh 221121_180930:cmsbot_crab_disableAutomaticOutputCollection - 1
outputFiles-check.sh 221121_180931:cmsbot_crab_outputFiles - 1
allowUndistributedCMSSW-check.sh 221121_180933:cmsbot_crab_allowUndistributedCMSSW - 1
maxMemoryMB-check.sh 221121_180934:cmsbot_crab_maxMemoryMB - 1
maxJobRuntimeMin-check.sh 221121_180936:cmsbot_crab_maxJobRuntimeMin - 1
numCores-check.sh 221121_180937:cmsbot_crab_numCores - 1
scriptExe-check.sh 221121_180939:cmsbot_crab_scriptExe - 1
scriptArgs-check.sh 221121_180940:cmsbot_crab_scriptArgs - 1
sendPythonFolder-check.sh 221121_180942:cmsbot_crab_sendPythonFolder - 1
sendExternalFolder-check.sh 221121_180952:cmsbot_crab_sendExternalFolder - 1
inputDBS-check.sh 221121_180953:cmsbot_crab_inputDBS - 1
useParent-check.sh 221121_180954:cmsbot_crab_useParent - 1
secondaryInputDataset-check.sh 221121_180956:cmsbot_crab_secondaryInputDataset - 1
lumiMaskFile-check.sh 221121_180957:cmsbot_crab_lumiMaskFile - 1
lumiMaskUrl-check.sh 221121_180959:cmsbot_crab_lumiMaskUrl - 1
outLFNDirBase-check.sh 221121_181002:cmsbot_crab_outLFNDirBase - 1
runRange-check.sh 221121_181003:cmsbot_crab_runRange - 1
ignoreLocality-check.sh 221121_181005:cmsbot_crab_ignoreLocality - 1
userInputFiles-check.sh 221121_181006:cmsbot_crab_userInputFiles - 1
whitelist-check.sh 221121_181008:cmsbot_crab_whitelist - 1
blacklist-check.sh 221121_181009:cmsbot_crab_blacklist - 1
ignoreGlobalBlacklist-check.sh 221121_181011:cmsbot_crab_ignoreGlobalBlacklist - 1
voRole-check.sh 221121_181012:cmsbot_crab_voRole - 1
voGroup-check.sh 221121_181013:cmsbot_crab_voGroup - 1
scheddName-check.sh 221121_181015:cmsbot_crab_scheddName - 1
collector-check.sh 221121_181017:cmsbot_crab_collector - 1
extraJDL-check.sh 221121_181018:cmsbot_crab_extraJDL - 1

[cmsbuild]

Test: Task Submission Status Tracking
Result: FULL-STATUS-UNKNOWN
Attempt: 0 out of 4. Will run again.
Finished at: 2022-11-21 20:13:10
Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_CheckTestResults/371/console

{'TN': '221121_181020:cmsbot_crab_20221121_191020', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 84, 'finished': 426, 'transferring': 489, 'running': 1}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221121_181022:cmsbot_crab_20221121_191021', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 5, 'transferring': 1, 'running': 1}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221121_181023:cmsbot_crab_20221121_191023', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 1}, 'combinedStatus': 'COMPLETED'}
{'TN': '221121_181025:cmsbot_crab_20221121_191024', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 2, 'transferring': 8}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221121_181026:cmsbot_crab_20221121_191026', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 42, 'transferring': 58}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221121_181031:cmsbot_crab_20221121_191027', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 6, 'transferring': 4}, 'combinedStatus': 'SUBMITTED'}

[cmsbuild]

Test: Task Submission Status Tracking
Result: FULL-STATUS-UNKNOWN
Attempt: 1 out of 4. Will run again.
Finished at: 2022-11-21 21:15:35
Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_CheckTestResults/373/console

{'TN': '221121_181020:cmsbot_crab_20221121_191020', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 85, 'finished': 614, 'transferring': 301}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221121_181022:cmsbot_crab_20221121_191021', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 7}, 'combinedStatus': 'COMPLETED'}
{'TN': '221121_181023:cmsbot_crab_20221121_191023', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 1}, 'combinedStatus': 'COMPLETED'}
{'TN': '221121_181025:cmsbot_crab_20221121_191024', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 4, 'transferring': 6}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221121_181026:cmsbot_crab_20221121_191026', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 70, 'transferring': 30}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221121_181031:cmsbot_crab_20221121_191027', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 8, 'transferring': 2}, 'combinedStatus': 'SUBMITTED'}

[cmsbuild]

Test: Task Submission Status Tracking
Result: FAILED
Attempt: 2 out of 4. Test failed. Investigate manually
Finished at: 2022-11-21 22:17:58
Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_CheckTestResults/376/console

{'TN': '221121_181020:cmsbot_crab_20221121_191020', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 85, 'finished': 904, 'transferring': 11}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221121_181022:cmsbot_crab_20221121_191021', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 7}, 'combinedStatus': 'COMPLETED'}
{'TN': '221121_181023:cmsbot_crab_20221121_191023', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 1}, 'combinedStatus': 'COMPLETED'}
{'TN': '221121_181025:cmsbot_crab_20221121_191024', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}
{'TN': '221121_181026:cmsbot_crab_20221121_191026', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 100}, 'combinedStatus': 'COMPLETED'}
{'TN': '221121_181031:cmsbot_crab_20221121_191027', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}