WMCore 2.2.6 production central services release
Despite not being yet available to the end users, initial changes for supporting partial pileup have been integrated into this release. The StepChain parentage CherryPy thread has been refactored to support partial block resolution and to be more resilient. MSUnmerged had some performance related improvements. Cleanup of unneeded rules (MSRuleCleaner) is no longer dependent on the status of the StepChain parentage resolution. Global WorkQueue will no longer fail workflows that have an incomplete input data placement and/or in cases where the data has not arrived at the final destination, instead it relies on a continuous data location daemon. Lastly, this release brings in many WMAgent and central services improvements.
Release date: 16 Jan 2024.
Changes since release: 2.2.4.
Central services
Software stack
- Replace svn by git in GitHub actions (Alan Malta Rodrigues) #11858
Features and/or feature changes
- Refactor StepChainParentage thread to resolve by workflow (Alan Malta Rodrigues) #11694
- Deal with partial block in the parentage fix (Alan Malta Rodrigues) #11757 #11779
- Remove unused Unified configuration and code; add schema and validation (Alan Malta Rodrigues) #11770
- MSUnmerged: Try to remove the base directory first and avoid recursive operations (#11781) (Todor Ivanov) #11781
- MSUnmerged: Handle gfal exceptions while listing baseDirEntry && Avoid extra stat operations during recursion. (#11794) (Todor Ivanov) #11794
- MSPileup: Add support for customName in MSPileup (Valentin Kuznetsov) #11765
- MSPileup: Adjust to use customName along with pileupName (Valentin Kuznetsov) #11769
- MSPileup: Introduce transition attribute in MSPileup record (Valentin Kuznetsov) #11802
- MSRuleCleaner: Moved ParentageResolved check from dispatch to archive (Dennis Lee) #11805
- MSTransferor: Ensure MSTransferor does not request more copies than RSEs available (Alan Malta Rodrigues) #11844
Bug Fixes
Enhancements
- Update test json templates with MINIAODSIM; fix DQMHarvest ; GPU StepChain; ReReco 2022C; etc (Alan Malta Rodrigues) #11238
- Fix for 11239: Replaced instances of logging.warn to logging.warning (Dennis Lee) #11785
- Add MSPileup and refactor MongoDB in the WM Schematic (Alan Malta Rodrigues) #11789
- New test campaigns added to parseUnifiedCampaigns (Alan Malta Rodrigues) #11806
- Added aborted-completed to the state transition diagram (Alan Malta Rodrigues) #11809
WMAgent
Features and/or feature changes
- Inherit siteLists from upper level task while creating WMBS subscriptions (Todor Trendafilov Ivanov) #11724
- Initial implementation for WorkflowUpdater component (Alan Malta Rodrigues) #11795 #11859
- Give priority to older workflows when fetching from database for JobSubmitter (German Giraldo) #11804
- Add loging and decode output of pre-scripts (German Giraldo) #11803
- Add a generic script for deploying wmagent inside a virtual environment. (Todor Ivanov) #11624
- Add runtime information json. (Kenyi Hurtado) #11812
Bug Fixes
- Fix logic for updating task-level site thresholds (Alan Malta Rodrigues) #11776
- Fix ChangeState logic for limiting number of docs to commit in bulk (Alan Malta Rodrigues) #11786
- Do not fail workflows due to empty location in global workqueue (Valentin Kuznetsov) #11810 #11862
- Deal with invalid blocks in WMBS for Dataset start policy (Alan Malta Rodrigues) #11838
Enhancements
- Bump deploy-wmagent script to version 2.2.5; insert T3_US_Ookami (Alan Malta Rodrigues) #11766
- Fix for 11239: Replaced instances of logging.warn to logging.warning (Dennis Lee) #11785
- Make DBS3Upload slightly more verbose (Alan Malta Rodrigues) #11768
- Fixed Inappropriate Logical Expression (fabihatasneem) #11808