WMCore 1.4.8 central services production release

This release contains mostly bug-fixes and a few python3 modernization changes.

Release date: 3 May 2021.
Changes since release: 1.4.7.

Central services

Software stack

Features and/or feature changes

• Force cmsweb-prod in DBS3Reader (Alan Malta Rodrigues) #10444

Bug Fixes

• Revert changes to src/python/WMCore/Configuration.py from slice 17 (Dario Mapelli) #10443
• Wrap sendAlert() in a try/except block. (Todor Ivanov) #10467
• Do not delete transfer documents if bystatus couchview response is empty (Alan Malta Rodrigues) #10471
• Properly evaluate stuck rules in MSTransferor (Alan Malta Rodrigues) #10474

Enhancements

• Remove py package from the requirements list (Alan Malta Rodrigues) #10462

WMAgent

Features and/or feature changes

• Force cmsweb-prod in DBS3Reader (Alan Malta Rodrigues) #10444
• Remove unused tweak file (Kenyi Hurtado) #10434

Bug Fixes

• Revert changes to src/python/WMCore/Configuration.py from slice 17 (Dario Mapelli) #10443
• Retrieve only a summary of the condor jobs in DrainStatusPoller (Alan Malta Rodrigues) #10458
• Only create Rucio connection in PileupFetcher if there is a pileup dataset (Alan Malta Rodrigues) #10470

Enhancements

• Create symbolic links in deploy-wmagent (tivanov) #10454
• Bump WMAgent deployment example to 1.4.7 series (Alan Malta Rodrigues) #10457

Python3 related changes

• [py2py3] fix some pylint --py3k warnings - #10440 - src (Dario Mapelli) #10441
• [py2py3] modernize ordering in DataStructs/Run.py and File.py - #10169 - src (Dario Mapelli) #10465

WMAgent 1.4.7.patch2 production release

This is a bug fix patch release. For full release cycle changes, please refer to: https://github.com/dmwm/WMCore/releases/tag/1.4.7.patch1

Changes since release: 1.4.7.patch1.
Release date: 19 April 2021.

Bug Fixes

• Temporary fix to let CouchDB replications go through the standard port - wmagent branch (Alan Malta Rodrigues) #10453

WMAgent 1.4.7.patch1 production release

First WMAgent production release on the 1.4.7 series.
This release has close to 100% of our codebase compatible with both python2 and python3. It is also important to mention that the PhEDExInjector component has been deprecated. Furthermore, many other developments and bug fixes went into this release as well. See all the changes compared to the previous WMAgent production release mentioned below.

Changes since release: 1.4.5.patch1.
Release date: 15 April 2021.

WMAgent

Software stack

• Bump httplib2 from 0.18.1 to 0.19.0 #10271

Features and/or feature changes

• Forward WMCore.Services.Requests class to port 8443 for cmsweb services. (Todor Ivanov) #10330 #10364
• Remove PhEDExInjector component and its config/deploy code (Alan Malta Rodrigues) #10366
• Remove datatier-based functionality from RucioInjector (Alan Malta Rodrigues) #10369
• Remove switch between Rucio and PhEDEx (Alan Malta Rodrigues) #10283
• Update production DBS URL to cmsweb-prod (Alan Malta Rodrigues) #10269

Bug Fixes

• Fix values for the "submitted" key in JobSummary (Alan Malta Rodrigues) #10325
• fix bug in selecting stagein for UL logcollect, fixes ##10294 (Dirk Hufnagel) #10296
• Check if DID exists before creating a replication rule (Alan Malta Rodrigues) #10265
• Trap SSL_connect error and Service Unavailable in DBSUploadPoller (Alan Malta Rodrigues) #10252

Enhancements

• added script to query Rucio and create mock data; added mock data (Alan Malta Rodrigues) #10342
• Add check and different path for wmagent component. (tivanov) #10327
• Remove PhEDEx backwards compatibility from the Rucio module (Alan Malta Rodrigues) #10273
• Make WMAgent job wrapper slightly more verbose; fix one error message (Alan Malta Rodrigues) #10261

Python3 related changes

• [py2py3] src/python/A/B/C slice 5 - issue #10116 - src (Dario Mapelli) #10262
• [py2py3] src/python/A/B/C slice 6 - issue #10127 - src (Dario Mapelli) #10277
• [py2py3] src/python/A/B/C slice 7 - issue #10131 - src (Dario Mapelli) #10284
• [py2py3] src/python/A/B/C slice 8 - issue #10132 - src (Dario Mapelli) #10289
• [py2py3] src/python/A/B/C slice 9 - issue #10133 - src (Dario Mapelli) #10290
• [py2py3] src/python/A/B/C slice 10 - issue #10134 - src (Dario Mapelli) #10295
• [py2py3] pycurl_manager imports encoder to Bytes from Utilities - src (Dario Mapelli) #10250
• [py2py3] Utils/Utilities.py - encoder Unicode to Bytes - src (Dario Mapelli) #10250
• [py2py3] src/python/A/B/C slice 11 - issue #10135 - src (Dario Mapelli) #10299
• [py2py3] src/python/A/B/C slice 12 - issue #10136 - src (Dario Mapelli) #10307
• [py2py3] src/python/A/B/C slice 13 - issue #10137 - src (Dario Mapelli) #10310
• [py2py3] src/python/A/B/C slice 14 - issue #10138 - src (Dario Mapelli) #10313
• [py2py3] src/python/A/B/C slice 15 - issue #10139 - src (Dario Mapelli) #10319
• [py2py3] src/python/A/B/C slice 16 - issue #10140 - src (Dario Mapelli) #10321
• [py2py3] src/python/A/B/C slice 17 - issue #10141 - src (Dario Mapelli) #10329
• [py2py3] src/python/A/B/C slice 18 - issue #10142 - src (Dario Mapelli) #10331
• [py2py3] src/python/A/B/C slice 19 - issue #10143 - src (Dario Mapelli) #10348
• [py2py3] src/python/A/B/C slice 20 - issue #10144 - src (Dario Mapelli) #10349
• [py2py3] src/python/A/B/C slice 21 - issue #10145 - src (Dario Mapelli) #10361
• [py2py3] src/python/A/B/C slice 24 - issue #10148 - src (Dario Mapelli) #10371
• [py2py3] src/python/A/B/C slice 22 - issue #10146 - src (Dario Mapelli) #10362
• [py2py3] src/python/A/B/C slice 23 - issue #10147 - src (Dario Mapelli) #10363
• [py2py3] src/python/A/B/C slice 25 - issue #10149 - src (Dario Mapelli) #10379

WMCore 1.4.7 central services and WMAgent production release

In this release, almost all the WMCore source code has been modernized to work with python2 and python3 (still a couple very minor modules to be carefully modernized, and third-party libraries).
It also brings in a new Prometheus AlertManager wrapper module, and its integration to the MicroServices. It's worth mentioning that WMCore services should now - at the very low pycurl layer - be using port 8443 for every HTTP request.
Last but not least, there has been many bug fixes to the central services.
NOTE: we are working in a new patch version for WMAgent, which will fix a bug originated from the python3 code modernization.

Changes since release: 1.4.6.
Release date: 3 Mar 2021.

Central services

Software stack

• Add py3-cmsmonitoring to the list of python3 requirements (Alan Malta Rodrigues) #10355

Features and/or feature changes

• Dont map pileup location to the campaign top level site lists (Alan Malta Rodrigues) #10338
• Add AlertManager API (Erik Gough) #10308 #10358
• Switch microservices to use AlertManager (Erik Gough) #10346
• Forward WMCore.Services.Requests class to port 8443 for cmsweb services. (Todor Ivanov) #10330 #10364
• Account for tape rule status in MSRulecleaner. (Todor Ivanov) #10370

Bug Fixes

• Fix unhashable types in the ReqMgr2 request API with mask (Alan Malta Rodrigues) #10312
• Fix to unhashable types in the filtered_requests REST API (Alan Malta Rodrigues) #10311
• Validate status transition for cascade option (Alan Malta Rodrigues) #10309
• Reject blocks without any Rucio files in GQ (Alan Malta Rodrigues) #10342
• MSTransferor: iterate over a dictionary as a list when it can change (Alan Malta Rodrigues) #10337
• Change regex when building list of PR numbers in buildrelease (Alan Malta Rodrigues) #10372

Enhancements

• added script to query Rucio and create mock data; added mock data (Alan Malta Rodrigues) #10342
• allow to run REST/Main in frontend via env.var. fix #10266 (belforte) #10339
• Reorganize microservices package structure; remove unused (Alan Malta Rodrigues) #10357 #10373
• Update CMS WM overview schematic (Alan Malta Rodrigues) #10438

WMAgent

Features and/or feature changes

• Forward WMCore.Services.Requests class to port 8443 for cmsweb services. (Todor Ivanov) #10330 #10364
• Remove PhEDExInjector component and its config/deploy code (Alan Malta Rodrigues) #10366
• Remove datatier-based functionality from RucioInjector (Alan Malta Rodrigues) #10369

Bug Fixes

• Fix values for the "submitted" key in JobSummary (Alan Malta Rodrigues) #10325

Enhancements

• Add check and different path for wmagent component. (tivanov) #10327

Python3 related changes

• [py2py3] src/python/A/B/C slice 11 - issue #10135 - src (Dario Mapelli) #10299
• [py2py3] src/python/A/B/C slice 12 - issue #10136 - src (Dario Mapelli) #10307
• [py2py3] src/python/A/B/C slice 13 - issue #10137 - src (Dario Mapelli) #10310
• [py2py3] src/python/A/B/C slice 14 - issue #10138 - src (Dario Mapelli) #10313
• [py2py3] src/python/A/B/C slice 15 - issue #10139 - src (Dario Mapelli) #10319
• [py2py3] src/python/A/B/C slice 16 - issue #10140 - src (Dario Mapelli) #10321
• [py2py3] src/python/A/B/C slice 17 - issue #10141 - src (Dario Mapelli) #10329
• [py2py3] src/python/A/B/C slice 18 - issue #10142 - src (Dario Mapelli) #10331
• [py2py3] src/python/A/B/C slice 19 - issue #10143 - src (Dario Mapelli) #10348
• [py2py3] src/python/A/B/C slice 20 - issue #10144 - src (Dario Mapelli) #10349
• [py2py3] src/python/A/B/C slice 21 - issue #10145 - src (Dario Mapelli) #10361
• [py2py3] src/python/A/B/C slice 24 - issue #10148 - src (Dario Mapelli) #10371
• [py2py3] src/python/A/B/C slice 22 - issue #10146 - src (Dario Mapelli) #10362
• [py2py3] src/python/A/B/C slice 23 - issue #10147 - src (Dario Mapelli) #10363
• [py2py3] src/python/A/B/C slice 25 - issue #10149 - src (Dario Mapelli) #10379

WMCore 1.4.6 central services production release

This release contains a major new feature in MSRuleCleaner, being able to delete input data placement rules for input data no longer locked. In addition to that, it comes with many python3 modernisation changes.

Release date: 3 Mar 2021.
Changes since release: 1.4.5.

Central services

Software stack

• Bump httplib2 from 0.18.1 to 0.19.0 #10271

Features and/or feature changes

• Include logic for cleaning input data rules. (Todor Ivanov) #10224 #10320
• Remove switch between Rucio and PhEDEx (Alan Malta Rodrigues) #10283
• Skip RSE quota check for RelVal input data placement; added rule meta data (Alan Malta Rodrigues) #10275
• Update production DBS URL to cmsweb-prod (Alan Malta Rodrigues) #10269

Bug Fixes

Enhancements

• add new dataset to mock in DBS (Alan Malta Rodrigues) #10283
• mock Rucio getPileupLockedAndAvailable API; patch PileupFetcher_t (Alan Malta Rodrigues) #10283
• Properly set rucio.cfg in our docker unittest setup (Alan Malta Rodrigues) #10292
• Remove reportWorkflowToDashboard method from StdBase (Alan Malta Rodrigues) #10281
• Completely remove SiteDB, including emulators and tests (Alan Malta Rodrigues) #10280
• Remove PhEDEx backwards compatibility from the Rucio module (Alan Malta Rodrigues) #10273
• Update snapshot of Unified campaigns; update a couple json templates (Alan Malta Rodrigues) #10242
• add back WMBS to crabtaskworker dependencies and remove asyncstageout (belforte) #10246

WMAgent

Features and/or feature changes

Bug Fixes

• fix bug in selecting stagein for UL logcollect, fixes ##10294 (Dirk Hufnagel) #10296
• Check if DID exists before creating a replication rule (Alan Malta Rodrigues) #10265
• Trap SSL_connect error and Service Unavailable in DBSUploadPoller (Alan Malta Rodrigues) #10252

Enhancements

• Make WMAgent job wrapper slightly more verbose; fix one error message (Alan Malta Rodrigues) #10261
• Update deployment script to accept python3 WMAgent service (Alan Malta Rodrigues) #10302

Python3 related changes

• Python3 requirements for WMAgent and WMAgent-Dev (Alan Malta Rodrigues) #10247 #10268
• [py2py3] src/python/A/B/C slice 10 - issue #10134 - src (Dario Mapelli) #10295
• [py2py3] src/python/A/B/C slice 9 - issue #10133 - src (Dario Mapelli) #10290
• [py2py3] src/python/A/B/C slice 7 - issue #10131 - src (Dario Mapelli) #10284
• [py2py3] src/python/A/B/C slice 8 - issue #10132 - src (Dario Mapelli) #10289
• [py2py3] src/python/A/B/C slice 5 - issue #10116 - src (Dario Mapelli) #10262
• [py2py3] src/python/A/B/C slice 6 - issue #10127 - src (Dario Mapelli) #10277
• [py2py3] pycurl_manager imports encoder to Bytes from Utilities - src (Dario Mapelli) #10250
• [py2py3] Utils/Utilities.py - encoder Unicode to Bytes - src (Dario Mapelli) #10250

WMAgent 1.4.5.patch1 production release

First WMAgent production release on the 1.4.5 series. This release is built on top of the system OpenSSL library and it contains a large amount of python3 modernization changes. Many other developments and bug fixes went into this release as well, see all the changes compared to the previous WMAgent production release.

Changes since release: 1.4.3.patch2.
Release date: 8 Feb 2021.

WMAgent

Software stack

• OpenSSL updated from 1.0.1r to version 1.0.2k-fips; and it now relies on the system-based libraries
• Python cx-Oracle library updated from 5.2.1 to 7.3.0
• Python rucio-clients library updated from 1.23.11 to 1.24.2.post1

Features and/or feature changes

• Reuse Rucio object in WorkQueue start policy modules (Alan Malta Rodrigues) #10099
• add all Rucio lfns2pfns args. to getPFN. Fix #10056 (belforte) #10122
• Add requiresApproval method to Rucio class. Manage RSE rule approval requirements in insertContainerRules (germanfgv) #10105
• add option for preloader for xrdcp stageout (Dirk Hufnagel) #10226
• Update production DBS URL to cmsweb-prod (Alan Malta Rodrigues) #10269

Bug Fixes

• Fix pileup location with multi-RSE rules (Alan Malta Rodrigues) #10194
• Fix servicekey permission in the agent deployment; remove central services url option (Alan Malta Rodrigues) #10163
• Add only merged parents to the ACDC docs (Alan Malta Rodrigues) #9997
• Fix JobAccountant missing parent dbsbuffer file (Alan Malta Rodrigues) #9997
• Only resolve RSE requires_approval for T0 data placement (Alan Malta Rodrigues) #10130
• Catch UnicodeDecodeError in JobAccountant (Alan Malta Rodrigues) #10152
• Trap SSL_connect error and Service Unavailable in DBSUploadPoller #10252
• Check if DID exists before creating a replication rule #10265

Enhancements

• Remove badly encoded single quotes from Rucio module (Alan Malta Rodrigues) #10124
• Minor updates to some json test templates (Alan Malta Rodrigues) #10113
• Add 2021 validation campaigns. (Todor Ivanov) #10210
• Make WMAgent job wrapper slightly more verbose; fix one error message #10261

Python3 related changes

• [py2py3] WMCore/WMException.py - issue 10173 - src (Dario Mapelli) #10174
• [py2py3] src/python/A/B/C slice 2 - issue 10098 - src (Dario Mapelli) #10103
• [py2py3] src/python/A/B/C slice 1 - issue 10055 - src (Dario Mapelli) #10057
• [py2py3] src/python/A/B/C slice 3 - issue 10114 - src (Dario Mapelli) #10117
• [py2py3] issue #10173, PR #10187 - src (Dario Mapelli) #10187
• [py2py3] WMCore/WMSpec/ConfigSectionTree.py - issue #10218 - src (Dario Mapelli) #10221
• [py2py3] src/python/A/B/C slice 4 - issue 10115 - src (Dario Mapelli) #10165
• [py2py3] fix pre-py2.6: type() == -> isinstance (Dario Mapelli) #10172
• [py2py3] Database/DBFormatter.py - src - fix #10235 (Dario Mapelli) #10238
• [py2py3] pycurl_manager - better handling of curl.setopts() input arguments #10250

WMCore 1.4.5 services and WMAgent production release

This release is built on top of the SLC7 OpenSSL library, thus no longer shipping our own CMS COMP OpenSSL package. The cx-Oracle library went through an upgrade from version 5.2.1 to 7.3.0 as well.
In addition to that, it comes with many python3 modernization changes.

Changes since release: 1.4.4.
Release date: 12 Jan 2021.

Central services

Features and/or feature changes

• All RPMs built on top of system-based OpenSSL version cms-sw/cmsdist#6440

Bug Fixes

• Catch UnboundLocalError exception when EmailAlert fails connecting to the SMTP server #10244
• MSTransferor: Convert Campaign pileup data location to PSNs to match against the SiteWhitelist #10243

Enhancements

WMAgent

Features and/or feature changes

• add option for preloader for xrdcp stageout #10226

Bug Fixes

Enhancements

• Add 2021 validation campaigns. #10210

Python3 related changes

• [py2py3] src/python/A/B/C slice 1 - issue 10055 - src #10057
• [py2py3] src/python/A/B/C slice 3 - issue 10114 - src #10117
• [py2py3] issue #10173, PR #10187 - src #10187
• [py2py3] WMCore/WMSpec/ConfigSectionTree.py - issue #10218 - src #10221
• [py2py3] src/python/A/B/C slice 4 - issue 10115 - src #10165
• [py2py3] fix pre-py2.6: type() == -> isinstance #10172
• [py2py3] Database/DBFormatter.py - src - fix #10235 #10238

WMCore 1.4.4 services production release

This release moves the workflow archival from ReqMgr2 to a newer microservice, MSRuleCleaner.

Changes since release: 1.4.3.
Release date: 2 Dec 2020.

Central services

Features and/or feature changes

• No longer resolve StepChain parentage for workflows in normal-archived #10201
• MSRuleCleaner: Add workflow archival logic to MSRuleCleaner #10153
• Add all Rucio lfns2pfns args. to getPFN. Fix #10056 #10122
• Add requiresApproval method to Rucio class. Manage RSE rule approval requirements in insertContainerRules #10105 #10130

Bug Fixes

• MSTransferor: do not consider Rucio blocks with 0 or None size #10205
• Fix pileup location with multi-RSE rules #10194
• Fix GlobalWorkQueue CherryPy threads initialization and logger #10099
• Fix regexp for enable cherrypy threads && Fix file permissions for dmwm-service-key.pem #10155

Enhancements

• Reuse Rucio object in WorkQueue start policy modules #10099
• Mock Rucio APIs #10099
• Remove DDM service wrapper module #10190

WMAgent

Features

• Add requiresApproval method to Rucio class. Manage RSE rule approval requirements in insertContainerRules #10105 #10130

Bug Fixes

• Fix servicekey permission in the agent deployment; remove central services url option #10163
• Remove badly encoded single quotes from Rucio module #10124
• Add only merged parents to the ACDC docs #9997
• Fix JobAccountant missing parent dbsbuffer file #9997
• Catch UnicodeDecodeError in JobAccountant #10152

Enhancements

• Reuse Rucio object in WorkQueue start policy modules #10099
• Minor updates to some json test templates #10113

Python3 related changes

• [py2py3] WMCore/WMException.py - issue 10173 - src #10174
• [py2py3] src/python/A/B/C slice 2 - issue 10098 - src #10103

WMCore 1.4.3 services and WMAgent production release

This release brings in a new MicroService called MSRuleCleaner (still in beta version), which is responsible for removing the no longer output data rules created by WMAgent (under wma_prod account).
It also comes with an up-to-date rucio-clients version - 1.23.11 - which has a more restrictive certificate permission checks.

Changes since release: 1.4.2.
Release date: 2 Dez 2020.

Central services

Features and/or feature changes

• Deprecate usage of useRucio parameter within MicroServices #10094
• Remove PhEDEx logic from MSOutput/MSMonitor #10094
• Added PyPI support for reqmon, reqmgr2ms, global-workqueue, acdcserver #10059
• Throttling re-factoring and added timerange for user activities under throttling #9211
• Removed CherryPy WMDataMining thread/code #10065
• Added new exception to the Rucio wrapper (WMRucioDIDNotFoundException) #10023
• MSRuleClener logic for output data rule cleanup #10023 #10079 #10080 #10100

Bug Fixes

Enhancements

• Switched validation script to Rucio (including rollback of python3 changes) #10045
• Removed DataStructs/DASDocument module #10096
• Update setup dependencies for CRAB services #10084
• Only inject test templates that can have MSTransferor making data placement #9784
• Update workflow json templates #9784
• Script to recursively parse cms-sw/cmsdist spec files #10036

WMAgent

Features and/or feature changes

• Removing file status condition from GetUnsubscribedDatasets query #10067

Bug Fixes

• Evaluate all WQEs when local resources are available #10072 #10102

Enhancements

Python3 related changes

• WMCore/DataStructs - futurize #10012
• WMCore/Algorithms - futurize #10012

WMCore 1.4.2 services production release

This release no longer uses any of the PhEDEx functionality and Rucio is enabled as the standard Data Management system. It also brings an alpha version of a new microservice (MSRuleCleaner), which will be responsible for cleaning unneeded input and output data rules. Last but not least, RucioInjector component now supports block deletion and output container data placement, meaning it now provides 100% of the expected functionality.

Changes since release: 1.4.1.
Release date: 5 Nov 2020.

Central services

Features and/or feature changes

• MSOutput: Add getTransferInfo to MSOutput and info request to MSManager. #9978
• MSOutput: treat NANOAOD/NANOAODSIM as any other datatier in the output data placement logic #9988
• MSOutput: create Disk rules with an weight attr, default to ddm_quota #9993
• MSOutput: use different lifetime configuration for RelVal Disk rules #10046
• If listDataRules API fail, raise a WMRucioException #9999
• MSTransferor: remove PhEDEx functionality #10002
• MSRulecleaner initial commit. #10003

Bug Fixes

• MSOutput: Skip Tape (too) data placement for Resubmission workflows #9983
• Properly validate RequestPriority parameter #9994
• MSTransferor: resolve RSE expression for pileup location #10041
• Fix pileup data location in global workqueue #10042

Enhancements

• Use list_replication_rules Rucio API to list rules #9963
• Add CMSMonitoring dependency for ReqMgr2 #10004
• Introduce deploy-centralvm.sh #10014
• List of json templates that can undergo input data placement #10034

WMAgent

Features

• Disable PhEDExInjector; Enable all tiers to be injected in Rucio #9980
• Treat NANOAOD/NANOAODSIM as any other datatier in the output data placement logic #9988
• Skip Custodial and make generic Disk NonCustodial data placement in RucioInjector #9989
• If listDataRules API fail, raise a WMRucioException #9999
• Implementation for the Rucio getPFN method #10029
• Increase polling cycle for WorkQueueManagerLocationPoller to 1h #9981
• Block deletion logic implemented in RucioInjector #9959 #10044
• Deal with T0 output data subscription in RucioInjector #10006

Bug Fixes

• Only add replicas if parent block has been properly added/attached into Rucio #10024
• Ignore availability for block level rules against the origin site/RSE #10031

Enhancements

• Bump deployment example to 1.4.1.patch3 #9968 #10005

Python3 related changes

• Revert stage2 fixers on DQMUpload module #9945
• [py2py3] WMCore/WMFactory - futurize #9971
• [py2py3] WMCore/Credential - futurize #10008