Releases: dmwm/WMCore
Releases · dmwm/WMCore
WMCore 2.1.1 central services production release
This release brings in a backward-compatible authorization layer for ReqMgr2. It also defines Nano workflows to use a Dataset level granularity (before it was set to Block). Input data placement for Nano workflows will also be done at dataset level and 2 copies will be request.
This version also comes with the very latest stable CouchDB release.
Release date: 5 August 2022.
Changes since release: 2.1.0.
Central services
Software stack
- Update Py3 CMSMonitoring to 0.6.5 (Alan Malta Rodrigues) cms-sw/cmsdist#8002
- Updated CouchDB to version 3.2.2 (Alan Malta Rodrigues) cms-sw/cmsdist#8001
Features and/or feature changes
- Adopt ReqMgr2 authorization based on the request status (Alan Malta Rodrigues) #11223
- Implement support for gzipped stream for RucioConMon (Valentin Kuznetsov) #11142
- Move some Rucio utils over to RucioUtils module (Alan Malta Rodrigues) #11218
- Set start policy to Dataset for workflows processing MINIAODSIM dataset (Alan Malta Rodrigues) #11219
- MS: Implement new object for Nano input data placement (Alan Malta Rodrigues) #11219
Bug Fixes
Enhancements
- Update CMSMonitoring version in the requirements file to 0.6.5 (Alan Malta Rodrigues) #11229
- Update DBS mock calls and data with MINIAODSIM (Alan Malta Rodrigues) #11219
WMAgent
Features and/or feature changes
- Allow list of multiple architectures (khurtado) #11228
Bug Fixes
Enhancements
- Add ExecuteDAO script. (Todor Ivanov) #11174
WMCore 2.1.0 central services production release
This release was focused on the migration from CouchDB 1.6 to CouchDB 3+, making the WMCore framework fully compatible with CouchDB 3+. It brings in a few bug fixes and continuous enhancements for the system.
Release date: 13 July 2022.
Changes since release: 2.0.4.
Central services
Software stack
- Update stomp.py package from 4.1.21 to 7.0.0 - required for CMSMonitoring (Alan Malta Rodrigues) cms-sw/cmsdist#7984
- Update Py3 CMSMonitoring to 0.6.4 (Alan Malta Rodrigues) cms-sw/cmsdist#7983
- Fix py3-pyjwt PyPi name (Alan Malta Rodrigues) cms-sw/cmsdist#7929
- New specs/version for Erlang and CouchDB 3.1.2 (Alan Malta Rodrigues) cms-sw/cmsdist#7218
- Many deployment configuration changes like (Alan Malta Rodrigues) dmwm/deployment#1154 , 1157, 1158, 1159 and 1162
Features and/or feature changes
- Remove all the CMS authz logic from the CouchDB validate functions (Alan Malta Rodrigues) #11191
- Bundle of fixes required for migration to CouchDB 3.x (Alan Malta Rodrigues) #11001
Bug Fixes
- Consistent return data type for closing workqueue elements (Alan Malta Rodrigues) #11193
Enhancements
- Create filter function to drop deleted documents (Alan Malta Rodrigues) #11196
- Give enough time for replication to complete in unit tests (Alan Malta Rodrigues) #11185
WMAgent
Features and/or feature changes
- Provide uniform way to handle DBS server errors (Valentin Kuznetsov) #11176
- Bundle of fixes required for migration to CouchDB 3.x (Alan Malta Rodrigues) #11001
Bug Fixes
- Load credentials when saving spec file in local workqueue (Alan Malta Rodrigues) #11204
- Ensure dataset_lifetime bind is defined for DBS3Buffer/NewSubscription (Alan Malta Rodrigues) #11179
Enhancements
WMAgent production patch release: 2.0.4.patch1
This is a major release upgrade because it's the first WMAgent stable release compatible with CouchDB 3.1 (and dependent on). It also has a few important features and bug-fixes required for data taking and the T0 agent. Lastly, it has the usual and continuous code and stability improvements.
Release date: 16 June 2022.
WMAgent changes since release: 2.0.2
WMAgent
Software stack
- Upgrade CouchDB to 3.1.2, Erlang to 22.3.4.19 (Alan Malta Rodrigues) cms-sw/cmsdist#7218
- Bring dbs3-client version to 4.0.8 (Alan Malta Rodrigues) #11138
- Removed pep8; added pycodestyle 2.8.0; updated pylint to 2.13.5 (Alan Malta Rodrigues) cms-sw/cmsdist#7846
- Update PyJWT from version 2.3.0 to 2.4.0 (snyk-bot) #11155
Features and/or feature changes
- When ScramArch is empty str/list/None, return any as required_os (Alan Malta Rodrigues) #11083
- If job requires rhel8 container, use slc7 python libraries (Alan Malta Rodrigues) #11077
- Support adding diskless resources into the database (Alan Malta Rodrigues) #11097
- Remove PFNs from a final document we send to WMArchive (Valentin Kuznetsov) #10998
- change fallback for missing system xrootd-client (Dirk Hufnagel) #11117
- Remove support for Unpacking user tarballs. No longer used by CRAB. (khurtado) #11114
- Enforce dataset_lifetime column to be integer and not null (Alan Malta Rodrigues) #11115
- speed up listDatasetFileDetails API (Valentin Kuznetsov) #11099
- Add support for IAM token in the underlying pycurl_manager module (Valentin Kuznetsov) #10939
- Split Deletable Blocks checks in two stages. (Todor Ivanov) #11127
- New CouchDB 3.x APIs for tracking replications; fix replication check in AgentStatusPoller (Alan Malta Rodrigues) #11039
- Add extra protection for T0 to prevent archival of workflows having blocks not yet deleted. (Todor Ivanov) #11169
Bug Fixes
- Assign the correct activity for Tape_Test RSEs (Jhonatan Amado) #11067
- [Tier0] Check CMSSW version before getting sim-datatier map (germanfgv) #11110
- Check if key exists when deleting standalone documents in Couch (Alan Malta Rodrigues) #11165
Enhancements
- Enhance logic to map ScramArch to OS (Alan Malta Rodrigues) #11088
- [PY2] Remove cPickle support (Erik Gough) #11096
- Update wmagent deploy script to add opportunistic resources one by one (Alan Malta Rodrigues) #11097
- Minor test json template updates (Alan Malta Rodrigues) #10864
- Add T3_US_Lancium. (Todor Ivanov) #11126
- Add buildAllDep option && Fix package verstion estimation for ParseSpecCmsswdist. (Todor Ivanov) #11132
WMCore 2.0.4 central services and WMAgent production release
This release brings in a policy change for pileup input data placement in MSTransferor. It also brings in most of the required changes to support the newest version of CouchDB. Finally, it comes with usual enhancements to the code and documentation.
Release date: 9 June 2022.
Changes since release: 2.0.3.
Central services
Software stack
- Update PyJWT from version 2.3.0 to 2.4.0 (snyk-bot) #11155
Features and/or feature changes
- [MSTransferor] Create rule for each pileup location (Alan Malta Rodrigues) #11143 #11153
- Remove CouchDB all_or_nothing option for bulk API (Alan Malta Rodrigues) #11011
- Remove CouchDB JobDump/highestJobID view (Alan Malta Rodrigues) #11161
Bug Fixes
- [Microservices] Fix logic to check for empty results in MS/Common (Alan Malta Rodrigues) #11148
- Avoid raising exception when ReqMgr2 request is not found (Alan Malta Rodrigues) #11159
Enhancements
- [MSTransferor] Reorganization and classification of workflow templates and placement policies (Alan Malta Rodrigues) #11143
WMAgent
Features and/or feature changes
- Split Deletable Blocks checks in two stages. (Todor Ivanov) #11127
- New CouchDB 3.x APIs for tracking replications; fix replication check in AgentStatusPoller (Alan Malta Rodrigues) #11039
- Add extra protection for T0 to prevent archival of workflows having blocks not yet deleted. (Todor Ivanov) #11169
Bug Fixes
- Check if key exists when deleting standalone documents in Couch (Alan Malta Rodrigues) #11165
Enhancements
WMCore 2.0.3 central services production release
This release brings in many awesome features, such as: some initial support to IAM tokens; full support to workflows with growing input dataset; new policy for Release Validation output data placement, with a distinction of pre-releases from the other samples.
It also includes a few bug fixes required for the DBSServer migration to the Go implementation. Finally, it comes with many enhancements to the code and documentation.
Release date: 7 April 2022.
Changes since release: 2.0.2.
Central services
Software stack
- Bring dbs3-client version to 4.0.8 (Alan Malta Rodrigues) #11138
- Removed pep8; added pycodestyle 2.8.0; updated pylint to 2.13.5 (Alan Malta Rodrigues) cms-sw/cmsdist#7846
Features and/or feature changes
- Add bin and dependencies to wmcore PyPI package (Erik Gough) #11103
- Remove block open logic from DBS3Reader and WorkQueue (Alan Malta Rodrigues) #11123
- Properly handle OpenRunningTimeout in WorkQueue (Alan Malta Rodrigues) #11119 #11135
- Initial script to create IAM token (Valentin Kuznetsov) #11093 #11134
- speed up listDatasetFileDetails API (Valentin Kuznetsov) #11099
- [MSTransferor] Lock the whole input container for growing workflows (Alan Malta Rodrigues) #11141
- [MSOutput] Apply different rule lifetime for RelVal pre-release samples (Alan Malta Rodrigues) #11139
- Add support for IAM token in the underlying pycurl_manager module (Valentin Kuznetsov) #10939
- Add support to OpenRunningTimeout for TaskChain/StepChain (Alan Malta Rodrigues) #11135
Bug Fixes
- Update ReqMgr2 CP thread to properly deal with aborted/force-complete workflows (Alan Malta Rodrigues) #11113
Enhancements
- move ckey/cert functions to Utils.CertTools (Valentin Kuznetsov) #11101
- Remove no longer needed code in the StatusChangeTasks ReqMgr2 thread (Alan Malta Rodrigues) #11095
- Create script to cancel GQEs (Alan Malta Rodrigues) #11113
- Switch from pep8 to pycodestyle (Erik Gough) #11128
WMAgent
Features and/or feature changes
- When ScramArch is empty str/list/None, return any as required_os (Alan Malta Rodrigues) #11083
- If job requires rhel8 container, use slc7 python libraries (Alan Malta Rodrigues) #11077
- Support adding diskless resources into the database (Alan Malta Rodrigues) #11097
- Remove PFNs from a final document we send to WMArchive (Valentin Kuznetsov) #10998
- change fallback for missing system xrootd-client (Dirk Hufnagel) #11117
- Remove support for Unpacking user tarballs. No longer used by CRAB. (khurtado) #11114
- Enforce dataset_lifetime column to be integer and not null (Alan Malta Rodrigues) #11115
- speed up listDatasetFileDetails API (Valentin Kuznetsov) #11099
- Add support for IAM token in the underlying pycurl_manager module (Valentin Kuznetsov) #10939
Bug Fixes
- Assign the correct activity for Tape_Test RSEs (Jhonatan Amado) #11067
- [Tier0] Check CMSSW version before getting sim-datatier map (germanfgv) #11110
Enhancements
- Enhance logic to map ScramArch to OS (Alan Malta Rodrigues) #11088
- [PY2] Remove cPickle support (Erik Gough) #11096
- Update wmagent deploy script to add opportunistic resources one by one (Alan Malta Rodrigues) #11097
- Minor test json template updates (Alan Malta Rodrigues) #10864
- Add T3_US_Lancium. (Todor Ivanov) #11126
- Add buildAllDep option && Fix package verstion estimation for ParseSpecCmsswdist. (Todor Ivanov) #11132
WMAgent patch release: 2.0.2.patch1
Release date: 12 April 2022.
Changes since release: 2.0.2.
In addition to the release notes provided under within the tag 2.0.2, this patch release brings in the following.
WMAgent
Features and/or feature changes
- Default job wrapper to use rhel7 python libraries, otherwise use the same as CMSSW (Alan Malta Rodrigues) #11090
Bug Fixes
- Assign the correct activity for Tape_Test RSEs (Jhonatan Amado) #11090
- When ScramArch is empty str/list/None, return any as required_os (Alan Malta Rodrigues) #11090
Enhancements
- Enhance logic to map ScramArch to OS (Alan Malta Rodrigues) #11090
WMCore 2.0.2 central services and WMAgent production release
This release has support to compressed data in HTTP requests, relying on the automatic
decoding algorithms built with libcurl. For WMAgent, it has a database schema change for
the processing version; and it has full support to EL8 workflows.
Other than that, many enhancements to the WMCore workflow, documentation and overall
improvements have been made.
Release date: 7 April 2022.
Changes since release: 2.0.1.
WMAgent changes since release: 1.5.7
Central services
Software stack
Features and/or feature changes
- Define gzip encoding by default in the headers; properly decompress it (Valentin Kuznetsov) #11041
- Add usage of gzip encoding (Valentin Kuznetsov) #11056
- Use Content-Encoding to automatically decode body in RequestHandler (Alan Malta Rodrigues) #11069 #11070
- [MSUnmerged] Fetch protectedLFNs for T0 (Todor Ivanov) #11050
Bug Fixes
- [MSTransferor] do not bypass Pileup rules in INJECT/REPLICATING state (Alan Malta Rodrigues) #11061
Enhancements
- Document the use of docstrings and best practices in the project (Alan Malta Rodrigues) #11052
- Document Services cache deletion on exit (Alan Malta Rodrigues) #10915
WMAgent
Features and/or feature changes
- Define gzip encoding by default in the headers; properly decompress it (Valentin Kuznetsov) #11041
- Adjust processing version to be integer (Valentin Kuznetsov) #11043 #11059
- [CouchDB] Do not sanitize couch url at the component layer (Alan Malta Rodrigues) #11045
- Add el8 flavor to the list of rhel8 container jobs (Alan Malta Rodrigues) #11060 #11062
- Add usage of gzip encoding (Valentin Kuznetsov) #11056
- Use Content-Encoding to automatically decode body in RequestHandler (Alan Malta Rodrigues) #11069 #11070
- Add parameters to rucio datasets in RucioInjector (Jhonatan Amado) #10989
- Add lifetime to RucioContainers shipping to disk sites (Jhonatan Amado) #10950
- make sure proxy is 2048 bit. Fix #11005 (belforte) #11006
- Explicitly create proxies with 2048 bits (Alan Malta Rodrigues) #11008
Bug Fixes
- Prevent excessive calls for agent config via ReqMgrAux (Erik Gough) #11033
- Fix a few DBS3Reader APIs, likely unused ones (Alan Malta Rodrigues) #10990
- DBS3Upload: check error message from the response body attribute (Alan Malta Rodrigues) #10961
Enhancements
- Better logging for DBS block insertion in DBS3Upload (Alan Malta Rodrigues) #11058
- Download script and insert US opportunistic resources during deployment (Alan Malta Rodrigues) #11047
- Update mocked DBS APIs and data (Alan Malta Rodrigues) #10990
- Enhancements to the GH issues and PRs workflows (Alan Malta Rodrigues) #11021
- Update deploy-wmagent for 1.5.7.patch1; remove deprecated py2 option (Alan Malta Rodrigues) #10966
- Update error message for jobs killed from site state transition (Alan Malta Rodrigues) #10958
WMCore 2.0.1 central services production release
This release brings in many new features and feature changes to the microservices. It also includes a few bug fixes required for the DBSServer migration to the Go implementation. Lastly, usual code and documentation enhancements have been provided.
Release date: 10 March 2022.
Changes since release: 2.0.0.
Central services
Software stack
- Bump dbs3-client to 4.0.7 (Alan Malta Rodrigues) cms-sw/cmsdist#7624
- Updated pymongo library to 4.0.1 (Todor Ivanov) cms-sw/cmsdist#7654
Features and/or feature changes
- Update requirements.txt and PyPI build scripts for py3 stack (Erik Gough) #10976
- Add parameters to rucio datasets (Jhonatan Amado) #10989
- [MSTransferor] Send alerts to prometheus when a workflow fails to be processed. Rename notifyLargeData method. Fixes #10215 (Kenyi Hurtado) #10959
- [MSUnmerged] Add rucioConMonErrors counter. (Todor Ivanov) #10993
- [MSUnmerged] Simplify counters logic && Substitute rucioConMonErrors with rucioConMonStatus. (Todor Ivanov) #10993
- [MSUnmerged] Swap NotMasterError with NotPrimaryError. (Todor Ivanov) #11016
- [MSUnmerged] Add rse=ALL_DOCS to the MSUnmerged info REST API. (Todor Ivanov) #11025
- [MSOutput] Make MSOutput Disk expression configurable (Alan Malta Rodrigues) #11018
- [MSOutput] MSOutput: read RelVal output data policy from configuration (Alan Malta Rodrigues) #11024
Bug Fixes
- Fix a few DBS3Reader APIs, likely unused ones (Alan Malta Rodrigues) #10990
- [MSUnmerged] Remove files from mongoProjection (Todor Ivanov) #10987
- [MSUnmerged] Allow MongoDB to be instantiated without mongomock library (Alan Malta Rodrigues) #11023
- [MSUnmerged] Reorder RSE's status and record age checks from RucioConMon. (Todor Ivanov) #11029
- [MSTransferor] Fix MSTransferor typo calling getName method (Alan Malta Rodrigues) #11031
- [MSTransferor] MSTransferor: fix positional and kwargs passed to sendAlert API (Alan Malta Rodrigues) #11032
Enhancements
- Update mocked DBS APIs and data (Alan Malta Rodrigues) #10990
- Remove leftover requirements_py3.txt; update contribution doc (Alan Malta Rodrigues) #10997
- fix uninitialized templates variable (Dirk Hufnagel) #11004
- Add mockMongoDB to MSUnmerged and MongoDB Class init options (Todor Ivanov) #11007
- Enhancements to the GH issues and PRs workflows (Alan Malta Rodrigues) #11021
WMAgent
Features and/or feature changes
- Add parameters to rucio datasets in RucioInjector (Jhonatan Amado) #10989
- Add lifetime to RucioContainers shipping to disk sites (Jhonatan Amado) #10950
- make sure proxy is 2048 bit. Fix #11005 (belforte) #11006
- Explicitly create proxies with 2048 bits (Alan Malta Rodrigues) #11008
Bug Fixes
- Fix a few DBS3Reader APIs, likely unused ones (Alan Malta Rodrigues) #10990
Enhancements
WMCore 2.0.0 central services production release
This is our first WMCore release which no longer supports Python2 stack, and we will keep gradually removing python2 from our baseline code (including the future library).
This release has a major bug fix for MSUnmerged as well as other bug-fixes and enhancements to the overall system.
Release date: 8 February 2022.
Changes since release: 1.5.7.
Central services
Software stack
- Update Python3 pycurl library from 7.43.0.3 to 7.43.0.6 (Alan Malta Rodrigues) cms-sw/cmsdist#7600
Features and/or feature changes
- Aggregate DBS results from Go server (Valentin Kuznetsov) #10963
Bug Fixes
- Fix MSUnmerged logic and MSUnmergedRSE format to account for deletion errors. (Todor Ivanov) #10687
Enhancements
- Check that sdir exists on a file system (Valentin Kuznetsov) #10938
- Added MSUnmerged to the WM schematic (Alan Malta Rodrigues) #10974
- Update diagrams and contribution documentation for end of support for Py2 (Alan Malta Rodrigues) #10968
WMAgent
Features and/or feature changes
Bug Fixes
- DBS3Upload: check error message from the response body attribute (Alan Malta Rodrigues) #10961
Enhancements
- Update deploy-wmagent for 1.5.7.patch1; remove deprecated py2 option (Alan Malta Rodrigues) #10966
- Update error message for jobs killed from site state transition (Alan Malta Rodrigues) #10958
Python3 related changes
WMCore 1.5.7.patch1 WMAgent production release
This is the last WMAgent stable release supporting both Python2 and Python3. This release brings in some important new features and bug fixes, in addition to the continuous minor enhancements. It also comes with a few very important software stack changes.
Release date: 27 January 2022.
Changes since release: 1.5.4.
WMAgent
Software stack
- Upgrade MariaDB spec from 10.1.21 to 10.6.5 (Alan Malta Rodrigues) #7205
- Update pylint version required from 2.6.0 to 2.7.0 (Alan Malta Rodrigues) #10913
- Bump dbs3-client to 4.0.5 (Alan Malta Rodrigues) #10931
- Bump dbs3-pycurl to 3.17.7 (Alan Malta Rodrigues) #10931
- Update CMSCouchapp version for python3 stack (Alan Malta Rodrigues) #10910
- Update python3 pip to 20.3.3 and setuptools to 51.3.3 (Alan Malta Rodrigues) #7418
Features and/or feature changes
- Remove trailing slash from DBS URL (Alan Malta Rodrigues) #10930
- Add two new CMSSW exit code: 8034 and 8501 (Stefano Belforte) #10850
- Remove PhEDEx library and any other use of PhEDEx (Alan Malta Rodrigues) #10856
- Stop using password for MariaDB localhost users (Alan Malta Rodrigues) #10932
- Use siteLocalConfig to get LogCollect stage out params (germanfgv) #10905
- Add --deep option to getReplicaInfoForBlocks. Add slicing to deleteBlocks (germanfgv) #10902
- Set request_GPUs classad according to RequiresGPU (Alan Malta Rodrigues) #10849
- Support different architectures during job submission and runtime (Alan Malta Rodrigues) #10853
Bug Fixes
- DBS3Upload: check error message from the response body attribute #10961
- Properly update workqueue elements from JobUpdater (Alan Malta Rodrigues) #10944
- Skip ErrorHandler cycle if agent configuration fails to be retrieved - take2 (Alan Malta Rodrigues) #10852
- Remove leading ../../ from logArchive tarballs (Alan Malta Rodrigues) #10868
- Check whether WMStep object has gpu attribute (Alan Malta Rodrigues) #10866
- Use the right COMP scramArch for PowerPC workflows (Alan Malta Rodrigues) #10871
- CredentialProxy - convert to bytes the input of hashlib.sha1() (Dario Mapelli) #10918
- REST/Server.py - encode to bytes input of hashlib.sha1 (Dario Mapelli) #10917
- Fix path for Py2/Py3 WMAgent logs (Alan Malta Rodrigues) #10897
Enhancements
- Update error message for jobs killed from site state transition #10958
- Create the MariaDB wmcore unittest database with the current unix user (Alan Malta Rodrigues) #10934
- fix DbsUrl in test modules and templates (Alan Malta Rodrigues) #10930
- Fix python version in subprocess call for inject-test-wfs (Alan Malta Rodrigues) #10847
- Update Rucio Int url (Alan Malta Rodrigues) #10855
- Update job priority only for processing and production task types (Kenyi Hurtado) #10812
- Updating workflow json templates; adding working PY3 templates (Alan Malta Rodrigues) #10761
- Update wmagent-deploy script to deploy Py3 WMAgent by default; update to cycle 1.5.4 (Alan Malta Rodrigues) #10859
- Update campaigns used in our test setup (Alan Malta Rodrigues) #10859