Add lifetime to RucioContainers shipping to disk sites #10950

jhonatanamado · 2022-01-07T15:00:39Z

Add a new lifetime parameter for each RucioContainer rule that is subscribed to disk site

Status

Tested with a replay

Description

For a given dataset (Rucio Container rule) processed by Tier0 and shipping to disk sites, add a new attribute of lifetime to the rucio container rule only if its subscribed at Disk sites

Is it backward compatible (if not, which system it affects?)

MAYBE

Related PRs

First discussed in #10747

External dependencies / deployment changes

No

cmsdmwmbot · 2022-01-07T15:11:36Z

Jenkins results:

Python2 Unit tests: failed
- 2 new failures
- 1 changes in unstable tests
Python3 Unit tests: failed
- 2 new failures
- 4 changes in unstable tests
Python2 Pylint check: failed
- 17 warnings and errors that must be fixed
- 13 warnings
- 89 comments to review
Python3 Pylint check: succeeded
Pylint py3k check: failed
- 1 warnings
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12716/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-01-10T14:31:55Z

Jenkins results:

Python2 Unit tests: failed
- 1 new failures
Python3 Unit tests: failed
- 1 new failures
- 1 tests no longer failing
- 1 changes in unstable tests
Python2 Pylint check: failed
- 18 warnings and errors that must be fixed
- 13 warnings
- 103 comments to review
Python3 Pylint check: succeeded
Pylint py3k check: failed
- 1 warnings
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12717/artifact/artifacts/PullRequestReport.html

amaltaro · 2022-01-20T13:34:28Z

@jhonatanamado Jhonatan, can you please link the relevant T0 changes to this PR? This will allow us to see all the required changes to have this feature fully functional.

This unit test:
WMCore_t.WMSpec_t.StdSpecs_t.StepChain_t.StepChainTests:testSubscriptions changed from success to failure

need to be looked at as well.

jhonatanamado · 2022-01-20T13:43:01Z

Hi @amaltaro, Sorry, indeed I forgot to add the relevant changes to the T0 code. Please check the following dmwm/T0#4642

About the unit test, I will look at it asap

cmsdmwmbot · 2022-01-21T06:58:37Z

Jenkins results:

Python2 Unit tests: succeeded
Python3 Unit tests: succeeded
- 1 tests no longer failing
- 2 changes in unstable tests
Python2 Pylint check: failed
- 26 warnings and errors that must be fixed
- 16 warnings
- 169 comments to review
Python3 Pylint check: succeeded
Pylint py3k check: failed
- 1 warnings
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12723/artifact/artifacts/PullRequestReport.html

jhonatanamado · 2022-01-22T00:40:09Z

Hi @amaltaro , Alan, the changes were tested with a replay without issues. Let me know if you have other suggestion.

amaltaro · 2022-01-24T23:36:05Z

@jhonatanamado @germanfgv could you please clarify whether this issue is supposed to fix #10558 as well? Or is #10558 actually requesting some additional rucio rule parameters?

jhonatanamado · 2022-01-25T02:03:24Z

Hi @amaltaro ,I did not remember that issue because I installed a cronjob to clean those 'block-level' rules but indeed, with some minor modifications in WMAgentConfig.py , adding that parameter in this function and applying that parameter to only the account tier0_replay, I believe that with the changes in this PR we could solve #10558 too.
Do you like to see those modifications included in this PR or in a different PR?

amaltaro · 2022-01-25T02:25:21Z

@jhonatanamado oh, I thought it was the same request. If it's slightly different, then I think working on that in a different PR will be the best.

amaltaro

@jhonatanamado these changes are looking good in general. However, I made some comments/requests along the code for your consideration. Thanks

amaltaro · 2022-02-11T19:41:09Z

src/python/WMComponent/DBS3Buffer/MySQL/Create.py

@@ -49,6 +49,7 @@ def __init__(self, logger=None, dbi=None, params=None):
                 subscribed             INTEGER      DEFAULT 0,
                 phedex_group           VARCHAR(100),
                 delete_blocks          INTEGER,
+                 dataset_lifetime       INTEGER     DEFAULT 46656000,


Jhonatan, if you are actually setting dataset_lifetime for every dataset from the RunConfigAPI, then having a default value of 0 looks more reasonable to me.
In the RucioInjector code, we could even have a check like:

if dataset_lifetime > 0: kwargs['lifetime'....

such that when it's 0, nothing happens. What do you think?

Indeed, is more reasonable. I also modified RucioInjector in the clause of add that parameter

amaltaro · 2022-02-11T19:43:52Z

src/python/WMCore/WMSpec/StdSpecs/StdBase.py

@@ -1079,6 +1079,9 @@ def getWorkloadCreateArgs():
                     "MergedLFNBase": {"default": "/store/data"},
                     "UnmergedLFNBase": {"default": "/store/unmerged"},
                     "DeleteFromSource": {"default": False, "type": strToBool},
+
+                     # Rucio rule subscription lifetime (used in ContainerRules by T0)
+                     "DatasetLifetime": {"default": 18*30*24*60*60, "type": int, "optional": True},


Can we make it an assignment only parameter? Just like the other PhEDEx-related parameters that we have encoded in the wofkflow?

src/python/WMCore/WMSpec/StdSpecs/StdBase.py

src/python/WMCore/WMSpec/WMTask.py

cmsdmwmbot · 2022-02-15T07:03:26Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 2 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: failed
- 1 warnings
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12783/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-02-15T07:21:34Z

Jenkins results:

Python3 Unit tests: succeeded
- 2 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: failed
- 1 warnings
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12784/artifact/artifacts/PullRequestReport.html

amaltaro

Jhonatan, thanks for providing these changes. There are still a few items that require attention, please check comments along the code. Thanks

src/python/WMComponent/DBS3Buffer/Oracle/Create.py

src/python/WMCore/WMSpec/StdSpecs/StdBase.py

src/python/WMCore/WMSpec/WMWorkload.py

test/python/WMCore_t/WMSpec_t/StdSpecs_t/StepChain_t.py

cmsdmwmbot · 2022-02-16T00:08:04Z

Jenkins results:

Python3 Unit tests: succeeded
- 2 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: failed
- 1 warnings
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12789/artifact/artifacts/PullRequestReport.html

Add a new lifetime parameter for each RucioContainer rule that is subscribed to disk site

cmsdmwmbot · 2022-02-16T06:38:42Z

Jenkins results:

Python3 Unit tests: succeeded
- 3 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: failed
- 1 warnings
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12790/artifact/artifacts/PullRequestReport.html

amaltaro

Thank you for applying those changes, Jhonatan. It looks good to go now.

amaltaro · 2022-02-16T20:02:07Z

@jhonatanamado Jhonatan, if you want to run a fresh and clean replay with this code, I have just cut tag 2.0.1.pre2.

jhonatanamado · 2022-02-16T20:04:20Z

Great, thanks for the tag @amaltaro. I will do it and give to you a feedback on how it goes.

src/python/WMComponent/RucioInjector/RucioInjectorPoller.py

jhonatanamado force-pushed the container-lifetime-jkins branch from 25ca441 to 0a891b0 Compare January 10, 2022 14:18

jhonatanamado marked this pull request as draft January 21, 2022 03:14

jhonatanamado force-pushed the container-lifetime-jkins branch from 0a891b0 to 6fd97e5 Compare January 21, 2022 06:42

jhonatanamado marked this pull request as ready for review January 22, 2022 00:33

amaltaro added the PR: Do not merge yet label Jan 27, 2022

amaltaro requested changes Feb 11, 2022

View reviewed changes

jhonatanamado marked this pull request as draft February 15, 2022 06:48

jhonatanamado force-pushed the container-lifetime-jkins branch from 6fd97e5 to b3f6010 Compare February 15, 2022 06:49

jhonatanamado force-pushed the container-lifetime-jkins branch from b3f6010 to 8c7673a Compare February 15, 2022 07:11

jhonatanamado marked this pull request as ready for review February 15, 2022 15:03

amaltaro requested changes Feb 15, 2022

View reviewed changes

jhonatanamado marked this pull request as draft February 15, 2022 23:54

jhonatanamado force-pushed the container-lifetime-jkins branch from 8c7673a to c982bfb Compare February 15, 2022 23:55

Add lifetime to RucioContainers shipping to disk sites

634c629

Add a new lifetime parameter for each RucioContainer rule that is subscribed to disk site

jhonatanamado marked this pull request as ready for review February 16, 2022 06:22

jhonatanamado force-pushed the container-lifetime-jkins branch from c982bfb to 634c629 Compare February 16, 2022 06:23

jhonatanamado requested a review from amaltaro February 16, 2022 06:39

amaltaro approved these changes Feb 16, 2022

View reviewed changes

amaltaro merged commit d7c04f3 into dmwm:master Feb 16, 2022

amaltaro removed the PR: Do not merge yet label Feb 16, 2022

jhonatanamado commented Feb 21, 2022

View reviewed changes

src/python/WMComponent/RucioInjector/RucioInjectorPoller.py Show resolved Hide resolved

todor-ivanov mentioned this pull request Apr 22, 2022

WorkQueue fails to create subscription due to ORACLE unique constraint violation #11108

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lifetime to RucioContainers shipping to disk sites #10950

Add lifetime to RucioContainers shipping to disk sites #10950

jhonatanamado commented Jan 7, 2022

cmsdmwmbot commented Jan 7, 2022

cmsdmwmbot commented Jan 10, 2022

amaltaro commented Jan 20, 2022

jhonatanamado commented Jan 20, 2022

cmsdmwmbot commented Jan 21, 2022

jhonatanamado commented Jan 22, 2022

amaltaro commented Jan 24, 2022

jhonatanamado commented Jan 25, 2022 •

edited

Loading

amaltaro commented Jan 25, 2022

amaltaro left a comment

amaltaro Feb 11, 2022

jhonatanamado Feb 15, 2022

amaltaro Feb 11, 2022

cmsdmwmbot commented Feb 15, 2022

cmsdmwmbot commented Feb 15, 2022

amaltaro left a comment

cmsdmwmbot commented Feb 16, 2022

cmsdmwmbot commented Feb 16, 2022

amaltaro left a comment

amaltaro commented Feb 16, 2022

jhonatanamado commented Feb 16, 2022

Add lifetime to RucioContainers shipping to disk sites #10950

Add lifetime to RucioContainers shipping to disk sites #10950

Conversation

jhonatanamado commented Jan 7, 2022

Status

Description

Is it backward compatible (if not, which system it affects?)

Related PRs

External dependencies / deployment changes

cmsdmwmbot commented Jan 7, 2022

cmsdmwmbot commented Jan 10, 2022

amaltaro commented Jan 20, 2022

jhonatanamado commented Jan 20, 2022

cmsdmwmbot commented Jan 21, 2022

jhonatanamado commented Jan 22, 2022

amaltaro commented Jan 24, 2022

jhonatanamado commented Jan 25, 2022 • edited Loading

amaltaro commented Jan 25, 2022

amaltaro left a comment

Choose a reason for hiding this comment

amaltaro Feb 11, 2022

Choose a reason for hiding this comment

jhonatanamado Feb 15, 2022

Choose a reason for hiding this comment

amaltaro Feb 11, 2022

Choose a reason for hiding this comment

cmsdmwmbot commented Feb 15, 2022

cmsdmwmbot commented Feb 15, 2022

amaltaro left a comment

Choose a reason for hiding this comment

cmsdmwmbot commented Feb 16, 2022

cmsdmwmbot commented Feb 16, 2022

amaltaro left a comment

Choose a reason for hiding this comment

amaltaro commented Feb 16, 2022

jhonatanamado commented Feb 16, 2022

jhonatanamado commented Jan 25, 2022 •

edited

Loading