Make container level data placement for DQMHarvest; update Start.Policy.Dataset to container too

[amaltaro]

Fixes #11892

Status

ready

Description

As the title says, it provides a change for MSTransferor and the relevant update to GlobalWorkqueue as well, such as:

• MSTransferor: when dealing with DQMHarvest workflows, make a Rucio rule for the whole input container.
• Global WorkQueue: the Dataset policy will resolve input location based on the container instead of each block (which could be falling back to the container level, but in an inefficient way)

Is it backward compatible (if not, which system it affects?)

YES (but DQMHarvest workflows stuck in the system might have to be aborted/recreated)

Related PRs

None

External dependencies / deployment changes

None

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 2 new failures
  • 1 tests no longer failing
• Python3 Pylint check: failed
  • 2 warnings and errors that must be fixed
  • 4 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 1 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14843/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 1 changes in unstable tests
• Python3 Pylint check: failed
  • 3 warnings and errors that must be fixed
  • 5 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 1 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14844/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 1 new failures
  • 1 tests no longer failing
  • 1 changes in unstable tests
• Python3 Pylint check: failed
  • 3 warnings and errors that must be fixed
  • 5 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 1 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14845/artifact/artifacts/PullRequestReport.html

[amaltaro]

test this please

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 2 new failures
• Python3 Pylint check: failed
  • 3 warnings and errors that must be fixed
  • 5 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 1 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14846/artifact/artifacts/PullRequestReport.html

[amaltaro]

test this please

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 1 tests no longer failing
• Python3 Pylint check: failed
  • 3 warnings and errors that must be fixed
  • 5 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 1 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14847/artifact/artifacts/PullRequestReport.html

[todor-ivanov]

looks good