handle inverted filters in TriggerObjectStandAlone

[missirol]

PR description:

This PR improves TriggerObjectStandAlone to handle cases where inverted filters (e.g. ~process.fooFilter, which returns the opposite of process.fooFilter) are used in the configuration file.

Addresses #37152 (comment) by @perrotta, and possibly solves #37157.

Note: I don't know TriggerObjectStandAlone well, and I would encourage PAT experts to have a look at the suggested fix.

FYI: @Martin-Grunewald

PR validation:

Ran runTheMatrix.py -l 11634.0, and checked that the warnings in question disappeared.

If this PR is a backport, please specify the original PR and why you need to backport that PR:

N/A

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37181/28749

• This PR adds an extra 20KB to repository

[cmsbuild]

A new Pull Request was created by @missirol (Marino Missiroli) for master.

It involves the following packages:

• DataFormats/PatCandidates (reconstruction)

@jpata, @cmsbuild, @clacaputo, @slava77 can you please review it and eventually sign? Thanks.
@gpetruc, @gouskos, @rovere, @hatakeyamak this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[perrotta]

please test
(Thank you Marino: let give it a try)

[perrotta]

Naive comment: shouldn't this be a tilde, instead (or in addition)?

[missirol]

So, what I did was simply print all these strings and it was clear that cms.ignore(process.foo) resolves to -foo, while ~process.foo resolves to !foo.

I don't actually know where the creation of these strings happens: it would be useful to know, so that we can figure out if other special cases/characters should be kept into account for EDFilters.

A naive look at the code base with git grep "'\!'" did not give me certainty that - and ! are the only two cases (maybe they are), e.g.

cmssw/FWCore/ParameterSet/python/SequenceTypes.py

Lines 576 to 628 in 6abbda5

	class _SequenceNegation(_UnarySequenceOperator):
	"""Used in the expression tree for a sequence as a stand in for the '!' operator"""
	def __init__(self, operand):
	super(_SequenceNegation,self).__init__(operand)
	def __str__(self):
	return '~%s' %self._operand
	def dumpSequenceConfig(self):
	return '!%s' %self._operand.dumpSequenceConfig()
	def dumpSequencePython(self, options=PrintOptions()):
	if self._operand.isOperation():
	return '~(%s)' %self._operand.dumpSequencePython(options)
	return '~%s' %self._operand.dumpSequencePython(options)
	def decoration(self):
	return '!'
	class _SequenceIgnore(_UnarySequenceOperator):
	"""Used in the expression tree for a sequence as a stand in for the '-' operator"""
	def __init__(self, operand):
	super(_SequenceIgnore,self).__init__(operand)
	def __str__(self):
	return 'ignore(%s)' %self._operand
	def dumpSequenceConfig(self):
	return '-%s' %self._operand.dumpSequenceConfig()
	def dumpSequencePython(self, options=PrintOptions()):
	return 'cms.ignore(%s)' %self._operand.dumpSequencePython(options)
	def decoration(self):
	return '-'
	class _SequenceWait(_UnarySequenceOperator):
	"""Used in the expression tree for a sequence as a stand in for the '|' operator"""
	def __init__(self, operand):
	super(_SequenceWait,self).__init__(operand)
	def __str__(self):
	return 'wait(%s)' %self._operand
	def dumpSequenceConfig(self):
	return '|%s' %self._operand.dumpSequenceConfig()
	def dumpSequencePython(self, options=PrintOptions()):
	return 'cms.wait(%s)' %self._operand.dumpSequencePython(options)
	def decoration(self):
	return '|'
	class _SequenceWaitAndIgnore(_UnarySequenceOperator):
	"""Used in the expression tree for a sequence as a stand in for the '+' operator"""
	def __init__(self, operand):
	super(_SequenceWaitAndIgnore,self).__init__(operand)
	def __str__(self):
	return 'wait(ignore(%s))' %self._operand
	def dumpSequenceConfig(self):
	return '+%s' %self._operand.dumpSequenceConfig()
	def dumpSequencePython(self, options=PrintOptions()):
	return 'cms.wait(cms.ignore(%s))' %self._operand.dumpSequencePython(options)
	def decoration(self):
	return '+'

cmssw/FWCore/Framework/src/StreamSchedule.cc

Lines 397 to 405 in 6abbda5

	if (name[0] == '!') {
	filterAction = WorkerInPath::Veto;
	} else if (name[0] == '-' or name[0] == '+') {
	filterAction = WorkerInPath::Ignore;
	}
	if (name[0] == '|' or name[0] == '+') {
	//cms.wait was specified so do not run concurrently
	doNotRunConcurrently = true;
	}

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2311fa/22960/summary.html
COMMIT: 1599c57
CMSSW: CMSSW_12_3_X_2022-03-08-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37181/22960/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 24 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3703700
• DQMHistoTests: Total failures: 14
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3703664
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
• Checked 0 log files, 0 edm output root files, 49 DQM output files

[missirol]

type bugfix

Looks like the warnings in question have disappeared as expected.

@makortel , it's not urgent but could you please have a look at #37181 (comment) , and let me know if I'm missing something obvious? Can we have EDFilters whose names are prepended by a special character other than - and !?

it's not urgent

Edit : I guess it became urgent.. #37181 (comment)

[perrotta]

urgent
@cms-sw/reconstruction-l2 could you please have a look, and sign if you agree with it?

[makortel]

@makortel , it's not urgent but could you please have a look at #37181 (comment) , and let me know if I'm missing something obvious? Can we have EDFilters whose names are prepended by a special character other than - and !?

I'd think these "decorations" to be more or less implementation details of the framework. + is applicable to EDFilters too (there is also |, but it wouldn't make much sense for EDFilters). It is also possible that we'll add more in the future.

The aim here appears to be stripping these off in order to get the ParameterSet of the module. Instead of extending the "stripping" logic here, perhaps we should provide a function to do that? @Dr15Jones, what do you think?

[clacaputo]

+reconstruction

• possible bugfix HLT menu does not map correctly in PAT when negated filters are present  #37157
• technical
• no reco changes observed or expected

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1

• It solves HLT menu does not map correctly in PAT when negated filters are present  #37157
• Further refinements of the stripping of the "decorations" of the modules can be investigated in future developments