DNN-based Tau-Id discrimians (94X) #25385
Conversation
- Defined base class for deep tau discriminators. - Removed weight files from home cms repository. Now using weights from cms-data. - Defined WP for both discriminators. Now all discriminators return the corresponding WP results. - Removed cfi files. Using fillDescriptions instead. - General code review and cleaning.
…ection with the new Tau-Ids
Integration of DPFIsolation and DeepTauId
A few cleanings to DNN tools
Made DeepTauId and DPFIsolation thread-safe
…es quantized - Added a new parameter 'version' on runTauIdMVA, used on DPFIsolation - Changes on DeepTauId to reduce memory consumption
…read and reduce the memory consuption - Creation of class DeepTauCache in DeepTauBase, in which now is created graph and session - Implementation of two new static methods inside the class DeepTauBase: initializeGlobalCache and globalEndJob. The graph and DeepTauCache object are created now inside initializeGlobalCache
TauWPThreshold class parses WP cut string (or value) provided in the python configuration. It is needed because the use of the standard StringObjectFunction class to parse complex expression results in an extensive memory usage (> 100 MB per expression).
- Implementation of global cache to avoid reloading graph for each thread - Creation of two new static methods inside the class DeepTauBase: initializeGlobalCache and globalEndJob. The graph and DeepTauCache object are created now inside initializeGlobalCache. The memory consumption of initializeGlobalCache for the original, quantized and files that are load using memory mapping method are in the memory_usage.pdf file - Implemented configuration to use new training files quantized, and set them as default - Implementation of configuration for load files using memory mapping. In our case there wasn't any improvement, respect at the memory consumption of this method, respect the quantized files, so this is not used, but set for future training files - General code review and cleaning.
|
+code-checks |
|
code-checks |
|
@mbluj as far as I can see the list of commits here is containing the unwanted big files, so we need to squash it. In case we do not want to miss the review history, I suggest to just open a new PR with the commit resulting from the squash. |
|
@fabiocos: please squash the history as done with 102X version of this PR if you you don't mind. The full history is anyway preserved for original PR to the master. Thank you. |
|
+1 The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic: |
|
Comparison job queued. |
|
The code-checks are being triggered in jenkins. |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-25385/7868
|
|
Comparison is ready Comparison Summary:
|
|
@mbluj @perrotta I would prefer to avoid losing the review history. But as 10 we need to squash anyway and 2) @smuzaffar will implement the possibility to instruct the bot for it, I will clone this PR into another one, and try the new possibility there. |
|
code-checks |
|
The code-checks are being triggered in jenkins. |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-25385/7927
|
|
@smuzaffar I have cross checked that the code in #26621 is identical to that in this PR, but code-checks is complaining in the new and not in the old version. If I just manually merge in a working area 25385 and run "scram b code-checks" I indeed find the complaints, and produces the patch as in #25621. So what is going on? Is the bot missing the differences in some case? |
|
It is indeed interesting issue to understand, but maybe in parallel it makes sense to fix the code issues here and prepare small PRs with similar fixes also for master and 102X? Or start with fix to master with backports to 102X and here (with cherry-pick)? |
|
In order to manage things in a ordered way, I would move forward with merging the present version, and let you add fixes where needed for all the possible releases, staring with master |
|
@mbluj as the squashed version has been merged, this one may be closed, and will stay as a a reference for the review. |
cmsbuild
added
pending-signatures
reconstruction-rejected
and removed
fully-signed
reconstruction-approved
labels
This pull request provides two new DNN-based Tau-Ids, DeepTau and DPFTau, to be produced for pat::Taus with MiniAOD.
It is a backport of #25016 to 94X for analyses based on 2016+2017 data and detailed description can be found therein.