Non-reproducibility in patMETs_slimmedMETsNoHF

[makortel]

Tests in #36550 (comment) showed a difference in 2026D60 28234.0 all_mini_OldVSNew for patMETs_slimmedMETsNoHF

The PR should be purely technical, and I didn't find any earlier mention of a possible non-reproducibility there.

[makortel]

assign reconstruction, xpog

[cmsbuild]

New categories assigned: xpog,reconstruction

@slava77,@jpata,@mariadalfonso,@gouskos,@clacaputo,@fgolf you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

Maybe not so rare, but recent, occurred again in #36550 (comment) and also in #36507 (comment).

[jpata]

Based on the PR tests in https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons I can first see this issue in CMSSW_12_3_X_2021-12-18-1100.
EDIT: actually, it's also there in CMSSW_12_3_X_2021-12-17-1100, investigating further...

[jpata]

Slava suggested it might be AVX in DeepMET, checking further in the logs...

[jpata]

Based on recent PRs, it looks like indeed an issue of executing TF on CPUs with different capabilities.

PR:

2022-01-05 17:40:07.040781: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA

baseline:

2022-01-05 12:25:59.300506: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA

This is a known issue with TF-based deepmet.

[jpata]

type jetmet

[vlimant]

related to #32628