Update tensorflow to 2.6.3

[smuzaffar]

No description provided.

[smuzaffar]

please test

[cmsbuild]

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_12_4_X/master.

@smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

[smuzaffar]

please test for el8_aarch64_gcc10

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a3f06c/24672/summary.html
COMMIT: 9354c72
CMSSW: CMSSW_12_4_X_2022-05-12-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7862/24672/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation warning when building: See details on the summary page.

[smuzaffar]

please test

[cmsbuild]

Pull request #7862 was updated.

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a3f06c/24675/summary.html
COMMIT: 7e6c9ba
CMSSW: CMSSW_12_4_X_2022-05-11-2300/el8_aarch64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7862/24675/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation warning when building: See details on the summary page.

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a3f06c/24673/summary.html
COMMIT: 7e6c9ba
CMSSW: CMSSW_12_4_X_2022-05-12-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7862/24673/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation warning when building: See details on the summary page.

[smuzaffar]

please test

[cmsbuild]

Pull request #7862 was updated.

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a3f06c/24699/summary.html
COMMIT: e85f650
CMSSW: CMSSW_12_4_X_2022-05-12-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7862/24699/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @cms-sw Update jupyter-server and notebook #7864

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a3f06c/24699/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a3f06c/24699/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test SiStripDAQ_O2O_test had ERRORS

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 478 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3697377
• DQMHistoTests: Total failures: 1122
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3696233
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 206 log files, 45 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[smuzaffar]

enable profiling

[smuzaffar]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a3f06c/24735/summary.html
COMMIT: e85f650
CMSSW: CMSSW_12_4_X_2022-05-15-2300/slc7_amd64_gcc10
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7862/24735/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-a3f06c/11634.301_TTbar_14TeV+2021_Run3FS+TTbar_14TeV_TuneCP5_GenSim+HARVESTNano

Summary:

• No significant changes to the logs found
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3741432
• DQMHistoTests: Total failures: 97
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3741312
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -0.004 KiB( 49 files compared)
• DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
• Checked 208 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[smuzaffar]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a3f06c/24751/summary.html
COMMIT: e85f650
CMSSW: CMSSW_12_5_X_2022-05-16-1100/slc7_amd64_gcc10
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7862/24751/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @cms-sw [SCRAM] Allow conditional dependency #7866
• @cms-sw [BuildRules] Do not create python/package symlink to src/package/python #7865

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a3f06c/24751/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a3f06c/24751/git-merge-result

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-a3f06c/11634.301_TTbar_14TeV+2021_Run3FS+TTbar_14TeV_TuneCP5_GenSim+HARVESTNano

Summary:

• No significant changes to the logs found
• Reco comparison results: 6 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3741432
• DQMHistoTests: Total failures: 98
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3741312
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 208 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[smuzaffar]

@cms-sw/reconstruction-l2 , can you please check the DQM comparison results here? There are some differences in Tracking for workflow 12434.0

[jpata]

Looks like TF ran with the same CPU options so it's not the usual topic related to different CPUs giving different results

new:      2022-05-16 22:38:42.260976: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
baseline: 2022-05-16 16:37:14.752072: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA

@leonardogiannini @cms-sw/tracking-pog-l2 could these changes in Tracking/TrackBuilding/pixelLessStep_quickAssociatorByHits be from the DNN in MkFit having small differences with a newer TF?
cms-sw/cmssw#37878

[jpata]

I'm noticing a similar DQM change in cms-sw/cmssw#37918, which seems not related to this TF upgrade at all.

So maybe there is some instability in the tracking DQM now. I will open an issue in cmssw.
EDIT: the issue is here cms-sw/cmssw#37970

[smuzaffar]

@cms-sw/reconstruction-l2 , as DQM comparison differences are not related to this update, should be go ahead and merge this? There are a lot of security alerts (flaged by guithub bot: https://github.com/cms-sw/cmsdist/security/dependabot ) which are due to TF 2.6.0. Updatign to 2.6.3 shoudl fix most of these issues. If no objections then I would like to merge it for 12.5.X

[jpata]

assign reconstruction

[jpata]

+reconstruction

• no physics changes from the TF update

[cmsbuild]

New categories assigned: reconstruction

@jpata,@slava77,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

[smuzaffar]

+externals
thanks @jpata

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_5_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)