[Tensorflow] Update to version 2.5.0 #7141

smuzaffar · 2021-07-15T20:56:40Z

Tensorflow 2.5.0
Bazel 3.7.2
Eigen commit https://gitlab.com/libeigen/eigen/-/commit/f612df273689a19d25b45ca4f8269463207c4fee as requited by TF
tensorboard 2.5.0
tensorboard-data-server 0.6.1
tensorflow-estimator 2.5.0

cmsbuild · 2021-07-15T20:57:03Z

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_12_0_X/master.

@cmsbuild, @smuzaffar, @mrodozov, @iarspider can you please review it and eventually sign? Thanks.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.
cms-bot commands are listed here

smuzaffar · 2021-07-15T20:59:50Z

please test

FYI @riga

cmsbuild · 2021-07-15T22:49:40Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-39f2e8/16866/summary.html
COMMIT: c18fb28
CMSSW: CMSSW_12_0_X_2021-07-15-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7141/16866/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ DEPS='py2-tensorboard-data-server '
+ '[' 'py2-tensorboard-data-server ' '!=' '' ']'
+ echo 'ERROR: Missing dependencies for py3-tensorboard (python3) found: py2-tensorboard-data-server '
ERROR: Missing dependencies for py3-tensorboard (python3) found: py2-tensorboard-data-server 
+ exit 1
error: Bad exit status from /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.y9NWRz (%build)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.y9NWRz (%build)

cmsbuild · 2021-07-16T04:40:34Z

Pull request #7141 was updated.

smuzaffar · 2021-07-16T04:40:46Z

please test

smuzaffar · 2021-07-16T04:41:31Z

please test for CMSSW_12_0_X/slc7_aarch64_gcc9

smuzaffar · 2021-07-16T04:42:01Z

please test for CMSSW_12_0_X/slc7_ppc64le_gcc9

cmsbuild · 2021-07-16T05:02:56Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-39f2e8/16877/summary.html
COMMIT: 2a06179
CMSSW: CMSSW_12_0_X_2021-07-15-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7141/16877/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ DEPS='py2-keras-nightly '
+ '[' 'py2-keras-nightly ' '!=' '' ']'
+ echo 'ERROR: Missing dependencies for py3-tensorflow (python3) found: py2-keras-nightly '
ERROR: Missing dependencies for py3-tensorflow (python3) found: py2-keras-nightly 
+ exit 1
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.q5cJMm (%build)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.q5cJMm (%build)

cmsbuild · 2021-07-16T06:13:01Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-39f2e8/16878/summary.html
COMMIT: 2a06179
CMSSW: CMSSW_12_0_X_2021-07-15-2300/slc7_ppc64le_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7141/16878/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ DEPS='py2-keras-nightly '
+ '[' 'py2-keras-nightly ' '!=' '' ']'
+ echo 'ERROR: Missing dependencies for py3-tensorflow (python3) found: py2-keras-nightly '
ERROR: Missing dependencies for py3-tensorflow (python3) found: py2-keras-nightly 
+ exit 1
error: Bad exit status from /scratch/cmsbuild/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.mqaM7X (%build)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /scratch/cmsbuild/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.mqaM7X (%build)

cmsbuild · 2021-07-16T06:29:10Z

Pull request #7141 was updated.

smuzaffar · 2021-07-16T06:29:32Z

please test

smuzaffar · 2021-07-16T06:30:09Z

please test for CMSSW_12_0_X/slc7_ppc64le_gcc9

smuzaffar · 2021-07-16T06:56:58Z

please test for CMSSW_12_0_X/slc7_aarch64_gcc9

cmsbuild · 2021-07-16T11:00:33Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-39f2e8/16882/summary.html
COMMIT: 05b9896
CMSSW: CMSSW_12_0_X_2021-07-15-2300/slc7_ppc64le_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7141/16882/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test_PrepareInputDb had ERRORS
---> test test_MpsWorkFlow had ERRORS
---> test TestHeterogeneousCoreSonicTritonProducerGPU had ERRORS
---> test testEigenGPUNoFit_t had ERRORS
and more ...

cmsbuild · 2021-07-16T12:45:25Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-39f2e8/16881/summary.html
COMMIT: 05b9896
CMSSW: CMSSW_12_0_X_2021-07-15-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7141/16881/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2786302
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2786279
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

smuzaffar · 2021-07-16T13:10:35Z

enable profiling

smuzaffar · 2021-07-16T13:11:02Z

please test

cmsbuild · 2021-07-16T16:31:58Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-39f2e8/16896/summary.html
COMMIT: 05b9896
CMSSW: CMSSW_12_0_X_2021-07-15-2300/slc7_aarch64_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7141/16896/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@davidlange6 remove python2 pip support and remaining python2 packages except for future #7112

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-39f2e8/16896/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-39f2e8/16896/git-merge-result

cmsbuild · 2021-07-16T21:45:07Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-39f2e8/16905/summary.html
COMMIT: 05b9896
CMSSW: CMSSW_12_0_X_2021-07-15-2300/slc7_amd64_gcc900
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7141/16905/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@davidlange6 remove python2 pip support and remaining python2 packages except for future #7112

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-39f2e8/16905/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-39f2e8/16905/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2786302
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2786273
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 37 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

smuzaffar · 2021-07-17T10:04:23Z

assign reconstruction
@cms-sw/reconstruction-l2 tensorflow 2.5.0 update looks good, can you please check the profiling results and sign this PR if all look good?

cmsbuild · 2021-07-17T10:04:46Z

New categories assigned: reconstruction

@slava77,@perrotta,@jpata you have been requested to review this Pull request/Issue and eventually sign? Thanks

smuzaffar · 2021-07-19T22:05:01Z

@slava77,@perrotta,@jpata any objection on going forward with TF 2.5.0 integration?

slava77 · 2021-07-19T22:40:39Z

@slava77,@perrotta,@jpata any objection on going forward with TF 2.5.0 integration?

https://cmssdt.cern.ch/circles/web/piechart.php?local=false&dataset=CMSSW_12_0_X_2021-07-15-2300%2Fslc7_amd64_gcc900%2F11834.21%2FPR-39f2e8%2F16905%2Fstep4_PAT_PU.resources&resource=time_thread&colours=default&groups=packages&threshold=0
vs
baseline
https://cmssdt.cern.ch/circles/web/piechart.php?local=false&dataset=CMSSW_12_0_X_2021-07-15-2300%2Fslc7_amd64_gcc900%2F11834.21%2Fstep4_PAT_PU.resources&resource=time_thread&colours=default&groups=packages&threshold=0

timing of deepTau and deepMet modules looks consistent (both are a few % slower with this PR tests, but some other times are up as well, which is most likely the measurement stability). Also, there are no differences.

So, it should be fine to merge for reco needs.

smuzaffar · 2021-07-20T06:30:06Z

+externals

[Tensorflow] Update to version 2.5.0

c18fb28

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Jul 15, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 15, 2021

cmsbuild added tests-rejected and removed tests-started labels Jul 15, 2021

added tensorboard data server

2a06179

cmsbuild added tests-pending and removed tests-rejected labels Jul 16, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 16, 2021

cmsbuild added tests-rejected and removed tests-started labels Jul 16, 2021

added keras-nightly as tensorflow deps

05b9896

cmsbuild added tests-pending and removed tests-rejected labels Jul 16, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 16, 2021

cmsbuild added tests-approved and removed tests-started labels Jul 16, 2021

cmsbuild added tests-started and removed tests-approved labels Jul 16, 2021

cmsbuild added tests-approved and removed tests-started labels Jul 16, 2021

cmsbuild added the reconstruction-pending label Jul 17, 2021

smuzaffar merged commit 9bef1b0 into IB/CMSSW_12_0_X/master Jul 20, 2021

cmsbuild added externals-approved and removed externals-pending labels Jul 20, 2021

This was referenced Jul 20, 2021

Update tag for RecoTracker-MkFit to V00-02-00 #7149

Merged

Use python3 to generate geometry cms-sw/cmssw#34554

Closed

Introduce conddb script unit tests cms-sw/cmssw#34392

Merged

[TF] patch it to use correct pybind11 paths #7151

Merged

smuzaffar deleted the tf-250 branch July 22, 2021 17:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tensorflow] Update to version 2.5.0 #7141

[Tensorflow] Update to version 2.5.0 #7141

smuzaffar commented Jul 15, 2021 •

edited

cmsbuild commented Jul 15, 2021

smuzaffar commented Jul 15, 2021

cmsbuild commented Jul 15, 2021

cmsbuild commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

smuzaffar commented Jul 17, 2021

cmsbuild commented Jul 17, 2021

smuzaffar commented Jul 19, 2021

slava77 commented Jul 19, 2021

smuzaffar commented Jul 20, 2021

[Tensorflow] Update to version 2.5.0 #7141

[Tensorflow] Update to version 2.5.0 #7141

Conversation

smuzaffar commented Jul 15, 2021 • edited

cmsbuild commented Jul 15, 2021

smuzaffar commented Jul 15, 2021

cmsbuild commented Jul 15, 2021

External Build

cmsbuild commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

External Build

cmsbuild commented Jul 16, 2021

External Build

cmsbuild commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

Unit Tests

cmsbuild commented Jul 16, 2021

Comparison Summary

smuzaffar commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

Comparison Summary

smuzaffar commented Jul 17, 2021

cmsbuild commented Jul 17, 2021

smuzaffar commented Jul 19, 2021

slava77 commented Jul 19, 2021

smuzaffar commented Jul 20, 2021

smuzaffar commented Jul 15, 2021 •

edited