cleaning up unused py2 tools ; restoring tensorflow as part of python_tools.spec #7019
Conversation
|
A new Pull Request was created by @davidlange6 (David Lange) for branch IB/CMSSW_12_0_X/devel. @cmsbuild, @smuzaffar, @mrodozov, @iarspider can you please review it and eventually sign? Thanks. |
|
test parameters:
|
|
please test |
|
-1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3ab08a/15916/summary.html External BuildI found compilation warning when building: See details on the summary page. |
|
Seems that I forgot to commit a file worth of changes. Will fix this afternoon
On Jun 12, 2021 9:42 AM, cmsbuild ***@***.***> wrote:
…-1
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3ab08a/15916/summary.html
COMMIT: 2d7cd22<2d7cd22>
CMSSW: CMSSW_12_0_DEVEL_X_2021-06-11-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7019/15916/install.sh to create a dev area with all the needed externals and cmssw changes.
External Build
I found compilation warning when building: See details on the summary page.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#7019 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGPFQ6RF5YHETHXK6SHEODTSMFSZANCNFSM46QSA5UQ>.
|
|
Pull request #7019 was updated. |
|
please test |
|
-1 Failed Tests: UnitTests The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
You can see more details here: Unit TestsI found errors in the following unit tests: ---> test test_PrepareInputDb had ERRORS ---> test test_CreateFileLists had ERRORS ---> test test_MpsWorkFlow had ERRORS ---> test EcalDAQ_O2O_test had ERRORS and more ... Comparison SummarySummary:
|
|
Pull request #7019 was updated. |
|
please test |
|
please test with cms-sw/cmssw#34106 |
|
please test with cms-sw/cmssw#34106 for CMSSW_12_0_DEVEL_X |
|
-1 Failed Tests: UnitTests The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
You can see more details here: Unit TestsI found errors in the following unit tests: ---> test GeometryDOMCount had ERRORS ---> test testTauEmbeddingProducers had ERRORS Comparison SummarySummary:
|
|
-1 Failed Tests: UnitTests The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
You can see more details here: Unit TestsI found errors in the following unit tests: ---> test TestFWCorePythonFrameworkDriver had ERRORS ---> test GeometryDOMCount had ERRORS ---> test materialBudgetHGCalPlots had ERRORS ---> test materialBudgetTrackerPlots had ERRORS and more ... Comparison SummarySummary:
|
|
I believe these failures are all in the mainstream IB or expected in DEVEL - @smuzaffar ?
… On Jun 15, 2021, at 12:12 AM, cmsbuild ***@***.***> wrote:
-1
Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3ab08a/15939/summary.html
COMMIT: 07e8776
CMSSW: CMSSW_12_0_DEVEL_X_2021-06-13-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7019/15939/install.sh to create a dev area with all the needed externals and cmssw changes.
The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
• #7021 @cms-sw: [BuildRules] use python3 in scripts
• #7025 @cms-sw: updated DetectorDescription-Schema=V02-03-00
• #7024 @belforte: move tested dev to prod
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3ab08a/15939/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3ab08a/15939/git-merge-result
Unit Tests
I found errors in the following unit tests:
---> test TestFWCorePythonFrameworkDriver had ERRORS
---> test GeometryDOMCount had ERRORS
---> test materialBudgetHGCalPlots had ERRORS
---> test materialBudgetTrackerPlots had ERRORS
and more ...
Comparison Summary
Summary:
• No significant changes to the logs found
• Reco comparison results: 5123 differences found in the comparisons
• DQMHistoTests: Total files compared: 38
• DQMHistoTests: Total histograms compared: 2862520
• DQMHistoTests: Total failures: 7
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2862491
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
• Checked 160 log files, 37 edm output root files, 38 DQM output files
• TriggerResults: no differences found
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
that is correct @davidlange6. What about comparison differences, do you think those are due to python3 used by cmsRun? |
|
The 7 dqm histogram failures are all message logger - so “normal”.
I’m puzzled how to have reco comparison differences without dqm histogram comparison differences. So, ok, something we have no DQM monitoring of - slimmedJetsPuppi..
The test done with the mainstream build did not have the comparison differences. How can I find what the baseline was for the reco comparisons for the build starting from the DEVEL branch? Was it not a recent DEVEL IB?
… On Jun 15, 2021, at 8:27 AM, Malik Shahzad Muzaffar ***@***.***> wrote:
that is correct @davidlange6. What about comparison differences, do you think those are due to python3 used by cmsRun?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
you can click on the "Details" link for |
|
Ok, thanks - so that is presumably a python3 difference (feature or bug - to be understood..). I will investigate that. But thats independent of this PR.
… On Jun 15, 2021, at 8:47 AM, Malik Shahzad Muzaffar ***@***.***> wrote:
How can I find what the baseline was for the reco comparisons for the build starting from the DEVEL branch
you can click on the "Details" link for cms/7019/DEVEL/slc7_amd64_gcc900/comparison status which should take you to comparison jenkins job ( in this case https://cmssdt.cern.ch/jenkins/job/compare-root-files-short-matrix/43455/ ) . As we ony generate baseline for 12.0.X so comparison is always done with normal 12.0.X IB.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
the puppi differences are due to different order of things the python config. Eg, 245924c245933
|
|
lets get this in for DEVEL |
|
@davidlange6 , 3 of DEVEL IBs unit tests can be fixed if they use
should we just fix this in CMSSW_DEVEL branch or define soem env e.g CMSRUN_PYTHON= |
|
I think the two in Validation can be changed also in mainstream. The one in FWCore needs to be consistent with the fwk python version- I can make a PR for the two in Validation.
… On Jun 16, 2021, at 9:23 AM, Malik Shahzad Muzaffar ***@***.***> wrote:
@davidlange6 , 3 of DEVEL IBs unit tests can be fixed if they use python3 e.g.
• https://github.com/cms-sw/cmssw/blob/master/FWCore/PythonFramework/test/runPythonTests.sh#L10-L15
• https://github.com/cms-sw/cmssw/blob/master/Validation/Geometry/test/genHGCalPlots.sh#L12
• https://github.com/cms-sw/cmssw/blob/master/Validation/Geometry/test/genTrackerPlots.sh#L11-L16
should we just fix this in CMSSW_DEVEL branch or define soem env e.g CMSRUN_PYTHON=python|python3 and use this in these scripts?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
Sorry, I’m wrong. The two in Validation use the FWCore/PythonFramework so have the same issue that the python version needs to make the python version of the fwk. (Which I don’t think we have an easy way to discover)
… On Jun 16, 2021, at 9:27 AM, David Lange ***@***.***> wrote:
I think the two in Validation can be changed also in mainstream. The one in FWCore needs to be consistent with the fwk python version- I can make a PR for the two in Validation.
> On Jun 16, 2021, at 9:23 AM, Malik Shahzad Muzaffar ***@***.***> wrote:
>
>
> @davidlange6 , 3 of DEVEL IBs unit tests can be fixed if they use python3 e.g.
>
> • https://github.com/cms-sw/cmssw/blob/master/FWCore/PythonFramework/test/runPythonTests.sh#L10-L15
> • https://github.com/cms-sw/cmssw/blob/master/Validation/Geometry/test/genHGCalPlots.sh#L12
> • https://github.com/cms-sw/cmssw/blob/master/Validation/Geometry/test/genTrackerPlots.sh#L11-L16
> should we just fix this in CMSSW_DEVEL branch or define soem env e.g CMSRUN_PYTHON=python|python3 and use this in these scripts?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
you can try but I think the tests in Validation should also use same python as used by FWK |
|
can we do something like the following to find the FWK python? or add a helper script cms-python which just runs the above and return either |
|
Yes, I think so - now is the limited lifetime of this script worthwhile to introduce? (Or maybe there is a longer term use case?)
But, eg, this works in a bash script - we could add that to the three unit tests
```
#!/bin/bash
{
python3 -c "from FWCore.PythonFramework.CmsRun import CmsRun" 2>/dev/null &&echo "using python3"
}||
{
python -c "from FWCore.PythonFramework.CmsRun import CmsRun" 2>/dev/null && echo "using python2"
}
```
… On Jun 16, 2021, at 9:38 AM, Malik Shahzad Muzaffar ***@***.***> wrote:
can we do something like the following to find the FWK python?
ldd `which cmsRun` | grep libpython
or add a helper script cms-python which just runs the above and return either python2 or python3
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
Ido not see any long term use case, so fixing these 3 sciprts should be enough. By the way, you want to use |
|
Ok, lets see if I can fix all three tests
… On Jun 16, 2021, at 10:21 AM, Malik Shahzad Muzaffar ***@***.***> wrote:
Ido not see any long term use case, so fixing these 3 sciprts should be enough. By the way, you want to use python2 instead of python asin PY3 IBs, python is python3 :-)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
(For DEVEL)