Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace boost_python parameterset reader with a python3 version #26377

Merged
merged 5 commits into from Apr 8, 2019

Conversation

davidlange6
Copy link
Contributor

PR description:

  • Remove boost_python version of cmsRun
  • rename pybind11+python2 parameterset interface to FWCore/PythonParameterSet
  • copy FWCore/PythonParameterSet to FWCore/PyDevParameterSet but with classes in different namespace for a pybind11+python3 parameterset interface

Despite the big changes, there is NO change to cmsRun itself aside from some package renaming. And, the big chunk of new code is a direct copy of old code - this duplication (which I don't see how to avoid, but interested in any ideas) will go away when we drop support for python2 in cmsRun

PR validation:

unit tests and basic runTheMatrix ok.

if this PR is a backport please specify the original PR:

@davidlange6
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 6, 2019

The code-checks are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 6, 2019

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26377/9121

  • This PR adds an extra 56KB to repository

  • Found files with invalid states:

    • FWCore/PythonParameterSet/BuildFile.xml:
    • FWCore/PythonParameterSet/test/makeprocess_t.cppunit.cc:
    • FWCore/PythonParameterSet/test/makepset_t.cppunit.cc:
    • FWCore/PythonParameterSet/test/BuildFile.xml:
    • FWCore/PythonParameterSet/python/test/testTask_cff.py:
    • FWCore/PythonParameterSet/test/ufip.txt:
    • FWCore/PythonParameterSet/test/fip.txt:
    • FWCore/PythonParameterSet/test/processbuilder_t.cppunit.cpp:
    • FWCore/PythonParameterSet/test/readpsetsfrom_t.cppunit.cc:

Code check has found code style and quality issues which could be resolved by applying a patch in https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26377/9121/git-diff.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26377/9121/git-diff.patch | patch -p1

You can run scram build code-checks to apply code checks directly

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 6, 2019

The code-checks are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 6, 2019

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26377/9122

  • This PR adds an extra 56KB to repository

  • Found files with invalid states:

    • FWCore/PythonParameterSet/BuildFile.xml:
    • FWCore/PythonParameterSet/test/makeprocess_t.cppunit.cc:
    • FWCore/PythonParameterSet/test/makepset_t.cppunit.cc:
    • FWCore/PythonParameterSet/test/BuildFile.xml:
    • FWCore/PythonParameterSet/python/test/testTask_cff.py:
    • FWCore/PythonParameterSet/test/ufip.txt:
    • FWCore/PythonParameterSet/test/fip.txt:
    • FWCore/PythonParameterSet/test/processbuilder_t.cppunit.cpp:
    • FWCore/PythonParameterSet/test/readpsetsfrom_t.cppunit.cc:

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 6, 2019

A new Pull Request was created by @davidlange6 (David Lange) for master.

It involves the following packages:

FWCore/Framework
FWCore/Integration
FWCore/ParameterSetReader
FWCore/PyDevParameterSet
FWCore/PythonFramework
FWCore/PythonParameterSet
Fireworks/FWInterface

@cmsbuild, @smuzaffar, @Dr15Jones, @alja can you please review it and eventually sign? Thanks.
@makortel, @alja, @wddgit, @Martin-Grunewald this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

@davidlange6
Copy link
Contributor Author

davidlange6 commented Apr 6, 2019 via email

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 6, 2019

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34037/console Started: 2019/04/06 20:58

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 6, 2019

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 6, 2019

Comparison job queued.

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 7, 2019

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26377/9128

  • This PR adds an extra 60KB to repository

  • Found files with invalid states:

    • FWCore/PythonParameterSet/BuildFile.xml:
    • FWCore/PythonParameterSet/test/makeprocess_t.cppunit.cc:
    • FWCore/PythonParameterSet/test/makepset_t.cppunit.cc:
    • FWCore/PythonParameterSet/test/BuildFile.xml:
    • FWCore/PythonParameterSet/python/test/testTask_cff.py:
    • FWCore/PythonParameterSet/test/ufip.txt:
    • FWCore/PythonParameterSet/test/fip.txt:
    • FWCore/PythonParameterSet/test/processbuilder_t.cppunit.cpp:
    • FWCore/PythonParameterSet/test/readpsetsfrom_t.cppunit.cc:

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 7, 2019

Pull request #26377 was updated. @cmsbuild, @smuzaffar, @Dr15Jones, @alja can you please check and sign again.

@@ -29,7 +29,7 @@ int main(int argc, char **argv) try {
config = std::string(argv[1]);
}

std::shared_ptr<edm::ParameterSet> parameterSet = edm::boost_python::readConfig(config);
std::shared_ptr<edm::ParameterSet> parameterSet = edm::readConfig(config);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, pybind11 is already inside ParameterSetReader, so no need to call it explicitly...

@fabiocos
Copy link
Contributor

fabiocos commented Apr 7, 2019

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 7, 2019

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34040/console Started: 2019/04/07 18:53

@davidlange6
Copy link
Contributor Author

davidlange6 commented Apr 7, 2019 via email

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 7, 2019

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 7, 2019

Comparison job queued.

@fabiocos
Copy link
Contributor

fabiocos commented Apr 7, 2019

@Dr15Jones the latest addition reintroduce the two utilities, the rest is unchanged. Do you confirm the signature?

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 7, 2019

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26377/34040/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 32
  • DQMHistoTests: Total histograms compared: 3140495
  • DQMHistoTests: Total failures: 1
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3140297
  • DQMHistoTests: Total skipped: 197
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
  • Checked 133 log files, 14 edm output root files, 32 DQM output files

@Dr15Jones
Copy link
Contributor

+1

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 7, 2019

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

@fabiocos
Copy link
Contributor

fabiocos commented Apr 8, 2019

+1

@cmsbuild cmsbuild merged commit 93d33b6 into cms-sw:master Apr 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants