Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removed usage of DDVectorGetter from production code #30069

Merged
merged 3 commits into from Jun 5, 2020

Conversation

Dr15Jones
Copy link
Contributor

PR description:

The DDVectorGetter was using a thread-unsafe singleton. The relevant information can be obtained directly from the DDCompactView or Views derived from it. This matches what is being done in the dd4hep migration.

This is needed to support concurrent EventSetup module running (and we've just been lucky it hasn't hurt us up to this point).

PR validation:

The code compiles. I don't know how to test this. The change should be strictly technical with not change in behavior.

The DDVectorGetter was using a thread-unsafe singleton. The relevant
information can be obtained directly from the DDCompactView or
Views derived from it. This matches what is being done in the
dd4hep migration.
@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2020

The code-checks are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2020

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30069/15795

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2020

A new Pull Request was created by @Dr15Jones (Chris Jones) for master.

It involves the following packages:

DetectorDescription/Core
Geometry/HGCalCommonData
Geometry/HcalCommonData
Geometry/MTDGeometryBuilder
Geometry/MTDNumberingBuilder
Geometry/TrackerGeometryBuilder
Geometry/TrackerNumberingBuilder

@civanch, @Dr15Jones, @makortel, @cvuosalo, @ianna, @mdhildreth, @cmsbuild, @kpedro88 can you please review it and eventually sign? Thanks.
@vargasa, @makortel, @VinInn, @ebrondol, @fabiocos, @venturia this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

@Dr15Jones
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2020

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6722/console Started: 2020/06/01 23:14

Comment on lines +67 to +68
const std::vector<double>& DDCompactView::vector(std::string_view iKey) const {
auto itFind = vectors_.find(std::string(iKey));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, why not

Suggested change
const std::vector<double>& DDCompactView::vector(std::string_view iKey) const {
auto itFind = vectors_.find(std::string(iKey));
const std::vector<double>& DDCompactView::vector(std::string const& iKey) const {
auto itFind = vectors_.find(iKey);

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hoping in C++20 I could drop the string copy :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, sounds good.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://en.cppreference.com/w/cpp/container/map/find which shows in C++14 they added the ability to do a find without having to create a key. Not sure why it didn't work here :(.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, yes... I've added the lockdown as well :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after the parsing is done

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the code snippet above, where did you get vnames?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -59,12 +59,15 @@ int main(int argc, char* argv[]) {
 
     DDErrorDetection ed(cpv);
     ed.report(cpv, std::cout);
+    cpv.lockdown();
 
     DDConstant::iterator<DDConstant> cit(DDConstant::begin()), ced(DDConstant::end());
     for (; cit != ced; ++cit) {
       cout << *cit << endl;
     }
 
+    std::vector<std::string> vnames;
+    
     DDVector::iterator<DDVector> vit;
     DDVector::iterator<DDVector> ved(DDVector::end());
     if (vit == ved)
@@ -72,6 +75,7 @@ int main(int argc, char* argv[]) {
     for (; vit != ved; ++vit) {
       if (vit->isDefined().second) {
         std::cout << vit->toString() << std::endl;
+       vnames.emplace_back(DDName(vit->name()).name());
         const std::vector<double>& tv = *vit;
         std::cout << "size: " << tv.size() << std::endl;
         for (double i : tv) {
@@ -80,12 +84,21 @@ int main(int argc, char* argv[]) {
         std::cout << std::endl;
       }
     }
-
-    std::vector<string> vnames;
-    DDVectorGetter::beginWith("Subdetector", vnames);
-    for (std::vector<string>::const_iterator sit = vnames.begin(); sit != vnames.end(); ++sit) {
-      std::cout << sit->c_str() << std::endl;
+    for(const auto& it : vnames) {
+      std::cout << it << std::endl;
+      auto v = cpv.vector(it);
+      std::cout << "size: " << v.size() << std::endl;
+      for (double i : v) {
+       std::cout << i << "\t";
+      }
+      std::cout << std::endl;
     }
+    

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[I should stop writing code before getting a response for stuff I ask others how they did it :)]

@cvuosalo
Copy link
Contributor

cvuosalo commented Jun 1, 2020

@Dr15Jones This PR is worrisome. It is changing core old-DD functionality at the same time as we are migrating to DD4hep and trying to validate against the old DD. If it is bug-free and perfect, that would be fine, but any hidden errors would be a bothersome diversion while we are trying to focus on DD4hep.
What is the schedule for the thread-safety improvements you are making? Could you just use DD4hep instead of the old DD for your thread-safety tests? We are aiming to have the migration completed for Run 3.

@Dr15Jones
Copy link
Contributor Author

The thread-safety changes are finishing up testing now and could be ready to deploy this week. They need to be tested well before Run 3 and are best to add early in the CMSSW_11_2 cycle.

@Dr15Jones
Copy link
Contributor Author

This change should also be bit-wise-identical to what we have now. If not, there is a problem with this code.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2020

+1
Tested at: 0ef36a7
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-949bc0/6722/summary.html
CMSSW: CMSSW_11_2_X_2020-06-01-1100
SCRAM_ARCH: slc7_amd64_gcc820

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-949bc0/6722/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-949bc0/6722/git-merge-result

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2020

Comparison job queued.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 2, 2020

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-949bc0/6722/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 2 differences found in the comparisons
  • DQMHistoTests: Total files compared: 36
  • DQMHistoTests: Total histograms compared: 2783662
  • DQMHistoTests: Total failures: 30
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 2783582
  • DQMHistoTests: Total skipped: 50
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
  • Checked 152 log files, 16 edm output root files, 36 DQM output files

@ianna
Copy link
Contributor

ianna commented Jun 2, 2020

@Dr15Jones - nicely done! I'd suggest to remove the DetectorDescription/​Core/​interface/​DDVectorGetter.h
To test it there is a regression test (that needs to be modified): https://cmssdt.cern.ch/lxr/source/DetectorDescription/RegressionTest/test/const_dump.cpp#0085

for (std::vector<string>::const_iterator sit = vnames.begin(); sit != vnames.end(); ++sit) {
std::cout << sit->c_str() << std::endl;
//This forces the 'vector' to be filled
cpv.lockdown();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ianna I did the lockdown here rather than right after the parsing in the case that we decide to clear the DDVector container in that call since such a clear would break the loop just before this one.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 2, 2020

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30069/15810

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 2, 2020

Pull request #30069 was updated. @civanch, @Dr15Jones, @makortel, @cvuosalo, @ianna, @mdhildreth, @cmsbuild, @kpedro88 can you please check and sign again.

@Dr15Jones
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 2, 2020

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6742/console Started: 2020/06/02 17:55

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 2, 2020

+1
Tested at: ce6ba82
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-949bc0/6742/summary.html
CMSSW: CMSSW_11_2_X_2020-06-02-1100
SCRAM_ARCH: slc7_amd64_gcc820

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 2, 2020

Comparison job queued.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 2, 2020

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-949bc0/6742/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 36
  • DQMHistoTests: Total histograms compared: 2783666
  • DQMHistoTests: Total failures: 1
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 2783615
  • DQMHistoTests: Total skipped: 50
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
  • Checked 152 log files, 16 edm output root files, 36 DQM output files

@cvuosalo
Copy link
Contributor

cvuosalo commented Jun 2, 2020

+1

@kpedro88
Copy link
Contributor

kpedro88 commented Jun 4, 2020

+upgrade

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 4, 2020

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

@silviodonato
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit ae40f9f into cms-sw:master Jun 5, 2020
@Dr15Jones Dr15Jones deleted the removeDDVectorGetter branch June 8, 2020 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants