Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix propagation of an earlier exception in Transformer #40879

Merged
merged 2 commits into from Mar 16, 2023

Conversation

makortel
Copy link
Contributor

PR description:

This PR proposes a possible fix for #40868.

PR validation:

FWCore/Integration unit tests pass

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40879/34349

  • This PR adds an extra 40KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @makortel (Matti Kortelainen) for master.

It involves the following packages:

  • FWCore/Framework (core)
  • FWCore/Integration (core)

@cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please review it and eventually sign? Thanks.
@missirol, @wddgit this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

@makortel
Copy link
Contributor Author

@cmsbuild, please test

@makortel
Copy link
Contributor Author

@Dr15Jones What do you think?

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ebc98f/30893/summary.html
COMMIT: 3f397f0
CMSSW: CMSSW_13_1_X_2023-02-24-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40879/30893/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially removed 6 lines from the logs
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 49
  • DQMHistoTests: Total histograms compared: 3528955
  • DQMHistoTests: Total failures: 1
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3528932
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
  • Checked 213 log files, 164 edm output root files, 49 DQM output files
  • TriggerResults: no differences found

@@ -27,6 +28,8 @@
process.maxEvents.input = 4

process.start = cms.EDProducer("IntProducer", ivalue = cms.int32(1))
if args.exception:
process.start = cms.EDProducer("FailingProducer")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the test failure is in what is consumed and not in the actual transform call its.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, basically from anywhere in the data dependence chain of the module registering the transformation call.

// products were actually produced and which not, because the
// transforming products are never produced by time of commit_()
// by construction.
if (edp) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the putProduct call always get a null wrapper? Or does the later call also call putProduct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The putProduct() call from Event::commit_()

for (auto index : iShouldPut) {
auto resolver = p.getProductResolverByIndex(index);
if (not resolver->productResolved()) {
dynamic_cast<ProductPutterBase const*>(resolver)->putProduct(std::unique_ptr<WrapperBase>());
}
}

will always have a null wrapper. The Transformer itself calls putProduct() via EventPrincipal::put()
dynamic_cast<ProductPutterBase const*>(phb)->putProduct(std::move(edp));

void EventForTransformer::put(ProductResolverIndex index,
std::unique_ptr<WrapperBase> edp,
BasicHandle const& iGetHandle) {
eventPrincipal_.put(index, std::move(edp), iGetHandle.provenance()->productProvenance()->parentageID());
}

Written that, I see now an alternative to customizing putProduct() that would be to customize (in some way) productResolved() to return true for Transformer. The productResolved() seems to be called only in {Event,LuminosityBlock,Run,ProcessBlock}::commit_(), but on a first thought "lying" in productResolved() sounds conceptually worse than leaving the status unaltered in putProduct().

@makortel
Copy link
Contributor Author

I'm coming to the conclusion that this is the least bad (quick) way, and given the "plan" to revisit the transformation ability in a context of more general memory space handling, and revisiting the Event system internals to make them more flexible, I would not spend much more time with this fix now.

@makortel makortel force-pushed the fixTransformerLostException branch from 3f397f0 to b70e365 Compare March 13, 2023 17:14
@makortel
Copy link
Contributor Author

Rebased to fix a conflict.

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40879/34596

  • This PR adds an extra 20KB to repository

@cmsbuild
Copy link
Contributor

Pull request #40879 was updated. @cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please check and sign again.

@makortel
Copy link
Contributor Author

@cmsbuild, please test

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ebc98f/31250/summary.html
COMMIT: b70e365
CMSSW: CMSSW_13_1_X_2023-03-13-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40879/31250/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially removed 12 lines from the logs
  • Reco comparison results: 11 differences found in the comparisons
  • DQMHistoTests: Total files compared: 49
  • DQMHistoTests: Total histograms compared: 3550756
  • DQMHistoTests: Total failures: 10
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3550724
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
  • Checked 213 log files, 164 edm output root files, 49 DQM output files
  • TriggerResults: no differences found

@makortel
Copy link
Contributor Author

Another incarnation of #39803

@makortel
Copy link
Contributor Author

+core

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

@perrotta
Copy link
Contributor

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants