Add option to dump offloaded tracks as an event file

[sethrj]

This adds a shared "offload writer" that writes one "event" per flush call, with all threads sharing a single output file (so that the separate events can be parallelized when reloading the file).

[sethrj]

@whokion Can you check out this branch, set options.offload_output_file = "cms-offload.hepmc3"; (and you can disable the SDs to speed it up a bit) and see if you can generate offload files from 100 ttbar events with CMSSW for the HLLHC and Run 3 configurations?

[whokion]

No problem. Just for my curiosity, is the cms-offload.hepmc3 different from celeritas-output.json with
options_->output_file = "celeritas-output.json"
or root output by cmsRun, for example, cms-hllhc-step2.root with
fileName = cms.untracked.string('file:cms-hllhc-step2.root')
inside cms.OutputModule("PoolOutputModule", ...)?

[sethrj]

• The JSON output file is for Celeritas internals (kernel diagnostics, etc.)
• The ROOT output is for processed Geant4 physics data
• The "offload" HepMC3 output file is for the tracks that Geant4 sends to celeritas

[whokion]

cmsRun with this branch runs successfully for both Run3 and HL-LHC configurations. Please see the size of file output with 100 ttbar events:
-rw-r--r-- 1 syjun g4gpu 3262865394 Jul 13 12:10 cms-offload-hllhc.hepmc3
-rw-r--r-- 1 syjun g4gpu 1136394364 Jul 13 11:22 cms-offload-run3.hepmc3

[whokion]

Temporarily, can down those output files by
wget https://g4cpt.fnal.gov/g4p/download/celeritas/cms-offload-run3.hepmc3
wget https://g4cpt.fnal.gov/g4p/download/celeritas/cms-offload-hllhc.hepmc3

[sethrj]

@whokion Thanks for those files; I've downloaded them so you can delete if you want to clean up space.

[amandalund]

Thanks @sethrj!

One minor observation: right now all the events in the output have event ID 0. This won't matter for us since we set our own sequential event IDs in the HepMC3 reader, but we might still want to add a evt.set_event_number(event_count_) in the EventWriter.

[amandalund]

Suggested change

	std::shared_ptr<detail::OffloadWriter> offload_writer_;
	SPOffloadWriter offload_writer_;

[sethrj]

Good point, we definitely want to do that! I didn't notice that property.

[sethrj]

I also realized that HepMC3PrimaryGenerator is giving all vertices the "event" position so I'll update and test that as well.

[sethrj]

I'm going to actually move the hepmc3/io fix to a separate PR since that's a problem that exists on the v0.3 release.