GEN crash at lumi-boundary change in 10_3 HIN campaign

[srimanob]

Not sure if it's related to #25708

In the new HIN campaign (with CMSSW_10_3), there is the failure in production, e.g.
https://cms-unified.web.cern.ch/cms-unified/report/cmsunified_task_HIN-HINPbPbAutumn18pLHE-00005__v1_T_200429_213517_9970

I've checked, and see that the crash on
Module: Pythia8HadronizerFilter:generator (crashed)
happens at the LS boundary.

How to reproduce the issue: One can use
https://cms-pdmv.cern.ch/mcm/public/restapi/requests/get_test/PPD-HINPbPbAutumn18GS-00005
i.e.

(1)
checkout CMSSW_10_3_2 and grid environment

(2) Checkout GEN fragment
curl -s --insecure https://cms-pdmv.cern.ch/mcm/public/restapi/requests/get_fragment/PPD-HINPbPbAutumn18GS-00005 --retry 2 --create-dirs -o Configuration/GenProduction/python/PPD-HINPbPbAutumn18GS-00005-fragment.py

(3) Here is the cmsDriver, skipping 90 events.
cmsDriver.py Configuration/GenProduction/python/PPD-HINPbPbAutumn18GS-00005-fragment.py --filein "dbs:/Ups1SMMCoh_5p02TeV_STARlight_PPDTest/HINPbPbAutumn18pLHE-pilot_103X_upgrade2018_realistic_HI_v11-v1/LHE" --fileout file:PPD-HINPbPbAutumn18GS-00005.root --mc --eventcontent RAWSIM --no_exec --datatier GEN-SIM --conditions 103X_upgrade2018_realistic_HI_v11 --beamspot RealisticPbPbCollision2018 --step GEN,SIM --nThreads 1 --scenario HeavyIons --geometry DB:Extended --era Run2_2018_pp_on_AA --python_filename PPD-HINPbPbAutumn18GS-00005_1_cfg.py --no_exec --customise Configuration/DataProcessing/Utils.addMonitoring -n 500 --customise_commands "process.source.skipEvents=cms.untracked.uint32(90)"

Full error report can be found at
/afs/cern.ch/user/s/srimanob/public/ForHIN/run.log

How this request passed McM validation:
When pLHE+GS are validated together, McM submits jobs to condor without setting no. of events per LS. Example:
https://cms-pdmv.cern.ch/mcm/public/restapi/chained_requests/get_test/PPD-chain_HINPbPbAutumn18pLHE_flowHINPbPbAutumn18GS_flowHINPbPbAutumn18DRNoPU-00002

[cmsbuild]

A new Issue was created by @srimanob Phat Srimanobhas.

@Dr15Jones, @silviodonato, @dpiparo, @smuzaffar, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

assign generator

[srimanob]

Thanks @Dr15Jones
Should be generators (with s)?

[Dr15Jones]

assign generators

[cmsbuild]

New categories assigned: generators

@alberto-sanchez,@SiewYan,@qliphy,@GurpreetSinghChahal,@mkirsano,@agrohsje you have been requested to review this Pull request/Issue and eventually sign? Thanks

[srimanob]

@alberto-sanchez
Maybe we just need #25850 to 10_3 as a workaround.

[srimanob]

Backport makes the GEN-SIM step works as expected. Here is the PR:
#30073

Thanks.