assure no DS collision LHE relvals

[franzoni]

• needs to go on top of Adding first LHE sequence to RelVal  #6626 (already rebased on top of CMSSW_7_4_X)
• enforces that no LHE workflow can produce the same /GEN dataset
• /GEN recycling command all point to the same /GEN dataset

[cmsbuild]

A new Pull Request was created by @franzoni (Giovanni Franzoni) for CMSSW_7_4_X.

assure no DS collision LHE relvals

It involves the following packages:

Configuration/PyReleaseValidation

@cmsbuild, @srimanob, @nclopezo, @boudoul, @franzoni can you please review it and eventually sign? Thanks.
@ghellwig this is something you requested to watch as well.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.

[franzoni]

+1
tests passed for all generator and extendedgen sets

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_4_X IBs unless changes or unless it breaks tests.

[cmsbuild]

+1
Tested at: e0d9b2e
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6692/1316/summary.html

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_4_X IBs unless changes (tests are also fine). This pull request will be automatically merged.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6692/1316/summary.html

[davidlange6]

Seems that these workflows are failing in the IBs - is that expected?
[and how does this work for someone without cvmfs - does one get a good error message from cmsDriver (cmsfs is not required for CMSSW in general)]

[franzoni]

hello David,

the crashes for the 6 new relvals happen because of the input dataset recycle used by default in the IB: until when we'll be able to run the workflows for the first time, the input won't be found.
I could remove the input command (PR needed), forcing temporarily that the step1 is actually executed.

Concerning cvmfs : if we remove the non existing input dataset, cvmfs will become necessary for the workflow to function. It is my understanding that cvmfs is installed throughout our computing infrastructure, GEN workflows exploit that assumption.

If you know for a fact that cmvf is not available on the IB machines (though are you sure ? Alessandro actually helped me to install it on my machine : https://twiki.cern.ch/twiki/bin/viewauth/CMS/CmsIbCvmfs),
there's no way we can solve these crashes. Otherwise, I can place the suggested PR in time for the 2am IB.

Cheers,
g.

[franzoni]

@bendavid @inugent

[bendavid]

On the general policy: we will try to avoid the cvmfs dependency for most GEN-SIM workflows, but for LHE workflows it is unavoidable, since the use of "alternate pdf weights" implies a large set of pdfs are used by the workflows, and they cannot all be included directly in the release.

[bendavid]

On the error message, this is tricky because it is not possible for cmsDriver or even the ExternalLHEProducer to definitively know a priori whether the particular workflow really requires cvmfs.

[franzoni]

Hi Josh, the error message is simple to digest:

Pythia at the step2 is trying to open this file :

29-Nov-2014 15:13:20 CET Closed file root://eoscms//eos/cms/store/data/Run2012B/MinimumBias/RAW/v1/000/194/533/0C1BB1C7-ACA2-E111-A831-003048D3750A.root?svcClass=default

:-P

[bendavid]

@franzoni, sorry I was referring to David's comment about the error message when cvmfs is missing.

[davidlange6]

Regarding cvmfs- my question is not for the IB infrastructure but for users.

On Nov 29, 2014, at 7:13 PM, Giovanni Franzoni <notifications@github.commailto:notifications@github.com> wrote:

hello David,

the crashes for the 6 new relvals happen because of the input dataset recycle used by default in the IB: until when we'll be able to run the workflows for the first time, the input won't be found.
I could remove the input command (PR needed), forcing temporarily that the step1 is actually executed.

Concerning cvmfs : if we remove the non existing input dataset, cvmfs will become necessary for the workflow to function. It is my understanding that cvmfs is installed throughout our computing infrastructure, GEN workflows exploit that assumption.

If you know for a fact that cmvf is not available on the IB machines (though are you sure ? Alessandro actually helped me to install it on my machine : https://twiki.cern.ch/twiki/bin/viewauth/CMS/CmsIbCvmfs),
there's no way we can solve these crashes. Otherwise, I can place the suggested PR in time for the 2am IB.

Cheers,
g.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/6692#issuecomment-64962064.

[davidlange6]

Hi Josh

Remind me how big these particular files are? (The ones used by relvals which I presume will now be essentially fixed)

On Nov 29, 2014, at 7:17 PM, Josh Bendavid <notifications@github.commailto:notifications@github.com> wrote:

On the general policy: we will try to avoid the cvmfs dependency for most GEN-SIM workflows, but for LHE workflows it is unavoidable, since the use of "alternate pdf weights" implies a large set of pdfs are used by the workflows, and they cannot all be included directly in the release.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/6692#issuecomment-64962172.

[ktf]

Well, also in the IB infrastructure, while CVMFS might be available on some machines, we do not guarantee (e.g. on mac / arm) it is actually there.