Force pickle protocol=0 for the spec creation to be compatible with Py2/Py3 #10628

amaltaro · 2021-06-28T10:01:23Z

Superseeds #10619 (please see whole discussion/investigation in this PR)
Fixes #10609

Status

Ready

Description

As discussed with Dario this morning, trying a variation of the current implementation in #10619
If it passes the unit tests, then we can run some real tests within WMCore.

NOTE: this patch is required for the Python2 1.4.7.patchX agents:

cmst1@vocms0262:/data/srv/wmagent/current $ diff -rup /data/srv/wmagent/v1.4.7.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.7.patch3/lib/python2.7/site-packages/WMCore/BossAir/Plugins/SimpleCondorPlugin.py{.bkp,}
--- /data/srv/wmagent/v1.4.7.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.7.patch3/lib/python2.7/site-packages/WMCore/BossAir/Plugins/SimpleCondorPlugin.py.bkp	2021-06-24 19:30:23.860759021 +0200
+++ /data/srv/wmagent/v1.4.7.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.7.patch3/lib/python2.7/site-packages/WMCore/BossAir/Plugins/SimpleCondorPlugin.py	2021-06-24 19:31:50.394372378 +0200
@@ -605,7 +605,7 @@ class SimpleCondorPlugin(BasePlugin):
             requiredOSes = self.scramArchtoRequiredOS(job.get('scramArch'))
             ad['My.REQUIRED_OS'] = classad.quote(requiredOSes)
             cmsswVersions = ','.join(job.get('swVersion'))
-            ad['My.CMSSW_Versions'] = classad.quote(cmsswVersions)
+            ad['My.CMSSW_Versions'] = classad.quote(str(cmsswVersions))

Is it backward compatible (if not, which system it affects?)

yes, hopefully!

Related PRs

none

External dependencies / deployment changes

none

cmsdmwmbot · 2021-06-28T10:14:59Z

Jenkins results:

Python2 Unit tests: failed
- 45 new failures
- 8 changes in unstable tests
Python3 Unit tests: failed
- 43 new failures
- 13 changes in unstable tests
Python2 Pylint check: failed
- 9 warnings and errors that must be fixed
- 1 warnings
- 6 comments to review
Python3 Pylint check: failed
- 9 warnings and errors that must be fixed
- 1 warnings
- 24 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 2 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12038/artifact/artifacts/PullRequestReport.html

src/python/WMCore/WMSpec/Persistency.py

…y2/Py3 pickle load auto detects the protocol

cmsdmwmbot · 2021-06-28T11:04:59Z

Jenkins results:

Python2 Unit tests: succeeded
Python3 Unit tests: succeeded
- 4 changes in unstable tests
Python2 Pylint check: failed
- 9 warnings and errors that must be fixed
- 1 warnings
- 6 comments to review
Python3 Pylint check: failed
- 9 warnings and errors that must be fixed
- 1 warnings
- 24 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 2 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12040/artifact/artifacts/PullRequestReport.html

amaltaro · 2021-06-28T15:49:48Z

I still have to check the outcome of my tests (might have to create another fix or two before I can get to that though), but a pre-review would be welcome ;-)

mapellidario

Well yes, this is what we agreed on and based on our current knowledge it is the best that we can do.

The results of you tests will greatly determine the fate of these changes, though 😏 Will see, but i am not too worried either, it should not take long to cherry pick changes from #10619 in case your tests fail because of pickle

amaltaro · 2021-06-28T21:49:50Z

Quick update on the tests I have been running for Python3 central services with these changes in.

PY3 central services are able to properly create the request spec pickle file
PY2 and PY3 WMAgent are able to acquire such workflows and load the spec pickle file (created with protocol 0)
PY2 SimpleCondorPlugin needs to be patch (see initial description)
PY3 WMAgent - based on 1.5.0.pre3 - required many fixes in order to get jobs submitted.
PY2 WMAgent had 100% failure rate. Failure happens during runtime with error [py2_wma]
PY3 WMAgent had 100% failure rate. Still to be debugged, but runtime error can be seen in [py3_wma]

Note that the Py2 WMAgent is still running 1.4.7.patch3, so it does not have the python decoupling developments.
A quick check around that CondorStatusService suggest that self.step.data._internal_name is actually unicode, which breaks the validation in CMSSW code itself. Debugging...

[py2_wma]

INFO:root:    Invoking command: /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_0/external/slc6_amd64_gcc530/bin/python2.7 -m WMCore.WMRuntime.ScriptInvoke WMTaskSpace.cmsRun1 SetupCMSSWPset 

INFO:root:Subprocess stdout was:
2021-06-28 23:14:52,519:INFO:ScriptInvoke:Invoking scripts in current directory: /afs/cern.ch/user/a/amaltaro/psetTweak2/job/WMTaskSpace/cmsRun1/CMSSW_8_1_0
2021-06-28 23:14:52,576:INFO:Bootstrap:Job Index = 10
Job Instance = {'ownerRole': u'unknown', 'agentName': 'vocms0261.cern.ch', 'jobgroup': 1, 'taskType': 'integration', 'owner': u'amaltaro', 'fwjr': None, 'id': 10, 'jobType': 'Production', 'estimatedJobTime': 28800, 'estimatedDiskUsage': 1000000, 'state_time': 1624886965, 'state': 'new', 'location': None, 'allowOpportunistic': False, 'agentNumber': 0, 'spec': '/data/srv/wmagent/v1.4.7.patch3/install/wmagent/WorkQueueManager/cache/amaltaro_TaskChain_ProdMinBias_June2021_Val_210628_150829_5725/WMSandbox/WMWorkload.pkl', 'workflow': 'amaltaro_TaskChain_ProdMinBias_June2021_Val_210628_150829_5725', 'ownerGroup': u'unknown', 'cache_dir': u'/data/srv/wmagent/v1.4.7.patch3/install/wmagent/JobCreator/JobCache/amaltaro_TaskChain_ProdMinBias_June2021_Val_210628_150829_5725/ProdMinBias/JobCollection_1_0/job_10', 'estimatedMemoryUsage': 1200.0, 'fwjr_path': None, 'input_files': [{'runs': set([]), 'last_event': 0, 'lfn': 'MCFakeFile-amaltaro_TaskChain_ProdMinBias_June2021_Val_210628_150829_5725-ProdMinBias-1fc1a33e4582105549b2d043a1939366', 'locations': set([]), 'first_event': 0, 'parents': set([]), 'checksums': {}, 'events': 9000, 'merged': True, 'size': 0}], 'task': u'/amaltaro_TaskChain_ProdMinBias_June2021_Val_210628_150829_5725/ProdMinBias', 'name': u'86fcac5d-2781-4c4f-99cd-879cd7c410fe-9', 'counter': 10, 'mask': {'LastRun': 1, 'LastLumi': 10, 'FirstRun': 1, 'inclusivemask': True, 'runAndLumis': {}, 'LastEvent': 410, 'FirstEvent': 370, 'jobID': 10, 'FirstLumi': 10}, 'numberOfCores': 1, 'ownerDN': u'/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amaltaro/CN=718748/CN=Alan Malta Rodrigues', 'retry_count': 0, 'sandbox': '/data/srv/wmagent/v1.4.7.patch3/install/wmagent/WorkQueueManager/cache/amaltaro_TaskChain_ProdMinBias_June2021_Val_210628_150829_5725/amaltaro_TaskChain_ProdMinBias_June2021_Val_210628_150829_5725-Sandbox.tar.bz2', 'outcome': 'failure'}

2021-06-28 23:14:52,724:INFO:SetupCMSSWPset:Executing SetupCMSSWPSet...
2021-06-28 23:14:55,127:INFO:SetupCMSSWPset:Tag chirp updates from CMSSW with step cmsRun1
Traceback (most recent call last):
  File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/python/2.7.11-giojec4/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/python/2.7.11-giojec4/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/afs/cern.ch/user/a/amaltaro/psetTweak2/job/WMCore.zip/WMCore/WMRuntime/ScriptInvoke.py", line 118, in <module>
RuntimeError: Error invoking script for step WMTaskSpace.cmsRun1
withe Script module: SetupCMSSWPset
_cmsRun1_ is not a valid <class 'FWCore.ParameterSet.Types.string'>Details:
  File "/afs/cern.ch/user/a/amaltaro/psetTweak2/job/WMCore.zip/WMCore/WMRuntime/ScriptInvoke.py", line 110, in <module>
  File "/afs/cern.ch/user/a/amaltaro/psetTweak2/job/WMCore.zip/WMCore/WMRuntime/ScriptInvoke.py", line 81, in invoke
  File "/build/dmwmbld/srv/state/dmwmbld/builds/comp_gcc630/w/tmp/BUILDROOT/a4c7a6fe65310a171692a1c6cb882348/build/dmwmbld/srv/state/dmwmbld/builds/comp_gcc630/w/slc7_amd64_gcc630/cms/wmagent/1.4.7.patch3/lib/python2.7/site-packages/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py", line 746, in __call__
    self.handleCondorStatusService()
  File "/build/dmwmbld/srv/state/dmwmbld/builds/comp_gcc630/w/tmp/BUILDROOT/a4c7a6fe65310a171692a1c6cb882348/build/dmwmbld/srv/state/dmwmbld/builds/comp_gcc630/w/slc7_amd64_gcc630/cms/wmagent/1.4.7.patch3/lib/python2.7/site-packages/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py", line 641, in handleCondorStatusService
    tag=cms.untracked.string("_%s_" % self.step.data._internal_name)))
  File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_0/python/FWCore/ParameterSet/Types.py", line 23, in __call__
    param = globals()[self.name](*value,**params)
  File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_0/python/FWCore/ParameterSet/Types.py", line 145, in __init__
    super(string,self).__init__(value)
  File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_0/python/FWCore/ParameterSet/Mixins.py", line 56, in __init__
    raise ValueError(str(value)+" is not a valid "+str(type(self)))

[py3_wma]

    JobPackageError (Exit Code: 11001)
        Failed to load JobPackage:/srv/job/WMSandbox/JobPackage.pcl
        unsupported pickle protocol: 5

amaltaro · 2021-06-29T09:27:09Z

We have some progress here. After patching SetupCMSSWPSet.py, I managed to get jobs running and successfully completing in our "old" python WMAgent 1.4.7.patch3. I'm going to create a patch soon.

Regarding the python3 agent, unfortunately I cannot test the whole chain, because jobs fail during runtime (there are other CVMFS dependencies that we need to sort first). However, the most critical part has been tested and this protocol seems to be fully compatible.

IMO, we should give it another try in a clean environment. So I'm going to merge and put this in a new tag.

Here is a new GH issue to upgrade the pickle protocol used: #10643
And here is the required patch for 1.4.7 agents: #10644

Again, thank you very much for your investigation and the discussion on this matter, Dario!

mapellidario · 2021-06-30T17:25:45Z

@amaltaro I have missed something, did you managed to fix the problems described in [py2_wma] and [py3_wma]? do we need to open new issues?

But more importantly, why is wmagent in py3 complaining that it cannot load protocol 5? Aren't we running py 3.8.2? It should be available there 😱

amaltaro · 2021-07-26T06:48:54Z

Hi @mapellidario , sorry for the very late reply.

@amaltaro I have missed something, did you managed to fix the problems described in [py2_wma] and [py3_wma]? do we need to open new issues?

Yes, those two issues have been resolved, at least in Py2 WMagent.

But more importantly, why is wmagent in py3 complaining that it cannot load protocol 5? Aren't we running py 3.8.2? It should be available there 😱

I mentioned this issue in our last chat - before I left on vacation - but I could not explain the reason back then. The issue is that WMAgent creates the job pickle file using the latest protocol (5), but in the worker node runtime, we are still sourcing a python2 environment and its latest protocol is 2. This is why things are not "totally" compatible, besides having our WMCore code dual stack. To be addressed with this issue: #10641

mapellidario · 2021-07-26T08:25:59Z

Aha, now I get it. Thanks Alan!

amaltaro added PR: Do not merge yet PR: Work in progress labels Jun 28, 2021

mapellidario reviewed Jun 28, 2021

View reviewed changes

src/python/WMCore/WMSpec/Persistency.py Outdated Show resolved Hide resolved

Force pickle protocol=0 for the spec creation to be compatible with P…

3acaed1

…y2/Py3 pickle load auto detects the protocol

amaltaro force-pushed the fix-10609 branch from f257224 to 3acaed1 Compare June 28, 2021 10:51

amaltaro requested a review from mapellidario June 28, 2021 15:49

mapellidario approved these changes Jun 28, 2021

View reviewed changes

amaltaro added Testing and removed PR: Work in progress Testing labels Jun 28, 2021

amaltaro removed the PR: Do not merge yet label Jun 29, 2021

amaltaro merged commit 8a66117 into dmwm:master Jun 29, 2021

belforte mentioned this pull request Jul 2, 2021

pickle CMSSW spec with protocolo=0 (and encode ?) dmwm/CRABClient#5054

Closed

mapellidario mentioned this pull request Aug 3, 2021

Dump pickle with py2-comatible protocol - WMCore-wide #10619

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Force pickle protocol=0 for the spec creation to be compatible with Py2/Py3 #10628

Force pickle protocol=0 for the spec creation to be compatible with Py2/Py3 #10628

amaltaro commented Jun 28, 2021 •

edited

Loading

cmsdmwmbot commented Jun 28, 2021

cmsdmwmbot commented Jun 28, 2021

amaltaro commented Jun 28, 2021

mapellidario left a comment

amaltaro commented Jun 28, 2021

amaltaro commented Jun 29, 2021 •

edited

Loading

mapellidario commented Jun 30, 2021

amaltaro commented Jul 26, 2021

mapellidario commented Jul 26, 2021

Force pickle protocol=0 for the spec creation to be compatible with Py2/Py3 #10628

Force pickle protocol=0 for the spec creation to be compatible with Py2/Py3 #10628

Conversation

amaltaro commented Jun 28, 2021 • edited Loading

Status

Description

Is it backward compatible (if not, which system it affects?)

Related PRs

External dependencies / deployment changes

cmsdmwmbot commented Jun 28, 2021

cmsdmwmbot commented Jun 28, 2021

amaltaro commented Jun 28, 2021

mapellidario left a comment

Choose a reason for hiding this comment

amaltaro commented Jun 28, 2021

amaltaro commented Jun 29, 2021 • edited Loading

mapellidario commented Jun 30, 2021

amaltaro commented Jul 26, 2021

mapellidario commented Jul 26, 2021

amaltaro commented Jun 28, 2021 •

edited

Loading

amaltaro commented Jun 29, 2021 •

edited

Loading