If job requires rhel8 container, use slc7 python libraries #11077

amaltaro · 2022-04-09T01:31:22Z

Fixes #11051
Complement to #11060 and #11062

Status

ready

Description

We do not have a build of COMP in EL8, so for jobs requesting rhel8, use our slc7_* python libraries.

We might consider using python and future library from one of the EL8 CMSSW releases as well, still unsure whether that would work out of the box though.

UPDATE: these changes triggered a discussion with the SI team, and they actually updated the container to be used for REQUIRED_OS=any from rhel6 to rhel7, on April 11, 2022. We need to check whether current production agents will behave properly.

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

Complement to #11060 and #11062

External dependencies / deployment changes

None

amaltaro · 2022-04-09T01:38:36Z

Testing it with vocms0192 and amaltaro_TC_EL8_Agent202_Val_220409_033647_4276.
FYI @smuzaffar

cmsdmwmbot · 2022-04-09T01:44:01Z

Jenkins results:

Python3 Unit tests: succeeded
- 7 tests added
- 2 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12995/artifact/artifacts/PullRequestReport.html

amaltaro · 2022-04-09T12:19:24Z

This doesn't work. Stage out fails with this error:

stderr: xrdcp: /cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/gcc/6.3.0/lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.5/3.5.60-2/el8-x86_64/usr/lib64/libXrdCl.so.3)
xrdfs: /cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/gcc/6.3.0/lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by xrdfs)
xrdfs: /cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/gcc/6.3.0/lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.5/3.5.60-2/el8-x86_64/usr/lib64/libXrdCl.so.3)
xrdfs: /cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/gcc/6.3.0/lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by xrdfs)
xrdfs: /cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/gcc/6.3.0/lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.5/3.5.60-2/el8-x86_64/usr/lib64/libXrdCl.so.3)
xrdfs: /cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/gcc/6.3.0/lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by xrdfs)
xrdfs: /cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/gcc/6.3.0/lib64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.5/3.5.60-2/el8-x86_64/usr/lib64/libXrdCl.so.3)

I guess we will have to do the same trick as we did for ppc64le, create a symlink to COMP python3 and py3-future from one of the EL8 CMSSW releases.

smuzaffar · 2022-04-09T12:27:06Z

yes, we need COMP python for el8 cmssw release. I can create the symlinks, let me know what paths WMCore would be expacting under /cvmfs/cms.cern.ch/COMP?

smuzaffar · 2022-04-09T12:40:33Z

by the way, starting with rhel8 can we simplify the logic here

WMCore/etc/submit_py3.sh

Lines 141 to 149 in 844ccaf

    
           if [ "$REQUIRED_OS" = "rhel7" ] || [ "$REQUIRED_OS" = "rhel8" ] 
        
           then 
        
               WMA_SCRAM_ARCH=slc7_${THIS_ARCH}_gcc630 
        
           elif [ "$THIS_ARCH" = "amd64" ] 
        
           then 
        
               WMA_SCRAM_ARCH=slc6_${THIS_ARCH}_gcc700 
        
           else 
        
               WMA_SCRAM_ARCH=slc6_${THIS_ARCH}_gcc493 
        
           fi

e.g.

if [ "$REQUIRED_OS" = "rhel7" ]
then
    WMA_SCRAM_ARCH=slc7_${THIS_ARCH}_gcc630
elif [ "$REQUIRED_OS" = "rhel6" ]
then
    if [ "$THIS_ARCH" = "amd64" ]
    then
        WMA_SCRAM_ARCH=slc6_${THIS_ARCH}_gcc700
    else
        WMA_SCRAM_ARCH=slc6_${THIS_ARCH}_gcc493
    fi
else
    WMA_SCRAM_ARCH=${REQUIRED_OS}_${THIS_ARCH}
fi

and we create rhelX_(amd64|aarch64|ppc64le) symlinks for rhel8 and above. In near future we have to test rhel9 too :-)

amaltaro · 2022-04-09T20:52:49Z

@smuzaffar your suggestion would be the ideal and most maintainable approach in my opinion. However, I don't feel comfortable with blindly using CMSSW-based builds to run the production job wrapper, reason is because there was no validation at all to running the WMCore runtime code with those libraries (python, future, and other packages that I see in cvmfs - gcc, zlib, bz2lib, etc)

amaltaro · 2022-04-09T21:33:48Z

Shahzad, I guess we can start with a path like:

/cvmfs/cms.cern.ch/COMP/rhel8_amd64_gcc630/

and maybe the same for the other 2 architectures (ppc64le and aarch64).

We need only python3 and py3-future packages.

smuzaffar · 2022-04-10T06:50:17Z

@smuzaffar your suggestion would be the ideal and most maintainable approach in my opinion. However, I don't feel comfortable with blindly using CMSSW-based builds to run the production job wrapper, reason is because there was no validation at all to running the WMCore runtime code with those libraries (python, future, and other packages that I see in cvmfs - gcc, zlib, bz2lib, etc)

I just wanted to have the code to support future archs. /cvmfs/cms.cern.ch/COMP/${REQUIRED_OS}_$(uname -m) could be symlink pointing to actual /cvmfs/cms.cern.ch/COMP/<os>_<arch>_gcc<version> . Whenever comp is ready to build/provide its own python3/future packages for these archs then we can just install those under /cvmfs/cms.cern.ch/COMP/<comp_os>_<arch>_<comp_gcc> and update the symlink to point to it. If comp does not have the arch support yet then we can create the symlink to point to cms based python3/future ( just like we do for ppc64le and now for rhel8).

For example, for existing WMA_SCRAM_ARCH , we could already create

rhel6_x86_64 -> slc6_amd64_gcc700
rhel6_ppc64le -> slc6_ppc64le_gcc493 (by the way, we do not have slc6_ppc64le releases, so no idea why we created this, may be it was during the ppc64le initial tests by Tommaso and submit.sh did not have the proper support for it. I think this can be deleted now)
rhel7_(x86_64|ppc64le) -> slc7_(amd64|ppc64le)_gcc630

and for new archs e.g. rhel8, rhel9 and aarch64

${REQUIRED_OS}_$(uname -m) - > pointing to either cms based python3/future archs OR comp based python3/future architectures

Once these symlinks are available then all you need is to use

WMA_SCRAM_ARCH="${REQUIRED_OS}_$(uname -m)"

smuzaffar · 2022-04-10T07:00:25Z

by the way, we are also in process of setting up jenkins jobs to test the system via crab jobs. This will allow us to automatically test new archs (e.g. rhel9) and find issues during our regular Integration Build process.

smuzaffar · 2022-04-11T09:35:53Z

@amaltaro , new archs are now available under /cvmfs/cms.cern.ch/COMP. Now paths are

cc8_aarch64_gcc9/external/py3-future/0.18.2
cc8_aarch64_gcc9/external/python3/3.8.2 -> /cvmfs/cms.cern.ch/cc8_aarch64_gcc9/external/python3/3.8.2
cc8_amd64_gcc9/external/py3-future/0.18.2
cc8_amd64_gcc9/external/python3/3.8.2 -> /cvmfs/cms.cern.ch/cc8_amd64_gcc9/external/python3/3.8.2-555576dae8d03b340ea3079ddb8abb0d
cc8_ppc64le_gcc9/external/py3-future/0.18.2
cc8_ppc64le_gcc9/external/python3/3.8.2 -> /cvmfs/cms.cern.ch/cc8_ppc64le_gcc9/external/python3/3.8.2
cs9_amd64_gcc11/external/py3-future/0.18.2
cs9_amd64_gcc11/external/python3/3.9.6 -> /cvmfs/cms.cern.ch/cs9_amd64_gcc11/external/python3/3.9.6-2c93591d796451daeaf67dc8130bfa35
slc7_aarch64_gcc820/external/py3-future/0.18.2 -> ../py2-future/0.18.2
slc7_aarch64_gcc820/external/python3/3.8.2 -> /cvmfs/cms.cern.ch/slc7_aarch64_gcc820/external/python3/3.8.2
slc7_ppc64le_gcc630/external/py3-future/0.18.2
slc7_ppc64le_gcc630/external/python3/3.8.2 -> /cvmfs/cms.cern.ch/slc7_ppc64le_gcc820/external/python3/3.8.2
slc7_ppc64le_gcc820/external/py3-future/0.18.2
slc7_ppc64le_gcc820/external/python3/3.8.2 -> /cvmfs/cms.cern.ch/slc7_ppc64le_gcc820/external/python3/3.8.2

I also have created short symlink, so if you prefer please use these in submit script

rhel5_x86_64 -> slc5_amd64_gcc434
rhel6_x86_64 -> slc6_amd64_gcc700
rhel7_aarch64 -> slc7_aarch64_gcc820
rhel7_ppc64le -> slc7_ppc64le_gcc820
rhel7_x86_64 -> slc7_amd64_gcc630
rhel8_aarch64 -> cc8_aarch64_gcc9
rhel8_ppc64le -> cc8_ppc64le_gcc9
rhel8_x86_64 -> cc8_amd64_gcc9
rhel9_x86_64 -> cs9_amd64_gcc11

amaltaro · 2022-04-11T13:05:16Z

@smuzaffar thank you very much, Shahzad!

I also have created short symlink, so if you prefer please use these in submit script

I am giving this a try in my last commit that I just pushed in into this branch. Let's see how it goes with grid tests.

smuzaffar · 2022-04-11T13:06:38Z

etc/submit_py3.sh

 then
-    WMA_SCRAM_ARCH=slc6_${THIS_ARCH}_gcc700
+    WMA_SCRAM_ARCH=slc7_${THIS_ARCH}


@amaltaro , may be rhel7 instead of slc7 here?

Good catch! Thanks!

cmsdmwmbot · 2022-04-11T13:12:32Z

Jenkins results:

Python3 Unit tests: succeeded
- 7 tests added
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/12999/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-04-11T13:22:28Z

Jenkins results:

Python3 Unit tests: succeeded
- 7 tests added
- 1 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13000/artifact/artifacts/PullRequestReport.html

amaltaro · 2022-04-11T15:16:46Z

Tests are still running, but I can already see a few 100s jobs all successfully completing.
@smuzaffar I couldn't explicitly ask for your review here, but feel free to leave any other concerns/questions/thumbs up that you might have.

smuzaffar · 2022-04-11T15:26:29Z

looks good to me.

vkuznet

I made one comment how to improve the it is up to you to decide if you want to make it. From the point of view of required changes everything is fine and I'll approve it.

vkuznet · 2022-04-11T15:26:39Z

etc/submit_py3.sh

 then
-    WMA_SCRAM_ARCH=slc6_${THIS_ARCH}_gcc700
+    WMA_SCRAM_ARCH=rhel7_${THIS_ARCH}


I would rather define separate default flavor variable, e.g. DEFAULT_OS=rhl7 somewhere on the top of the script and then use it here. This makes it easy to maintain and avoid changing defaults somewhere down in a code. I usually prefer to have common definitions on top of the script and then have a line saying DO NOT EDIT below this line to make clear separation between definitions and logic itself.

That's a good suggestion, thanks Valentin.

todor-ivanov

looks good to me

amaltaro · 2022-04-11T19:13:04Z

With these changes in, now the Cleanup jobs fail because I just noticed they do not define REQUIRED_OS=any, but to an empty string. To be followed up in: #11081

Thanks for the reviews everyone!
@smuzaffar these changes to the simplified schema of rhelX_arch seems to be working fine, which is great!

Trying the new map of OS_ARCH replace slc7 by rhel7 apply Valentins suggestion

cmsdmwmbot · 2022-04-11T20:03:33Z

Jenkins results:

Python3 Unit tests: succeeded
- 7 tests added
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13006/artifact/artifacts/PullRequestReport.html

amaltaro · 2022-04-12T10:53:39Z

Having the agent patched with #11083 does the tricky and there are successful Cleanup jobs now. I think it's safe to get this merged and packaged in a new release, however, I am trying to understand some random errors for:

merge jobs at T2_CH_CERN_HLT (only for the EL8 workflow, and it might be not related only to merge)
cleanup jobs at T1_US_FNAL, still using rhel6 container.

Discussion is ongoing over the SI mattermost channel.

amaltaro · 2022-04-12T21:11:41Z

other than the two transient issues mentioned above, this looks good to go and will be backported to the agent branch

amaltaro added the PR: Further tests needed label Apr 9, 2022

smuzaffar reviewed Apr 11, 2022

View reviewed changes

amaltaro force-pushed the fix-11051-take3 branch from 689facc to bdea13d Compare April 11, 2022 13:13

amaltaro requested review from todor-ivanov and vkuznet April 11, 2022 15:15

amaltaro added the PR: squashing needed label Apr 11, 2022

smuzaffar approved these changes Apr 11, 2022

View reviewed changes

vkuznet approved these changes Apr 11, 2022

View reviewed changes

todor-ivanov approved these changes Apr 11, 2022

View reviewed changes

amaltaro mentioned this pull request Apr 11, 2022

Cleanup jobs requesting an empty OS at the job classad #11081

Closed

If job requires rhel8 container, use slc7 python libraries

765a720

Trying the new map of OS_ARCH replace slc7 by rhel7 apply Valentins suggestion

amaltaro force-pushed the fix-11051-take3 branch from bdea13d to 765a720 Compare April 11, 2022 19:50

amaltaro mentioned this pull request Apr 11, 2022

When ScramArch is empty str/list/None, return any as required_os #11083

Merged

amaltaro added PR: Need wmagent branch and removed PR: squashing needed PR: Further tests needed labels Apr 11, 2022

amaltaro mentioned this pull request Apr 12, 2022

Support to EL8 workflows/jobs - wmagent branch #11071

Closed

amaltaro merged commit 9bad6a8 into dmwm:master Apr 12, 2022

amaltaro mentioned this pull request Apr 12, 2022

Bundle of patches for WMAgent 2.0.2.patch1 #11090

Merged

amaltaro mentioned this pull request May 2, 2022

Support ARM64 architecture at the job description and runtime #11063

Closed

mapellidario mentioned this pull request Jul 21, 2022

CRAB jobwrapper should be python3-compatible dmwm/CRABServer#7210

Closed

2 tasks

todor-ivanov mentioned this pull request Feb 12, 2024

CleanupJobs fail due to xrdfs shared library version mismatch. #11893

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

If job requires rhel8 container, use slc7 python libraries #11077

If job requires rhel8 container, use slc7 python libraries #11077

amaltaro commented Apr 9, 2022 •

edited

Loading

amaltaro commented Apr 9, 2022

cmsdmwmbot commented Apr 9, 2022

amaltaro commented Apr 9, 2022

smuzaffar commented Apr 9, 2022

smuzaffar commented Apr 9, 2022

amaltaro commented Apr 9, 2022

amaltaro commented Apr 9, 2022

smuzaffar commented Apr 10, 2022 •

edited

Loading

smuzaffar commented Apr 10, 2022 •

edited

Loading

smuzaffar commented Apr 11, 2022

amaltaro commented Apr 11, 2022

smuzaffar Apr 11, 2022 •

edited

Loading

amaltaro Apr 11, 2022

cmsdmwmbot commented Apr 11, 2022

cmsdmwmbot commented Apr 11, 2022

amaltaro commented Apr 11, 2022

smuzaffar commented Apr 11, 2022

vkuznet left a comment

vkuznet Apr 11, 2022

amaltaro Apr 11, 2022

todor-ivanov left a comment

amaltaro commented Apr 11, 2022

cmsdmwmbot commented Apr 11, 2022

amaltaro commented Apr 12, 2022

amaltaro commented Apr 12, 2022

If job requires rhel8 container, use slc7 python libraries #11077

If job requires rhel8 container, use slc7 python libraries #11077

Conversation

amaltaro commented Apr 9, 2022 • edited Loading

Status

Description

Is it backward compatible (if not, which system it affects?)

Related PRs

External dependencies / deployment changes

amaltaro commented Apr 9, 2022

cmsdmwmbot commented Apr 9, 2022

amaltaro commented Apr 9, 2022

smuzaffar commented Apr 9, 2022

smuzaffar commented Apr 9, 2022

amaltaro commented Apr 9, 2022

amaltaro commented Apr 9, 2022

smuzaffar commented Apr 10, 2022 • edited Loading

smuzaffar commented Apr 10, 2022 • edited Loading

smuzaffar commented Apr 11, 2022

amaltaro commented Apr 11, 2022

smuzaffar Apr 11, 2022 • edited Loading

Choose a reason for hiding this comment

amaltaro Apr 11, 2022

Choose a reason for hiding this comment

cmsdmwmbot commented Apr 11, 2022

cmsdmwmbot commented Apr 11, 2022

amaltaro commented Apr 11, 2022

smuzaffar commented Apr 11, 2022

vkuznet left a comment

Choose a reason for hiding this comment

vkuznet Apr 11, 2022

Choose a reason for hiding this comment

amaltaro Apr 11, 2022

Choose a reason for hiding this comment

todor-ivanov left a comment

Choose a reason for hiding this comment

amaltaro commented Apr 11, 2022

cmsdmwmbot commented Apr 11, 2022

amaltaro commented Apr 12, 2022

amaltaro commented Apr 12, 2022

amaltaro commented Apr 9, 2022 •

edited

Loading

smuzaffar commented Apr 10, 2022 •

edited

Loading

smuzaffar commented Apr 10, 2022 •

edited

Loading

smuzaffar Apr 11, 2022 •

edited

Loading