more python tool updates

[davidlange6]

(I still have local problems with pip dependencies that are unrelated to things I've changed.. so this may not work

[davidlange6]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5166/console Started: 2020/03/13 09:11

[cmsbuild]

A new Pull Request was created by @davidlange6 (David Lange) for branch IB/CMSSW_11_1_X/master.

@cmsbuild, @smuzaffar, @mrodozov, @tulamor can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

[cmsbuild]

+1
Tested at: 1452ed3
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dfdf56/5166/summary.html
CMSSW: CMSSW_11_1_X_2020-03-12-2300
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dfdf56/5166/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 2680593
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2680272
• DQMHistoTests: Total skipped: 319
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
• Checked 147 log files, 16 edm output root files, 34 DQM output files

[smuzaffar]

+externals

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[smuzaffar]

@davidlange6 , how can I reproduce the issue you get locally?

[davidlange6]

My attempt at reproducing in a simple way didn’t work, so can only say what I do and its errors.... For me setenv ARCH slc7_amd64_gcc820 setenv PKGTOOLS_TAG V00-32-XX setenv CMSDIST_TAG dl200313<https://github.com/davidlange6/cmsdist/tree/dl200313> git clone -b $CMSDIST_TAG git@github.com<mailto:git@github.com>:davidlange6/cmsdist.git CMSDIST git clone -b $PKGTOOLS_TAG git@github.com<mailto:git@github.com>:cms-sw/pkgtools.git PKGTOOLS PKGTOOLS/cmsBuild --repo cms.week1 -a $ARCH -j 6 --builders 5 build python_tools Fails typically with errors like Successfully installed hyperas-0.4.1 ++ pip3 show hyperas --disable-pip-version-check ++ sed 's|^Name: *||;s| ||g' ++ grep '^Name:' + PKG_NAME=hyperas ++ grep '^hyperas *0.4.1 *requires ' ++ pip3 check --disable-pip-version-check ++ tr '\n' ' ' ++ sed 's|,.*||;s|.* |py2-|' + DEPS='py2-hyperopt ' + '[' 'py2-hyperopt ' '!=' '' ']' + echo 'ERROR: Missing dependencies for py2-hyperas (python3) found: py2-hyperopt ' ERROR: Missing dependencies for py2-hyperas (python3) found: py2-hyperopt + exit 1 error: Bad exit status from /build/dlange/200312t2/tmp/rpm-tmp.5ZVoq4 (%build) I can instead do PKGTOOLS/cmsBuild --repo cms.week1 -a $ARCH -j 6 --builders 5 build py2-hyperas But even after I do that, then PKGTOOLS/cmsBuild --repo cms.week1 -a $ARCH -j 6 --builders 5 build python_tools Still fails in similar ways.. On Mar 13, 2020, at 4:11 PM, Malik Shahzad Muzaffar <notifications@github.com<mailto:notifications@github.com>> wrote: @davidlange6<https://github.com/davidlange6> , how can I reproduce the issue you get locally? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#5639 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGPFQZLEXLJ554ELNEL4ULRHJELZANCNFSM4LG4AOKA>.

[smuzaffar]

do you have PYTHONPATH set? Sometime it happens if our pip sees packages outside our build area.

[davidlange6]

I don’t… I forgot to add that I was able to reproduce that pip was not seeing hyperopt in one of my areas in a shell - even if it was installed and in my python3path On Mar 13, 2020, at 4:39 PM, Malik Shahzad Muzaffar <notifications@github.com<mailto:notifications@github.com>> wrote: do you have PYTHONPATH set? Sometime it happens if our pip sees packages outside our build area. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#5639 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGPFQ4N2SMEDZ4NCDLHCVDRHJHTLANCNFSM4LG4AOKA>.

[smuzaffar]

something wronmg with hyperopt as it installed docs version 0.2.3 instead of hyperopt==0.2.3

/build/muz/p3/w/tmp/BUILDROOT/f7c939747d57a31eced47fbccaa5716d/opt/cmssw/slc7_amd64_gcc820/external/py2-hyperopt/0.2.3-cms/lib/python2.7/site-packages/hyperopt/base.py to base.pyc
    byte-compiling /build/muz/p3/w/tmp/BUILDROOT/f7c939747d57a31eced47fbccaa5716d/opt/cmssw/slc7_amd64_gcc820/external/py2-hyperopt/0.2.3-cms/lib/python2.7/site-packages/hyperopt/algobase.py to algobase.pyc
    running install_egg_info
    Copying docs.egg-info to /build/muz/p3/w/tmp/BUILDROOT/f7c939747d57a31eced47fbccaa5716d/opt/cmssw/slc7_amd64_gcc820/external/py2-hyperopt/0.2.3-cms/lib/python2.7/site-packages/docs-0.2.3-py2.7.egg-info
    running install_scripts
    Installing hyperopt-mongo-worker script to /build/muz/p3/w/tmp/BUILDROOT/f7c939747d57a31eced47fbccaa5716d/opt/cmssw/slc7_amd64_gcc820/external/py2-hyperopt/0.2.3-cms/bin
    writing list of installed files to '/build/muz/p3/w/BUILD/slc7_amd64_gcc820/external/py2-hyperopt/0.2.3-cms/cmsdist-tmp/pip-jG30hW-record/install-record.txt'
    Running setup.py install for docs: finished with status 'done'
  Removing source in /build/muz/p3/w/BUILD/slc7_amd64_gcc820/external/py2-hyperopt/0.2.3-cms/cmsdist-tmp/pip-ZfzMFF-build
Successfully installed docs-0.2.3

[davidlange6]

thanks @smuzaffar - any idea why this seems not to affect the bot?

[smuzaffar]

no idea :-(

[davidlange6]

So picking up this fix apparently moves the error to py2-qtpy... On Mar 14, 2020, at 2:28 PM, Malik Shahzad Muzaffar <notifications@github.com<mailto:notifications@github.com>> wrote: no idea :-( — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#5639 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGPFQ6MR27RV3FPS3XNKVLRHOBAHANCNFSM4LG4AOKA>.

[smuzaffar]

what is py2-qtpy?

[davidlange6]

Sorry - I meant py2-qtpy. On Mar 15, 2020, at 10:35 AM, Malik Shahzad Muzaffar <notifications@github.com<mailto:notifications@github.com>> wrote: what is py2-qypy? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#5639 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGPFQ2P76YZHEKA2MJZEATRHSOONANCNFSM4LG4AOKA>.

[smuzaffar]

We have py2-qtconsole or pyqt. I do not see any issue with any of these . if you are talking about pyqt then it does not use build-with-pip and only depend on python, sip and qt so fix should not break it.

[davidlange6]

Successfully installed qtconsole-4.7.1 ++ pip3 show qtconsole --disable-pip-version-check ++ grep '^Name:' ++ sed 's|^Name: *||;s| ||g' + PKG_NAME=qtconsole ++ pip3 check --disable-pip-version-check ++ grep '^qtconsole *4.7.1 *requires ' ++ sed 's|,.*||;s|.* |py2-|' ++ tr '\n' ' ' + DEPS='py2-qtpy ' + '[' 'py2-qtpy ' '!=' '' ']' + echo 'ERROR: Missing dependencies for py2-qtconsole (python3) found: py2-qtpy ' Let me see if it reproduces or not.. (in my area) On Mar 15, 2020, at 12:04 PM, Malik Shahzad Muzaffar <notifications@github.com<mailto:notifications@github.com>> wrote: We have py2-qtconsole or pyqt. I do not see any issue with any of these . if you are talking about pyqt then it does not use build-with-pip and only depend on python, sip and qt so fix should not break it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#5639 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGPFQ3CHFUBPKUBMILK33TRHSY5LANCNFSM4LG4AOKA>.

[smuzaffar]

thisis telling the qtconsole now depends on qtpy (which is not in our distribution). As we are going to drop qt , I would suggest that we install jupyterlab instead of jupyter (which requires qtconsole).

[davidlange6]

ha - so its real this time..
ok -for the moment, I'll just not update qtconsole as a first step.