[Sonic] Add OpenCV external #5747

mrodozov · 2020-04-15T15:47:44Z

cmsbuild · 2020-04-15T15:48:05Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5705/console Started: 2020/04/15 18:13

cmsbuild · 2020-04-15T15:48:06Z

A new Pull Request was created by @mrodozov (Mircho Rodozov) for branch IB/CMSSW_11_1_X/master.

@cmsbuild, @smuzaffar, @mrodozov, @tulamor can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

cmsbuild · 2020-04-15T20:08:19Z

-1

Tested at: 3511c39

CMSSW: CMSSW_11_1_X_2020-04-14-2300
SCRAM_ARCH: slc7_amd64_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6311d9/5705/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests RelVals AddOn

Unit Tests:

I found errors in the following unit tests:

---> test testPhase2PixelNtuple had ERRORS
---> test materialBudgetTrackerPlots had ERRORS
---> test testAlignmentOfflineValidation had ERRORS
---> test testTauEmbeddingProducers had ERRORS

RelVals:

When I ran the RelVals I found an error in the following workflows:
101.0 step1

runTheMatrix-results/101.0_SingleElectronE120EHCAL+SingleElectronE120EHCAL/step1_SingleElectronE120EHCAL+SingleElectronE120EHCAL.log

250202.181 step2

runTheMatrix-results/250202.181_TTbar_13UP18+TTbar_13UP18+PREMIXUP18_PU25+DIGIPRMXLOCALUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25/step2_TTbar_13UP18+TTbar_13UP18+PREMIXUP18_PU25+DIGIPRMXLOCALUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25.log

AddOn:

I found errors in the following addon tests:

cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_Fake1 --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_25ns --fileout file:RelVal_Raw_Fake1_MC.root : FAILED - time: date Wed Apr 15 21:56:36 2020-date Wed Apr 15 21:49:12 2020 s - exit: 34304
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_2018 --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018 --fileout file:RelVal_Raw_2018_MC.root : FAILED - time: date Wed Apr 15 22:03:13 2020-date Wed Apr 15 21:49:13 2020 s - exit: 34304
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run3_mc_GRun --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run3 --fileout file:RelVal_Raw_GRun_MC.root : FAILED - time: date Wed Apr 15 22:02:48 2020-date Wed Apr 15 21:49:17 2020 s - exit: 34304
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run3_mc_PRef --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run3 --fileout file:RelVal_Raw_PRef_MC.root : FAILED - time: date Wed Apr 15 21:57:37 2020-date Wed Apr 15 21:49:28 2020 s - exit: 34304
cmsDriver.py TTbar_Tauola_8TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run1_mc_Fake --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --fileout file:RelVal_Raw_Fake_MC.root : FAILED - time: date Wed Apr 15 21:56:45 2020-date Wed Apr 15 21:49:38 2020 s - exit: 34304
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run3_mc_HIon --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run3_pp_on_PbPb --fileout file:RelVal_Raw_HIon_MC.root : FAILED - time: date Wed Apr 15 21:58:45 2020-date Wed Apr 15 21:49:43 2020 s - exit: 34304
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_Fake2 --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2016 --fileout file:RelVal_Raw_Fake2_MC.root : FAILED - time: date Wed Apr 15 21:57:21 2020-date Wed Apr 15 21:49:45 2020 s - exit: 34304
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run3_mc_PIon --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run3 --fileout file:RelVal_Raw_PIon_MC.root : FAILED - time: date Wed Apr 15 21:59:52 2020-date Wed Apr 15 21:54:37 2020 s - exit: 34304

cmsbuild · 2020-04-15T20:08:21Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

cmsbuild · 2020-04-15T20:15:03Z

@mrodozov , without adding toolfile and dependency in cmssw-tool-conf , cmsBuild is not going to build this. Please provide a toolfile for this and add opencv-toolfile in cmssw-tool-conf.

Did you try building it on aarch64 and ppc64le?

mrodozov · 2020-04-15T20:27:33Z

hm, right. there is no existing reference. I'll add the toolfile
no I didn' build it on arm and ppc yet but it's simple enough and I'm not expecting any trouble there.

davidlange6 · 2020-04-16T14:09:18Z

how does this work with python3?

mrodozov · 2020-04-16T14:43:58Z

let me post the cmake config for short:

-- Found PythonInterp: /build/mrodozov/Sonic_externals/build_grpc/slc7_amd64_gcc820/external/python/2.7.15-bcolbf/bin/python2.7 (found suitable version "2.7.15", minimum required is "2.7")           
-- Found PythonLibs: /build/mrodozov/Sonic_externals/build_grpc/slc7_amd64_gcc820/external/python/2.7.15-bcolbf/lib/libpython2.7.so (found suitable exact version "2.7.15+")                           
Traceback (most recent call last):                                                                                                                                                                     
  File "<string>", line 1, in <module>                                                                                                                                                                 
ImportError: No module named numpy.distutils                                                                                                                                                           
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.8", minimum required is "3.2")                                                                                                    
-- Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS) (Required is exact version "3.6.8")                                                                                       
Traceback (most recent call last):                                                                                                                                                                     
  File "<string>", line 1, in <module>                                                                                                                                                                 
ModuleNotFoundError: No module named 'numpy'

there are cmake vars that can be added to the config to make it find both py2 and 3 , but because the config from @kpedro88 aimed to build opencv libs I didn't add them.
If the concern is 'how do we make this work only with py3' I actually don't know.
It only needed to find py2 to build it so I didn't check

davidlange6 · 2020-04-16T14:54:07Z

seems more cmake options are needed to pick up the right python environment.

…eps for

mrodozov · 2020-04-21T16:33:32Z

please test
I added all libs I found in the lib dir to the toolfile.
works for both py2 and 3 with import cv2 to use it

cmsbuild · 2020-04-21T16:43:00Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5806/console Started: 2020/04/21 19:14

cmsbuild · 2020-04-21T16:43:01Z

Pull request #5747 was updated.

smuzaffar · 2020-05-22T14:56:10Z

+externals
we will merge it after cmssw 11.1.0.pre8

cmsbuild · 2020-05-22T14:56:33Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild · 2020-05-22T18:43:18Z

Pull request #5747 was updated.

smuzaffar · 2020-05-22T18:43:29Z

please test

cmsbuild · 2020-05-22T18:43:36Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6523/console Started: 2020/05/22 20:47

cmsbuild · 2020-05-22T21:17:10Z

+1
Tested at: 6191986
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6311d9/6523/summary.html
CMSSW: CMSSW_11_1_X_2020-05-22-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-05-22T21:17:12Z

Comparison job queued.

cmsbuild · 2020-05-22T22:49:24Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6311d9/6523/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2780792
DQMHistoTests: Total failures: 16
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2780726
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
Checked 152 log files, 16 edm output root files, 36 DQM output files

[Sonic] Add OpenCV external

3511c39

cmsbuild added comparison-pending externals-pending orp-pending pending-signatures tests-pending labels Apr 15, 2020

cmsbuild added tests-started and removed tests-pending labels Apr 15, 2020

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Apr 15, 2020

Build everything possible opencv is trying to build and we have the d…

d03cdf3

…eps for

cmsbuild added comparison-pending tests-pending and removed comparison-notrun tests-rejected labels Apr 21, 2020

cmsbuild removed the tests-pending label Apr 21, 2020

cmsbuild added the comparison-available label May 20, 2020

cmsbuild added externals-approved and removed externals-pending pending-signatures labels May 22, 2020

cmsbuild added the fully-signed label May 22, 2020

Merge branch 'IB/CMSSW_11_1_X/master' into add_opencv_external

6191986

cmsbuild added comparison-pending externals-pending pending-signatures tests-pending and removed comparison-available externals-approved fully-signed tests-approved labels May 22, 2020

cmsbuild added tests-started and removed tests-pending labels May 22, 2020

smuzaffar merged commit 37d5106 into cms-sw:IB/CMSSW_11_1_X/master May 22, 2020

cmsbuild added tests-approved and removed tests-started labels May 22, 2020

cmsbuild added comparison-available and removed comparison-pending labels May 22, 2020

kpedro88 mentioned this pull request May 26, 2020

Triton inference server #5868

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Sonic] Add OpenCV external #5747

[Sonic] Add OpenCV external #5747

mrodozov commented Apr 15, 2020 •

edited

cmsbuild commented Apr 15, 2020 •

edited

cmsbuild commented Apr 15, 2020

cmsbuild commented Apr 15, 2020

cmsbuild commented Apr 15, 2020

cmsbuild commented Apr 15, 2020

mrodozov commented Apr 15, 2020

davidlange6 commented Apr 16, 2020

mrodozov commented Apr 16, 2020

davidlange6 commented Apr 16, 2020

mrodozov commented Apr 21, 2020 •

edited

cmsbuild commented Apr 21, 2020 •

edited

cmsbuild commented Apr 21, 2020

smuzaffar commented May 22, 2020

cmsbuild commented May 22, 2020

cmsbuild commented May 22, 2020

smuzaffar commented May 22, 2020

cmsbuild commented May 22, 2020 •

edited

cmsbuild commented May 22, 2020

cmsbuild commented May 22, 2020

cmsbuild commented May 22, 2020

[Sonic] Add OpenCV external #5747

[Sonic] Add OpenCV external #5747

Conversation

mrodozov commented Apr 15, 2020 • edited

cmsbuild commented Apr 15, 2020 • edited

cmsbuild commented Apr 15, 2020

cmsbuild commented Apr 15, 2020

cmsbuild commented Apr 15, 2020

cmsbuild commented Apr 15, 2020

mrodozov commented Apr 15, 2020

davidlange6 commented Apr 16, 2020

mrodozov commented Apr 16, 2020

davidlange6 commented Apr 16, 2020

mrodozov commented Apr 21, 2020 • edited

cmsbuild commented Apr 21, 2020 • edited

cmsbuild commented Apr 21, 2020

smuzaffar commented May 22, 2020

cmsbuild commented May 22, 2020

cmsbuild commented May 22, 2020

smuzaffar commented May 22, 2020

cmsbuild commented May 22, 2020 • edited

cmsbuild commented May 22, 2020

cmsbuild commented May 22, 2020

cmsbuild commented May 22, 2020

mrodozov commented Apr 15, 2020 •

edited

cmsbuild commented Apr 15, 2020 •

edited

mrodozov commented Apr 21, 2020 •

edited

cmsbuild commented Apr 21, 2020 •

edited

cmsbuild commented May 22, 2020 •

edited