Add mxnet python bindings #5667

mrodozov · 2020-03-18T22:32:25Z

please test
both python 2 and 3

cmsbuild · 2020-03-18T22:32:45Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5264/console Started: 2020/03/18 23:36

cmsbuild · 2020-03-18T22:32:46Z

A new Pull Request was created by @mrodozov (Mircho Rodozov) for branch IB/CMSSW_11_1_X/master.

@cmsbuild, @smuzaffar, @mrodozov, @tulamor can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

cmsbuild · 2020-03-19T00:14:34Z

+1
Tested at: a6732ef
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-952791/5264/summary.html
CMSSW: CMSSW_11_1_X_2020-03-18-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-03-19T00:14:37Z

Comparison job queued.

cmsbuild · 2020-03-19T01:40:14Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-952791/5264/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2692437
DQMHistoTests: Total failures: 35
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2692083
DQMHistoTests: Total skipped: 319
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

smuzaffar · 2020-03-19T12:40:10Z

mxnet-predict.spec

+## INITENV +PATH PYTHON27PATH %{i}/lib64/python`echo $PYTHON_VERSION | cut -d. -f 1,2`/site-packages
+## INITENV +PATH PYTHON3PATH %{i}/lib64/python`echo $PYTHON3_VERSION | cut -d. -f 1,2`/site-packages
+## INITENV +PATH LD_LIBRARY_PATH %{i}/lib64
+
 %define tag 337cf1b54cc02bde94f459c89863a18187b0aada
 %define branch 1.5.0-cms-mod
 %define github_user cms-externals
 Source: git+https://github.com/%{github_user}/incubator-mxnet.git?obj=%{branch}/%{tag}&export=%{n}-%{realversion}&submodules=1&output=/%{n}-%{realversion}-%{tag}.tgz

 BuildRequires: cmake ninja ccache


@mrodozov , please drop this dependency and use our gcc compiler

smuzaffar · 2020-03-19T12:40:55Z

mxnet-predict.spec

@@ -17,6 +21,8 @@ rm -rf ../build; mkdir ../build; cd ../build
 # use LAPACK functions in OpenBLAS:
 # manually set MXNET_USE_LAPACK=1 and turn off USE_LAPACK in cmake
 export CFLAGS="-DMXNET_USE_LAPACK=1 -DMXNET_THREAD_LOCAL_ENGINE=1"
+export PYTHONV=$(echo $PYTHON_VERSION | cut -f1,2 -d.)
+export PYTHON3V=$(echo $PYTHON3_VERSION | cut -f1,2 -d.)

 cmake ../%{n}-%{realversion} -GNinja \
    -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache \


@mrodozov , we also need to clean this up, we should use our compiler

ok let me see about that

I'm trying to understand what is ccache doing as a "compiler launcher". we don't use cuda with mxnet so I'll guess I have to change it to gcc (I did it and it's building), but also because we don't use cuda do we need this flag ?

just drop ccache dependency and remove the 3 LAUNCHER lines from cmake command and you should be fine.

smuzaffar · 2020-03-19T13:21:40Z

@mrodozov , can you please also add a unit test in cmssw to text mxnet ( https://mxnet.apache.org/get_started/validate_mxnet#python ) ?

mrodozov · 2020-03-19T13:57:01Z

sure. I'll get the test as it is

cmsbuild · 2020-03-19T16:14:29Z

Pull request #5667 was updated.

smuzaffar · 2020-03-19T16:17:46Z

@mrodozov , please add mxnet-predict in to python_tools.spec otherwise PYTHON*PATH will not be properly set in cmssw env.

mrodozov · 2020-03-19T16:43:13Z

please test

smuzaffar · 2020-03-20T12:08:45Z

mxnet-predict.spec

@@ -1,12 +1,16 @@
 ### RPM external mxnet-predict 1.5.0
+## INITENV +PATH PYTHON27PATH %{i}/lib64/python`echo $PYTHON_VERSION | cut -d. -f 1,2`/site-packages
+## INITENV +PATH PYTHON3PATH %{i}/lib64/python`echo $PYTHON3_VERSION | cut -d. -f 1,2`/site-packages


@mrodozov , please use PYTHON3_LIB_SITE_PACKAGES here

smuzaffar · 2020-03-20T12:08:58Z

mxnet-predict.spec

@@ -1,12 +1,16 @@
 ### RPM external mxnet-predict 1.5.0
+## INITENV +PATH PYTHON27PATH %{i}/lib64/python`echo $PYTHON_VERSION | cut -d. -f 1,2`/site-packages


use PYTHON_LIB_SITE_PACKAGES

smuzaffar · 2020-03-20T12:09:45Z

mxnet-predict.spec


 cmake ../%{n}-%{realversion} -GNinja \
-    -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache \
-    -DCMAKE_C_COMPILER_LAUNCHER=ccache \
-    -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \


pass -DCMAKE_INSTALL_LIBDIR=lib to get lib instead of lib64. fix -toolfile to pointto lib

smuzaffar · 2020-03-20T12:10:11Z

mxnet-predict.spec

-rm %{i}/lib64/*.a
-
+rm %{i}/lib64/*.a %{i}/*.so
+mv %{i}/python* %{i}/lib64


once cmake is updated to use lib, then fix this one too

cmsbuild · 2020-03-20T13:16:03Z

Pull request #5667 was updated.

mrodozov · 2020-03-20T13:16:46Z

please test

cmsbuild · 2020-03-20T13:17:05Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5290/console Started: 2020/03/20 14:26

cmsbuild · 2020-03-20T15:23:38Z

+1
Tested at: 23f7bde
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-952791/5290/summary.html
CMSSW: CMSSW_11_1_X_2020-03-19-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-03-20T15:23:41Z

Comparison job queued.

cmsbuild · 2020-03-20T16:52:22Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-952791/5290/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2692493
DQMHistoTests: Total failures: 60
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2692114
DQMHistoTests: Total skipped: 319
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

smuzaffar · 2020-03-20T16:55:03Z

+externals

cmsbuild · 2020-03-20T16:55:25Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

Add mxnet python bindings

a6732ef

cmsbuild added comparison-pending externals-pending orp-pending pending-signatures tests-pending labels Mar 18, 2020

cmsbuild added tests-started and removed tests-pending labels Mar 18, 2020

cmsbuild added tests-approved and removed tests-started labels Mar 19, 2020

cmsbuild added comparison-available and removed comparison-pending labels Mar 19, 2020

smuzaffar reviewed Mar 19, 2020

View reviewed changes

mrodozov mentioned this pull request Mar 19, 2020

Add test for mxnet python interface cms-sw/cmssw#29243

Merged

Remove ccache dep and cmake flags using it

917ba95

cmsbuild added comparison-pending tests-pending and removed comparison-available tests-approved labels Mar 19, 2020

smuzaffar reviewed Mar 20, 2020

View reviewed changes

Add cmake flag for lib install dir and change lib64 to lib

23f7bde

cmsbuild added comparison-pending tests-pending and removed comparison-available tests-approved labels Mar 20, 2020

cmsbuild added tests-started and removed tests-pending labels Mar 20, 2020

cmsbuild added tests-approved and removed tests-started labels Mar 20, 2020

cmsbuild added comparison-available and removed comparison-pending labels Mar 20, 2020

smuzaffar merged commit 4be22b5 into IB/CMSSW_11_1_X/master Mar 20, 2020

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Mar 20, 2020

smuzaffar deleted the mxnet-python-pkgs branch March 21, 2020 21:40

smuzaffar mentioned this pull request Mar 21, 2020

Build python packages for MXNet and ONNXRuntime #5637

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add mxnet python bindings #5667

Add mxnet python bindings #5667

mrodozov commented Mar 18, 2020

cmsbuild commented Mar 18, 2020 •

edited

cmsbuild commented Mar 18, 2020

cmsbuild commented Mar 19, 2020

cmsbuild commented Mar 19, 2020

cmsbuild commented Mar 19, 2020

smuzaffar Mar 19, 2020

smuzaffar Mar 19, 2020

mrodozov Mar 19, 2020

mrodozov Mar 19, 2020

smuzaffar Mar 19, 2020

smuzaffar commented Mar 19, 2020

mrodozov commented Mar 19, 2020

cmsbuild commented Mar 19, 2020

smuzaffar commented Mar 19, 2020

mrodozov commented Mar 19, 2020

smuzaffar Mar 20, 2020

smuzaffar Mar 20, 2020

smuzaffar Mar 20, 2020

smuzaffar Mar 20, 2020

cmsbuild commented Mar 20, 2020

mrodozov commented Mar 20, 2020

cmsbuild commented Mar 20, 2020 •

edited

cmsbuild commented Mar 20, 2020

cmsbuild commented Mar 20, 2020

cmsbuild commented Mar 20, 2020

smuzaffar commented Mar 20, 2020

cmsbuild commented Mar 20, 2020

		@@ -1,12 +1,16 @@
		### RPM external mxnet-predict 1.5.0
		## INITENV +PATH PYTHON27PATH %{i}/lib64/python`echo $PYTHON_VERSION \| cut -d. -f 1,2`/site-packages

Add mxnet python bindings #5667

Add mxnet python bindings #5667

Conversation

mrodozov commented Mar 18, 2020

cmsbuild commented Mar 18, 2020 • edited

cmsbuild commented Mar 18, 2020

cmsbuild commented Mar 19, 2020

cmsbuild commented Mar 19, 2020

cmsbuild commented Mar 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smuzaffar commented Mar 19, 2020

mrodozov commented Mar 19, 2020

cmsbuild commented Mar 19, 2020

smuzaffar commented Mar 19, 2020

mrodozov commented Mar 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmsbuild commented Mar 20, 2020

mrodozov commented Mar 20, 2020

cmsbuild commented Mar 20, 2020 • edited

cmsbuild commented Mar 20, 2020

cmsbuild commented Mar 20, 2020

cmsbuild commented Mar 20, 2020

smuzaffar commented Mar 20, 2020

cmsbuild commented Mar 20, 2020

cmsbuild commented Mar 18, 2020 •

edited

cmsbuild commented Mar 20, 2020 •

edited