Building MxNet with CPP_PACKAGE on Windows10 (2019-07-23)

[visionNoob]

This document gives a information about Building MxNet with CPP_PACKAGE on Windows10.
If you think it is inappropriate, please close the issue.

Hello. I'm visionNoob.
I think many people are struggling with building MxNet on win10, including me.
Finally, I (partially) succeed in building MxNet on win10.

Environment

Windows 10 Education
Visual Studio 2015
python 3.6(7.5.0)
Nvidia CUDA 9.0(V9.0.176)
Intel® MKL 2019(V2019.4.245)
OpenCV 4.1.0(released in 2019-04-08)

contents

1. MxNet : https://github.com/Vigilans/incubator-mxnet/tree/patch-1 (not merged yet)

I cloned MxNet source from https://github.com/Vigilans/incubator-mxnet/tree/patch-1
see PR #15144
this branch handle several issues( #14116, #15143 that make failure in generated op.h.)

2. Visual Studio 2015(vc14)

I think CUDA Visual Studio Integration is also needed for stable building for cuda.
And update 3 also needed.

3. OpenCV

I use "OpenCV 4.1.0 official release version" not build from source. I tried to use OpenCV that builds myself with extra-modules but with this, fail to build mxnet. I don't know why.

4. Cmake version

I use latest version of Cmake, 3.15.0. it's fine.

5. Environment Variables

and my env variables is below

6. Cmake configuration

my configuation is below

you have to set some paths and disable/enable some options.
please refer from the screen shots below.

7. In my case, I generated "op.h" manually .

it's very simple
OpWrapperGenerator.py generate "op.h" in "...\incubator-mxnet\cpp-package\include\mxnet-cpp\op.h"
The location of OpWrapperGenerator.py is
...\incubator-mxnet\cpp-package\scripts\OpWrapperGenerator.py

go to the directory and

python OpWrapperGenerator.py "\your\build\Release\libmxnet.dll"

then "op.h" is generated in "...\incubator-mxnet\cpp-package\include\mxnet-cpp\op.h"

If you see this sentences(below) during generation, You fail to generate "op.h".
I say it again. I downloaded mxnet source from here(https://github.com/Vigilans/incubator-mxnet/tree/patch-1). and it maybe solve your problems.

1>argument "lrs" of operator "multi_sgd_update" has unknown type ", required"
1>argument "wds" of operator "multi_sgd_update" has unknown type ", required"
1>argument "lrs" of operator "multi_sgd_mom_update" has unknown type ", required"
1>argument "wds" of operator "multi_sgd_mom_update" has unknown type ", required"
1>argument "lrs" of operator "multi_mp_sgd_update" has unknown type ", required"
1>argument "wds" of operator "multi_mp_sgd_update" has unknown type ", required"
1>argument "lrs" of operator "multi_mp_sgd_mom_update" has unknown type ", required"
1>argument "wds" of operator "multi_mp_sgd_mom_update" has unknown type ", required"

8. Test your build!

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Build

[zachgk]

@aaronmarkham Do we want to update some of the windows build docs?

[aaronmarkham]

@zachgk This is good for workaround info, but it's pretty convoluted for putting into instructions. We need to figure out why it doesn't work out of the box. It shouldn't be this hard.

[QueensGambit]

Hello @visionNoob, @zachgk and @aaronmarkham
thanks to the official as well as your instructions (@visionNoob) I managed to build the MXNet cpp-package with Intel-MKL as well as CUDA support.
I summarized my findings here:

• https://github.com/QueensGambit/CrazyAra-Engine/wiki/Compile-instructions-for-Windows

On one system I was able to generate the op.h with the python script, but on another system I encountered a character encoding error. Thus I ended up using op.h from an external source.

• https://github.com/dmlc/MXNet.cpp/blob/master/include/mxnet-cpp/op.h

@zachgk @aaronmarkham
Moreover, building for Windows was overall more complex and time consuming than for Linux:
The building process with CUDA support for Windows took more than 8 hours on my machine and generated a larger shared object file:
libmxnet.so - 245 mb
libmxnet.dll - 1.6 GB

@zachgk @aaronmarkham
Any ideas what could be causing this issue?

Best regards,
~QueensGambit

[JiaoPaner]

facing the same error. Is there any other solution?

[QueensGambit]

Hello @JiaoPaner.
Yes, the reason for the higher memory footprint is because the -DCMAKE_BUILD_TYPE=Release argument is missing in the build instructions. I created a pull-request about this 8 days ago, but it is not merged yet: #16144

Moreover, I managed to build the CPP_PACKAGE with TensorRT support on Linux. Unfortunately, official instruction are still missing there as well. I might provide official instructions for this later.

However, when you use TensorRT, the graph optimization will be run on every model.bind() call. This will hopefully be improved in future versions: #16173

[JiaoPaner]

@QueensGambit
I read the #15144 ,modified the line " #if !defined(_MSC_VER)" which is 744 in "include/mxnet/tuple.h" to "#if !(defined(_MSC_VER) && _MSC_VER < 1900)" .Then, compile again that config includes USE_CPP_PACKAGE, my operating system is win10 x64 and IDE is Visual Studio 2015.
After that the compile result has no any errors and no any above errors.
And the examples can run without error , the solution does work.