scala-package 1.1.0 build instruction Windows VS2015

[smpawlowski]

Description

I'd like to build mxnet for Scala on Windows from source. I can build libmxnet.dll successfully, but there are no instructions how to build scala-package on Windows. Instruction page
refers to make only.

Could you provide instructions relevant for windows or mxnet-scala.dll?

Environment info (Required)

Windows 10.
mxnet 1.1.0

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio):
Visual Studio 14 2015 Win64

MXNet commit hash: "07a83a0325a3d782513a04f47d711710972cb144"

Build config:
mxnet_option(USE_CPP_PACKAGE "Build C++ Package" ON)

Steps to reproduce

1. Build libmxnet.dll
2. Try to build scala-package??

What have you tried to solve it?

1. I tried to compile dll from files in
   scala-package\native\src
   but failed to resolve all dependencies correctly

[Roshrini]

@nswamy Could you please add labels-
Scala, Windows, Installation

[sometao]

any update here.

[lanking520]

Unfortunately, Scala is not supported on Windows for now. We are prioritize Windows support for Scala this month as many users requested that. Will try to provide support that as soon as possible. However, there is still more time needed for a stable release.

[aaronmarkham]

Please add the label for Call for Contribution.

[nswamy]

#8075

[nswamy]

@evanthomas we'll be tracking this issue here and update here when we have Windows support added.

[saudet]

The JavaCPP Presets for MXNet now bundle the Scala API and they work on Windows too:
https://github.com/bytedeco/javacpp-presets/tree/master/mxnet
What I ended up doing is generating the missing files that we can't generate with CMake on Windows using the Makefiles on Linux, and that's pretty much it. I think those files should be checked in the source code repository as it would help with this issue. Anyway, to give the current presets a try, run mvn install in the directory for MXNet or use the prebuilt binaries from Maven Central.

[lanking520]

@saudet That's amazing! We are planning to officially support MXNet very soon from 1.4.x and later version. Will you be interested in doing that? Currently we greatly optimized the building process and it should be very straight-forward to do so.

[saudet]

@lanking520 The presets use their own build system for Java classes to unify how they are packaged and loaded w.r.t. other native libraries such as OpenCV, so I'm not sure if any of it could be reused here. Could you take a look and let me know?

Anyway, there isn't a lot that needs to be changed for the build system in MXNet to work as well. It's basically just making sure everything can be built with CMake and then it looks like Windows would pretty much work.

[marcoabreu]

Awesome, thank you @saudet ! We'd be happy to assist you to get this ported into mainline. Please feel free to open a thread on dev@ as well :)

[saudet]

@marcoabreu It's more than just about MXNet on Mac or Windows though. It also about CUDA/cuDNN, OpenCV, MKL/MKL-DNN, Caffe, Deeplearning4j, TensorFlow, and (eventually) PyTorch, and how they interact. For details, read http://bytedeco.org/news/2019/01/11/importance-of-a-distribution/. How are the builds of MXNet for Java/Scala currently deployed to support CUDA, OpenCV, and MKL on Linux, Mac, and Windows? They don't seem to be available...

[lanking520]

@smpawlowski You may be interested in our publish pipeline: https://github.com/apache/incubator-mxnet/tree/master/ci/publish. We do a static-link on most of the dependencies and that will also include MKLDNN very soon. We require users to set up their own CUDA path.

[saudet]

@lanking520 For MKL-DNN, you'll be bundling MKLML which is ~30 MB, probably doubling the size of the artifacts, but what if the user wants to use the full version of MKL in their application? What about the features that require OpenCV, etc? Someone has to manage these dependencies. If MXNet does everything by themselves, it would prevent using these and other native libraries that conflict with the statically linked versions...

[lanking520]

@smpawlowski https://github.com/apache/incubator-mxnet/tree/master/tools/dependencies we have a set of libraries static-linked including OpenCV. We are keeping the up-to-date and aviod influencing the users' environment

[saudet]

@lanking520 How can Java/Scala users access the functions from the statically linked version? If they can't, they will try to use another version that will most likely conflict, especially if GPU support is enabled.

[lanking520]

@lanking520 How can Java/Scala users access the functions from the statically linked version? If they can't, they will try to use another version that will most likely conflict, especially if GPU support is enabled.

Recently we enable the Scala-Java users load libmxnet.so directly, this library contains all we need for the dependencies. We no longer require users to dynamic link them. However, we only do this for the nightly-snapshot and official versions from 1.4.x and above. https://repository.apache.org/#nexus-search;gav~org.apache.mxnet

Apart from that, will you be interested in joining our Slack channel? We can discuss there.
http://mxnet.incubator.apache.org/versions/master/community/contribute.html#slack

[saudet]

@lanking520 I don't see in your documentation where it explains how users can call from Java/Scala the functions of OpenCV that are in libmxnet.so. In any case, I'm trying to explain that this isn't just about builds for Mac or Windows. This is also about, at least, CUDA, MKL, and OpenCV: They are not just dependencies that can be buried in MXNet. I would like to expand the discussion, but is this something you think people on dev@ (as per @marcoabreu) or on Slack would be interested in?

[marcoabreu]

I think this is a great discussion point. These are powerful libraries that don't only serve mxnet but also other use cases. Changing the build system to allow access to them could potentially help especially in environments with restricted space requirements. Also, that way, people don't have to figure out the dependency clojure twice (once by us for mxnet and once by the user for their pipeline).