Update Wiki Build Environments (VS 2015, MSYS2 changes) #298

wongjoel · 2016-10-03T15:03:08Z

Following the wiki instructions on "Build Environments" (Windows section) no longer appears to work.

I attempted to build javacpp-presets according to the instructions at https://github.com/bytedeco/javacpp-presets/wiki/Build-Environments, but it failed to compile successfully, and many of the steps did not work as written (at least for me).

I had hoped to submit an update to the wiki myself, but I have so far failed to get a working build environment.

For the Windows section:

Link to Visual Studio Community 2013 now takes you to the download for Visual Studio Community 2015 (not sure if that causes any issues)
running "pacman -S base-devel tar patch make git unzip zip nasm yasm pkg-config mingw-w64-x86_64-cmake mingw-w64-x86_64-gcc mingw-w64-i686" in a MSYS2 shell will fail, claiming conflicting files (can be forced with --force flag, but not sure if that's the correct thing to do?)
c:\msys64\mingw64_shell.bat no longer exists with the current installer of msys2 (I believe the equivalent command for a new install is "C:\msys64\msys2_shell.cmd -mingw64")

currently I can run the bash cppbuild.sh script, but I get a cmake issue, and can't work out if the issue is with my Visual Studio install, or my MSYS2 install:
-- The CXX compiler identification is unknown
-- The C compiler identification is unknown

Sorry if that's too many things put into one issue, the main concern I have here is that there doesn't seem to be a working set of instructions on how configure a clean install of windows to be able to build javacpp-presets.

wongjoel · 2016-10-04T02:25:28Z

Okay, so I found the cause of the cmake issue - when using Visual Studio Community 2015, the cppbuild.sh scripts need to be updated to point to Visual Studio 2015:

{"$CMAKE" -G "Visual Studio 12 2013 Win64"} should be replaced by {"$CMAKE" -G "Visual Studio 14 2015 Win64"}
{cp -r ../x86/vc12/lib ..} and {cp -r ../x86/vc12/bin ..} should be replaced by {cp -r ../x86/vc14/lib ..} and {cp -r ../x86/vc14/bin ..}

Still having issues running "mvn clean install -Djavacpp.platform=windows-x86_64", but hopefully if I get that working too, I can write up instructions on setting up a build environment on Windows 10, with Visual Studio 2015 and MSYS2

saudet · 2016-10-04T02:58:54Z

Thanks for testing this out! If you could check that everything works with Visual Studio 2015, that would be great. If there is anything that doesn't work, we can still download old versions here:
https://www.visualstudio.com/vs/older-downloads/

wongjoel · 2016-10-08T15:35:09Z

Thanks Samuel - I didn't realise that the older builds of Visual Studio were still available for free.

These are my notes from attempting to build javacpp-presets on a fresh install of Windows using Visual Studio 2015 and the latest build of msys2. Looking at the results, I think the main problem is actually the way I've set up msys2, though I have no idea what's wrong with Maven trying to build the presets for OpenCV. I don't really have much more time to look into it deeper right now, but hopefully these notes save someone some time in the future.

Environment setup (in the listed order):

Fresh install of Windows 10 rs1_release 14393 (Redstone 1 aka Anniversary Update)
Install JDK 8u101 (x64)
Install Maven 3.3.9 (Add JAVA_HOME variable, add Maven to PATH)
Install msys2 (x64, 2016-09-21 installer, update pacman immediately)
Install Visual Studio Community 2015 Update 3 (All Language options, Visual C++ in particular)
Install CUDA 8.0.44

Packages in msys2

there is an issue with the packages prefixed "mingw-*", which will give the following message - error: failed to commit transaction (conflicting files)
- non mingw-* packages are fine, so "pacman -S base-devel tar patch make git unzip zip nasm yasm pkg-config"
- need gcc, so "pacman -S gcc"
- need fortran for OpenBLAS, so "pacman -S gcc-fortran"
- install cmake on windows directly (because can't use "pacman -S mingw-w64-x86_64-cmake")

Misc Changes

edit batch "c:\msys64\msys2_shell.cmd" by uncommenting line "set MSYS2_PATH_TYPE=inherit"
from VS2015 x64 Native Tools Command Prompt run "c:\msys64\msys2_shell.cmd -mingw64"
edit opencv\cppbuild.sh for Visual Studio 2015: {"$CMAKE" -G "Visual Studio 14 2015 Win64"} and {cp -r ../x64/vc14/lib ..} and {cp -r ../x64/vc14/bin ..}
Using javacpp version 1.2.5-SNAPSHOT
Trying to build javacpp-presets 1.2.5-SNAPSHOT
Build results
running each individually with "bash cppbuild.sh -platform windows-x86_64 install " and "mvn install --projects .,"

Project	cppbuild.sh	maven	error message
opencv	works	fails	jnicvkernels.obj : error lnk2001: unresolved external symbol getprocessmemoryinfo
ffmpeg	fails	-	Cannot export gzopen_w: symbol not defined
flycapture	fails	-	Please install FlyCapture under the default installation directory
libdc1394	fails	-	cp: cannot stat '/c/Program Files (x86)/CMU/1394Camera/lib64/x64/1394camera.dll': No such file or directory
libfreenect	fails	-	"javacpp-presets\libfreenect\cppbuild\windows-x86_64\pthreads-w32-2-9-1-release\Pre-built.2\include\pthread.h(320): error C2011: 'timespec': 'struct' type redefinition"
videoinput	works	fails	jnivideoInputLib.obj : error LNK2001: unresolved external symbol GetProcessMemoryInfo
artoolkitplus	works	works	-
chilitags	works	fails	Could not find artifact org.bytedeco.javacpp-presets:opencv:jar:3.1.0-1.2.5-SNAPSHOT
flandmark	fails	-	C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\stdint.h(17): error C2371: 'int8_t': redefinition; different basic types
openblas	works	fails	collect2: error: ld returned 1 exit status
fftw	works	works	-
gsl	works	works	-
llvm	fails	-	Error: Platform "windows-x86_64" is not supported
leptonica	fails	-	Cannot export gzopen_w: symbol not defined
tesseract	fails	-	checking for leptonica... configure: error: leptonica not found
caffe	fails	-	Error: Platform "windows-x86_64" is not supported
cuda	works	fails	[ERROR] Failed to execute JavaCPP Builder: Could not parse "cudnn.h": File does not exist
mxnet	fails	-	Error: Platform "windows-x86_64" is not supported
tensorflow	fails	-	Error: Platform "windows-x86_64" is not supported

saudet · 2016-10-08T15:46:29Z

Thanks! MSYS2 not installing properly, that's strange. Let me know if you figure out why this is happening. This is the first time I hear anyone having problems installing MSYS2.

vb216 · 2016-10-08T17:57:57Z

Hey FWIW, I can give a bit more info on some of these.. The unresolved symbol for GetProcessMemoryInfo is weird, it needs psapi.lib to supply this. For some reason, calling the cppbuild.sh gets this dependency OK, but with the mvn build trying the same it doesn't. It could be added in as explicit dependencies I think, but I'm thinking of using the flags for the mvn build to say don't do the cppbuild part in mvn.

The errors to do with gzopen_w - have you install zlib? That might fix those.

There's something strange with ffmpeg, libvpx seems to have a bug on version 1.6 with gcc 6.2, if you go back to 1.5 it seems to build OK.

And then the last few error platform not supported are OK - the mvn install step will just create empty jars.

With that, there might only be the odd one or two not building. I think I'm using the latest msys2, but I have deviated a bit from the wiki instructions. At the last check the one I'm stuck on is still CUDA, but lets see now I've figured out the ffmpeg problem :)

wongjoel · 2016-10-09T06:10:34Z

That's great! OpenCV and ffmpeg were the ones I was most concerned about, so knowing what's causing the issues for those is really helpful.

I had installed zlib via "pacman -S zlib", after seeing that gzopen_w message, but it doesn't/didn't change the output or the error message, so it's probably some other supporting library/package that's missing. (Unless there's another version-specific bug with one of the current packages)

…f Visual Studio (issue bytedeco/javacpp-presets#298)

saudet · 2016-10-09T14:27:51Z

So it looks like we need to link with psapi.lib manually for some versions of Visual Studio? I've added that to the commit above. It should work fine now. Thanks for reporting!

Neiko2002 · 2016-10-24T13:24:00Z

I can confirm the current msys version (msys2-i686-20160921.exe) needs the force option when installing mingw. Another option is to create the mingw directories mkdir /mingw32 /mingw64 beforehand.

This is a bug which is already fixed in the next msys version.
msys2/MSYS2-packages#689

Neiko2002 · 2016-10-24T13:47:33Z

I was able to build and install openblas for maven with the following step. This is a clean installation.

• Download and install msys2-x86_64-20160921 to C:\msys64
http://msys2.github.io/

• Start msys2 console and run:
pacman -Sy pacman

• close msys2 and start again to run:
pacman -Syu

• close msys2 and start again to run:
pacman -Su

• Next install some necessary packages (press Enter afterwards to accept the installation of all modules):
pacman -S base-devel tar patch make git unzip zip nasm yasm pkg-config 

• Now we can install mingw
mkdir /mingw32 /mingw64
pacman -S mingw-w64-x86_64-cmake mingw-w64-x86_64-gcc mingw-w64-i686-gcc mingw-w64-x86_64-gcc-fortran mingw-w64-i686-gcc-fortran mingw-w64-x86_64-libwinpthread-git mingw-w64-i686-libwinpthread-git

• Inside the C:\msys64 edit the new files mingw32.ini and mingw64.ini. Remove the # at line 4 resulting in:
MSYS2_PATH_TYPE=inherit

• Install Visual Studio Community 2013
https://www.visualstudio.com/en-us/news/releasenotes/vs2013-community-vs

• Start the command line tool "VS2013 x64 Native Tools Command Prompt" inside the directory:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\Shortcuts

• Navigate to the C:\msys64 directory and run mingw64.exe

• Check the path (look for /c/Program Files (x86)/Microsoft Visual Studio 12.0/Common7/Tools)
echo $PATH

• Check the Visual Studio enviroment variables
echo $VisualStudioVersion

• The msys2 console will open. Navigate to a directory where you would like to build the libs. e.g.
cd /c/Lang/Java/javacpp/

• There run
git clone https://github.com/bytedeco/javacpp.git
git clone https://github.com/bytedeco/javacpp-presets.git
cd javacpp
mvn clean install

• Change the exec-maven-plugin version to 1.5 in the javacpp-preset/pom.xml file

• At the end of the process the javacpp library has been installed into your local maven repository
C:\Users\%USERNAME%\.m2\repository\org\bytedeco\javacpp\

• Now build javacpp-openblas
cd ..
cd javacpp-presets
mvn clean install -Djavacpp.platform=windows-x86_64 --projects .
bash cppbuild.sh -platform windows-x86_64 install openblas
mvn install -Djavacpp.platform=windows-x86_64 --projects openblas

Neiko2002 · 2016-10-24T13:50:41Z

The important part is Change the exec-maven-plugin version to 1.5 in the javacpp-preset/pom.xml file without these changes I got the following error:

[DEBUG] Executing command line: [bash, C:\Lang\Java\javacpp\javacpp-presets\openblas/../cppbuild.sh, clean, openblas]
    Error: 0x80070057
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD FAILURE
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 1.623 s
    [INFO] Finished at: 2016-10-24T13:20:38+02:00
    [INFO] Final Memory: 20M/491M
    [INFO] ------------------------------------------------------------------------
    [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (javacpp.cppbuild.clean) on project openblas: Command execution failed. Process exited with an error: -1 (Exit value: -1) -> [Help 1]

@wongjoel you might wanna try to change the version.

Neiko2002 · 2016-10-24T14:29:39Z

Another problem. Installing Visual Studio 2013 before or alongside Visual Studio 2015 will corrupt the default values (e.g. VCTargetsPath) used for MSBuild in the Windows registry. This will lead to error MSB4019 errors when building any Visual Studio project inside msys2.

I've tried bash cppbuild.sh -platform windows-x86_64 install opencv and got the MSB4019 error. Afterwards I opened the OpenCV.sln file with Visual Studio 2013 and build the projects without any problems.

@saudet I would like to replace the current Building-on-Windows wiki with my new version and also remove all windows related steps from the general installation guide. Is that okey for you?

vb216 · 2016-10-24T17:08:53Z

Cool! Looks like openblas works for me now. Did you change ffmpeg to use an older version of libvpx or did it just work for you? Also, did hdf5 build?

Neiko2002 · 2016-10-24T17:39:53Z

@vb216 Sorry I only looked at openblas and opencv in order to build mxnet. After adding a PR for mxnet tomorrow, I will have a look at ffmpeg.

vb216 · 2016-10-24T21:41:32Z

No problem.. looks pretty promising I think.. I double checked ffmpeg, still an issue with libvpx, maybe it'll be fixed in next version but for now I just downgrade to 1.5 and all is good. And from a recent commit, it looks like I need to download hdf5 for windows before trying the build. Apart from that, everything built fine, I'll try hdf5 fix and then it should all just build from a simple mvn install..

Neiko2002 · 2016-10-25T13:48:11Z

There is a new msys2 version (msys2-x86_64-20161025.exe) which does not have the mingw problems.

Neiko2002 · 2016-10-25T14:15:08Z

Visual Studio 2015

It is possible to build the projects (e.g. opencv) with Visual Studio 2015 and like @wongjoel mentioned one have to edit the opencv\cppbuild.sh file {"$CMAKE" -G "Visual Studio 14 2015 Win64"} and {cp -r ../x64/vc14/lib ..} and {cp -r ../x64/vc14/bin ..}.

Depending on the update version of the studio it might be necessary to add some more include and library directories. Execute these commands before running the build process. You might have to change the version "10.0.10240.0"

INCLUDE="C:\Program Files (x86)\Windows Kits\10\Include\10.0.10240.0\ucrt;$INCLUDE"
LIB="C:\Program Files (x86)\Windows Kits\10\Lib\10.0.10240.0\ucrt\x64;$LIB"

I got some "stdio.h": No such file or directory errors without those two lines.

Neiko2002 · 2016-10-25T14:21:37Z

@vb216 HDF5 works for me I've installed https://support.hdfgroup.org/HDF5/release/obtain5110.html and run the steps I've described yesterday.

vb216 · 2016-10-25T14:27:46Z

Yeah looks good to me too. Just libvpx in ffmpeg that has an issue, I think that might be their code tho, latest from their git repo built fine and the 1.5 earlier version too, so maybe some bug introduced and then fixed.

Seems a pretty straightforward way to get a build env up and running now!

Neiko2002 · 2016-10-25T15:12:44Z

True libvpx makes problems:

In file included from ./vp9/common/vp9_entropymode.h:15:0,
                 from ./vp9/common/vp9_blockd.h:23,
                 from ./vp9/common/vp9_scan.h:18,
                 from test/dct16x16_test.cc:24:
./vp9/common/vp9_entropymv.h: In function 'int use_mv_hp(const MV*)':
./vp9/common/vp9_entropymv.h:35:25: error: invalid operands of types '__gnu_cxx::__enable_if<true, double>::__type {aka double}' and 'int' to binary 'operator>>'
   return (abs(ref->row) >> 3) < COMPANDED_MVREF_THRESH &&
           ~~~~~~~~~~~~~~^~~~
./vp9/common/vp9_entropymv.h:36:25: error: invalid operands of types '__gnu_cxx::__enable_if<true, double>::__type {aka double}' and 'int' to binary 'operator>>'
          (abs(ref->col) >> 3) < COMPANDED_MVREF_THRESH;
           ~~~~~~~~~~~~~~^~~~
make[1]: *** [Makefile:160: test/dct16x16_test.cc.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:17: .DEFAULT] Error 2

This seems to be a problem with GCC6 and is fixed with:
https://chromium-review.googlesource.com/#/c/388931/
Apply the patch there and it works.

saudet · 2016-10-30T14:06:55Z

@Neiko2002 Please do update the wiki page, but do not replace it entirely:

Don't remove information that is there, only update it. You can for example direct users that don't like to use the "Start Menu" to "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\Shortcuts", but most users will just use the "Start Menu", so don't remove that information. We also need CUDA,etc.
Don't include workarounds for bugs in MSYS that are already fixed, or soon to be fixed. That is only going to cause confusion.
No need to document updating the pom.xml file when we can just update it, which I've already done for the exec-maven-plugin. Please send pull requests instead.
The instructions should be more general, not just for OpenBLAS.

That said, thank you very much your contributions!

saudet · 2017-04-08T16:00:48Z

I've finally updated the wiki page for Visual Studio 2015 and the latest of MSYS2.

I've also reinstalled MSYS2 from scratch, but have not encountered any of the issues described here.

As noted by @wongjoel, I had to update a few things as per the commit above, but that's it.

@Neiko2002 Please feel free to update https://github.com/bytedeco/javacpp-presets/wiki/Building-on-Windows in any way you wish, but do not remove the general guidelines from https://github.com/bytedeco/javacpp-presets/wiki/Build-Environments.

Thank you all for all the information and the great feedback!

Neiko2002 · 2017-04-11T12:56:29Z

Thanks you saudet. I've extended the "Building on Windows" wiki page and solved a problem ("stdio.h": No such file or directory) that often occurred to me because of the missing ucrt directory in the Windows Kit. Precautionally installing the Windows SDK prevents such problems in the future for any other user who might want to use javacpp under Windows.

saudet · 2017-04-11T23:18:34Z

@Neiko2002 Thanks! Although for me the installer of Visual Studio 2015 automatically installs Windows 10 SDK too.

Neiko2002 · 2017-04-12T08:46:54Z

I have VS2013 and VS2015 installed. Using the newer one, the include variable always pointed to this directory "C:\Program Files (x86)\Windows Kits\10\Lib\10.0.14393.0". Which does not have any ucrt libs. Otherwise the 10.0.10240.0 folder of VS2013 and the 10.0.15063.0 folder of the Windows 10 SDK contains the ucrt files. Microsoft might have some problems when updating and installing multiple versions.

saudet · 2017-04-15T15:11:44Z

Great, thanks! Added a link at the bottom here:
https://github.com/bytedeco/javacpp-presets/wiki/Build-Environments#windows-x86-and-x86_64

saudet added a commit to bytedeco/javacpp that referenced this issue Oct 9, 2016

* Add missing platform.link to psapi required by some versions o…

b7b1e5d

…f Visual Studio (issue bytedeco/javacpp-presets#298)

Neiko2002 mentioned this issue Oct 25, 2016

Build MXNet under windows #309

Merged

saudet added enhancement help wanted labels Dec 9, 2016

saudet added a commit that referenced this issue Apr 8, 2017

* Update presets for Visual Studio 2015 (issue #298)

7dfdee3

saudet removed the help wanted label Apr 8, 2017

saudet closed this as completed Apr 8, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Wiki Build Environments (VS 2015, MSYS2 changes) #298

Update Wiki Build Environments (VS 2015, MSYS2 changes) #298

wongjoel commented Oct 3, 2016

wongjoel commented Oct 4, 2016 •

edited

Loading

saudet commented Oct 4, 2016

wongjoel commented Oct 8, 2016

saudet commented Oct 8, 2016

vb216 commented Oct 8, 2016

wongjoel commented Oct 9, 2016

saudet commented Oct 9, 2016

Neiko2002 commented Oct 24, 2016 •

edited

Loading

Neiko2002 commented Oct 24, 2016 •

edited

Loading

Neiko2002 commented Oct 24, 2016

Neiko2002 commented Oct 24, 2016 •

edited

Loading

vb216 commented Oct 24, 2016

Neiko2002 commented Oct 24, 2016 •

edited

Loading

vb216 commented Oct 24, 2016

Neiko2002 commented Oct 25, 2016

Neiko2002 commented Oct 25, 2016 •

edited

Loading

Neiko2002 commented Oct 25, 2016

vb216 commented Oct 25, 2016

Neiko2002 commented Oct 25, 2016 •

edited

Loading

saudet commented Oct 30, 2016

saudet commented Apr 8, 2017 •

edited

Loading

Neiko2002 commented Apr 11, 2017

saudet commented Apr 11, 2017

Neiko2002 commented Apr 12, 2017

saudet commented Apr 15, 2017

Update Wiki Build Environments (VS 2015, MSYS2 changes) #298

Update Wiki Build Environments (VS 2015, MSYS2 changes) #298

Comments

wongjoel commented Oct 3, 2016

wongjoel commented Oct 4, 2016 • edited Loading

saudet commented Oct 4, 2016

wongjoel commented Oct 8, 2016

saudet commented Oct 8, 2016

vb216 commented Oct 8, 2016

wongjoel commented Oct 9, 2016

saudet commented Oct 9, 2016

Neiko2002 commented Oct 24, 2016 • edited Loading

Neiko2002 commented Oct 24, 2016 • edited Loading

Neiko2002 commented Oct 24, 2016

Neiko2002 commented Oct 24, 2016 • edited Loading

vb216 commented Oct 24, 2016

Neiko2002 commented Oct 24, 2016 • edited Loading

vb216 commented Oct 24, 2016

Neiko2002 commented Oct 25, 2016

Neiko2002 commented Oct 25, 2016 • edited Loading

Visual Studio 2015

Neiko2002 commented Oct 25, 2016

vb216 commented Oct 25, 2016

Neiko2002 commented Oct 25, 2016 • edited Loading

saudet commented Oct 30, 2016

saudet commented Apr 8, 2017 • edited Loading

Neiko2002 commented Apr 11, 2017

saudet commented Apr 11, 2017

Neiko2002 commented Apr 12, 2017

saudet commented Apr 15, 2017

wongjoel commented Oct 4, 2016 •

edited

Loading

Neiko2002 commented Oct 24, 2016 •

edited

Loading

Neiko2002 commented Oct 24, 2016 •

edited

Loading

Neiko2002 commented Oct 24, 2016 •

edited

Loading

Neiko2002 commented Oct 24, 2016 •

edited

Loading

Neiko2002 commented Oct 25, 2016 •

edited

Loading

Neiko2002 commented Oct 25, 2016 •

edited

Loading

saudet commented Apr 8, 2017 •

edited

Loading