New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail when building xgboost using 64-bit mingw g++ #1049

Closed
Willamette-OR opened this Issue Mar 27, 2016 · 35 comments

Comments

Projects
None yet
@Willamette-OR

Willamette-OR commented Mar 27, 2016

I would like to compile xgboost for Anaconda Python 3.x in 64-bit Windows 10, so I followed the steps here - https://xgboost.readthedocs.org/en/latest/build.html#building-on-windows and here - Install xgboost under python with 64-bit msys failing. Using the 64-bit mingw g++, I got the following error message when "make -j4" using mingw64.mk:

g++: error: dmlc-core/libdmlc.a: No such file or directory
g++: error: rabit/lib/librabit_empty.a: No such file or directory
Makefile:120: recipe for target 'lib/libxgboost.dll' failed
mingw32-make: *** [lib/libxgboost.dll] Error 1

More details of the log messages can be found here:
log.txt

Can you please let me know what's going on and how I can resolve this?

Thanks a lot.

@baibaiw5

This comment has been minimized.

Show comment
Hide comment
@baibaiw5

baibaiw5 Mar 28, 2016

You can copy mingw32-make.exe to git directory(eg. C:\Program Files\Git\usr\bin),and rename it to make.exe.Then rerun make -j4 in git shell.

baibaiw5 commented Mar 28, 2016

You can copy mingw32-make.exe to git directory(eg. C:\Program Files\Git\usr\bin),and rename it to make.exe.Then rerun make -j4 in git shell.

@jfpuget

This comment has been minimized.

Show comment
Hide comment
@jfpuget

jfpuget Mar 28, 2016

You need to build dmlc-core and rabit directly

cd dmlc-core
make -j4

cd ../rabit
make -j4

but this one fails in my installation, trying to understand why.

jfpuget commented Mar 28, 2016

You need to build dmlc-core and rabit directly

cd dmlc-core
make -j4

cd ../rabit
make -j4

but this one fails in my installation, trying to understand why.

@jfpuget

This comment has been minimized.

Show comment
Hide comment
@jfpuget

jfpuget Mar 28, 2016

OK, I found it.

Using mingw64, you need to have mingw32-make in your path. Alias it to make as explained in the installation guide.
You then go to xgboost directory and do the following
cd dmlc-core
make -j4
cd ../rabit
make lib/librabit_empty -j4
cd ..
cp make/mingw64.mk config.mk
make -j4

jfpuget commented Mar 28, 2016

OK, I found it.

Using mingw64, you need to have mingw32-make in your path. Alias it to make as explained in the installation guide.
You then go to xgboost directory and do the following
cd dmlc-core
make -j4
cd ../rabit
make lib/librabit_empty -j4
cd ..
cp make/mingw64.mk config.mk
make -j4

@jfpuget

This comment has been minimized.

Show comment
Hide comment
@jfpuget

jfpuget Mar 28, 2016

typo:
in the previous comment, one line should read
make lib/librabit_empty.a -j4
instead of
make lib/librabit_empty -j4

jfpuget commented Mar 28, 2016

typo:
in the previous comment, one line should read
make lib/librabit_empty.a -j4
instead of
make lib/librabit_empty -j4

@Willamette-OR

This comment has been minimized.

Show comment
Hide comment
@Willamette-OR

Willamette-OR Mar 28, 2016

Thanks a lot jfpuget!! It finally works with your steps above!

Willamette-OR commented Mar 28, 2016

Thanks a lot jfpuget!! It finally works with your steps above!

@jfpuget

This comment has been minimized.

Show comment
Hide comment
@thirdwing

This comment has been minimized.

Show comment
Hide comment
@thirdwing

thirdwing Mar 29, 2016

Member

@jfpuget Maybe you can help to improve our instructions. Any PR will be truly welcome.

Member

thirdwing commented Mar 29, 2016

@jfpuget Maybe you can help to improve our instructions. Any PR will be truly welcome.

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Mar 29, 2016

Member

@jfpuget This looks great, can you make a PR to improve the old instruction. Thanks!

Member

tqchen commented Mar 29, 2016

@jfpuget This looks great, can you make a PR to improve the old instruction. Thanks!

@jfpuget

This comment has been minimized.

Show comment
Hide comment
@jfpuget

jfpuget Mar 30, 2016

Sure, will do ASAP (couple of days as I am just back from vacation and people have thought of me at work ;) )

jfpuget commented Mar 30, 2016

Sure, will do ASAP (couple of days as I am just back from vacation and people have thought of me at work ;) )

@Zhongjiong

This comment has been minimized.

Show comment
Hide comment
@Zhongjiong

Zhongjiong Mar 30, 2016

Hi all. When 'make -j4' in the root directory, I have this problem. Do you have some advice? Thx

A subdirectory or file build already exists.
Error occurred while processing: build.
Makefile:97: recipe for target 'build/logging.o' failed
make: *** [build/logging.o] Error 1
make: *** Waiting for unfinished jobs....
The syntax of the command is incorrect.
Makefile:97: recipe for target 'build/common/common.o' failed
make: *** [build/common/common.o] Error 1
The syntax of the command is incorrect.
Makefile:97: recipe for target 'build/c_api/c_api_error.o' failed
make: *** [build/c_api/c_api_error.o] Error 1

Zhongjiong commented Mar 30, 2016

Hi all. When 'make -j4' in the root directory, I have this problem. Do you have some advice? Thx

A subdirectory or file build already exists.
Error occurred while processing: build.
Makefile:97: recipe for target 'build/logging.o' failed
make: *** [build/logging.o] Error 1
make: *** Waiting for unfinished jobs....
The syntax of the command is incorrect.
Makefile:97: recipe for target 'build/common/common.o' failed
make: *** [build/common/common.o] Error 1
The syntax of the command is incorrect.
Makefile:97: recipe for target 'build/c_api/c_api_error.o' failed
make: *** [build/c_api/c_api_error.o] Error 1

@Zhongjiong

This comment has been minimized.

Show comment
Hide comment
@Zhongjiong

Zhongjiong Mar 30, 2016

I got it...I was using git cmd.. not git bash... That's why I encountered the problem.

Zhongjiong commented Mar 30, 2016

I got it...I was using git cmd.. not git bash... That's why I encountered the problem.

@khotilov

This comment has been minimized.

Show comment
Hide comment
@khotilov

khotilov Mar 30, 2016

Member

I can also share my notes on how I build xgboost R and python packages under windows.

Some of my workarounds were similar to Jean-François', but there are some subtle differences because I'm using the mingw build toolchain bundled with Rtools. Essentially, windows is a mess with a huge variety of mingw-based bundles out there nowadays of varying degree of completeness, stability and up-to-dateness. Even in my situation that I describe below there are three mingw-based toolsets that have to work together:

  • a minimal no-compiler set of command line gnu tools with windows git;
  • a gcc-4.7 compiler but no-make set with Anaconda;
  • and a fairly usable set which has gcc-4.6.3 and make that comes with Rtools.

The latter one is sufficient for the C++11 needs of xgboost, and I don't see any reason so far to involve yet another mingw distribution into this mess. In such situations it's critical to be very careful about what goes into PATH and in what order.

A few notes on Rtools:

I use the latest Rtools bundle version 3.3. However (IMPORTANT), during Rtools installation, I do not install the 3.3 toolchain with gcc 4.9 - it still has some issues. The 3.2 64-bit toolchain is sufficient and works fine. Since I use R more, I have Rtools' paths added to my system PATH, but not Miniconda's. Whenever I need python while not using Anaconda's shortcuts, I just set the needed paths manually.

R package build is fairly trivial:
  • git the source
  • launch R
setwd('wherever xgboost R-package directory is')
install.packages('.', repos = NULL, type='source')
setwd('tests')
source('testthat.R')
Building xgboost exe & libraries needs some workarounds:
  • assuming git-bash, within xgboost root directory
  • make sure that which gcc points to /c/Rtools/gcc-4.6.3/bin/gcc (depending on where Rtools were installed into)
  • edit rabit/Makefile and add the -m64 flag to the CFLAGS definition line (shouldn't it be there in the first place?): export CFLAGS = -m64 -O3 -msse2 $(WARNFLAGS)
  • the following should work now (need to run make in multiple stages here due to some mingw bundle interactions/quirks/bugs):
cp make/mingw64.mk config.mk
cp make/mingw64.mk dmlc-core/config.mk
cd rabit
make lib/librabit_empty.a -j4
cd ../dmlc-core
make -j4
cd ..
make -j4
Python package:
export PATH=/c/Miniconda/Scripts:/c/Miniconda:/C/Miniconda/MinGW/bin:$PATH
cd python-package/
python setup.py install
python tests/python/test_basic_models.py
Member

khotilov commented Mar 30, 2016

I can also share my notes on how I build xgboost R and python packages under windows.

Some of my workarounds were similar to Jean-François', but there are some subtle differences because I'm using the mingw build toolchain bundled with Rtools. Essentially, windows is a mess with a huge variety of mingw-based bundles out there nowadays of varying degree of completeness, stability and up-to-dateness. Even in my situation that I describe below there are three mingw-based toolsets that have to work together:

  • a minimal no-compiler set of command line gnu tools with windows git;
  • a gcc-4.7 compiler but no-make set with Anaconda;
  • and a fairly usable set which has gcc-4.6.3 and make that comes with Rtools.

The latter one is sufficient for the C++11 needs of xgboost, and I don't see any reason so far to involve yet another mingw distribution into this mess. In such situations it's critical to be very careful about what goes into PATH and in what order.

A few notes on Rtools:

I use the latest Rtools bundle version 3.3. However (IMPORTANT), during Rtools installation, I do not install the 3.3 toolchain with gcc 4.9 - it still has some issues. The 3.2 64-bit toolchain is sufficient and works fine. Since I use R more, I have Rtools' paths added to my system PATH, but not Miniconda's. Whenever I need python while not using Anaconda's shortcuts, I just set the needed paths manually.

R package build is fairly trivial:
  • git the source
  • launch R
setwd('wherever xgboost R-package directory is')
install.packages('.', repos = NULL, type='source')
setwd('tests')
source('testthat.R')
Building xgboost exe & libraries needs some workarounds:
  • assuming git-bash, within xgboost root directory
  • make sure that which gcc points to /c/Rtools/gcc-4.6.3/bin/gcc (depending on where Rtools were installed into)
  • edit rabit/Makefile and add the -m64 flag to the CFLAGS definition line (shouldn't it be there in the first place?): export CFLAGS = -m64 -O3 -msse2 $(WARNFLAGS)
  • the following should work now (need to run make in multiple stages here due to some mingw bundle interactions/quirks/bugs):
cp make/mingw64.mk config.mk
cp make/mingw64.mk dmlc-core/config.mk
cd rabit
make lib/librabit_empty.a -j4
cd ../dmlc-core
make -j4
cd ..
make -j4
Python package:
export PATH=/c/Miniconda/Scripts:/c/Miniconda:/C/Miniconda/MinGW/bin:$PATH
cd python-package/
python setup.py install
python tests/python/test_basic_models.py
@jfpuget

This comment has been minimized.

Show comment
Hide comment
@jfpuget

jfpuget Mar 30, 2016

You are right: if you already have a working g++ compiler, then you don't need to install yet another one.

jfpuget commented Mar 30, 2016

You are right: if you already have a working g++ compiler, then you don't need to install yet another one.

@tqchen

This comment has been minimized.

Show comment
Hide comment
@tqchen

tqchen Mar 31, 2016

Member

with #1071, things should be easier

Member

tqchen commented Mar 31, 2016

with #1071, things should be easier

@khotilov

This comment has been minimized.

Show comment
Hide comment
@khotilov

khotilov Mar 31, 2016

Member

BTW, I've submitted #1071 which would allow to simply run mingw32-make.exe -j from the project root to compile everything at once, when using that recent Mingw distribution. The make in Rtools still needs workarounds...

@jfpuget I'm curious: did you have any specific reason to choose the POSIX threads and SEH exception handling option? E.g., I've just tried installing Win32 and SJLJ (just because that was what gcc in Rtools used) and that seems to work.

An interesting observation on gcc 5.3: it seems to generate the came 64 bit code no matter whether there was the -m64 flag set for the rabit library or not. That's why you had no error linking it. Did gcc switch to -m64 as a default at some version?

On modification of the PATH within python: if you have just modified the system PATH, Python from anaconda in windows sometimes (depending on how you start it?) would only pick it up after a reboot/re-login. I didn't check it right now, but I had some similar issues some time ago.

Member

khotilov commented Mar 31, 2016

BTW, I've submitted #1071 which would allow to simply run mingw32-make.exe -j from the project root to compile everything at once, when using that recent Mingw distribution. The make in Rtools still needs workarounds...

@jfpuget I'm curious: did you have any specific reason to choose the POSIX threads and SEH exception handling option? E.g., I've just tried installing Win32 and SJLJ (just because that was what gcc in Rtools used) and that seems to work.

An interesting observation on gcc 5.3: it seems to generate the came 64 bit code no matter whether there was the -m64 flag set for the rabit library or not. That's why you had no error linking it. Did gcc switch to -m64 as a default at some version?

On modification of the PATH within python: if you have just modified the system PATH, Python from anaconda in windows sometimes (depending on how you start it?) would only pick it up after a reboot/re-login. I didn't check it right now, but I had some similar issues some time ago.

@jfpuget

This comment has been minimized.

Show comment
Hide comment
@jfpuget

jfpuget Mar 31, 2016

@khotilov I just left the default settings in the mingw64-gw installer.

Modifying the system path wasn't good enough, even if I rebooted python (actually, I restarted the kernel from within a notebook). I'll check again why I needed to modify the os path variable inside Python.

My goal was to have xgboost up and running, hence I stopped investigating when I was done.

jfpuget commented Mar 31, 2016

@khotilov I just left the default settings in the mingw64-gw installer.

Modifying the system path wasn't good enough, even if I rebooted python (actually, I restarted the kernel from within a notebook). I'll check again why I needed to modify the os path variable inside Python.

My goal was to have xgboost up and running, hence I stopped investigating when I was done.

@thirdwing

This comment has been minimized.

Show comment
Hide comment
@thirdwing

thirdwing Mar 31, 2016

Member

@khotilov I am not sure which MinGW-w64 with GCC 5.3 you are using.

But in some distribution (maybe most), there is no multiarch support.

Member

thirdwing commented Mar 31, 2016

@khotilov I am not sure which MinGW-w64 with GCC 5.3 you are using.

But in some distribution (maybe most), there is no multiarch support.

@yunzhou

This comment has been minimized.

Show comment
Hide comment
@yunzhou

yunzhou Jun 10, 2016

In xgboost\dmlc-core\include\dmlc\base.h
line 113
#if (defined MINGW32) || (defined MINGW64)
//#define fopen64 std::fopen
#endif
if you are using MINGW64, you should comment this fopen64 define since
mingw64 has defined its own fopen64 in the stdio.h

yunzhou commented Jun 10, 2016

In xgboost\dmlc-core\include\dmlc\base.h
line 113
#if (defined MINGW32) || (defined MINGW64)
//#define fopen64 std::fopen
#endif
if you are using MINGW64, you should comment this fopen64 define since
mingw64 has defined its own fopen64 in the stdio.h

@qingdatascience

This comment has been minimized.

Show comment
Hide comment
@qingdatascience

qingdatascience Jun 27, 2016

@yunzhou Thank you. After messed around for a whole night, your trick really did the job.

qingdatascience commented Jun 27, 2016

@yunzhou Thank you. After messed around for a whole night, your trick really did the job.

@qingdatascience

This comment has been minimized.

Show comment
Hide comment
@qingdatascience

qingdatascience Jun 27, 2016

Actually my errors did say something wrong with the 'base.h'.

qingdatascience commented Jun 27, 2016

Actually my errors did say something wrong with the 'base.h'.

@harishneela

This comment has been minimized.

Show comment
Hide comment
@harishneela

harishneela Nov 2, 2016

@jfpuget : I tried installing using your link but i am getting following error
Makefile:136: recipe for target 'xgboost' failed

I am sure i did everything as per your article. Can you please help me? Thanks

harishneela commented Nov 2, 2016

@jfpuget : I tried installing using your link but i am getting following error
Makefile:136: recipe for target 'xgboost' failed

I am sure i did everything as per your article. Can you please help me? Thanks

@jfpuget

This comment has been minimized.

Show comment
Hide comment
@jfpuget

jfpuget Nov 5, 2016

Hi,
 
there is no way you can get help unless you provide more information.  What is the error during the build?  What you provide is not the error message, it is the summary.
 
Please send the log of the build without any editing.
 
Best regards/Cordialement,JFJean-Francois Puget, PhDDistinguished Engineer, Machine Learning and Optimization, IBM AnalyticsMobile: +33(0)6 7000 8815 Email: j-f.puget@fr.ibm.comBlog: http://bit.ly/jfpuget Twitter: @jfpuget
 
 
----- Original message -----From: harishneela notifications@github.comTo: dmlc/xgboost xgboost@noreply.github.comCc: Jean Francois Puget/France/IBM@IBMFR, Mention mention@noreply.github.comSubject: Re: [dmlc/xgboost] Fail when building xgboost using 64-bit mingw g++ (#1049)Date: Wed, Nov 2, 2016 7:51 PM 
@jfpuget : I tried installing using your link but i am getting following errorMakefile:136: recipe for target 'xgboost' failed
I am sure i did everything as per your article. Can you please help me? Thanks
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

 Sauf indication contraire ci-dessus:/ Unless stated otherwise above:
Compagnie IBM France
Siège Social : 17 avenue de l'Europe, 92275 Bois-Colombes Cedex
RCS Nanterre 552 118 465
Forme Sociale : S.A.S.
Capital Social : 657.364.587 €
SIREN/SIRET : 552 118 465 03644 - Code NAF 6202A

jfpuget commented Nov 5, 2016

Hi,
 
there is no way you can get help unless you provide more information.  What is the error during the build?  What you provide is not the error message, it is the summary.
 
Please send the log of the build without any editing.
 
Best regards/Cordialement,JFJean-Francois Puget, PhDDistinguished Engineer, Machine Learning and Optimization, IBM AnalyticsMobile: +33(0)6 7000 8815 Email: j-f.puget@fr.ibm.comBlog: http://bit.ly/jfpuget Twitter: @jfpuget
 
 
----- Original message -----From: harishneela notifications@github.comTo: dmlc/xgboost xgboost@noreply.github.comCc: Jean Francois Puget/France/IBM@IBMFR, Mention mention@noreply.github.comSubject: Re: [dmlc/xgboost] Fail when building xgboost using 64-bit mingw g++ (#1049)Date: Wed, Nov 2, 2016 7:51 PM 
@jfpuget : I tried installing using your link but i am getting following errorMakefile:136: recipe for target 'xgboost' failed
I am sure i did everything as per your article. Can you please help me? Thanks
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

 Sauf indication contraire ci-dessus:/ Unless stated otherwise above:
Compagnie IBM France
Siège Social : 17 avenue de l'Europe, 92275 Bois-Colombes Cedex
RCS Nanterre 552 118 465
Forme Sociale : S.A.S.
Capital Social : 657.364.587 €
SIREN/SIRET : 552 118 465 03644 - Code NAF 6202A

@df865017

This comment has been minimized.

Show comment
Hide comment
@df865017

df865017 Dec 17, 2016

Mac 安装 xgboost

1、升级brew

由于把Mac升级到10.12后,brew没有更新,我在更新brew的时候发现几个坑,现分析一下。不要直接使用 brew update 更新brew,会报错。使用 brew upgrade 进行软件更新,在更新其他软件的同时会主动更新brew。

2、安装最新版本的gcc (gcc-6)

brew install gcc --without-multilib

注意:如果部升级 brew 就不能安装 gcc。

3、从Git上下载源码

Git clone --recursive https://github.com/dmlc/xgboost

下载后 cd xgboost; cp make/minimum.mk ./config.mk; make -j4 编译xgboost。到目前为止,只是编译了xgboost而已。(多尝试几次,可能下次就成功了哈)

4、安装python版xgboost

因为xgboost分Python版和R语言版。由于R的性能瓶颈,我使用的python,因此安装python版xgboost。

进入xgboost目录下,然后再进入xgboost的 python-package 目录,使用命令:cd python-package/

在 python-package 目录下执行:sudo python setup.py install,如果不加sudo则会报错:

5.完成,我自己安装成功了,希望对其他人有用哈

df865017 commented Dec 17, 2016

Mac 安装 xgboost

1、升级brew

由于把Mac升级到10.12后,brew没有更新,我在更新brew的时候发现几个坑,现分析一下。不要直接使用 brew update 更新brew,会报错。使用 brew upgrade 进行软件更新,在更新其他软件的同时会主动更新brew。

2、安装最新版本的gcc (gcc-6)

brew install gcc --without-multilib

注意:如果部升级 brew 就不能安装 gcc。

3、从Git上下载源码

Git clone --recursive https://github.com/dmlc/xgboost

下载后 cd xgboost; cp make/minimum.mk ./config.mk; make -j4 编译xgboost。到目前为止,只是编译了xgboost而已。(多尝试几次,可能下次就成功了哈)

4、安装python版xgboost

因为xgboost分Python版和R语言版。由于R的性能瓶颈,我使用的python,因此安装python版xgboost。

进入xgboost目录下,然后再进入xgboost的 python-package 目录,使用命令:cd python-package/

在 python-package 目录下执行:sudo python setup.py install,如果不加sudo则会报错:

5.完成,我自己安装成功了,希望对其他人有用哈

@abhijithch

This comment has been minimized.

Show comment
Hide comment
@abhijithch

abhijithch Mar 8, 2017

@jfpuget just curious to know if you are aware that these instructions are valid on a POWER8 machine. I am trying to build XGBoost on POWER8 and centos 7. I am getting the same error as the OP.

abhijithch commented Mar 8, 2017

@jfpuget just curious to know if you are aware that these instructions are valid on a POWER8 machine. I am trying to build XGBoost on POWER8 and centos 7. I am getting the same error as the OP.

@jfpuget

This comment has been minimized.

Show comment
Hide comment
@jfpuget

jfpuget Mar 8, 2017

I don't know.

jfpuget commented Mar 8, 2017

I don't know.

@elessar001

This comment has been minimized.

Show comment
Hide comment
@elessar001

elessar001 May 14, 2017

Hello @jfpuget

Thank you for your blog. However I need some guidance to complete the installation. I get an error once I type "python setup.py install"

It says "cannot find xgboost library in the candidate path". Does this mean I have an error in installation prior to this point. I did not get any errors when I followed the steps.

Kind Regards

elessar001 commented May 14, 2017

Hello @jfpuget

Thank you for your blog. However I need some guidance to complete the installation. I get an error once I type "python setup.py install"

It says "cannot find xgboost library in the candidate path". Does this mean I have an error in installation prior to this point. I did not get any errors when I followed the steps.

Kind Regards

@zhengwang522

This comment has been minimized.

Show comment
Hide comment
@zhengwang522

zhengwang522 Jun 11, 2017

I followed jfpuget's instructions to make for submodules, then here is an error after the last make,

dmlc-core/libdmlc.a: error adding symbols: Archive has no index; run ranlib to add one
collect2.exe: error: ld returned 1 exit status
Makefile:174: recipe for target 'xgboost' failed
make: *** [xgboost] Error 1

Anyone has ideas?

Thanks!

zhengwang522 commented Jun 11, 2017

I followed jfpuget's instructions to make for submodules, then here is an error after the last make,

dmlc-core/libdmlc.a: error adding symbols: Archive has no index; run ranlib to add one
collect2.exe: error: ld returned 1 exit status
Makefile:174: recipe for target 'xgboost' failed
make: *** [xgboost] Error 1

Anyone has ideas?

Thanks!

@sajitroshan

This comment has been minimized.

Show comment
Hide comment
@sajitroshan

sajitroshan Jul 12, 2017

@jfpuget I tried following your instruction on my windows OS several times, but I get this error every time:
$ cd dmlc-core
$ mingw32-make -j4
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o line_split.o src/io/line_split.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o recordio_split.o src/io/recordio_split.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o input_split_base.o src/io/input_split_base.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o io.o src/io.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o local_filesys.o src/io/local_filesys.cc
In file included from include/dmlc/./././parameter.h:25:0,
from include/dmlc/././registry.h:14,
from include/dmlc/./data.h:16,
from include/dmlc/threadediter.h:20,
from src/io/cached_input_split.h:16,
from src/io.cc:13:
include/dmlc/././././optional.h: In instantiation of 'dmlc::optional::~optional() [with T = int]':
include/dmlc/./././parameter.h:513:7: required from here
include/dmlc/././././optional.h:60:7: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
reinterpret_cast<T*>(&val)->~T();
^~~~~~~~~~~~~~~~
include/dmlc/././././optional.h: In instantiation of 'const T& dmlc::optional::value() const [with T = int]':
include/dmlc/./././parameter.h:872:9: required from here
include/dmlc/././././optional.h:106:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return reinterpret_cast<const T>(&val);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o data.o src/data.cc
include/dmlc/././././optional.h: In instantiation of 'const T& dmlc::optional::operator*() const [with T = int]':
include/dmlc/././././optional.h:134:11: required from 'std::ostream& dmlc::operator<<(std::ostream&, const dmlc::optional&) [with T = int; std::ostream = std::basic_ostream]'
include/dmlc/./././parameter.h:877:13: required from here
include/dmlc/././././optional.h:97:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return reinterpret_cast<const T>(&val);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o recordio.o src/recordio.cc
In file included from src/data/././text_parser.h:11:0,
from src/data/./libsvm_parser.h:13,
from src/data/disk_row_iter.h:19,
from src/data.cc:12:
include/dmlc/omp.h:15:81: note: #pragma message: Warning: OpenMP is not available, project will be compiled into single-thread code. Use OpenMP-enabled compiler to get benefit of multi-threading.
"Use OpenMP-enabled compiler to get benefit of multi-threading.")
^
In file included from include/dmlc/././parameter.h:25:0,
from include/dmlc/./registry.h:14,
from include/dmlc/data.h:16,
from src/data.cc:5:
include/dmlc/./././optional.h: In instantiation of 'dmlc::optional::~optional() [with T = int]':
include/dmlc/././parameter.h:513:7: required from here
include/dmlc/./././optional.h:60:7: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
reinterpret_cast<T*>(&val)->~T();
^~~~~~~~~~~~~~~~
include/dmlc/./././optional.h: In instantiation of 'const T& dmlc::optional::value() const [with T = int]':
include/dmlc/././parameter.h:872:9: required from here
include/dmlc/./././optional.h:106:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return reinterpret_cast<const T>(&val);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/dmlc/./././optional.h: In instantiation of 'const T& dmlc::optional::operator*() const [with T = int]':
include/dmlc/./././optional.h:134:11: required from 'std::ostream& dmlc::operator<<(std::ostream&, const dmlc::optional&) [with T = int; std::ostream = std::basic_ostream]'
include/dmlc/././parameter.h:877:13: required from here
include/dmlc/./././optional.h:97:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return reinterpret_cast<const T>(&val);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o config.o src/config.cc
ar cr libdmlc.a line_split.o recordio_split.o input_split_base.o io.o local_filesys.o data.o recordio.o config.o

sajitroshan commented Jul 12, 2017

@jfpuget I tried following your instruction on my windows OS several times, but I get this error every time:
$ cd dmlc-core
$ mingw32-make -j4
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o line_split.o src/io/line_split.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o recordio_split.o src/io/recordio_split.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o input_split_base.o src/io/input_split_base.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o io.o src/io.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o local_filesys.o src/io/local_filesys.cc
In file included from include/dmlc/./././parameter.h:25:0,
from include/dmlc/././registry.h:14,
from include/dmlc/./data.h:16,
from include/dmlc/threadediter.h:20,
from src/io/cached_input_split.h:16,
from src/io.cc:13:
include/dmlc/././././optional.h: In instantiation of 'dmlc::optional::~optional() [with T = int]':
include/dmlc/./././parameter.h:513:7: required from here
include/dmlc/././././optional.h:60:7: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
reinterpret_cast<T*>(&val)->~T();
^~~~~~~~~~~~~~~~
include/dmlc/././././optional.h: In instantiation of 'const T& dmlc::optional::value() const [with T = int]':
include/dmlc/./././parameter.h:872:9: required from here
include/dmlc/././././optional.h:106:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return reinterpret_cast<const T>(&val);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o data.o src/data.cc
include/dmlc/././././optional.h: In instantiation of 'const T& dmlc::optional::operator*() const [with T = int]':
include/dmlc/././././optional.h:134:11: required from 'std::ostream& dmlc::operator<<(std::ostream&, const dmlc::optional&) [with T = int; std::ostream = std::basic_ostream]'
include/dmlc/./././parameter.h:877:13: required from here
include/dmlc/././././optional.h:97:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return reinterpret_cast<const T>(&val);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o recordio.o src/recordio.cc
In file included from src/data/././text_parser.h:11:0,
from src/data/./libsvm_parser.h:13,
from src/data/disk_row_iter.h:19,
from src/data.cc:12:
include/dmlc/omp.h:15:81: note: #pragma message: Warning: OpenMP is not available, project will be compiled into single-thread code. Use OpenMP-enabled compiler to get benefit of multi-threading.
"Use OpenMP-enabled compiler to get benefit of multi-threading.")
^
In file included from include/dmlc/././parameter.h:25:0,
from include/dmlc/./registry.h:14,
from include/dmlc/data.h:16,
from src/data.cc:5:
include/dmlc/./././optional.h: In instantiation of 'dmlc::optional::~optional() [with T = int]':
include/dmlc/././parameter.h:513:7: required from here
include/dmlc/./././optional.h:60:7: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
reinterpret_cast<T*>(&val)->~T();
^~~~~~~~~~~~~~~~
include/dmlc/./././optional.h: In instantiation of 'const T& dmlc::optional::value() const [with T = int]':
include/dmlc/././parameter.h:872:9: required from here
include/dmlc/./././optional.h:106:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return reinterpret_cast<const T>(&val);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/dmlc/./././optional.h: In instantiation of 'const T& dmlc::optional::operator*() const [with T = int]':
include/dmlc/./././optional.h:134:11: required from 'std::ostream& dmlc::operator<<(std::ostream&, const dmlc::optional&) [with T = int; std::ostream = std::basic_ostream]'
include/dmlc/././parameter.h:877:13: required from here
include/dmlc/./././optional.h:97:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return reinterpret_cast<const T>(&val);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o config.o src/config.cc
ar cr libdmlc.a line_split.o recordio_split.o input_split_base.o io.o local_filesys.o data.o recordio.o config.o

@jfpuget

This comment has been minimized.

Show comment
Hide comment
@jfpuget

jfpuget Jul 12, 2017

You are compiling with g++, not mingw.

jfpuget commented Jul 12, 2017

You are compiling with g++, not mingw.

@sajitroshan

This comment has been minimized.

Show comment
Hide comment
@sajitroshan

sajitroshan Jul 13, 2017

@jfpuget
Based on your prescription available at https://www.ibm.com/developerworks/community/blogs/jfp/entry/Installing_XGBoost_For_Anaconda_on_Windows?lang=en, I carried out the following steps:

  1. I installed Git for windows.

  2. I launched Git Bash from start menu

  3. I changed directory to /c/Users/Roshan/code

  4. I ran the commands:
    $ git clone --recursive https://github.com/dmlc/xgboost
    $ cd xgboost
    $ git submodule init
    $ git submodule update

  5. I downloaded MinGW-W64 from the url mentioned in your post

  6. I checked and found mingw32-make in "C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev0\mingw64\bin"

  7. I added "C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev0\mingw64\bin" to my path system variable

  8. I ran the command "which mingw32-make" which returned "/c/Program Files/mingw-w64/x86_64-7.1.0-posix-seh-rt_v5-rev0/mingw64/bin/mingw32-make"

  9. I ran the command: alias make='mingw32-make'

  10. I changed the directory to "/c/Users/Roshan/code/xgboost/dmlc-core"

  11. Then, I ran the command: make -j4

How did I end up compiling with g++ instead of mingw. I followed each one of your instructions, almighty. Why am I still burning in hell?

sajitroshan commented Jul 13, 2017

@jfpuget
Based on your prescription available at https://www.ibm.com/developerworks/community/blogs/jfp/entry/Installing_XGBoost_For_Anaconda_on_Windows?lang=en, I carried out the following steps:

  1. I installed Git for windows.

  2. I launched Git Bash from start menu

  3. I changed directory to /c/Users/Roshan/code

  4. I ran the commands:
    $ git clone --recursive https://github.com/dmlc/xgboost
    $ cd xgboost
    $ git submodule init
    $ git submodule update

  5. I downloaded MinGW-W64 from the url mentioned in your post

  6. I checked and found mingw32-make in "C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev0\mingw64\bin"

  7. I added "C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev0\mingw64\bin" to my path system variable

  8. I ran the command "which mingw32-make" which returned "/c/Program Files/mingw-w64/x86_64-7.1.0-posix-seh-rt_v5-rev0/mingw64/bin/mingw32-make"

  9. I ran the command: alias make='mingw32-make'

  10. I changed the directory to "/c/Users/Roshan/code/xgboost/dmlc-core"

  11. Then, I ran the command: make -j4

How did I end up compiling with g++ instead of mingw. I followed each one of your instructions, almighty. Why am I still burning in hell?

@sajitroshan

This comment has been minimized.

Show comment
Hide comment
@sajitroshan

sajitroshan Jul 13, 2017

@jfpuget I cleaned up everything and tried all of these steps again. This time, I installed mingw in C drive (as one of the websites recommended avoiding Program Files, as this folder has a space in its name). While installing, I also selected the version as 5.3.0 (instead of the default 7 something). But that did not help either.

sajitroshan commented Jul 13, 2017

@jfpuget I cleaned up everything and tried all of these steps again. This time, I installed mingw in C drive (as one of the websites recommended avoiding Program Files, as this folder has a space in its name). While installing, I also selected the version as 5.3.0 (instead of the default 7 something). But that did not help either.

@sajitroshan

This comment has been minimized.

Show comment
Hide comment
@sajitroshan

sajitroshan Jul 14, 2017

@qingdatascience : What specifically did you do to make it work?

sajitroshan commented Jul 14, 2017

@qingdatascience : What specifically did you do to make it work?

@datascienceqing

This comment has been minimized.

Show comment
Hide comment
@datascienceqing

datascienceqing Jul 14, 2017

I did not remember. It looks like just editing the bash script file like @yunzhou suggested. If it did not work for you, I would suggest you to try LightGBM, which is from MS and faster.

datascienceqing commented Jul 14, 2017

I did not remember. It looks like just editing the bash script file like @yunzhou suggested. If it did not work for you, I would suggest you to try LightGBM, which is from MS and faster.

@sajitroshan

This comment has been minimized.

Show comment
Hide comment
@sajitroshan

sajitroshan Jul 23, 2017

Many thanks to @datascienceqing as well as @jfpuget

Since I could not install xgboost, I installed LightGBM. I followed the instructions provided at the following URL. The entire installation (start to finish) took less than 30 minutes:
https://github.com/Microsoft/LightGBM/tree/master/python-package

I have windows OS. I do not have Visual Studio. I could install LightGBM in 5 steps:

  1. Download and install cmake from https://cmake.org/download/
  2. Download and install MS Build from https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017
  3. Download and install Git from https://git-for-windows.github.io/?cm_mc_uid=41805252005114998995219&cm_mc_sid_50200000=1500774547
  4. Start Git Bash
  5. Run these commands:
    git clone --recursive https://github.com/Microsoft/LightGBM
    cd LightGBM/python-package
    python setup.py install

sajitroshan commented Jul 23, 2017

Many thanks to @datascienceqing as well as @jfpuget

Since I could not install xgboost, I installed LightGBM. I followed the instructions provided at the following URL. The entire installation (start to finish) took less than 30 minutes:
https://github.com/Microsoft/LightGBM/tree/master/python-package

I have windows OS. I do not have Visual Studio. I could install LightGBM in 5 steps:

  1. Download and install cmake from https://cmake.org/download/
  2. Download and install MS Build from https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017
  3. Download and install Git from https://git-for-windows.github.io/?cm_mc_uid=41805252005114998995219&cm_mc_sid_50200000=1500774547
  4. Start Git Bash
  5. Run these commands:
    git clone --recursive https://github.com/Microsoft/LightGBM
    cd LightGBM/python-package
    python setup.py install
@sajitroshan

This comment has been minimized.

Show comment
Hide comment
@sajitroshan

sajitroshan Jul 23, 2017

I could finally install xgboost, after several rounds of trial and error. The instructions at the following location was very helpful: http://www.picnet.com.au/blogs/guido/post/2016/09/22/xgboost-windows-x64-binaries-for-download/

Thanks to @gatapia @jfpuget and @qingdatascience

sajitroshan commented Jul 23, 2017

I could finally install xgboost, after several rounds of trial and error. The instructions at the following location was very helpful: http://www.picnet.com.au/blogs/guido/post/2016/09/22/xgboost-windows-x64-binaries-for-download/

Thanks to @gatapia @jfpuget and @qingdatascience

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment