[python] [setup] improving installation

python-package/MANIFEST.in

@@ -5,5 +5,6 @@ recursive-include lightgbm *.py *.txt *.so
 recursive-include lightgbm/Release *.dll
 recursive-include lightgbm/include *
 recursive-include lightgbm/src *
-global-exclude *.pyo
-global-exclude *.pyc
+recursive-include lightgbm/windows LightGBM.sln LightGBM.vcxproj LightGBM.vcxproj.filters
+recursive-include lightgbm/windows/x64/DLL *.dll
+global-exclude *.py[co]

python-package/README.rst

@@ -3,7 +3,6 @@ LightGBM Python Package
 
 |PyPI version|
 
-
 Installation
 ------------
 
@@ -16,63 +15,76 @@ For Mac OS X users, gcc with OpenMP support must be installed first. Refer to `w
 
 Note: 32-bit python is not supported. Please install 64-bit version.
 
-Install from pip
-''''''''''''''''
+Install from `PyPI <https://pypi.python.org/pypi/lightgbm>`_ using ``pip``
+''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+For Windows users, `VC runtime <https://go.microsoft.com/fwlink/?LinkId=746572>`_ is needed if Visual Studio (2013, 2015 or 2017) is not installed.
+
+Install `wheel <http://pythonwheels.com>`_ via ``pip install wheel`` first. After that download the wheel file and install from it:
+
+.. code:: sh
+
+    pip install lightgbm
 
-Install `wheel <http://pythonwheels.com>`_ via ``pip install wheel`` first. For Windows user, `VC runtime <https://go.microsoft.com/fwlink/?LinkId=746572>`_ is needed if Visual Studio (2013, 2015 or 2017) is not installed.
+Build from sources
+******************
 
+.. code:: sh
 
-``pip install lightgbm``
+    pip install --no-binary :all: lightgbm
 
+For Linux and Mac OS X users, installation from sources requires installed `CMake <https://cmake.org/>`_.
 
-Install source package from pip
-*******************************
+For Mac OS X users, you need to specify compilers by runnig ``export CXX=g++-7 CC=gcc-7`` first.
 
-``pip install --no-binary :all: lightgbm``
+For Windows users, Visual Studio (or `MS Build <https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017>`_) is needed. If you get any errors during installation, you may need to install `CMake <https://cmake.org/>`_ (version 3.8 or higher).
 
+Build GPU version
+~~~~~~~~~~~~~~~~~
 
-Note: Installation from source package require installing `CMake <https://cmake.org/>`_ first.
+.. code:: sh
 
-For Windows user, Visual Studio (or `MS Build <https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017>`_) is needed, and `CMake <https://cmake.org/>`_ must be version 3.8 or higher.
+    pip install lightgbm --install-option=--gpu
 
-For OSX user, you need to run ```export CXX=g++-7 CC=gcc-7``` before running ```pip install ... ```.
+For Windows users, `CMake <https://cmake.org/>`_ (version 3.8 or higher) is strongly required in this case.
 
-Install GPU version:
+Note: Boost and OpenCL are needed: details for installation can be found in `gpu-support <https://github.com/Microsoft/LightGBM/wiki/Installation-Guide#with-gpu-support>`_. You need to add ``OpenCL_INCLUDE_DIR`` to the environmental variable **'PATH'** and export ``BOOST_ROOT`` before installation.
 
-``pip install lightgbm --install-option=--gpu``
+Build with MinGW-w64 on Windows
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Note: Boost and OpenCL are needed: details for installation can be found in `gpu-support <https://github.com/Microsoft/LightGBM/wiki/Installation-Guide#with-gpu-support>`_. Need to add OpenCL_INCLUDE_DIR to PATH and export BOOST_ROOT before installation.
+.. code:: sh
 
-Install with MinGW on Windows:
+    pip install lightgbm --install-option=--mingw
 
-``pip install lightgbm --install-option=--mingw``
+Note: `CMake <https://cmake.org/>`_ and `MinGW-w64 <https://mingw-w64.org/>`_ should be installed first.
 
 Install from GitHub
 '''''''''''''''''''
 
-Installation from GitHub require installing `CMake <https://cmake.org/>`_ first.
+For Linux and Mac OS X users, installation from GitHub requires installed `CMake <https://cmake.org/>`_.
 
-For Windows user, Visual Studio (or `MS Build <https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017>`_) is needed, and `CMake <https://cmake.org/>`_ must be version 3.8 or higher.
+For Windows users, Visual Studio (or `MS Build <https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017>`_) is needed. If you get any errors during installation and there is the warning ``WARNING:LightGBM:Compilation with MSBuild from existing solution file failed.`` in the log, you should install `CMake <https://cmake.org/>`_ (version 3.8 or higher).
 
 .. code:: sh
 
-    git clone --recursive https://github.com/Microsoft/LightGBM
+    git clone --recursive https://github.com/Microsoft/LightGBM.git
     cd LightGBM/python-package
-    # export CXX=g++-7 CC=gcc-7 # for OSX
+    # export CXX=g++-7 CC=gcc-7  # for Mac OS X users only
     python setup.py install
 
-``sudo`` (or administrator rights in Windows) may is needed to perform ``python setup.py install``.
+Note: ``sudo`` (or administrator rights in Windows) may be needed to perform the command.
 
-Use ``python setup.py install --mingw`` to use MinGW in Windows.
+Run ``python setup.py install --mingw`` if you want to use MinGW-w64 on Windows instead of Visual Studio. `CMake <https://cmake.org/>`_ and `MinGW-w64 <https://mingw-w64.org/>`_ should be installed first.
 
-Use ``python setup.py install --gpu`` to enable GPU support. Boost and OpenCL are needed: details for installation can be found in `gpu-support <https://github.com/Microsoft/LightGBM/wiki/Installation-Guide#with-gpu-support>`_.
+Run ``python setup.py install --gpu`` to enable GPU support. For Windows users, `CMake <https://cmake.org/>`_ (version 3.8 or higher) is strongly required in this case. Boost and OpenCL are needed: details for installation can be found in `gpu-support <https://github.com/Microsoft/LightGBM/wiki/Installation-Guide#with-gpu-support>`_.
+
+If you get any errors during installation or due to any other reason, you may want to build dynamic library from sources by any method you prefer (see `Installation-Guide <https://github.com/Microsoft/LightGBM/wiki/Installation-Guide>`_) and then run ``python setup.py install --precompile``.
 
 Examples
 --------
 
 Refer to the walk through examples in `python-guide folder <https://github.com/Microsoft/LightGBM/tree/master/examples/python-guide>`_.
 
-
 Troubleshooting
 ---------------
 

python-package/lightgbm/libpath.py

@@ -19,6 +19,7 @@ def find_lib_path():
     dll_path = [curr_path, os.path.join(curr_path, '../../'), os.path.join(curr_path, '../../lib/')]
     if os.name == 'nt':
         dll_path.append(os.path.join(curr_path, './Release/'))
+        dll_path.append(os.path.join(curr_path, './windows/x64/DLL/'))
         dll_path.append(os.path.join(curr_path, '../../Release/'))
         dll_path.append(os.path.join(curr_path, '../../windows/x64/DLL/'))
         dll_path = [os.path.join(p, 'lib_lightgbm.dll') for p in dll_path]
@@ -27,5 +28,5 @@ def find_lib_path():
     lib_path = [p for p in dll_path if os.path.exists(p) and os.path.isfile(p)]
     if not lib_path:
         dll_path = [os.path.realpath(p) for p in dll_path]
-        raise Exception('Cannot find lightgbm Library in following paths: ' + ','.join(dll_path))
+        raise Exception('Cannot find lightgbm library in following paths: ' + '\n'.join(dll_path))
     return lib_path

python-package/setup.py

@@ -4,9 +4,11 @@
 from __future__ import absolute_import
 
 import distutils
+import logging
 import os
 import shutil
 import struct
+import subprocess
 import sys
 
 from setuptools import find_packages, setup
@@ -22,7 +24,7 @@ def find_lib():
     exec(compile(open(libpath_py, "rb").read(), libpath_py, 'exec'), libpath, libpath)
 
     LIB_PATH = [os.path.relpath(path, CURRENT_DIR) for path in libpath['find_lib_path']()]
-    print("Install lib_lightgbm from: %s" % LIB_PATH)
+    logging.info("Installing lib_lightgbm from: %s" % LIB_PATH)
     return LIB_PATH
 
 
@@ -40,20 +42,31 @@ def copy_files_helper(folder_name):
     if not os.path.isfile('./_IS_SOURCE_PACKAGE.txt'):
         copy_files_helper('include')
         copy_files_helper('src')
+        copy_files_helper('windows')
         if use_gpu:
             copy_files_helper('compute')
         distutils.file_util.copy_file("../CMakeLists.txt", "./lightgbm/")
         distutils.file_util.copy_file("../LICENSE", "./")
 
 
 def clear_path(path):
-    contents = os.listdir(path)
-    for file in contents:
-        file_path = os.path.join(path, file)
-        if os.path.isfile(file_path):
-            os.remove(file_path)
-        else:
-            shutil.rmtree(file_path)
+    if os.path.isdir(path):
+        contents = os.listdir(path)
+        for file in contents:
+            file_path = os.path.join(path, file)
+            if os.path.isfile(file_path):
+                os.remove(file_path)
+            else:
+                shutil.rmtree(file_path)
+
+
+def silent_call(cmd):
+    try:
+        with open(os.devnull, "w") as shut_up:
+            subprocess.check_output(cmd, stderr=shut_up)
+            return 0
+    except Exception:
+        return 1
 
 
 def compile_cpp(use_mingw=False, use_gpu=False):
@@ -63,33 +76,56 @@ def compile_cpp(use_mingw=False, use_gpu=False):
     os.makedirs("build_cpp")
     os.chdir("build_cpp")
 
-    cmake_cmd = "cmake "
-    build_cmd = "make _lightgbm"
+    logger.info("Starting to compile the library.")
+
+    cmake_cmd = ["cmake", "../lightgbm/"]
     if use_gpu:
-        cmake_cmd += " -DUSE_GPU=ON "
+        cmake_cmd.append("-DUSE_GPU=ON")
     if os.name == "nt":
         if use_mingw:
-            cmake_cmd += " -G \"MinGW Makefiles\" "
-            os.system(cmake_cmd + " ../lightgbm/")
-            build_cmd = "mingw32-make.exe _lightgbm"
+            logger.info("Starting to compile with CMake and MinGW.")
+            status = silent_call(cmake_cmd + ["-G", "MinGW Makefiles"])
+            status += silent_call(["mingw32-make.exe", "_lightgbm"])
+            if status != 0:
+                raise Exception('Please install CMake and MinGW first')
         else:
-            vs_versions = ["Visual Studio 15 2017 Win64", "Visual Studio 14 2015 Win64", "Visual Studio 12 2013 Win64"]
-            try_vs = 1
-            for vs in vs_versions:
-                tmp_cmake_cmd = "%s -G \"%s\"" % (cmake_cmd, vs)
-                try_vs = os.system(tmp_cmake_cmd + " ../lightgbm/")
-                if try_vs == 0:
-                    cmake_cmd = tmp_cmake_cmd
-                    break
-                else:
-                    clear_path("./")
-            if try_vs != 0:
-                raise Exception('Please install Visual Studio or MS Build first')
-
-            build_cmd = "cmake --build . --target _lightgbm  --config Release"
-    print("Start to compile library.")
-    os.system(cmake_cmd + " ../lightgbm/")
-    os.system(build_cmd)
+            status = 1
+            lib_path = "../lightgbm/windows/x64/DLL/lib_lightgbm.dll"
+            if not use_gpu:
+                logger.info("Starting to compile with MSBuild from existing solution file.")
+                platform_toolsets = ("v141", "v140", "v120")
+                for pt in platform_toolsets:
+                    status = silent_call(["MSBuild", "../lightgbm/windows/LightGBM.sln",
+                                          "/p:Configuration=DLL",
+                                          "/p:Platform=x64",
+                                          "/p:PlatformToolset={0}".format(pt)])
+                    if status == 0 and os.path.exists(lib_path):
+                        break
+                    else:
+                        clear_path("../lightgbm/windows/x64")
+                if status != 0 or not os.path.exists(lib_path):
+                    logger.warning("Compilation with MSBuild from existing solution file failed.")
+            if status != 0 or not os.path.exists(lib_path):
+                vs_versions = ("Visual Studio 15 2017 Win64", "Visual Studio 14 2015 Win64", "Visual Studio 12 2013 Win64")
+                for vs in vs_versions:
+                    logger.info("Starting to compile with %s." % vs)
+                    status = silent_call(cmake_cmd + ["-G", vs])
+                    if status == 0:
+                        break
+                    else:
+                        clear_path("./")
+                if status != 0:
+                    raise Exception('Please install Visual Studio or MS Build first')
+
+                status = silent_call(["cmake", "--build", ".", "--target", "_lightgbm", "--config", "Release"])
+                if status != 0:
+                    raise Exception('Please install CMake first')
+    else:  # Linux, Darwin (OS X), etc.
+        logger.info("Starting to compile with CMake.")
+        status = silent_call(cmake_cmd)
+        status += silent_call(["make", "_lightgbm"])
+        if status != 0:
+            raise Exception('Please install CMake first')
     os.chdir("..")
 
 
@@ -154,6 +190,9 @@ def run(self):
 
     sys.path.insert(0, '.')
 
+    logging.basicConfig(level=logging.INFO)
+    logger = logging.getLogger('LightGBM')
+
     setup(name='lightgbm',
           version=version,
           description='LightGBM Python Package',