added info and possibility to install Python-package HDFS version (#1252

)

docs/Installation-Guide.rst

@@ -275,6 +275,78 @@ Docker
 
 Refer to `GPU Docker folder <https://github.com/Microsoft/LightGBM/tree/master/docker/gpu>`__.
 
+Build HDFS Version
+~~~~~~~~~~~~~~~~~~
+
+Windows
+^^^^^^^
+
+Visual Studio (or MSBuild)
+**************************
+
+1. Install `Git for Windows`_, `CMake`_ (3.8 or higher) and `MSBuild`_ (**MSBuild** is not needed if **Visual Studio** (2015 or newer) is installed).
+
+2. Run the following commands:
+
+   .. code::
+
+     git clone --recursive https://github.com/Microsoft/LightGBM
+     cd LightGBM
+     mkdir build
+     cd build
+     cmake -DCMAKE_GENERATOR_PLATFORM=x64 -DUSE_HDFS=ON ..
+     cmake --build . --target ALL_BUILD --config Release
+
+MinGW64
+*******
+
+1. Install `Git for Windows`_, `CMake`_ and `MinGW-w64`_.
+
+2. Run the following commands:
+
+   .. code::
+
+     git clone --recursive https://github.com/Microsoft/LightGBM
+     cd LightGBM
+     mkdir build
+     cd build
+     cmake -G "MinGW Makefiles" -DUSE_HDFS=ON ..
+     mingw32-make.exe -j4
+
+Linux
+^^^^^
+
+LightGBM uses **CMake** to build. Run the following commands:
+
+.. code::
+
+  git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
+  mkdir build ; cd build
+  cmake -DUSE_HDFS=ON ..
+  make -j4
+
+macOS
+^^^^^
+
+LightGBM depends on **OpenMP** for compiling, which isn't supported by Apple Clang.
+
+Please install **gcc/g++** by using the following commands:
+
+.. code::
+
+  brew install cmake
+  brew install gcc
+
+Then install LightGBM:
+
+.. code::
+
+  git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
+  export CXX=g++-7 CC=gcc-7
+  mkdir build ; cd build
+  cmake -DUSE_HDFS=ON ..
+  make -j4
+
 Build Java Wrapper
 ~~~~~~~~~~~~~~~~~~
 

python-package/README.rst

@@ -82,6 +82,15 @@ All available options:
 
 For more details see `FindBoost <https://cmake.org/cmake/help/v3.8/module/FindBoost.html>`__ and `FindOpenCL <https://cmake.org/cmake/help/v3.8/module/FindOpenCL.html>`__.
 
+Build HDFS Version
+~~~~~~~~~~~~~~~~~~
+
+.. code:: sh
+
+    pip install lightgbm --install-option=--hdfs
+
+For Windows users, `CMake <https://cmake.org/>`_ (version 3.8 or higher) is strongly required in this case.
+
 Build with MinGW-w64 on Windows
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -113,6 +122,8 @@ Run ``python setup.py install --mingw`` if you want to use MinGW-w64 on Windows
 
 Run ``python setup.py install --gpu`` to enable GPU support. For Windows users, `CMake <https://cmake.org/>`_ (version 3.8 or higher) is strongly required in this case. Boost and OpenCL are needed: details for installation can be found in `Installation Guide <https://github.com/Microsoft/LightGBM/blob/master/docs/Installation-Guide.rst#build-gpu-version>`__. You can pass additional options to CMake: ``python setup.py install --gpu --opencl-include-dir=/usr/local/cuda/include/``, see `Build GPU Version <#build-gpu-version>`__ for complete list of them.
 
+Run ``python setup.py install --hdfs`` to enable HDFS support. For Windows users, `CMake <https://cmake.org/>`_ (version 3.8 or higher) is strongly required in this case.
+
 If you get any errors during installation or due to any other reason, you may want to build dynamic library from sources by any method you prefer (see `Installation Guide <https://github.com/Microsoft/LightGBM/blob/master/docs/Installation-Guide.rst>`__) and then run ``python setup.py install --precompile``.
 
 Examples

python-package/setup.py

@@ -78,7 +78,7 @@ def silent_call(cmd, raise_error=False, error_msg=''):
         return 1
 
 
-def compile_cpp(use_mingw=False, use_gpu=False, use_mpi=False,
+def compile_cpp(use_mingw=False, use_gpu=False, use_mpi=False, use_hdfs=False,
                 boost_root=None, boost_dir=None, boost_include_dir=None,
                 boost_librarydir=None, opencl_include_dir=None,
                 opencl_library=None):
@@ -107,6 +107,8 @@ def compile_cpp(use_mingw=False, use_gpu=False, use_mpi=False,
             cmake_cmd.append("-DOpenCL_LIBRARY={0}".format(opencl_library))
     if use_mpi:
         cmake_cmd.append("-DUSE_MPI=ON")
+    if use_hdfs:
+        cmake_cmd.append("-DUSE_HDFS=ON")
     if os.name == "nt":
         if use_mingw:
             if use_mpi:
@@ -119,7 +121,7 @@ def compile_cpp(use_mingw=False, use_gpu=False, use_mpi=False,
         else:
             status = 1
             lib_path = "../compile/windows/x64/DLL/lib_lightgbm.dll"
-            if not use_gpu:
+            if not use_gpu and not use_hdfs:
                 logger.info("Starting to compile with MSBuild from existing solution file.")
                 platform_toolsets = ("v141", "v140")
                 for pt in platform_toolsets:
@@ -147,7 +149,7 @@ def compile_cpp(use_mingw=False, use_gpu=False, use_mpi=False,
                                     log_notice)))
                 silent_call(["cmake", "--build", ".", "--target", "_lightgbm", "--config", "Release"], raise_error=True,
                             error_msg='Please install CMake first')
-    else:  # Linux, Darwin (OS X), etc.
+    else:  # Linux, Darwin (macOS), etc.
         logger.info("Starting to compile with CMake.")
         silent_call(cmake_cmd, raise_error=True, error_msg='Please install CMake and all required dependencies first')
         silent_call(["make", "_lightgbm"], raise_error=True,
@@ -172,6 +174,7 @@ class CustomInstall(install):
         ('mingw', 'm', 'Compile with MinGW'),
         ('gpu', 'g', 'Compile GPU version'),
         ('mpi', None, 'Compile MPI version'),
+        ('hdfs', 'h', 'Compile HDFS version'),
         ('precompile', 'p', 'Use precompiled library'),
         ('boost-root=', None, 'Boost preferred installation prefix'),
         ('boost-dir=', None, 'Directory with Boost package configuration file'),
@@ -192,13 +195,14 @@ def initialize_options(self):
         self.opencl_include_dir = None
         self.opencl_library = None
         self.mpi = 0
+        self.hdfs = 0
         self.precompile = 0
 
     def run(self):
         open(path_log, 'wb').close()
         if not self.precompile:
             copy_files(use_gpu=self.gpu)
-            compile_cpp(use_mingw=self.mingw, use_gpu=self.gpu, use_mpi=self.mpi,
+            compile_cpp(use_mingw=self.mingw, use_gpu=self.gpu, use_mpi=self.mpi, use_hdfs=self.hdfs,
                         boost_root=self.boost_root, boost_dir=self.boost_dir,
                         boost_include_dir=self.boost_include_dir, boost_librarydir=self.boost_librarydir,
                         opencl_include_dir=self.opencl_include_dir, opencl_library=self.opencl_library)