Add documentation for building using CMake and for building Python pa…

…ckage using standard instruments.. #830, #2278, Fix #2187
catboost · May 11, 2023 · 7c50803 · 7c50803
1 parent beedf6b
commit 7c50803
Show file tree

Hide file tree

Showing 36 changed files with 963 additions and 469 deletions.
diff --git a/...ncludes/work_src/reusage-installation/build-model-interface-shared-artifacts.md b/...ncludes/work_src/reusage-installation/build-model-interface-shared-artifacts.md
@@ -0,0 +1,5 @@
+|OS|Files|
+|--|-----|
+|Linux|`libcatboostmodel.so`|
+|macOS|`libcatboostmodel.dylib`|
+|Windows|`catboostmodel.lib` and `catboostmodel.dll`|
diff --git a/...des/work_src/reusage-installation/build-model-interface-static-old-artifacts.md b/...des/work_src/reusage-installation/build-model-interface-static-old-artifacts.md
@@ -0,0 +1,2 @@
+- `liblibcatboostmodel.o`. This part contains symbols that require forced initialization.
+- `libcatboostmodel.a`. This part contains all other symbols.
diff --git a/catboost/docs/en/_includes/work_src/reusage-installation/catboost-src-root.md b/catboost/docs/en/_includes/work_src/reusage-installation/catboost-src-root.md
@@ -0,0 +1 @@
+Later in this document `$CATBOOST_SRC_ROOT` refers to the root dir of the local working copy of the source code cloned from the GitHub {{ product }} repository.
diff --git a/catboost/docs/en/_includes/work_src/reusage-installation/cmake-platforms.md b/catboost/docs/en/_includes/work_src/reusage-installation/cmake-platforms.md
@@ -0,0 +1,12 @@
+**Host platform** refers to the operating system and CPU architecture you run build on.
+
+**Target platform** refers to the operating system and CPU architecture you run build for (where you intend to run built artifacts like executable CLI application, dynamic library, Python extension library etc.)
+
+Possible host and target platform combinations:
+
+| Host platform | Target platform |
+| ------- | --------- |
+| Linux x86_64 | Linux or Android |
+| Linux non x86_64 | Linux |
+| macOS x86_64 or arm64 | macOS x86_64 or arm64 or universal binaries |
+| Windows x86_64 | Windows x86_64 |
diff --git a/...cs/en/_includes/work_src/reusage-installation/cmake-unix-makefiles-generator.md b/...cs/en/_includes/work_src/reusage-installation/cmake-unix-makefiles-generator.md
@@ -0,0 +1 @@
+`Unix Makefiles` CMake generator usage on macOS and Linux is possible but not recommended because of some issues with properly taking dependencies into account.
diff --git a/...ocs/en/_includes/work_src/reusage-installation/cmake-visual-studio-generator.md b/...ocs/en/_includes/work_src/reusage-installation/cmake-visual-studio-generator.md
@@ -0,0 +1 @@
+Alternatively, on Windows you could also use `Visual Studio 16 2019` generator for CMake. In this case also specify the required toolset version when calling CMake by adding `-T version=14.28` to the command line.
diff --git a/.../docs/en/_includes/work_src/reusage-installation/get-source-code-from-github.md b/.../docs/en/_includes/work_src/reusage-installation/get-source-code-from-github.md
@@ -0,0 +1,3 @@
+{{ product }} source code is stored as a [Git](https://git-scm.com/) repository on GitHub at <https://github.com/catboost/catboost/>. You can obtain a local copy of this Git repository by running the following command from a command line interpreter (you need to have Git command line tools installed):
+
+    {{ installation--git-clone }}
diff --git a/..._includes/work_src/reusage-installation/setup-build-environment-alternatives.md b/..._includes/work_src/reusage-installation/setup-build-environment-alternatives.md
@@ -0,0 +1,10 @@
+{% note warning %}
+
+{% include [ya-make-to-cmake-switch](ya-make-to-cmake-switch.md) %}
+
+Select the appropriate build environment setup below accordingly.
+
+{% endnote %}
+
+  - [For CMake](../../../installation/build-environment-setup-for-cmake.md)
+  - [For Ya Make](../../../installation/build-environment-setup-for-ya-make.md)
diff --git a/catboost/docs/en/_includes/work_src/reusage-installation/windows-build-setup.md b/catboost/docs/en/_includes/work_src/reusage-installation/windows-build-setup.md
@@ -16,43 +16,3 @@
     ```no-highlight
     -gencode arch=compute_20,code=compute_20
     ```
-
-1. {% if audience == "internal" %} {% include [arcadia_users_step](../../../yandex_specific/_includes/arcadia_users_step.md) %} {% endif %} Install [Visual Studio Community 2019](https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2019).
-
-    - Choose the **Windows Platform development** and **Desktop Development with C++** options in the **workloads** tab.
-    - Choose a suitable version of the MSVC compiler. It is advised to install VC++ 2019 version 16.11.11 v14.28 and CUDA Toolkit 11.0 or newer.
-
-    {% note info %}
-
-    Visual Studio forcibly installs the latest version of the compiler upon each update. The latest MSVC compiler may not be suitable for compiling {{ product }}, especially with CUDA.
-
-    {% cut "Identify the set version of the compiler" %}
-
-
-    1. Open the properties window for any `cpp` file of the project.
-    1. Ensure the absence of the `/nologo` option in the compiler's command-line (for example, by adding the deprecated `/nologo-` option in the **Command Line/Additional Options** box).
-    1. Compile this source file (**Ctrl** + **F7**).
-
-    The set version of the compiler is printed to the **Output window**. {{ product }} can not be compiled with 19.14.* versions.
-
-    {% endcut %}
-
-    {% cut "Change the version of the compiler" %}
-
-    Use one of the following methods to set the recommended version of the compiler:
-    - Enable the required version as described in the [Visual C++ Team Blog](https://devblogs.microsoft.com/cppblog/side-by-side-minor-version-msvc-toolsets-in-visual-studio-2019/).
-
-    - Run the environment setter from the command line with the `vcvars_ver` option (the path to the script depends on the installation settings):
-
-    ```bash
-    call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat" -vcvars_ver=14.28
-    ```
-
-    Then open the solution:
-    ```no-highlight
-    start msvs\arcadia.sln
-    ```
-    {% endcut %}
-
-    {% endnote %}
-
diff --git a/...oost/docs/en/_includes/work_src/reusage-installation/ya-make-to-cmake-switch.md b/...oost/docs/en/_includes/work_src/reusage-installation/ya-make-to-cmake-switch.md
@@ -0,0 +1 @@
+{{ product }} uses [CMake](https://cmake.org/)-based build process since [this commit](https://github.com/catboost/catboost/commit/c5c642ca0b8e093336d0229ac4b14c78db3915bb). Previously `Ya Make` (Yandex's build system) had been used.
diff --git a/catboost/docs/en/concepts/build-from-source.md b/catboost/docs/en/concepts/build-from-source.md
@@ -0,0 +1,47 @@
+# Build from source
+
+{% include [get-source-code-from-github](../_includes/work_src/reusage-installation/get-source-code-from-github.md) %}
+
+{% note warning %}
+
+{% include [ya-make-to-cmake-switch](../_includes/work_src/reusage-installation/ya-make-to-cmake-switch.md) %}
+
+Select the appropriate build method below accordingly.
+
+{% endnote %}
+
+## Build using CMake {#build-cmake}
+
+- [Build environment setup](../installation/build-environment-setup-for-cmake.md)
+- [Build native artifacts](../installation/build-native-artifacts.md)
+- By component:
+    - [Build Python package](../installation/python-installation-method-build-from-source.md)
+    - [Build CatBoost for Apache Spark](../installation/spark-installation-build-from-source-maven.md)
+    - Build R package:
+        - [Install from GitHub](../installation/r-installation-github-installation.md)
+{% if audience == "external" %}
+        - [Install from a local Git repository](../installation/r-installation-local-copy-installation.md)
+{% endif %}
+
+{% if audience == "external" %}
+    - [Build Command-line binary](../installation/cli-installation-local-copy-installation.md)
+{% endif %}
+
+## Build using Ya Make (for previous versions) {#build-ya-make}
+
+- [Build environment setup](../installation/build-environment-setup-for-ya-make.md)
+- By component:
+    - Build Python package
+        - [In-place build on Linux and macOS](../installation/python-installation-method-build-from-source-linux-macos-using-ya-make.md)
+        - [In-place build on Windows](../installation/python-installation-method-build-from-source-windows-using-ya-make.md)
+        - [Build a wheel using mk_wheel.py](../installation/python-installation-method-build-a-wheel-package.md#mk-wheel)
+    - [Build CatBoost for Apache Spark](../installation/spark-installation-build-from-source-maven.md)
+    - Build R package:
+        - [Install from GitHub](../installation/r-installation-github-installation.md)
+{% if audience == "external" %}
+        - [Install from a local Git repository](../installation/r-installation-local-copy-installation.md)
+{% endif %}
+
+{% if audience == "external" %}
+    - [Build Command-line binary](../installation/cli-installation-local-copy-installation.md)
+{% endif %}
diff --git a/catboost/docs/en/concepts/c-plus-plus-api_dynamic-c-pluplus-wrapper.md b/catboost/docs/en/concepts/c-plus-plus-api_dynamic-c-pluplus-wrapper.md
@@ -2,72 +2,137 @@
 
 This is the fastest way to evaluate a model. The library provides a [C API](#c-api) and a simple [C++ wrapper API](#c-plus-plus-wrapper). The C API interface can be accessed from any programming language.
 
-Perform the following steps to build the library:
-1. Clone the repository:
+## Build
 
-    ```
-    {{ installation--git-clone }}
-    ```
+{% note warning %}
+
+{% include [ya-make-to-cmake-switch](../_includes/work_src/reusage-installation/ya-make-to-cmake-switch.md) %}
+
+Select the appropriate build method below accordingly.
+
+{% endnote %}
+
+### Source code
+
+{% include [get-source-code-from-github](../_includes/work_src/reusage-installation/get-source-code-from-github.md) %}
+
+### Build using CMake  {#cmake}
+
+{% list tabs %}
+
+- Shared library
+
+    Build `catboostmodel` target.
+
+    See [Build native artifacts](../installation/build-native-artifacts.md).
+
+    Built artifacts will be in `$CMAKE_BINARY_DIR/catboost/libs/model_interface`:
+
+    {% include [build-model-interface-shared-artifacts](../_includes/work_src/reusage-installation/build-model-interface-shared-artifacts.md) %}
+
+- Static library
+
+    Build `catboostmodel_static` target.
+
+    See [Build native artifacts](../installation/build-native-artifacts.md).
+
+    Built library will consist of two parts:
+        - `global` part. This part contains symbols that require forced initialization.
+        - non-`global` part. All other symbols.
+
+    Built artifacts will be in `$CMAKE_BINARY_DIR/catboost/libs/model_interface/static`:
+
+    |OS|Files|
+    |--|-----|
+    |Linux or macOS|`libcatboostmodel_static.a`, `libcatboostmodel_static.global.a`|
+    |Windows|`catboostmodel_static.lib`, `catboostmodel_static.global.lib`|
+
+{% endlist %}
+
+### Build using Ya Make  {#ya-make}
+
+1. [Setup build environment](../installation/build-environment-setup-for-ya-make.md)
 
 1. Open the `catboost` directory from the local copy of the {{ product }} repository.
 
-1. Choose the preferred way to use the evaluation library and compile it accordingly:
+1. Run the following command:
 
     {% list tabs %}
 
     - Shared library
 
-        ```bash
-        ./ya make -r catboost/libs/model_interface
         ```
+        ./ya make -r [optional parameters] catboost/libs/model_interface
+        ```
+
+        The output directory `catboost/libs/model_interface` will contain:
 
-        Or (Linux-only):
+        {% include [build-model-interface-shared-artifacts](../_includes/work_src/reusage-installation/build-model-interface-shared-artifacts.md) %}
 
-        ```bash
-        export CXX=/path/to/clang++
-        export CC=/path/to/clang
+    - Static library (only for Linux and macOS)
 
-        make -f make/model_interface.CLANG50-LINUX-X86_64.makefile
+        ```
+        ./ya make -r [optional parameters] catboost/libs/model_interface/static
         ```
 
-        The output directory `catboost/libs/model_interface` will contain:
+        The output directory `catboost/libs/model_interface/static` will contain a pair of artifacts:
 
-        |OS|Files|
-        |--|-----|
-        |Linux|`libcatboostmodel.so`|
-        |macOS|`libcatboostmodel.dylib`|
-        |Windows|`catboostmodel.lib` and `catboostmodel.dll`|
+        {% include [build-model-interface-static-old-artifacts](../_includes/work_src/reusage-installation/build-model-interface-static-old-artifacts.md) %}
 
-    - Static library (Linux or macOS only)
+    {% endlist %}
 
-        ```bash
-        ya make -r catboost/libs/model_interface/static
-        ```
+    Useful parameters:
 
-        Or (Linux-only):
+    Parameter | Description
+    ----- | -----
+    `-DCUDA_ROOT` | The path to CUDA. This parameter is required to support training on GPU.
+    `-DHAVE_CUDA=no` | Disable CUDA support. This speeds up compilation.<br/><br/>By default, the package is built with CUDA support if CUDA Toolkit is installed.
 
-        ```bash
-        export CXX=/path/to/clang++
-        export CC=/path/to/clang
+### Build using Make (Linux-only)  {#make}
 
-        make -f make/model_interface_static.CLANG50-LINUX-X86_64.makefile
-        ```
+{% note warning %}
 
-        The output directory `catboost/libs/model_interface/static` will contain a pair of artifacts:
+This approach will work only for versions prior to [this commit](https://github.com/catboost/catboost/commit/c5c642ca0b8e093336d0229ac4b14c78db3915bb).
 
-        - `liblibcatboostmodel.o`. This part contains symbols that require forced initialization.
-        - `libcatboostmodel.a`. This part contains all other symbols.
+For newer versions use [Build with CMake](#cmake)
 
-    {% endlist %}
+{% endnote %}
 
-The {{ product }} model can be loaded from a file or initialized from the buffer memory.
+Choose the preferred way to use the evaluation library and compile it accordingly:
 
+{% list tabs %}
 
-#### Related information
+- Shared library
 
-[Source code and a CMake usage example](https://github.com/catboost/catboost/tree/master/catboost/libs/model_interface/cmake_example)
+    ```bash
+    export CXX=/path/to/clang++
+    export CC=/path/to/clang
+
+    make -f make/model_interface.CLANG50-LINUX-X86_64.makefile
+    ```
+
+    The output directory `catboost/libs/model_interface` will contain `libcatboostmodel.so`.
+
+- Static library
+
+    ```bash
+    export CXX=/path/to/clang++
+    export CC=/path/to/clang
+
+    make -f make/model_interface_static.CLANG50-LINUX-X86_64.makefile
+    ```
+
+    The output directory `catboost/libs/model_interface/static` will contain a pair of artifacts:
 
-## C API {#c-api}
+    {% include [build-model-interface-static-old-artifacts](../_includes/work_src/reusage-installation/build-model-interface-static-old-artifacts.md) %}
+
+{% endlist %}
+
+## Usage
+
+The {{ product }} model can be loaded from a file or initialized from the buffer memory.
+
+### C API {#c-api}
 
 Perform the following steps to use this API:
 
@@ -122,7 +187,36 @@ Perform the following steps to use this API:
 
         The shared library must be accessible from the dynamic library loader search path. See your operating system documentation for the details.
 
-    - Static library (Linux or macOS only)
+    - Static library built using CMake
+
+        Add both `global` and non-`global` parts to the linker input. `global` part requires passing special platform-specific flags to force the required initialization of symbols.
+
+        See per-platform examples below:
+
+        - Linux
+
+            On Linux additional libraries `libdl` and `libpthread` have to be added to the linker input as well.
+
+            ```
+            clang++ <your sources and options> -nodefaultlibs -lpthread -ldl -Wl,--whole-archive <catboost_lib_dir>/libcatboostmodel_static.global.a -Wl,--no-whole-archive <catboost_lib_dir>/libcatboostmodel_static.a
+            ```
+
+        - macOS
+
+            ```
+            clang++ <your sources and options> <catboost_lib_dir>/libcatboostmodel_static.a
+            -Wl,-force_load,<catboost_lib_dir>/libcatboostmodel_static.global.a
+            ```
+
+       - Windows
+
+            When using `c_api.h` with the static library the additional define `CATBOOST_API_STATIC_LIB` is required.
+
+            ```
+            cl.exe <your sources and options> /DCATBOOST_API_STATIC_LIB /link /WHOLEARCHIVE:<catboost_lib_dir>\catboostmodel_static.global.lib <catboost_lib_dir>\catboostmodel_static.lib
+            ```
+
+    - Static library built using Ya Make or Make (Linux or macOS only)
 
         Add both `liblibcatboostmodel.o` and  `libcatboostmodel.a` to the linker input.
 
@@ -136,7 +230,7 @@ Perform the following steps to use this API:
 
     {% endlist %}
 
-## C++ wrapper API {#c-plus-plus-wrapper}
+### C++ wrapper API {#c-plus-plus-wrapper}
 
 A C++ wrapper for the C API interface is also available.
 
@@ -151,3 +245,7 @@ std::cout << calcer.Calc(floatFeatures, catFeatures) << std::endl;
 ```
 
 `ModelCalcerWrapper` also has a constructor to read data from the memory buffer.
+
+### Related information
+
+[Source code and a CMake usage example](https://github.com/catboost/catboost/tree/master/catboost/libs/model_interface/cmake_example)