Merge remote-tracking branch 'rapidsai/branch-24.04' into add-ivf_fla…

…t-and-ivf_pq
rapidsai · Mar 18, 2024 · f5aeffc · f5aeffc
2 parents 58dc152 + 964f686
commit f5aeffc
Show file tree

Hide file tree

Showing 77 changed files with 1,563 additions and 1,175 deletions.
diff --git a/.gitignore b/.gitignore
@@ -61,6 +61,7 @@ _xml
 # sphinx
 _html
 _text
+docs/source/_static/rust
 
 # clang tooling
 compile_commands.json

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -79,12 +79,6 @@ repos:
                 verbose: true
                 require_serial: true
                 exclude: .*/thirdparty/.*
-              - id: copyright-check
-                name: copyright-check
-                entry: python ./ci/checks/copyright.py --git-modified-only --update-current-year
-                language: python
-                pass_filenames: false
-                additional_dependencies: [gitpython]
               - id: include-check
                 name: include-check
                 entry: python ./cpp/scripts/include_checker.py cpp/bench cpp/include cpp/test
@@ -107,6 +101,22 @@ repos:
         rev: v4.5.0
         hooks:
               - id: check-json
+      - repo: https://github.com/rapidsai/pre-commit-hooks
+        rev: v0.0.1
+        hooks:
+          - id: verify-copyright
+            files: |
+              (?x)
+                  [.](cmake|cpp|cu|cuh|h|hpp|sh|pxd|py|pyx|rs)$|
+                  CMakeLists[.]txt$|
+                  CMakeLists_standalone[.]txt$|
+                  meta[.]yaml$|
+                  setup[.]cfg$
+            exclude: |
+              (?x)
+                  cpp/include/cuvs/neighbors/detail/faiss_select/|
+                  docs/source/sphinxext/github_link\.py|
+                  cpp/cmake/modules/FindAVX\.cmake|
 
 default_language_version:
       python: python3
diff --git a/README.md b/README.md
@@ -1,5 +1,12 @@
 # <div align="left"><img src="https://rapids.ai/assets/images/rapids_logo.png" width="90px"/>&nbsp;cuVS: Vector Search and Clustering on the GPU</div>
 
+> [!note]
+> cuVS is a new library mostly derived from the approximate nearest neighbors and clustering algorithms in the [RAPIDS RAFT](https://github.com/rapidsai) library of data mining primitives. RAPIDS RAFT currently contains the most fully-featured versions of the approximate nearest neighbors and clustering algorithms in cuVS. We are in the process of migrating the algorithms from RAFT to cuVS, but if you are unsure of which to use, please consider the following:
+> 1. RAFT contains C++ and Python APIs for all of the approximate nearest neighbors and clustering algorithms.
+> 2. cuVS contains a growing support for different languages, including C, C++, Python, and Rust. We will be adding more language support to cuVS in the future but will not be improving the language support for RAFT.
+> 3. Once all of RAFT's approximate nearest neighbors and clustering algorithms are moved to cuVS, the RAFT APIs will be deprecated and eventually removed altogether. Once removed, RAFT will become a lightweight header-only library. In the meantime, there's no harm in using RAFT if support for additional languages is not needed.
+
+
 ## Contents
 
 1. [Useful Resources](#useful-resources)
@@ -11,10 +18,10 @@
 
 ## Useful Resources
 
-- [cuVS Reference Documentation](https://docs.rapids.ai/api/cuvs/stable/): API Documentation.
-- [cuVS Getting Started](./docs/source/quick_start.md): Getting started with RAFT.
-- [Build and Install cuVS](./docs/source/build.md): Instructions for installing and building cuVS.
-- [Example Notebooks](./notebooks): Example jupyer notebooks
+- [Code Examples](https://github.com/rapidsai/cuvs/tree/HEAD/examples): Self-contained Code Examples.
+- [API Reference Documentation](https://docs.rapids.ai/api/cuvs/nightly/api_docs): API Documentation.
+- [Getting Started Guide](https://docs.rapids.ai/api/cuvs/nightly/getting_started): Getting started with RAFT.
+- [Build and Install Guide](https://docs.rapids.ai/api/cuvs/nightly/build): Instructions for installing and building cuVS.
 - [RAPIDS Community](https://rapids.ai/community.html): Get help, contribute, and collaborate.
 - [GitHub repository](https://github.com/rapidsai/cuvs): Download the cuVS source code.
 - [Issue tracker](https://github.com/rapidsai/cuvs/issues): Report issues or request features.
@@ -23,18 +30,13 @@
 
 cuVS contains state-of-the-art implementations of several algorithms for running approximate nearest neighbors and clustering on the GPU. It can be used directly or through the various databases and other libraries that have integrated it. The primary goal of cuVS is to simplify the use of GPUs for vector similarity search and clustering.
 
-**Please note** that cuVS is a new library mostly derived from the approximate nearest neighbors and clustering algorithms in the [RAPIDS RAFT](https://github.com/rapidsai) library of data mining primitives. RAPIDS RAFT currently contains the most fully-featured versions of the approximate nearest neighbors and clustering algorithms in cuVS. We are in the process of migrating the algorithms from RAFT to cuVS, but if you are unsure of which to use, please consider the following:
-1. RAFT contains C++ and Python APIs for all of the approximate nearest neighbors and clustering algorithms.
-2. cuVS contains a growing support for different languages, including C, C++, Python, and Rust. We will be adding more language support to cuVS in the future but will not be improving the language support for RAFT.
-3. Once all of RAFT's approximate nearest neighbors and clustering algorithms are moved to cuVS, the RAFT APIs will be deprecated and eventually removed altogether. Once removed, RAFT will become a lightweight header-only library. In the meantime, there's no harm in using RAFT if support for additional languages is not needed.
-
 ## Installing cuVS
 
 cuVS comes with pre-built packages that can be installed through [conda](https://conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-python). Different packages are available for the different languages supported by cuVS:
 
-| Python | C++ | C | Rust |
-|--------|-----|---|------|
-| `pycuvs`| `libcuvs` | `libcuvs_c` | `cuvs-rs` |
+| Python | C++ | C |
+|--------|-----|---|
+| `pycuvs`| `libcuvs` | `libcuvs_c` |
 
 ### Stable release
 
@@ -170,7 +172,7 @@ fn cagra_example() -> Result<()> {
 
 ## Contributing
 
-If you are interested in contributing to the cuVS library, please read our [Contributing guidelines](docs/source/contributing.md). Refer to the [Developer Guide](docs/source/developer_guide.md) for details on the developer guidelines, workflows, and principals.
+If you are interested in contributing to the cuVS library, please read our [Contributing guidelines](docs/source/contributing.md). Refer to the [Developer Guide](docs/source/developer_guide.md) for details on the developer guidelines, workflows, and principles.
 
 ## References
 

diff --git a/build.sh b/build.sh
@@ -18,13 +18,14 @@ ARGS=$*
 # scripts, and that this script resides in the repo dir!
 REPODIR=$(cd $(dirname $0); pwd)
 
-VALIDARGS="clean libcuvs python docs tests examples clean --uninstall  -v -g -n --compile-static-lib --allgpuarch --no-nvtx --show_depr_warn --incl-cache-stats --time -h"
+VALIDARGS="clean libcuvs python rust docs tests examples --uninstall  -v -g -n --compile-static-lib --allgpuarch --no-nvtx --show_depr_warn --incl-cache-stats --time -h"
 HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<tool>] [--limit-tests=<targets>] [--build-metrics=<filename>]
  where <target> is:
    clean            - remove all existing build artifacts and configuration (start over)
    libcuvs          - build the cuvs C++ code only. Also builds the C-wrapper library
                       around the C++ code.
    python           - build the cuvs Python package
+   rust             - build the cuvs Rust bindings
    docs             - build the documentation
    tests            - build the tests
    examples         - build the examples
@@ -48,13 +49,14 @@ HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<to
                                  Results can be interpreted with cpp/scripts/analyze_nvcc_log.py
    -h                          - print this text
 
- default action (no args) is to build libcuvs, tests, cuvs and cuvs-dask targets
+ default action (no args) is to build libcuvs, tests and cuvs targets
 "
 LIBCUVS_BUILD_DIR=${LIBCUVS_BUILD_DIR:=${REPODIR}/cpp/build}
 SPHINX_BUILD_DIR=${REPODIR}/docs
 DOXYGEN_BUILD_DIR=${REPODIR}/cpp/doxygen
 PYTHON_BUILD_DIR=${REPODIR}/python/cuvs/_skbuild
-BUILD_DIRS="${LIBCUVS_BUILD_DIR} ${PYTHON_BUILD_DIR} ${CUVS_DASK_BUILD_DIR}"
+RUST_BUILD_DIR=${REPODIR}/rust/target
+BUILD_DIRS="${LIBCUVS_BUILD_DIR} ${PYTHON_BUILD_DIR} ${RUST_BUILD_DIR}"
 
 # Set defaults for vars modified by flags to this script
 CMAKE_LOG_LEVEL=""
@@ -76,6 +78,7 @@ CLEAN=0
 UNINSTALL=0
 DISABLE_DEPRECATION_WARNINGS=ON
 CMAKE_TARGET=""
+EXTRA_CMAKE_ARGS=""
 
 # Set defaults for vars that may not have been defined externally
 INSTALL_PREFIX=${INSTALL_PREFIX:=${PREFIX:=${CONDA_PREFIX:=$LIBCUVS_BUILD_DIR/install}}}
@@ -218,28 +221,6 @@ if hasArg --uninstall; then
         echo "Could not uninstall cuvs from pip or conda. cuvs package will need to be manually uninstalled"
       fi
     fi
-
-    if hasArg cuvs-dask || (( ${NUMARGS} == 1 )); then
-      echo "Uninstalling cuvs-dask package..."
-      if [ -e ${CUVS_DASK_BUILD_DIR}/install_manifest.txt ]; then
-          xargs rm -fv < ${CUVS_DASK_BUILD_DIR}/install_manifest.txt > /dev/null 2>&1
-      fi
-
-      # Try to uninstall via pip if it is installed
-      if [ -x "$(command -v pip)" ]; then
-        echo "Using pip to uninstall cuvs-dask"
-        pip uninstall -y cuvs-dask
-
-      # Otherwise, try to uninstall through conda if that's where things are installed
-      elif [ -x "$(command -v conda)" ] && [ "$INSTALL_PREFIX" == "$CONDA_PREFIX" ]; then
-        echo "Using conda to uninstall cuvs-dask"
-        conda uninstall -y cuvs-dask
-
-      # Otherwise, fail
-      else
-        echo "Could not uninstall cuvs-dask from pip or conda. cuvs-dask package will need to be manually uninstalled."
-      fi
-    fi
     exit 0
 fi
 
@@ -261,24 +242,14 @@ if hasArg --allgpuarch; then
     BUILD_ALL_GPU_ARCH=1
 fi
 
-
-# Append `-DFIND_CUVS_CPP=ON` to EXTRA_CMAKE_ARGS unless a user specified the option.
-if [[ "${EXTRA_CMAKE_ARGS}" != *"DFIND_CUVS_CPP"* ]]; then
-    EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DFIND_CUVS_CPP=ON"
-fi
-
 if hasArg tests || (( ${NUMARGS} == 0 )); then
     BUILD_TESTS=ON
     CMAKE_TARGET="${CMAKE_TARGET};${TEST_TARGETS}"
 
     # Force compile library when needed test targets are specified
-    if [[ $CMAKE_TARGET == *"CLUSTER_TEST"* || \
-          $CMAKE_TARGET == *"DISTANCE_TEST"* || \
-          $CMAKE_TARGET == *"NEIGHBORS_ANN_CAGRA_TEST"* || \
-          $CMAKE_TARGET == *"NEIGHBORS_ANN_IVF_TEST"* || \
-          $CMAKE_TARGET == *"NEIGHBORS_ANN_NN_DESCENT_TEST"* || \
-          $CMAKE_TARGET == *"NEIGHBORS_TEST"* || \
-          $CMAKE_TARGET == *"STATS_TEST"* ]]; then
+    if [[ $CMAKE_TARGET == *"CAGRA_C_TEST"* || \
+          $CMAKE_TARGET == *"INTEROP_TEST"* || \
+          $CMAKE_TARGET == *"NEIGHBORS_ANN_CAGRA_TEST"* ]]; then
       echo "-- Enabling compiled lib for gtests"
       COMPILE_LIBRARY=ON
     fi
@@ -304,13 +275,10 @@ if [[ ${CMAKE_TARGET} == "" ]]; then
     CMAKE_TARGET="all"
 fi
 
-# Append `-DFIND_CUVS_CPP=ON` to EXTRA_CMAKE_ARGS unless a user specified the option.
-
-
 
 SKBUILD_EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS}"
 if [[ "${EXTRA_CMAKE_ARGS}" != *"DFIND_CUVS_CPP"* ]]; then
-    SKBUILD_EXTRA_CMAKE_ARGS="${SKBUILD_EXTRA_CMAKE_ARGS} -DFIND_CUVS_CPP=ON"
+    SKBUILD_EXTRA_CMAKE_ARGS="${SKBUILD_EXTRA_CMAKE_ARGS};-DFIND_CUVS_CPP=ON"
 fi
 
 # If clean given, run it prior to any other steps
@@ -353,7 +321,6 @@ if (( ${NUMARGS} == 0 )) || hasArg libcuvs || hasArg docs || hasArg tests || has
           -DCMAKE_INSTALL_PREFIX=${INSTALL_PREFIX} \
           -DCMAKE_CUDA_ARCHITECTURES=${CUVS_CMAKE_CUDA_ARCHITECTURES} \
           -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
-          -DCUVS_COMPILE_LIBRARY=${COMPILE_LIBRARY} \
           -DBUILD_C_LIBRARY=${COMPILE_LIBRARY} \
           -DCUVS_NVTX=${NVTX} \
           -DCUDA_LOG_COMPILE_TIME=${LOG_COMPILE_TIME} \
@@ -422,24 +389,37 @@ fi
 
 # Build and (optionally) install the cuvs Python package
 if (( ${NUMARGS} == 0 )) || hasArg python; then
-    SKBUILD_CONFIGURE_OPTIONS="${SKBUILD_EXTRA_CMAKE_ARGS}" \
+    SKBUILD_CMAKE_ARGS="${SKBUILD_EXTRA_CMAKE_ARGS}" \
         SKBUILD_BUILD_OPTIONS="-j${PARALLEL_LEVEL}" \
-        python -m pip install --no-build-isolation --no-deps ${REPODIR}/python/cuvs
+        python -m pip install --no-build-isolation --no-deps -vvv ${REPODIR}/python/cuvs
 fi
 
+# Build the cuvs Rust bindings
+if (( ${NUMARGS} == 0 )) || hasArg rust; then
+    cd ${REPODIR}/rust
+    cargo build --examples --lib
+    cargo test
+fi
+
+export RAPIDS_VERSION="$(sed -E -e 's/^([0-9]{2})\.([0-9]{2})\.([0-9]{2}).*$/\1.\2.\3/' "${REPODIR}/VERSION")"
+export RAPIDS_VERSION_MAJOR_MINOR="$(sed -E -e 's/^([0-9]{2})\.([0-9]{2})\.([0-9]{2}).*$/\1.\2/' "${REPODIR}/VERSION")"
+
 if hasArg docs; then
     set -x
     cd ${DOXYGEN_BUILD_DIR}
     doxygen Doxyfile
     cd ${SPHINX_BUILD_DIR}
     sphinx-build -b html source _html
+    cd ${REPODIR}/rust
+    cargo doc -p cuvs --no-deps
+    rsync -av ${RUST_BUILD_DIR}/doc/ ${SPHINX_BUILD_DIR}/_html/_static/rust
 fi
 
 ################################################################################
 # Initiate build for c++ examples (if needed)
 
 if hasArg examples; then
-    pushd ${REPODIR}/cpp/examples
+    pushd ${REPODIR}/examples/cpp
     ./build.sh
     popd
 fi
diff --git a/ci/build_docs.sh b/ci/build_docs.sh
@@ -12,19 +12,27 @@ rapids-dependency-file-generator \
   --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
 
 rapids-mamba-retry env create --force -f env.yaml -n docs
+
+# seeing failures on activating the environment here on unbound locals
+# apply workaround from https://github.com/conda/conda/issues/8186#issuecomment-532874667
+set +eu
 conda activate docs
+set -eu
 
 rapids-print-env
 
 rapids-logger "Downloading artifacts from previous jobs"
 CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
-#PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)
+PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)
 
 rapids-mamba-retry install \
   --channel "${CPP_CHANNEL}" \
-  libcuvs
+  --channel "${PYTHON_CHANNEL}" \
+  libcuvs cuvs
 
-export RAPIDS_VERSION_NUMBER="24.04"
+export RAPIDS_VERSION="$(rapids-version)"
+export RAPIDS_VERSION_MAJOR_MINOR="$(rapids-version-major-minor)"
+export RAPIDS_VERSION_NUMBER="$RAPIDS_VERSION_MAJOR_MINOR"
 export RAPIDS_DOCS_DIR="$(mktemp -d)"
 
 rapids-logger "Build CPP docs"

diff --git a/ci/build_rust.sh b/ci/build_rust.sh
@@ -35,6 +35,5 @@ rapids-mamba-retry install \
   libcuvs  \
   libraft
 
-# build and test the rust bindings
-cd rust
-cargo test
+export EXTRA_CMAKE_ARGS=""
+bash ./build.sh rust
diff --git a/ci/build_wheel_cuvs.sh b/ci/build_wheel_cuvs.sh
@@ -4,6 +4,6 @@
 set -euo pipefail
 
 # Set up skbuild options. Enable sccache in skbuild config options
-export SKBUILD_CONFIGURE_OPTIONS="-DDETECT_CONDA_ENV=OFF -DFIND_CUVS_CPP=OFF"
+export SKBUILD_CMAKE_ARGS="-DDETECT_CONDA_ENV=OFF -DFIND_CUVS_CPP=OFF"
 
 ci/build_wheel.sh cuvs python/cuvs