Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cuDF devcontainers #14015

Merged
merged 98 commits into from
Nov 20, 2023
Merged

Conversation

trxcllnt
Copy link
Contributor

@trxcllnt trxcllnt commented Aug 31, 2023

Description

This PR adds some devcontainers to help simplify building the cuDF C++ and Python libraries.

It also adds an optional job to the pr.yaml to build the cuDF libs in each devcontainer, so the build caches are populated for devs by CI.

A devcontainer can be launched by clicking the "Reopen in Container" button that VSCode shows when opening the repo (or by using the "Rebuild and Reopen in Container" command from the command palette):
image

Clicking this button will cause VSCode to prompt the user to select one of these devcontainer variants:
image

On startup, the devcontainer creates or updates the conda/pip environment using cudf/dependencies.yaml. The envs/package caches are cached on the host via volume mounts, which are described in more detail in .devcontainer/README.md.

The container includes convenience functions to clean, configure, and build the various cuDF components:

$ clean-cudf-cpp # only cleans the C++ build dir
$ clean-cudf-python # only cleans the Python build dir
$ clean-cudf # cleans both C++ and Python build dirs

$ configure-cudf-cpp # only configures cudf C++ lib

$ build-cudf-cpp # only builds cudf C++ lib
$ build-cudf-python # only builds cudf Python lib
$ build-cudf # builds both C++ and Python libs
  • The C++ build script is a small wrapper around cmake -S ~/cudf/cpp -B ~/cudf/cpp/build and cmake --build ~/cudf/cpp/build
  • The Python build script is a small wrapper around pip install --editable ~/cudf/cpp

Unlike build.sh, these convenience scripts don't install the libraries after building them. Instead, they automatically inject the correct arguments to build the C++ libraries from source and use their build dirs as package roots:

$ cmake -S ~/cudf/cpp -B ~/cudf/cpp/build
$ CMAKE_ARGS="-Dcudf_ROOT=~/cudf/cpp/build" \ # <-- this argument is automatic
  pip install -e ~/cudf/cpp

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@trxcllnt
Copy link
Contributor Author

/ok to test

@trxcllnt
Copy link
Contributor Author

/ok to test

rapids-bot bot pushed a commit to rapidsai/cuspatial that referenced this pull request Sep 25, 2023
Removes the unified and isolated devcontainers in favor of just the single one. The unified devcontainers are [still available](https://github.com/rapidsai/devcontainers/blob/d970ac17a894f5d1047fcfd17cffa6d14fe66ec8/.devcontainer/cuda12.0-conda/devcontainer.json#L23-L31) in the [`rapidsai/devcontainers`](https://github.com/rapidsai/devcontainers) repository.


Adds an optional job to the `pr.yaml` to [build the cuGraph libs in each devcontainer](https://github.com/trxcllnt/cuspatial/blob/fea/devcontainers/.github/workflows/pr.yaml#L106-L111), so the build caches are populated for devs by CI.

Other PRs:

* rapidsai/rmm#1328
* rapidsai/kvikio#273
* rapidsai/cudf#14015
* rapidsai/raft#1791
* rapidsai/cumlprims_mg#149
* rapidsai/cuml#5568
* rapidsai/cugraph-ops#538
* rapidsai/cugraph#3838

Authors:
  - Paul Taylor (https://github.com/trxcllnt)
  - H. Thomson Comer (https://github.com/thomcom)

Approvers:
  - Mark Harris (https://github.com/harrism)
  - Jake Awe (https://github.com/AyodeAwe)

URL: #1263
@trxcllnt
Copy link
Contributor Author

/ok to test

@bdice bdice changed the base branch from branch-23.10 to branch-24.02 November 11, 2023 15:59
@trxcllnt
Copy link
Contributor Author

/ok to test

@trxcllnt
Copy link
Contributor Author

/ok to test

@bdice
Copy link
Contributor

bdice commented Nov 14, 2023

We'll have to wait a bit for some of cudf's dependencies to have nightly builds. It's a bit late for this to target 23.12 but we'll be able to finalize this early in the 24.02 development cycle. In the meantime, I'm testing out the unified devcontainers with version 23.12 of everything.

@github-actions github-actions bot removed the Python Affects Python cuDF API. label Nov 16, 2023
@bdice
Copy link
Contributor

bdice commented Nov 16, 2023

/ok to test

@bdice
Copy link
Contributor

bdice commented Nov 17, 2023

/ok to test

@bdice
Copy link
Contributor

bdice commented Nov 17, 2023

/ok to test

.devcontainer/Dockerfile Outdated Show resolved Hide resolved
@bdice
Copy link
Contributor

bdice commented Nov 19, 2023

/ok to test

@bdice
Copy link
Contributor

bdice commented Nov 19, 2023

/merge

@bdice
Copy link
Contributor

bdice commented Nov 20, 2023

Blocked by #14452. I requested an admin-merge in the meantime.

@AyodeAwe AyodeAwe merged commit 58387ff into rapidsai:branch-24.02 Nov 20, 2023
66 of 67 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Ready for Review Ready for review by team CMake CMake build issue improvement Improvement / enhancement to an existing function libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

5 participants