-
-
Notifications
You must be signed in to change notification settings - Fork 8.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[CI] Add nightly CI job to test against dev version of deps (#10351)
* [CI] Add nightly CI job to test against dev version of deps * Update build-containers.sh * Add build step * Wait for build artifact * Try pinning dask * Address reviewers' comments * Fix unbound variable error * Specify dev version exactly * Pin dask=2024.1.1
- Loading branch information
Showing
14 changed files
with
120 additions
and
18 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
# Nightly CI pipeline, to test against dev versions of dependencies | ||
|
||
env: | ||
DOCKER_CACHE_ECR_ID: "492475357299" | ||
DOCKER_CACHE_ECR_REGION: "us-west-2" | ||
DISABLE_RELEASE: "1" | ||
# Skip uploading artifacts to S3 bucket | ||
# Also, don't build all CUDA archs; just build sm_75 | ||
USE_DEPS_DEV_VER: "1" | ||
# Use dev versions of RAPIDS and other dependencies | ||
steps: | ||
#### -------- CONTAINER BUILD -------- | ||
- label: ":docker: Build containers" | ||
commands: | ||
- "tests/buildkite/build-containers.sh gpu_build_centos7" | ||
- "tests/buildkite/build-containers.sh gpu_dev_ver" | ||
key: build-containers | ||
agents: | ||
queue: linux-amd64-cpu | ||
- wait | ||
|
||
- label: ":console: Build CUDA" | ||
command: "tests/buildkite/build-cuda.sh" | ||
key: build-cuda | ||
agents: | ||
queue: linux-amd64-cpu | ||
- wait | ||
- label: ":console: Test Python package, single GPU" | ||
command: "tests/buildkite/test-python-gpu.sh gpu" | ||
key: test-python-gpu | ||
agents: | ||
queue: linux-amd64-gpu | ||
- label: ":console: Test Python package, 4 GPUs" | ||
command: "tests/buildkite/test-python-gpu.sh mgpu" | ||
key: test-python-mgpu | ||
agents: | ||
queue: linux-amd64-mgpu |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
# Container to test XGBoost against dev versions of dependencies | ||
|
||
ARG CUDA_VERSION_ARG | ||
FROM nvidia/cuda:$CUDA_VERSION_ARG-runtime-ubuntu22.04 | ||
ARG CUDA_VERSION_ARG | ||
ARG RAPIDS_VERSION_ARG | ||
# Should be first 4 digits of the dev version (e.g. 24.06) | ||
ARG NCCL_VERSION_ARG | ||
|
||
# Environment | ||
ENV DEBIAN_FRONTEND noninteractive | ||
SHELL ["/bin/bash", "-c"] # Use Bash as shell | ||
|
||
# Install all basic requirements | ||
RUN \ | ||
apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub && \ | ||
apt-get update && \ | ||
apt-get install -y wget unzip bzip2 libgomp1 build-essential openjdk-8-jdk-headless && \ | ||
# Python | ||
wget -nv -O conda.sh https://github.com/conda-forge/miniforge/releases/download/24.3.0-0/Miniforge3-24.3.0-0-Linux-x86_64.sh && \ | ||
bash conda.sh -b -p /opt/mambaforge | ||
|
||
ENV PATH=/opt/mambaforge/bin:$PATH | ||
|
||
# Create new Conda environment with dev versions of cuDF, Dask, and cuPy | ||
RUN \ | ||
export NCCL_SHORT_VER=$(echo "$NCCL_VERSION_ARG" | cut -d "-" -f 1) && \ | ||
mamba create -y -n gpu_test -c rapidsai-nightly -c conda-forge -c nvidia \ | ||
python=3.10 "cudf=$RAPIDS_VERSION_ARG.*" "rmm=$RAPIDS_VERSION_ARG.*" cudatoolkit=$CUDA_VERSION_ARG \ | ||
"nccl>=${NCCL_SHORT_VER}" \ | ||
dask=2024.1.1 \ | ||
"dask-cuda=$RAPIDS_VERSION_ARG.*" "dask-cudf=$RAPIDS_VERSION_ARG.*" cupy \ | ||
numpy pytest pytest-timeout scipy scikit-learn pandas matplotlib wheel python-kubernetes urllib3 graphviz hypothesis \ | ||
"pyspark>=3.4.0" cloudpickle cuda-python && \ | ||
mamba clean --all --yes && \ | ||
conda run --no-capture-output -n gpu_test pip install buildkite-test-collector | ||
|
||
ENV GOSU_VERSION 1.10 | ||
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/ | ||
|
||
# Install lightweight sudo (not bound to TTY) | ||
RUN set -ex; \ | ||
wget -nv -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-amd64" && \ | ||
chmod +x /usr/local/bin/gosu && \ | ||
gosu nobody true | ||
|
||
# Default entry-point to use if running locally | ||
# It will preserve attributes of created files | ||
COPY entrypoint.sh /scripts/ | ||
|
||
WORKDIR /workspace | ||
ENTRYPOINT ["/scripts/entrypoint.sh"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters