Repository overhaul for RAPIDS v23.08

[raydouglass]

docker repo (RAPIDS end-user images) overhaul

This issue outlines how the RAPIDS end-user images in the docker repo will be overhauled to improve functionality and performance.

Images

Requirements

These are the requirements for all images produced by this repo.

1. Must use multi-architectural images that publish amd64 and aarch64/arm64 images to the same docker repository
   a. This eliminates the arm64 variant repositories such as https://hub.docker.com/r/rapidsai/rapidsai-core-arm64
2. Must install all conda package into the base conda environment
3. Default user must be non-root user named rapids
4. Must define the WORKDIR as the rapids user's home directory
5. Must use a tag in the format: ${RAPIDS_VER}-cuda${CUDA_VER}-py${PYTHON_VER}
   a. Nightly builds will append a to the RAPIDS_VER (eg 23.08a)
6. Must support the current matrix of CUDA and Python versions (currently defined here)

Considerations

1. Consider defining any dependencies not inherited from RAPIDS libraries in a dependencies.yaml consumable by dependency-file-generator
   a. If there aren't many, might be simpler to directly install them (eg mamba install rapids ipython)
2. Consider implementing sudoers allowing the rapids user to sudo

Base image

This image is the basic docker image that installs the RAPIDS suite of libraries.

Requirements

1. Must FROM rapidsai/mambaforge-cuda
2. Must use default CMD to launch an ipython REPL
3. Must install the rapids meta package
4. Must use dask/distributed/dask-sql nightly conda builds for nightly image builds
5. Must include ipython dependency
6. Must be published to rapidsai/base

Notebooks image

This image extends the base image to include Jupyter and RAPIDS notebooks.

Requirements

1. Must FROM the base image define above
2. Must use default CMD to launch (in the foreground) a jupyter server
3. Must install the required "notebook" dependencies from all RAPIDS repositories (eg https://github.com/rapidsai/cudf/blob/branch-23.04/dependencies.yaml#L304)
4. Must clone the RAPIDS notebooks into the WORKDIR
5. Must be published to rapidsai/notebooks

Implementation

Building the base and notebooks images should utilize a multi-stage docker build.

Stage 1

1. Clone RAPIDS repositories for the current branch
2. Generate conda environment files for each repositories' "notebook" dependencies
3. Merge the conda environment files
4. Collect the notebooks directory of each RAPIDS repository and associated datasets if required

Stage 2

1. Build the base image defined above

Stage 3

1. Pull the merged conda environment file from Stage 1
2. Copy the notebooks directories from Stage 1
3. Build the notebooks image defined above using the merged conda environment file for dependencies

Testing

Base image

The base image will be tested using basic RAPIDS smoke tests being developed in their respective repositories. The full suite of unit tests will not be run.

Notebooks image

The notebooks image will be tested using the "notebook tests" which basically executes the notebooks provided by each repositories (consolidated in the notebooks repo).

It will be necessary to standardize the mechanism for testing each notebook and for exempting a notebook from testing. cugraph provides a reliable way for doing this: https://github.com/rapidsai/cugraph/blob/branch-23.04/ci/notebook_list.py.

Breaking changes

These are the breaking changes that will occur when v23.06 of this docker repository is released:

1. rapidsai/rapidsai-core is deprecated and superceded by rapidsai/base and rapidsai/notebooks
2. "Runtime" images (eg rapidsai/rapidsai:23.02-cuda11.8-runtime-ubuntu22.04-py3.10) are replaced by the rapidsai/notebooks docker repo
3. IMAGE_TYPE & LINUX_VER will be removed from image tags. Change ${RAPIDS_VER}-cuda${CUDA_VER}-${IMAGE_TYPE}-${LINUX_VER}-py${PYTHON_VER} to ${RAPIDS_VER}-cuda${CUDA_VER}-py${PYTHON_VER}
4. arm64 image repositories (eg rapidsai/rapidsai-arm64) are deprecated and superceded by multi-architectural images in rapidsai/base and rapidsai/notebooks
5. Images will not longer default to the root user. Instead they will use non-root user rapids
   a. To use previous behavior, supply --user root to the docker run command
6. The base image will start an ipython REPL instead of a bash shell
   a. To use previous behavior, supply a docker run CMD of bash (eg docker run -it ${IMAGE} bash)
7. The notebooks image will start jupyter-lab in the foreground instead of running in the background and starting a bash shell.
   a. To use previous behavior, supply a docker run CMD of bash (eg docker run -it ${IMAGE} bash) and run jupyter-lab in the background
8. The notebooks directory in the notebooks image will not longer contain the full source code git clone of RAPIDS repositories
   a. To use previous behavior, start the container and git clone ${RAPIDS_GIT_URL} for each required RAPIDS repository
9. The rapids conda environment is removed and replaced by the base conda environment
10. The environment variables EXTRA_APT_PACKAGES & EXTRA_YUM_PACKAGES will not longer be supported.
11. The devel images will be removed and replaced by devcontainers (such as Add cuspatial devcontainers cuspatial#960 for example)
12. LINUX_VER will be removed from the image tags. The Ubuntu version is implied by the CUDA version.
13. The notebooks directory is moved from /rapids/notebooks to /home/rapids/notebooks
14. centos7 & rockylinux8 OSes are dropped.
    a. Users can manually build the images with these OSes by supplying --build-arg LINUX_VER=centos7, but this not tested/supported
15. Nightly images are no longer uploaded to a separate docker repo. They are uploaded to rapidsai/base and rapidsai-notebooks with an alpha tag for the RAPIDS_VER (eg 23.08a).

Deprecated docker repositories

These docker repositories will be deprecated:

• https://hub.docker.com/r/rapidsai/rapidsai
• https://hub.docker.com/r/rapidsai/rapidsai-core
• https://hub.docker.com/r/rapidsai/rapidsai-core-arm64
• https://hub.docker.com/r/rapidsai/rapidsai-core-dev
• https://hub.docker.com/r/rapidsai/rapidsai-core-dev-arm64
• https://hub.docker.com/r/rapidsai/rapidsai-core-dev-nightly
• https://hub.docker.com/r/rapidsai/rapidsai-core-dev-nightly-arm64
• https://hub.docker.com/r/rapidsai/rapidsai-core-nightly
• https://hub.docker.com/r/rapidsai/rapidsai-core-nightly-arm64
• https://hub.docker.com/r/rapidsai/rapidsai-dev
• https://hub.docker.com/r/rapidsai/rapidsai-dev-nightly
• https://hub.docker.com/r/rapidsai/rapidsai-nightly

nightly and arm64 repos are folded into the rapidsai/base & rapidsai/notebooks repos. The dev/devel repos are replaced by devcontainers.

rapidsai/rapidsai-core is replaced by rapidsai/base

rapidsai/rapidsai is replaced by rapidsai/notebooks

[raydouglass]

Added a Breaking changes section

[aravenel]

Note that this change is now being targeted for RAPIDS 23.08 release.