This repository has been archived by the owner on Nov 17, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 6.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
) * Refactor CD to support newer cuda versions (11.0-11.7) (#21123) * WIP to add cuda build versions. * WIP to add cuda build versions. * Remove sudo install, moved to CD specific dockerfile. * Allow passing of linker flags for Distribution build type. * Update distribution cmake configs, add new configs for newer cuda versions. * Update cuda versions to build in CD. * Update base images for GPU, add new Cuda 11.6 container. * Correctly set LD_LIBRARY_PATH. * Provide cmake hints in dependency install scripts. * Refactor Cuda dependency installation to simplify and support newer versions. * Add new Dockerfile for CD builds. * Use new CD-specific container for building MXNet static library. * Add Cuda verions. * Upgrade to Python 3.8 in CentOS 7 containers. * Update base images. * Install requirements only if file exists. * Clean up dockerfile * Do not pin Cython, relax scipy version. * Install all build dependencies. * Add documentation. * Add documentation. * Set LD_LIBRARY_PATH to include stubs. * Build cmake from source for portability. * Install hdf5 headers during python install, as it is required for h5py module. * Install any dependencies via yum for cmake build. * Update libtiff to version that builds on aarch64. * Build libtiff and protobuf from source so we can statically link mxnet on aarch64. * Change centos7_aarch64_cpu container to install software using common scripts for consistency. Remove installing protobuf and other depedency libraries so we properly statically link to them. * Install pre-built cmake packages. * Use common method to install cmake. * Update pipelines to use supported cuda versions for static build tests. * Ensure required build tools are installed. * Install required headers for building all R packages. * Add/update make configs for newer Cuda versions. * Install gfortran as build dependency in CD image. * Use ldd to find actual path of dynamically linked libraries instead of guessing. * Add additional Cuda versions for CI testing. * Set minimum OSX version to support via C/CXXFLAGS to match what we build MXNet for. * Don't specify minimum OS version when building MXNet for OSX. * Turns out we can set target OSX version, but recently libtiff introduced zstd support, which doesn't link properly. Disabling support via --disable-zstd works. * Disable zlib, as it was previously. * Disable webp support in libtiff (present only in newer version) * [v1.9.x] Refactor dockerfiles in CI, migrate some ubuntu docker containers to use docker-compose. Update CI to use Cuda 11.7 (#21126) * Remove deprecated dockerfiles. * Update documentation to use different image. * Install Scala in centos7 CD container and build tools. * Update static scala build to use CD container, change julia container. * Removed deprecated Jenkins pipeline files, remove old disabled build steps. * Add new base Dockerfile for docker-compose. * Migrate ubuntu cuda containers to docker-compose. * Build python from source on ubuntu for portability. * Remove old dockerfiles, upgrade nightly gpu image to cuda 11.7. * Remove Cuda versions from runtime function names to simplify. * Update Jenkins pipelines to use newer Cuda containers. * Install LLVM before TVM. * Fix ubuntu TVM install script (was failing but returning true.) * Move cmake install into unified script. * Move cmake install for ubuntu into centralized script. * Update cudnn version passed to builds. * Consolidate installation of packages for efficiency. * Remove unused containers from docker-compose config. * Fix pylint. * Set LD_LIBRARY_PATH on ubuntu_gpu images to find libcuda.so. * Set CUB_IGNORE_DEPRECATED_CPP_DIALECT to prevent build failures with gcc-4.8 + Cuda 11.7. * Install sqlite headers/library before building python on ubuntu. * Revert "Remove unused containers from docker-compose config." This reverts commit 5de82df. * Revert "Set CUB_IGNORE_DEPRECATED_CPP_DIALECT to prevent build failures with gcc-4.8 + Cuda 11.7." This reverts commit e649660. * Allow building CUB with c++11 to prevent failures on newer cuda versions. * Set variable only on gpu make builds. * Use docker-compose to also build ubuntu_cpu image. * We no longer need to enable python3.8 on aarch64 since we are building from source now. * Add Cuda 11.1 and 11.3 centos7 images which is used by CD testing phase. * Don't install python-opencv, we are installing the module via pip instead. * Change Makefile to set CUB_IGNORE_DEPRECATED_CPP_DIALECT when using Cuda, not only for < 11.0. * Don't pin down h5py (old versions do not work on aarch64.) * Conditionally install different versions of h5py dependending on architecture. * Fix value for platform_machine. * Don't install h5py on aarch64 at all. * Set USE_LAPATH_PATH to correct path on ubuntu 18.04. * Rearrange dockerfiles to build more efficiently when small changes occur. Split python install into 2 steps: building python and install requirements. * Since we are not using multi-stage builds, do not specify target to ensure docker cache works as expected. * When building docker-compose based containers, pull the latest version for caching before building. * When pulling docker-compose images, pass quiet option to squell CI logs. * When pulling docker-compose images, pass quiet option to squell CI logs. * Clean up docker cache build code. * [v1.9.x] Restore Cuda 10.x CD builds (#21128) * Create Dockerfile for ubuntu CD, add ccache, install cuda repos in base container instead of adding dynamically and requiring more sudo permissions. * Prevent hanging for user input on package installation. * Update build configs for cuda 10.0, 10.1 and 10.2 to work with centos7 CD. * Update links to other versions to include all supported cuda releases. * Update supported cuda version list. * Add back support for cuda 10.x, change installation design to require cuda repos to be already setup and accessible in the base containers for simplicity. * Use correct script name for installing ccache. * No need to use non-exact matches for variants. * Standardize name for ccache installation script. * Update ccache version and clean up install scripts. * Install libtool in ubuntu CD container. * Restore Cuda 10.x builds for CD. * Dynamically determine which dockerfiles are used by docker-compose (instead of having a hard-coded list) so docker cache refresh will finish successfully. * Remove debug line. * Define python executable path for tensorrt build. * Remove old hacks for changing permissions to /usr/local/bin. * Install libtool in ubuntu r container. * Update permissions to allow CI tasks to run. * Recursively set permissions on deps directory.
- Loading branch information
1 parent
702e475
commit e2ed553
Showing
121 changed files
with
2,324 additions
and
1,887 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
# -*- mode: dockerfile -*- | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. | ||
# | ||
# Dockerfile to build and run MXNet on CentOS 7 for CPU | ||
|
||
FROM centos:7 | ||
|
||
WORKDIR /work/deps | ||
|
||
COPY install/centos7_base.sh /work/ | ||
RUN /work/centos7_base.sh | ||
|
||
COPY install/centos7_scala.sh /work/ | ||
RUN /work/centos7_scala.sh | ||
|
||
# Install cmake | ||
COPY install/centos7_cmake.sh /work/ | ||
RUN /work/centos7_cmake.sh | ||
|
||
COPY install/centos7_ccache.sh /work/ | ||
RUN /work/centos7_ccache.sh | ||
|
||
# Install tools for static dependency builds | ||
RUN yum install -y sudo patchelf nasm automake libtool file gcc-c++ gcc gcc-gfortran which | ||
|
||
# Allow jenkins user to use sudo for installing cuda libraries | ||
RUN echo "jenkins_slave ALL=(root) NOPASSWD: /usr/bin/yum" >> /etc/sudoers.d/10_jenkins_slave | ||
|
||
COPY install/centos7_python.sh /work/ | ||
RUN /work/centos7_python.sh | ||
COPY install/requirements /work/ | ||
RUN pip3 install -r /work/requirements | ||
|
||
ARG USER_ID=0 | ||
COPY install/centos7_adduser.sh /work/ | ||
RUN /work/centos7_adduser.sh | ||
|
||
ENV PYTHONPATH=./python/ | ||
WORKDIR /work/mxnet | ||
|
||
# setup cuda repos | ||
RUN yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/$(uname -m)/cuda-rhel7.repo && \ | ||
rpm --import http://developer.download.nvidia.com/compute/machine-learning/repos/rhel7/$(uname -m)/7fa2af80.pub && \ | ||
yum-config-manager --add-repo https://developer.download.nvidia.com/compute/machine-learning/repos/rhel7/$(uname -m) | ||
|
||
COPY runtime_functions.sh /work/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.