Fix PyTorch init time due to inefficient libdrm search for amdgpu.ids #1412
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The libdrm replacement for ROCm images will first search the typical location for the amdgpu.ids file. If failed, it will search as it did before in the python install location.
jithunnair-amd
changed the title
jithunnair-amd
approved these changes
Jun 1, 2023
|
CUDA image failed to build. Unrelated to these changes. |
|
@atalman Please merge this PR to update the ROCm docker images |
malfet
approved these changes
Jun 5, 2023
|
Oops, sorry about the botched commit description |
jithunnair-amd
added a commit
to ROCm/builder
that referenced
this pull request
Aug 7, 2023
* Add 12.1 workflow for docker image build (pytorch#1367) * add 12.1 workflow for docker image build * add github workflow * update cuDNN to 8.8.1 and location for archive * Do not use ftp (pytorch#1369) * Do not use ftp `s#ftp://#https://#` * Remove no-longer relevant comment * add magma build for CUDA12.1 (pytorch#1368) * add magma build for CUDA12.1 * copy and fix CMake.patch; drop sm_37 for CUDA 12.1 * remove CUDA 11.6 builds (pytorch#1366) * remove CUDA 11.6 builds * remove more 11.6 builds * build libtorch and manywheel for 12.1 (pytorch#1373) * enable nightly CUDA 12.1 builds (pytorch#1374) * enable nightly CUDA 12.1 builds * fix version typo * Fix typo (bracket) in DEPS_LIST setting (pytorch#1377) * Remove special cases for Python 3.11 (pytorch#1381) * Remove special case for Python 3.11 * Remove install torch script * Windows CUDA 12.1 changes (pytorch#1376) * Windows CUDA 12.1 changes * add CUDA version checks for Windows MAGMA builds * use magma branch without fermi arch * fix check for 12.1 (pytorch#1383) * Update CUDA_UPGRADE_GUIDE.MD (pytorch#1384) * add pytorch-cuda constraints for CUDA 12.1 (pytorch#1385) * Fix `cuda-pytorch/meta.yaml` And add `12.1` to the matrix Test plan: `conda build . -c nvidia` and observe https://anaconda.org/malfet/pytorch-cuda/files * [aarch64] acl build script updates for multi isa build with armv8-a base architecture (pytorch#1370) * Compile with C++17 standard in check_binary.sh To unblock pytorch/pytorch#98209 * Use robocopy fix 256char limit (pytorch#1386) * add constraints for pytorch-cuda (pytorch#1391) * [Conda] Update MacOS target to 11.0 As 10.9 was release a decade ago and for that reason yet not supported C++17 standard. Similar to pytorch/pytorch#99857 * Update MACOSX_DEPLOYMENT_TARGET to 10.13 To fix builds, though we should really target 11.0 at the very least * add nvjitlink for Windows builds for CUDA 12.1 (pytorch#1393) * pytorch-cuda: Added `nvjitlink` as dependency * Add `fsspec` to list of packages Added by pytorch/pytorch#99768 * Add ffmpeg build to audio aarch64 (pytorch#1396) * [S3] Add all `cu117.with.pypi` deps to nightly CUDA index * Fix nvjitlink inclusion in 12.1 wheels (pytorch#1397) * Fix nvjitlink inclusion in 12.1 wheels * Fix typo * update winserver driver (pytorch#1388) * Add pyyaml as PyTorch runtime dep (pytorch#1394) Companion PR to pytorch/pytorch#100166 * Fix typo * Temp: Comment out VS2019 installation As it should be part of the AMI * Use VS2022 for libtorch windows tests * Revert "Use VS2022 for libtorch windows tests" This reverts commit 2922d7d. * Revert "Temp: Comment out VS2019 installation" This reverts commit e919e17. * Do not cleanup MSVC and CUDA on Windows non-ephemeral runners (pytorch#1398) * Update CUDA_UPGRADE_GUIDE.MD (pytorch#1399) Update Related PR's * Attempt to fix infinite copy into existing folder Looks like robocopy is confused what to do about symlinks * Handle symlink when using robocopy (pytorch#1400) * Use /xjd instead of /sl when dealing with symlink on Windows https://superuser.com/questions/1642106/how-does-robocopy-handle-file-system-links-symbolic-links-hard-links-and-junct * Skip uninstalling other MSVC versions if they are found (pytorch#1402) * Update driver for cuda 12.1 (pytorch#1403) * Don't prepend system-wide PATH when installing Python for binary smoke test (pytorch#1404) The PATH has already been set and restored manually in the script * Pin MSVC version to 2019 (pytorch#1405) * Pin numpy for windows builds (pytorch#1406) * Pin numpy for windows builds * bump mkl version, remove conda-forge * Change python 3.9 mkl version * Use pinned mkl * Upgrade nightly wheels to rocm5.5 (pytorch#1407) * Add MIOpen db files to wheel * Update magma commits for various branches to include header path updates * Add ROCm5.5 support with Navi31-tuned MIOpen branch * Upgrade nightly wheels to rocm5.5 * Update build_docker.sh for gfx1100 * Update build_docker.sh for gfx1100 --------- Co-authored-by: Jithun Nair <jithun.nair@amd.com> Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com> * Update CUDA_UPGRADE_GUIDE.MD with CI update instructions * Pin numpy to 1.21.3 for python 3.10 (pytorch#1409) * Update cuda matrix CUDA_UPGRADE_GUIDE.MD * Pin delocate (pytorch#1411) * Fix pytorch#1410. (pytorch#1412) The libdrm replacement for ROCm images will first search the typical location for the amdgpu.ids file. If failed, it will search as it did before in the python install location. * One-step ROCm manywheel/libtorch docker build (pytorch#1418) * Use rocm/dev-centos-7:<rocm_version>-complete image with full ROCm install * Remove ROCm install step and reinstate magma and MIOpen build steps * Install full package for MIOpen, including headers and db files Retained some of the disk-cleanup-related code from pytorch@cb0912c * Use rocm/dev-ubuntu-20.04:<rocm_version>-complete image with full ROCm install * Remove ROCm install and reinstate magma build from source * Use --offload-arch instead of --amdgpu-target to silence warnings * Use beefier runner instance for ROCm docker builds * Typo * Simplify ROCm targets * Update wheel build scripts for ROCm5.5 (pytorch#1413) * Fix lib search logic and lib list * Add libhipsolver.so dependency for upstream Pytorch (needed since PyTorch PR 97370) * Add MIOpen db files to share path only for ROCm5.5 and above * Add cu12 packages to allow list for download.pytorch.org (pytorch#1420) * Fix ROCm5.4.2 builds breakage (pytorch#1421) * Use MIOpen db logic for ROCm5.5 or later * Remove moved lines * Fix bash logic to add elements of array to existing array * Add nvidia-nvjitlink-cu12 to s3 manage (pytorch#1422) * Add nvidia-nvjitlink-cu12 to s3 manage (pytorch#1423) * Pin openssl for python 3.9 conda package (pytorch#1424) Fixes pytorch/pytorch#103436 Test PR: pytorch/pytorch#103437 * Update CUDA_UPGRADE_GUIDE.MD * Pin openssl for py3.8 (pytorch#1425) * Add pytorch-triton to small wheel (pytorch#1426) * Add poetry and pypi tests (pytorch#1419) * smoke test poetry Add a little more tests test Test poetry test Test poetry on python 3.10 Add more poetry tests Test en us test test Try verboose testing testing try quiet install Code refactooring test move linux pipy validation to workflow test test Fix path try test pipy More torch installations test testing test test test new fix install 2 try poetry nightly test nightly test test Test poetry validation test test_new test * Put back executing this on pull * [Manywheel] Add Python-3.12.0b2 (pytorch#1427) To enable initial experiments with PyTorch builds * smoke test poetry (pytorch#1428) Add a little more tests test Test poetry test Test poetry on python 3.10 Add more poetry tests Test en us test test Try verboose testing testing try quiet install Code refactooring test move linux pipy validation to workflow test test Fix path try test pipy More torch installations test testing test test test new fix install 2 try poetry nightly test nightly test test Test poetry validation test test_new test Put back executing this on pull Print matrix variable test Fix conditional for pypi poetry tests add quptes Add nightly as supplemental requirement Make sure we clone module only for first time Fix python test validate binaries Add repo existance checks test Disable runtime error before final validation fix typo fix cwd * smoke test poetry (pytorch#1429) * s/master/main/ * Update aarch64 Scripts for CI worflow (pytorch#1431) * Run release validation testing once a day, run nightly a little later (pytorch#1434) * Remove CUDA 11.7 builds (pytorch#1408) * remove CUDA 11.7 builds * remove CUDA 11.7 from MAGMA builds * add pytorch-cuda back for 11.7 * add 11.7 back to pytorch-cuda * Add safe directory to all dockerfiles (pytorch#1435) * update cuDNN to 8.9.2.26 for CUDA 12.1 (pytorch#1436) * update NCCL to 2.18.1 (pytorch#1437) Co-authored-by: Nikita Shulga <nshulga@meta.com> * Update CI for aarch64 (pytorch#1438) * Fix wheel macos arm64 validations (pytorch#1441) * Fix wheel validations * Try using upgrade flag instead * try uninstall * test * Try using python3 * use python3 vs python for validation * Fix windows vs other os python execution * Uninstall fix * Remove previous installations on macos-arm64 before smoke testing (pytorch#1444) More arm64 changes test run under environment sleep 15min allow investigate add sleep test test Test test test Arm64 use python fix test testing test tests testing test test * Update build_docker.sh (pytorch#1446) Use [`nvidia/cuda:11.4.3-devel-centos7`](https://hub.docker.com/layers/nvidia/cuda/11.4.3-devel-centos7/images/sha256-e2201a4954dfd65958a6f5272cd80b968902789ff73f26151306907680356db8?context=explore) because `nvidia/cuda:10.2-devel-centos7` was deleted in accordance with [Nvidia's Container Support Policy](https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/support-policy.md): > After a period of Six Months time, the EOL tags WILL BE DELETED from Docker Hub and Nvidia GPU Cloud (NGC). This deletion ensures unsupported tags (and image layers) are not left lying around for customers to continue using after they have long been abandoned. Also delete redundant DEVTOOLSET=7 clause * Fix aarch64 nightly (pytorch#1449) * Update Docker base images for conda and libtorch (pytorch#1448) Followup after pytorch#1446 CUDA-10.2 and moreover CUDA-9.2 docker images are gone per [Nvidia's Container Support Policy](https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/support-policy.md): > After a period of Six Months time, the EOL tags WILL BE DELETED from Docker Hub and Nvidia GPU Cloud (NGC). This deletion ensures unsupported tags (and image layers) are not left lying around for customers to continue using after they have long been abandoned. Also, as all our Docker script install CUDA toolkit anyway, what's the point of using `nvidia/cuda` images at all instead of `centos:7`/`ubuntu:18.04` that former are based on, according to https://gitlab.com/nvidia/container-images/cuda/-/blob/master/dist/11.4.3/centos7/base/Dockerfile Explicitly install `g++` to `libtorch/Docker` base image, as it's needed by `patchelf` Please note, that `libtorch/Docker` can not be completed without buildkit, as `rocm` step depends on `python3` which is not available in `cpu` image * Fix magma installation inside docker container (pytorch#1447) Not sure, what weird version of `wget` is getting installed, but attempt to download https://anaconda.org/pytorch/magma-cuda121/2.6.1/download/linux-64/magma-cuda121-2.6.1-1.tar.bz2 fails with: ``` --2023-07-06 03:18:38-- https://anaconda.org/pytorch/magma-cuda121/2.6.1/download/linux-64/magma-cuda121-2.6.1-1.tar.bz2 Resolving anaconda.org (anaconda.org)... 104.17.93.24, 104.17.92.24, 2606:4700::6811:5d18, ... Connecting to anaconda.org (anaconda.org)|104.17.93.24|:443... connected. ERROR: cannot verify anaconda.org's certificate, issued by ‘/C=US/O=Let's Encrypt/CN=E1’: Issued certificate has expired. To connect to anaconda.org insecurely, use `--no-check-certificate'. ``` Also, switch from NVIDIA container to a stock `centos:7` one, to make containers slimmer and fit on standard GitHub Actions runners. * [Manywheel] Add `/usr/local/cuda` symlink And add `nvcc` to path Regression introduced by pytorch#1447 when NVIDIA image was dropped in favor of base `centos` image * Do not build PyTorch with LLVM (pytorch#1445) As NNC is dead, and llvm dependency has not been updated in last 4 years First step towards fixing pytorch/pytorch#103756 * Remove `DESIRED_CUDA` logic from `check_binary.sh` As [`pytorch/manylinux-builder`](https://hub.docker.com/r/pytorch/manylinux-builder) containers has only one version of CUDA, there is no need to select any Nor setup `LD_LIBRARY_PATH` as it does not match the setup users might have on their system (but keep it for libtorch tests) Should fix crash due to different minor version of cudnn installed in docker container and specified as dependency to a small wheel package, seen here https://github.com/pytorch/pytorch/actions/runs/5478547018/jobs/9980463690 * Revert "Remove `DESIRED_CUDA` logic from `check_binary.sh`" This reverts commit ed9a2ae. * Update CUDA_UPGRADE_GUIDE.MD to add small wheel update Cudnn step * Set CUDA_VERSION in conda Docker environment * Rebuild docker images on release builds (pytorch#1451) * Rebuild docker images on release * Include with-push * Create `/usr/local/cuda` in libtorch builds I.e. applying the same changes as in pytorch@4a7ed14 to libtorch docker builds * Revert "Rebuild docker images on release builds (pytorch#1451)" (pytorch#1452) This reverts commit 2ba03df as it essentially broke all the builds on trunk (fix is coming) * Reland "Rebuild docker images on release builds" This is a reland of pytorch#1451 with an important to branches filter: entries in multi-line array definition should start with `-` otherwise it were attempting to match branch name `main\nrelease/*` I.e. just copy-n-paste example from https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#using-filters Test plan: actionlint .github/workflows/build-manywheel-images.yml Original PR description: Rebuild docker images on release builds. It should also tag images for release here: https://github.com/pytorch/builder/blob/3fc310ac21c9ede8d0ce13ec71096820a41eb9f4/conda/build_docker.sh#L58-L60 This is first step in pinning docker images for release. * Let's try to force the path this way * Remove `DESIRED_CUDA` logic from `check_binary.sh` As [`pytorch/manylinux-builder`](https://hub.docker.com/r/pytorch/manylinux-builder) containers has only one version of CUDA, there is no need to select any Nor setup `LD_LIBRARY_PATH` as it does not match the setup users might have on their system (but keep it for libtorch tests for now) Should fix crash due to different minor version of cudnn installed in docker container and specified as dependency to a small wheel package, seen here https://github.com/pytorch/pytorch/actions/runs/5478547018/jobs/9980463690 * Advance libgfortran version (pytorch#1453) * Update builder images to ROCm5.6 (pytorch#1443) * Update manywheel and libtorch images to rocm5.6 * Add MIOpen branch for ROCm5.6 * Pin miniconda install to py310_23.5.2 for macos and windows (pytorch#1460) * Cleanup unused builder files (pytorch#1459) * Remove unused builder files (pytorch#1461) * Add support for ROCm5.6 for nightly wheels (pytorch#1442) * Add msccl-algorithms directory to PyTorch wheel * Bundle msccl-algorithms into wheel * Use correct src path for msccl-algorithms (cherry picked from commit 95b5af3) * Add hipblaslt dependency for ROCm5.6 onwards * Update build_all_docker.sh to ROCm5.6 * [aarch64][build] Aarch64 lapack fix and ARMCL version update (pytorch#1462) * Fix lapack missing and armcl update * update ARMCL version * Remove unused parameter to limit-win-builds from validation workflows (pytorch#1464) * Run git update-index --chmod=+x on aarch64_ci_build.sh (pytorch#1466) * Fix erroneous logic that was skipping msccl files even for ROCm5.6; update msccl path for ROCm5.7 (cherry picked from commit 36c10cc) --------- Co-authored-by: ptrblck <ptrblck@users.noreply.github.com> Co-authored-by: Nikita Shulga <nshulga@fb.com> Co-authored-by: Andrey Talman <atalman@fb.com> Co-authored-by: Nikita Shulga <nshulga@meta.com> Co-authored-by: snadampal <87143774+snadampal@users.noreply.github.com> Co-authored-by: Richard Zou <zou3519@users.noreply.github.com> Co-authored-by: Huy Do <huydhn@gmail.com> Co-authored-by: Bo Li <110066325+BLOrange-AMD@users.noreply.github.com> Co-authored-by: Jeff Daily <jeff.daily@amd.com> Co-authored-by: Mike Schneider <104035434+xncqr@users.noreply.github.com> Co-authored-by: Eli Uriegas <1700823+seemethere@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1410.
The libdrm replacement for ROCm images will first search the typical location for the amdgpu.ids file. If failed, it will search as it did before in the python install location.