[rocm6.3_internal_testing] Changes to support UB 24.04 build #1633

pruthvistony · 2024-10-14T05:30:44Z

Changes are based on branch by @iupaikov-amd - https://github.com/ROCm/pytorch/tree/ypajkov/ubuntu_noble_build

rocm-mici · 2024-10-14T06:45:34Z

Jenkins build for ed0e6e5e9af1fe15a82aa2fa53f510c97318b6b6 commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

pruthvistony · 2024-10-14T14:44:02Z

If I remove the torch_cpu HSA additional line, not sure why this dependency.
https://github.com/ROCm/pytorch/pull/1633/files#diff-c5ee05f1e918772792ff6f2a3f579fc2f182e57b1709fd786ef6dc711fd68b27R1420

I get this error -

[2024-10-14T06:44:26.129Z] FAILED: bin/BackoffTest
[2024-10-14T06:44:26.129Z] : && /opt/cache/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic -Wl,-rpath-link,/usr/lib/x86_64-linux-gnu -Wl,--no-as-needed test_cpp_c10d/CMakeFiles/BackoffTest.dir/BackoffTest.cpp.o -o bin/BackoffTest -L/lib/intel64 -L/lib/intel64_win -L/lib/win-x64 -Wl,-rpath,/lib/intel64:/lib/intel64_win:/lib/win-x64:/var/lib/jenkins/pytorch/build/lib:/opt/conda/envs/py_3.12/lib lib/libtorch_cpu.so lib/libgtest_main.a -lpthread lib/libprotobuf.a lib/libc10.so /opt/conda/envs/py_3.12/lib/libmkl_intel_lp64.so /opt/conda/envs/py_3.12/lib/libmkl_gnu_thread.so /opt/conda/envs/py_3.12/lib/libmkl_core.so -fopenmp /usr/lib/x86_64-linux-gnu/libpthread.a -lm /usr/lib/x86_64-linux-gnu/libdl.a lib/libgtest.a && /opt/conda/envs/py_3.12/bin/cmake -E __run_co_compile --lwyu="ldd;-u;-r" --source=bin/BackoffTest && :
[2024-10-14T06:44:26.129Z] /usr/bin/ld: /opt/rocm/lib/libamdhip64.so.6: undefined reference to hsa_amd_vmem_map@ROCR_1' [2024-10-14T06:44:26.129Z] /usr/bin/ld: /opt/rocm/lib/libamdhip64.so.6: undefined reference to hsa_amd_vmem_address_reserve@ROCR_1'
[2024-10-14T06:44:26.129Z] /usr/bin/ld: /opt/rocm/lib/libamdhip64.so.6: undefined reference to hsa_amd_vmem_address_free@ROCR_1' [2024-10-14T06:44:26.129Z] /usr/bin/ld: /opt/rocm/lib/libamdhip64.so.6: undefined reference to hsa_amd_vmem_handle_release@ROCR_1'
[2024-10-14T06:44:26.129Z] /usr/bin/ld: /opt/rocm/lib/libamdhip64.so.6: undefined reference to hsa_amd_vmem_export_shareable_handle@ROCR_1' [2024-10-14T06:44:26.129Z] /usr/bin/ld: /opt/rocm/lib/libamdhip64.so.6: undefined reference to hsa_amd_vmem_import_shareable_handle@ROCR_1'
[2024-10-14T06:44:26.130Z] /usr/bin/ld: /opt/rocm/lib/libamdhip64.so.6: undefined reference to hsa_amd_vmem_handle_create@ROCR_1' [2024-10-14T06:44:26.130Z] /usr/bin/ld: /opt/rocm/lib/libamdhip64.so.6: undefined reference to hsa_amd_vmem_unmap@ROCR_1'
[2024-10-14T06:44:26.130Z] /usr/bin/ld: /opt/rocm/lib/libamdhip64.so.6: undefined reference to hsa_amd_vmem_set_access@ROCR_1' [2024-10-14T06:44:26.130Z] /usr/bin/ld: /opt/rocm/lib/libamdhip64.so.6: undefined reference to hsa_amd_vmem_get_access@ROCR_1'

for [2024-10-14T06:44:26.130Z] FAILED: bin/test_edge_op_registration

link - http://rocm-ci.amd.com/job/mainline-framework-pytorch-2.4-ub24-py3.12-ci/21/

jeffdaily

Why is ROCM_HSART_LIB suddenly needed?

rocm-mici · 2024-10-14T18:52:28Z

Jenkins build for ed0e6e5e9af1fe15a82aa2fa53f510c97318b6b6 commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

pruthvistony · 2024-10-14T23:52:12Z

docker image - compute-artifactory.amd.com:5000/rocm-plus-docker/framework/compute-rocm-dkms-no-npi-hipclang:14929_ubuntu24.04_py3.12_pytorch_rocm6.3_ub24_ed0e6e5

jithunnair-amd

Suggested some changes and another look at the ROCM_HSART_LIB changes

jithunnair-amd · 2024-10-14T15:39:48Z

.ci/docker/common/install_conda.sh

  fi

+  if [ "$ANACONDA_PYTHON_VERSION" = "3.12" ] ; then
+    conda_install_through_forge libstdcxx-ng=14


Is the libstdcxx-ng version dependent on Ubuntu version or python version?

If I recall correctly this is the lowest available for ub24, python doesn't have a strict requirement for this version. But I'm not 100% sure, worked on this some time ago.

Sounds like at the minimum, this condition needs to be on OS version. I assume the reason is similar to pytorch#121556 where we get symbol version errors when building PyTorch?

This is dependent on UB 24.04 and minimum python version is 3.12, so I will update the condition.

Updated the condition.

jithunnair-amd · 2024-10-14T15:42:27Z

.ci/docker/common/install_rocm.sh

+    if [[ $UBUNTU_VERSION == 24.04 ]]; then
+        apt-get install -y --no-install-recommends gpg-agent
+        echo -e 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600' \
+            | sudo tee /etc/apt/preferences.d/rocm-pin-600


According to DevOps, this is needed for Ubuntu 22.04 onwards (as also confirmed by https://github.com/ROCm/ROCm-docker/blob/master/build_all.sh). So we should:

update this condition to be applicable for 22.04 onwards,

cherry-pick this change into all release branches and

Remove the corresponding patch logic in DevOps's build_pytorch.bash

jithunnair-amd · 2024-10-14T15:47:01Z

.ci/docker/common/install_user.sh

+if [[ $UBUNTU_VERSION == 24.04 ]]; then
+    # touch is used to disable harmless error message
+    touch /var/mail/ubuntu && chown ubuntu /var/mail/ubuntu && userdel -r ubuntu
+fi


This would seem to be a problem for upstream PyTorch as well. Can we file an upstream GitHub issue with logs and error snippets so that they're aware of this and might come up with a different way to address this? It's okay to merge this patch in rocm6.3_internal_testing though.

@iupaikov-amd Can you please file a github issue on pytorch/pytorch for this? Upstream PyTorch team would like some more details. Please discuss with Pruthvi or me if you have questions regarding what info to mention on the issue.

Upstream issue about install_user.sh: pytorch#138812

jithunnair-amd · 2024-10-14T15:49:14Z

aten/src/ATen/native/transformers/cuda/attention_backward.cu

  using aotriton::v2::flash::attn_bwd;
  using sdp::aotriton_adapter::mk_aotensor;
  using sdp::aotriton_adapter::cast_dtype;
+  using sdp::aotriton_adapter::mk_aoscalartensor;


This change is already in rocm6.3_internal_testing via 7ac294f.
Need a rebase.

jithunnair-amd · 2024-10-14T15:49:34Z

aten/src/ATen/native/transformers/hip/flash_attn/flash_api.hip

    using aotriton::v2::flash::attn_bwd;
    using sdp::aotriton_adapter::mk_aotensor;
    using sdp::aotriton_adapter::cast_dtype;
+    using sdp::aotriton_adapter::mk_aoscalartensor;


This change is already in rocm6.3_internal_testing via 7ac294f. Need a rebase.

jithunnair-amd · 2024-10-15T03:25:31Z

caffe2/CMakeLists.txt

 target_link_libraries(torch_cpu PUBLIC ${Caffe2_PUBLIC_DEPENDENCY_LIBS})
 target_link_libraries(torch_cpu PRIVATE ${Caffe2_DEPENDENCY_LIBS})
 target_link_libraries(torch_cpu PRIVATE ${Caffe2_DEPENDENCY_WHOLE_LINK_LIBS})
+target_link_libraries(torch_cpu PUBLIC ${ROCM_HSART_LIB})


The other ROCM_HSART_LIB usages seem like they should be avoidable as well, but this one seems the most egregious, being a torch_cpu dependency? I wonder if @naromero77amd's latest refactor in https://github.com/pytorch/pytorch/pull/137112/files might help with all the ROCM_HSART_LIB occurrences because it uses CMake targets instead of paths to .so files.

@jithunnair-amd ,
torch_cpu is already dependent on libamdhip64.so even before this change.

root@ctr-ubbsmc12:/var/lib/jenkins/pytorch/build/lib# ldd libtorch_cpu.so
linux-vdso.so.1 (0x00007fff799f6000)
libc10.so (0x00007fd3ae78e000)
libgcc_s.so.1 => /opt/conda/envs/py_3.10/lib/libgcc_s.so.1 (0x00007fd3ae775000)
libmkl_intel_lp64.so.1 => /opt/conda/envs/py_3.10/lib/libmkl_intel_lp64.so.1 (0x00007fd3adbd6000)
libmkl_gnu_thread.so.1 => /opt/conda/envs/py_3.10/lib/libmkl_gnu_thread.so.1 (0x00007fd3ac04b000)
libmkl_core.so.1 => /opt/conda/envs/py_3.10/lib/libmkl_core.so.1 (0x00007fd3a7bdb000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd3a7ae6000)
libgomp.so.1 => /opt/conda/envs/py_3.10/lib/libgomp.so.1 (0x00007fd3a7aad000)
libroctracer64.so.4 => /opt/rocm/lib/libroctracer64.so.4 (0x00007fd3a7a44000)
libamdhip64.so.6 => /opt/rocm/lib/libamdhip64.so.6 (0x00007fd3a6181000)
libstdc++.so.6 => /opt/conda/envs/py_3.10/lib/libstdc++.so.6 (0x00007fd3a5fcd000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd3a5da2000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd3bb09b000)
libnuma.so.1 => /lib/x86_64-linux-gnu/libnuma.so.1 (0x00007fd3a5d95000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd3a5d90000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd3a5d8b000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd3a5d86000)
libhsa-runtime64.so.1 => /opt/rocm/lib/libhsa-runtime64.so.1 (0x00007fd3a5a49000)
librocprofiler-register.so.0 => /opt/rocm/lib/librocprofiler-register.so.0 (0x00007fd3a59c7000)
libamd_comgr.so.2 => /opt/rocm/lib/libamd_comgr.so.2 (0x00007fd39b4e1000)
libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fd39b4c3000)
libdrm.so.2 => /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2 (0x00007fd39b4a9000)
libdrm_amdgpu.so.1 => /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1 (0x00007fd39b497000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd39b47b000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007fd39b3ac000)
libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fd39b37a000)

.ci/docker/common/install_base.sh

pruthvistony · 2024-10-15T04:45:38Z

There is a hip sample - /opt/rocm/share/hip/samples/2_Cookbook/15_static_library/host_functions/CMakeLists.txt

target_link_libraries(test_opt_static PRIVATE amdhip64 amd_comgr hsa-runtime64::hsa-runtime64)

removed amd_comgr and hsa-runtime64::hsa-runtime64, but build of test_opt_static was fine. So it seems this problem is NOT universal.

rocm-mici · 2024-10-18T06:50:00Z

Jenkins build for ed0e6e5e9af1fe15a82aa2fa53f510c97318b6b6 commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

rocm-mici · 2024-10-22T22:00:12Z

Jenkins build for 243f1f9e0b400a38c09d0a59b74adb1b77e99b3f commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

pruthvistony · 2024-10-23T05:43:28Z

Successful build - http://rocm-ci.amd.com/job/mainline-framework-pytorch-2.4-ub24-py3.12-ci/37/

@iupaikov-amd

Changes are based on branch by @iupaikov-amd - https://github.com/ROCm/pytorch/tree/ypajkov/ubuntu_noble_build

jithunnair-amd · 2025-01-02T20:35:02Z

docker image - compute-artifactory.amd.com:5000/rocm-plus-docker/framework/compute-rocm-dkms-no-npi-hipclang:14929_ubuntu24.04_py3.12_pytorch_rocm6.3_ub24_ed0e6e5

Generated by CI build: http://rocm-ci.amd.com/job/mainline-framework-pytorch-2.4-ub24-py3.12-ci/22/

@iupaikov-amd

Changes are based on branch by @iupaikov-amd - https://github.com/ROCm/pytorch/tree/ypajkov/ubuntu_noble_build

@iupaikov-amd

Changes are based on branch by @iupaikov-amd - https://github.com/ROCm/pytorch/tree/ypajkov/ubuntu_noble_build Validation: http://rocm-ci.amd.com/view/Release-6.4/job/framework-pytorch-2.3-ub24-py3.12-ci_rel-6.4/3/ Fixes https://ontrack-internal.amd.com/browse/SWDEV-520708 Co-authored-by: Pruthvi Madugundu <pruthvigithub@gmail.com>

@iupaikov-amd

Changes are based on branch by @iupaikov-amd - https://github.com/ROCm/pytorch/tree/ypajkov/ubuntu_noble_build

pruthvistony requested review from jeffdaily and jithunnair-amd as code owners October 14, 2024 05:30

pruthvistony requested a review from iupaikov-amd October 14, 2024 05:31

jeffdaily approved these changes Oct 14, 2024

View reviewed changes

jithunnair-amd requested changes Oct 15, 2024

View reviewed changes

pruthvistony mentioned this pull request Oct 16, 2024

Problem with building PyTorch from source for ROCm on Ubuntu 24.04 pytorch/pytorch#137858

Closed

pruthvistony added 9 commits October 22, 2024 12:07

Initial changes

66412eb

Bump cmake version 3.28

ec1b230

Add hsa-runtime64 lib into torch_hip dependencies

a0126a2

Add HSAruntime dependency

e8ec496

Add HSAruntime to test binaries

2b39e58

Add HSA dependency to torch_cpu

943377a

HSA dependency to SHM manager

df60210

install libstdcxx-ng 14 from conda forge

ebca049

Correct the shm dependencies

243f1f9

pruthvistony force-pushed the rocm6.3_ub24 branch from ed0e6e5 to 243f1f9 Compare October 22, 2024 17:10

Update the condition to install conda libc++ng 14 package

d98825f

pruthvistony merged commit 8e3ef93 into rocm6.3_internal_testing Oct 23, 2024

pruthvistony deleted the rocm6.3_ub24 branch October 23, 2024 05:45

pruthvistony added a commit that referenced this pull request Oct 23, 2024

Changes to support UB 24.04 build (#1633)

e671542

Changes are based on branch by @iupaikov-amd - https://github.com/ROCm/pytorch/tree/ypajkov/ubuntu_noble_build

jithunnair-amd mentioned this pull request Jan 6, 2025

[release/2.5] Added missing logic for OS, noble=24.04 #1812

Closed

jithunnair-amd changed the title ~~Changes to support UB 24.04 build~~ [rocm6.3_internal_testing] Changes to support UB 24.04 build Jan 6, 2025

jithunnair-amd mentioned this pull request Jan 8, 2025

[release/2.5] Changes to support UB 24.04 build #1816

Merged

ethanwee1 pushed a commit that referenced this pull request Mar 11, 2025

Changes to support UB 24.04 build (#1633)

90deb1b

Changes are based on branch by @iupaikov-amd - https://github.com/ROCm/pytorch/tree/ypajkov/ubuntu_noble_build

jithunnair-amd pushed a commit that referenced this pull request Mar 17, 2025

Changes to support UB 24.04 build (#1633)

3758087

Changes are based on branch by @iupaikov-amd - https://github.com/ROCm/pytorch/tree/ypajkov/ubuntu_noble_build

[rocm6.3_internal_testing] Changes to support UB 24.04 build #1633

[rocm6.3_internal_testing] Changes to support UB 24.04 build #1633

Uh oh!

Conversation

pruthvistony commented Oct 14, 2024

Uh oh!

rocm-mici commented Oct 14, 2024

Uh oh!

pruthvistony commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeffdaily left a comment

Choose a reason for hiding this comment

Uh oh!

rocm-mici commented Oct 14, 2024

Uh oh!

pruthvistony commented Oct 14, 2024

Uh oh!

jithunnair-amd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pruthvistony Oct 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pruthvistony commented Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-mici commented Oct 18, 2024

Uh oh!

rocm-mici commented Oct 22, 2024

Uh oh!

pruthvistony commented Oct 23, 2024

Uh oh!

jithunnair-amd commented Jan 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pruthvistony commented Oct 14, 2024 •

edited

Loading

pruthvistony Oct 22, 2024 •

edited

Loading

pruthvistony commented Oct 15, 2024 •

edited

Loading