Building PyTorch with ROCm

[briansp2020]

❓ Questions and Help

Please note that this issue tracker is not a help form and this issue will be closed.

I'm trying to build PyTorch to run on ROCm (Ubuntu 18.04) and am having issues. I tried the following.

1. I followed https://github.com/ROCmSoftwarePlatform/pytorch/wiki/Building-PyTorch-for-ROCm but it seems to have failed at pyyaml (https://gist.github.com/briansp2020/114bd75ff0182197cf7efc7af265e89c)
   I got over the error by installing wheel. However, the build still failed later (https://gist.github.com/briansp2020/2719353d626968082410011dc36608cf)

2. I tried build it in tensorflow docker and I get https://gist.github.com/briansp2020/2a109c0f1d40b45299cb73a76a255767

It seems the wiki is old and I needed to get latest rocSPARSE (https://github.com/ROCmSoftwarePlatform/rocSPARSE/releases) to get past the CMake phase. Unfortunately, build still failed(https://gist.github.com/briansp2020/52047cf73d8d59ddd72f730d779b952c)...

Do you have up to date instruction on how to build PyTorch with ROCm? My goal is to run fast.ai on Vega FE with ROCm.

Thanks!

[iotamudelta]

We are in the process of releasing a new version of ROCm for PyTorch soon. The wiki and docker file will be updated then. Stay tuned!

[jithunnair-amd]

@briansp2020 Please don't forget to run Step 6 of https://github.com/ROCmSoftwarePlatform/pytorch/wiki/Building-PyTorch-for-ROCm before you start the build process. It seems you didn't "hipify" using that step, which is why your code is still referencing CUDA headers.

[briansp2020]

I was able to make some more progress. I did ran hipify but it failed because I did not set environment variables for language. After setting the environment variable and installing hipSPARSE & rocSPARSE to ROCm/TensorFlow docker, I was able to make progress. The build process reached 87% (https://gist.github.com/briansp2020/159742c61da5bb205b7214ac980ff092).

Some errors I still get are

In file included from /root/pytorch/aten/src/THC/THCTensorMathReduce.cu:1:
/root/pytorch/aten/src/THC/THCTensorMathReduce.cuh:121:61: error: '__builtin_inff': no overloaded function has restriction specifiers that are compatible with the ambient context 'THCTensor_kernel_renorm'
if (THCNumerics::eq(value, scalar_cast<AccT, float>(INFINITY))) {
^
/usr/include/x86_64-linux-gnu/bits/inf.h:26:34: note: expanded from macro 'INFINITY'

LLVM ERROR: Cannot select: 0xb7325640: ch = store<(store 8 into %ir.dense.coerce.fca.0.gep16, addrspace 5), trunc to i64> 0xb773aa58, 0xb5979530, FrameIndex:i32<0>, undef:i32
0xb5979530: i32 = srl 0xb622f9f0, Constant:i32<16>
0xb622f9f0: i32,ch = load<(dereferenceable invariant load 4 from i16 addrspace(4)* undef, align 16, addrspace 4)> 0x3e0d508, 0xb5bfbd00, undef:i64
0xb5bfbd00: i64,ch = CopyFromReg 0x3e0d508, Register:i64 %47
0x668b4f50: i64 = Register %47
0xb575d268: i64 = undef
0xb7fbc7c0: i32 = Constant<16>
0xb5b29038: i32 = FrameIndex<0>
0xb5c0d4b8: i32 = undef
In function: ZN2at6native5apply23sparseElementwiseKernelI12TensorCAddOpINS_4HalfEEmS4_EEvT_NS_4cuda6detail10TensorInfoIT1_T0_EENS9_IlSB_EESC_SB
Generating AMD GCN kernel failed in llc for target: gfx906
LLVM ERROR: Cannot select: 0x6688ad48: ch = store<(store 8 into %ir.dense.coerce.fca.0.gep16, addrspace 5), trunc to i64> 0xb405a170, 0x66b1be88, FrameIndex:i32<0>, undef:i32
0x66b1be88: i32 = srl 0xb54e5028, Constant:i32<16>
0xb54e5028: i32,ch = load<(dereferenceable invariant load 4 from i16 addrspace(4)* undef, align 16, addrspace 4)> 0x3eb5498, 0x6b130040, undef:i64
0x6b130040: i64,ch = CopyFromReg 0x3eb5498, Register:i64 %47
0x6aecdfc8: i64 = Register %47
0xb6fbef08: i64 = undef
0x6b71b7b8: i32 = Constant<16>
0xb3fde388: i32 = FrameIndex<0>
0x66924668: i32 = undef
In function: ZN2at6native5apply23sparseElementwiseKernelI12TensorCAddOpINS_4HalfEEmS4_EEvT_NS_4cuda6detail10TensorInfoIT1_T0_EENS9_IlSB_EESC_SB
Generating AMD GCN kernel failed in llc for target: gfx900
LLVM ERROR: Cannot select: 0xb91ced38: ch = store<(store 8 into %ir.dense.coerce.fca.0.gep16, addrspace 5), trunc to i64> 0xb6d397d8, 0xb55c15d0, FrameIndex:i32<0>, undef:i32
0xb55c15d0: i32 = srl 0x67a21678, Constant:i32<16>
0x67a21678: i32,ch = load<(dereferenceable invariant load 4 from i16 addrspace(4)* undef, align 16, addrspace 4)> 0x3ef94f8, 0xb6d5a770, undef:i64
0xb6d5a770: i64,ch = CopyFromReg 0x3ef94f8, Register:i64 %47
0xb6a24280: i64 = Register %47
0xb53ad3f0: i64 = undef
0x66205d58: i32 = Constant<16>
0xb91dacc8: i32 = FrameIndex<0>
0x6c344ed8: i32 = undef
In function: ZN2at6native5apply23sparseElementwiseKernelI12TensorCAddOpINS_4HalfEEmS4_EEvT_NS_4cuda6detail10TensorInfoIT1_T0_EENS9_IlSB_EESC_SB
Generating AMD GCN kernel failed in llc for target: gfx803
clang-7: error: linker command failed with exit code 7 (use -v to see invocation)

Any advice would be appreciated. I'll keep looking.

Thanks!

[briansp2020]

FYI. I saw that the Dockerfile was updated and tried to build pytorch & fast.ai. I was able to build them eventually but noticed that rocm_agent_enumerator installed in the docker image does not match the latest (https://github.com/RadeonOpenCompute/rocminfo/blob/master/rocm_agent_enumerator) and fails to run under Python 3.6.

[briansp2020]

After building docker image using updated docker file (https://github.com/ROCmSoftwarePlatform/pytorch/wiki/Building-PyTorch-for-ROCm). I needed to uninstall hip-thrust and get the latest hip-thrust from git (https://github.com/ROCmSoftwarePlatform/Thrust.git) before building PyTorch. I summarized steps I took to build PyTorch using Python3.6 here (https://gist.github.com/briansp2020/717f5cab006889f056b36eacee7dd5d7).

After building PyTorch successfully in Python3.6, I tried fast.ai but ran into issues. Running the lesson1 notebook, Fine-tuning and differential learning rate annealing step shows very low accuracy. This screenshot shows the comparison between ROCm output and the output from the notebook that came with fast.ai repo. https://imgur.com/a/g4f5InS The ROCm version shows .55 accuracy when it should be .99.

Any ideas?

[iotamudelta]

Could you elaborate what you mean by/observe when you say:

• rocm_agent_enumerator failing under python 3.6
• hip-thrust package not working

It's hard to comment on the convergence/accuracy issue you observe directly without knowing what model/kernels are running.

[briansp2020]

The rocm_agent_enumerator file in docker image has no parenthesis around print statement at line 141. The latest file in git has them. It seems to work fine in Python 2 but it generates error when using Python 3.6.

SyntaxError: Missing parentheses in call to 'print'. Did you mean print("gfx000")?
File "/opt/rocm/bin/rocm_agent_enumerator", line 141
print "gfx000"
^

I tried building pytorch again without updating thrust and it worked this time...

[briansp2020]

After building PyTorch, how do I make sure it is working ok? I ran some scripts under test directory and got bunch of errors (https://gist.github.com/briansp2020/c532898bb4a63a65d37281536df850e2). I'm not sure what the expected behavior is.

[briansp2020]

Figured out what the issue was with thrust. I tried both https://github.com/pytorch/pytorch.git and https://github.com/ROCmSoftwarePlatform/pytorch.git. Code from pytorch repo requires updated thrust.

[briansp2020]

It looks like https://github.com/pytorch/pytorch.git works better for me. When using ROCmSoftwarePlatform/pytorch.git, all training in fast.ai lesson1 fails. Using pytorch/pytorch.git, training seems to work until all the layers are unfrozen as shown here (https://imgur.com/a/g4f5InS).

[iotamudelta]

@briansp2020 before going to more complex tests, could you run
PYTORCH_TEST_WITH_ROCM=1 python test/run_test.py --verbose (or python3 if you are running that)? If your compilation is good, there will be no errors.

[briansp2020]

Test output from pytorch/pytorch at https://gist.github.com/briansp2020/8ddfdce9b1bc7da335cd5a09f166a8a9

[jithunnair-amd]

@iotamudelta the failing test in Brian's run is skipped for python2, which is what our CI runs on. @briansp2020 can you rerun with python2 and report the results?

[briansp2020]

Test output from pytorch/pytorch using Python2.
https://gist.github.com/briansp2020/15c5792174c3cee11dfc03c8d999225f

It ran more tests before failing.

[briansp2020]

Test and build output from ROCmSoftwarePlatform/pytorch using Python3.
https://gist.github.com/briansp2020/af9610196f5cb36c1b6dcec126aa2658

I see that more tests are run and passed. However, test_multinomial_invalid_probs_cuda still fails. Since test_multinomial is skipped on ROCm, test_multinomial_invalid_probs_cuda should be skipped as well.

I also ran into a build error. It seems compiler crashed while compiling unique_ops_hip.cc.

1. parser at end of file
2. Per-module optimization passes
3. Running pass 'Function Pass Manager' on module '/root/pytorch/caffe2/operators/hip/unique_ops_hip.cc'.
4. Running pass 'Combine redundant instructions' on function '@_ZN6thrust6system4cuda6detail4cub_28DeviceRadixSortUpsweepKernelINS3_21DeviceRadixSortPolicyIliiE9Policy700ELb0ELb0EliEEvPKT2_PT3_SB_iiNS3_13GridEvenShareISB_EE'
   /opt/rocm/hcc/bin/clang-7(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamE+0x2a)[0x17678fa]
   /opt/rocm/hcc/bin/clang-7(_ZN4llvm3sys17RunSignalHandlersEv+0x4c)[0x1765c2c]
   /opt/rocm/hcc/bin/clang-7[0x1765d97]
   /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f2c56230390]
   /opt/rocm/hcc/bin/clang-7[0x1378daa]
   /opt/rocm/hcc/bin/clang-7[0x1379a08]
   /opt/rocm/hcc/bin/clang-7(_ZN4llvm24InstructionCombiningPass13runOnFunctionERNS_8FunctionE+0x232)[0x137a042]
   /opt/rocm/hcc/bin/clang-7(_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE+0x2ca)[0x125d39a]
   /opt/rocm/hcc/bin/clang-7(_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE+0x33)[0x125d463]
   /opt/rocm/hcc/bin/clang-7(_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE+0x32c)[0x125cf0c]
   /opt/rocm/hcc/bin/clang-7(_ZN5clang17EmitBackendOutputERNS_17DiagnosticsEngineERKNS_19HeaderSearchOptionsERKNS_14CodeGenOptionsERKNS_13TargetOptionsERKNS_11LangOptionsERKN4llvm10DataLayoutEPNSE_6ModuleENS_13BackendActionESt10unique_ptrINSE_17raw_pwrite_streamESt14default_deleteISM_EEb+0xc47)[0x195cbb7]
   /opt/rocm/hcc/bin/clang-7[0x2088602]
   /opt/rocm/hcc/bin/clang-7(_ZN5clang8ParseASTERNS_4SemaEbb+0x370)[0x2814310]
   /opt/rocm/hcc/bin/clang-7(_ZN5clang13CodeGenAction13ExecuteActionEv+0x37)[0x2087ba7]
   /opt/rocm/hcc/bin/clang-7(_ZN5clang14FrontendAction7ExecuteEv+0x126)[0x1d5e7c6]
   /opt/rocm/hcc/bin/clang-7(_ZN5clang16CompilerInstance13ExecuteActionERNS_14FrontendActionE+0x146)[0x1d281c6]
   /opt/rocm/hcc/bin/clang-7(_ZN5clang25ExecuteCompilerInvocationEPNS_16CompilerInstanceE+0x96c)[0x1df9c0c]
   /opt/rocm/hcc/bin/clang-7(_Z8cc1_mainN4llvm8ArrayRefIPKcEES2_Pv+0xa18)[0x8f9248]
   /opt/rocm/hcc/bin/clang-7(main+0x17e3)[0x8974c3]
   /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f2c54f8d830]
   /opt/rocm/hcc/bin/clang-7(_start+0x29)[0x8f6a19]
   clang-7: error: unable to execute command: Segmentation fault (core dumped)
   clang-7: error: clang frontend command failed due to signal (use -v to see invocation)
   HCC clang version 7.0.0 (ssh://gerritgit/compute/ec/hcc-tot/clang 37396a09268c7b90c63b91dbd66fe6c3b765bfee) (ssh://gerritgit/compute/ec/hcc-tot/llvm 4144ab076c5720d34d0f36c158ef3b094fe0d339) (based on HCC 1.2.18383-7685003-37396a0-4144ab0 )
   Target: x86_64-unknown-linux-gnu
   Thread model: posix
   InstalledDir: /opt/rocm/hcc/bin
   clang-7: note: diagnostic msg: PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
   [ 83%] Building HIPCC object caffe2/CMakeFiles/caffe2_hip.dir/sgd/hip/caffe2_hip_generated_fp16_momentum_sgd_op_hip.cc.o
   [ 83%] Building HIPCC object caffe2/CMakeFiles/caffe2_hip.dir/sgd/hip/caffe2_hip_generated_momentum_sgd_op_hip.cc.o
   clang-7: note: diagnostic msg:

---

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-7: note: diagnostic msg: /tmp/unique_ops_hip-b16c04.cpp
clang-7: note: diagnostic msg: /tmp/unique_ops_hip-efcfe7.cpp
clang-7: note: diagnostic msg: /tmp/unique_ops_hip-b16c04.sh
clang-7: note: diagnostic msg:

---

Full build output up to the error message is in the gist as well. Do I need updated compiler to build it properly?

Edit: fixed the link to gist