[ROCm/xformers] error generated when compiling for gfx1100 #1026
Comments
|
Text is better than screenshots when reporting issues. It looks like you have ROCm 6.0.1 but you are installing torch built for ROCm 5.7. What happens when you try with the other index-url? git clone https://github.com/ROCm/xformers
cd xformers
conda create -n xformers python=3.9
conda activate xformers
- pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.7
+ pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0
pip install ninja
pip install ./ |
No help. The same errors. Thanks anyway! |
|
It's a bit difficult to read which symbol it's failing on. If you paste the text from the error then it will be searchable for people in the future. It looks like it might be a similar error to: #978 (comment) Basically, the ROCm support is experimental. The code might have been merged a bit early / prematurely. It looks like there are some improvements to get it to work. You bug report is an important part of that. Please consider editing your first comment to remove the screenshots and replace it with the text. Thanks 馃檹 |
|
Can you try this Dockerfile that builds the ROCm fork? FROM rocm/pytorch:latest
WORKDIR /workspace
RUN python3 -m pip install --upgrade pip
RUN apt-get update \
&& apt-get install -y \
wget \
git \
build-essential \
ninja-build \
git-lfs \
libaio-dev \
&& rm -rf /var/lib/apt/lists/*
RUN pip install -U pip \
&& pip install ninja packaging pytest numpy
# See https://github.com/ROCm/xformers/pull/1/files#r1494885190
env PYTORCH_ROCM_ARCH=gfx1100
env MAX_JOBS=12
RUN pip wheel -v --no-build-isolation git+https://github.com/ROCm/xformers.git@main#egg=xformers
RUN find . -name xformers\*.whlYou can build it with: docker build -t xformers . --progress=plain |
|
|
|
|
FYI, I have a W7900 (gfx1100) that I am trying to build with and getting errors with both upstream and using the ROCm fork. PyTorch/HIP version: Errors: A lot of similar errors so I've elided but definitely busted. Maybe @tenpercent has some feedback/thoughts on getting the ROCm fork running? I am on a clean mamba environment with Ubuntu 22.04 LTS HWE and ROCm installed via: https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/native-install/ubuntu.html |
|
My setup on arch is failing, but the docker example worked for me. |
|
It's not supposed to work on gfx10xx or gfx11xx yet (as we haven't had developers' bandwidth to support it). The root cause is different matrix core instructions being used between different gfx families, as the compiler error is trying to hint. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
馃悰 Bug
I put up some screenshots that I think it is important:
And
Command
Environment
PyTorch version: 2.2.0+rocm5.7
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 5.7.31921-d1770ee1b
OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35
Python version: 3.9.18 (main, Sep 11 2023, 13:41:44) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.5.0-21-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: Radeon RX 7900 XTX (gfx1100)
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 5.7.31921
MIOpen runtime version: 2.20.0
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: GenuineIntel
Model name: Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz
CPU family: 6
Model: 158
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 1
Stepping: 13
CPU max MHz: 4700.0000
CPU min MHz: 800.0000
BogoMIPS: 6000.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp vnmi md_clear flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 256 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 2 MiB (8 instances)
L3 cache: 12 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT disabled
Vulnerability Retbleed: Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds: Mitigation; Microcode
Vulnerability Tsx async abort: Mitigation; TSX disabled
Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] pytorch-triton-rocm==2.2.0
[pip3] torch==2.2.0+rocm5.7
[pip3] torchaudio==2.2.0+rocm5.7
[pip3] torchvision==0.17.0+rocm5.7
[conda] numpy 1.24.1 pypi_0 pypi
[conda] pytorch-triton-rocm 2.2.0 pypi_0 pypi
[conda] torch 2.2.0+rocm5.7 pypi_0 pypi
[conda] torchaudio 2.2.0+rocm5.7 pypi_0 pypi
[conda] torchvision 0.17.0+rocm5.7 pypi_0 pypi
Additional context
@tenpercent @qianfengz
The text was updated successfully, but these errors were encountered: