Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MIOpen using wrong ISA for certain convolution operations #117

Closed
89Mods opened this issue Aug 29, 2019 · 6 comments
Closed

MIOpen using wrong ISA for certain convolution operations #117

89Mods opened this issue Aug 29, 2019 · 6 comments
Assignees
Labels

Comments

@89Mods
Copy link

89Mods commented Aug 29, 2019

I've just updated MIOpen to the latest version (package version 2.0.1.7405), which now apparently uses the wrong ISA for my GPU. I normally work with PyTorch for all my projects, however, this issue occurs while MIOpen is tuning the Perf. Db before even running my python code. As soon as it tries to do so, it repeatedly runs into errors similar to this:

`<instantiation>:4:3: error: instruction not supported on this GPU
                s_mul_hi_u32 s[tiles_w], div_const_1_4, s[tiles_w]
                ^
<instantiation>:2:2: note: while in macro instantiation
        _s_div_const_u32_u16 s[tiles_w], s[tiles_w], 4
        ^
<stdin>:796:2: note: while in macro instantiation
        _s_ceil_u32 s[tiles_w], s[S], %xformx_f_size
        ^
<instantiation>:4:3: error: instruction not supported on this GPU
                s_mul_hi_u32 s[tiles_h], div_const_1_4, s[tiles_h]
                ^
<instantiation>:2:2: note: while in macro instantiation
        _s_div_const_u32_u16 s[tiles_h], s[tiles_h], 4
        ^
<stdin>:797:2: note: while in macro instantiation
        _s_ceil_u32 s[tiles_h], s[R], %xformy_f_size
        ^
<instantiation>:8:2: error: instruction not supported on this GPU
        v_sub_u32         v[vtmp+1+3],   0,            v[vtmp+1+1]
        ^
<stdin>:857:2: note: while in macro instantiation
        ceil_2_32_div_u16 v[vtmp], v[vtmp], vtmp+1, stmp`

This goes on for several pages, full log is here: https://pastebin.com/dughVHes
I'm running an RX 470, though the error also occurs on a RX 560.
The only way I've found to replicate this is to try to run a convolution with exactly 3 output channels and 128 input channels, otherwise, this error occurs seemingly at random, as most operations complete successfully and only some trigger the error (as seen in the full log).

@daniellowell
Copy link
Contributor

@zjing14 Please take a look.

@zjing14
Copy link
Contributor

zjing14 commented Aug 29, 2019

@daniellowell The Conv3x3AsmWrW is not supported on RX4/560 (i.e., Fiji/Polaris Arch). We need to disable the kernel on Fiji platform. Will create a PR.

@daniellowell
Copy link
Contributor

@89Mods We will fix this in the next release. In the mean time can you see if setting the environment variable:
MIOPEN_DEBUG_GCN_ASM_KERNELS=0
unblocks you.

@89Mods
Copy link
Author

89Mods commented Sep 14, 2019

@daniellowell I tried setting the environment variable as you suggested, but while it does hide the errors, nothing works correctly anymore. I also tried downgrading back to MIOpen 2.0.0, but doing that now causes a Segfault in PyTorch, so I'm pretty much stuck with the broken 2.0.1 release.

@daniellowell
Copy link
Contributor

Resolved in 2.1.0

@daniellowell
Copy link
Contributor

@89Mods please try the latest release. If your issue is not resolved please feel free to reopen this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants