Compiler error

[gahdritz]

I tried compiling the latest build, but setup.py crashed with the following error:

/mnt/ceph/users/dberenberg/gustaf_stuff/repos/flash-attention/csrc/flash_attn/src/fmha_fprop_fp16_kernel.sm80.cu:62:385: internal compiler error: in maybe_undo_parenthesized_ref, at cp/semantics.c:1740
     BOOL_SWITCH(launch_params.is_dropout, IsDropoutConst, [&] {

[tridao]

Hi Gustaf, what are your nvcc and gcc versions?

I've just tried with nvcc 11.3, 11.6, 11.7 (and gcc 9.3), and all 3 versions of nvcc were able to compile it.

[gahdritz]

nvcc:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_Oct_11_21:27:02_PDT_2021
Cuda compilation tools, release 11.4, V11.4.152
Build cuda_11.4.r11.4/compiler.30521435_0

gcc:

gcc (Spack GCC) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Maybe my gcc is a little old? It seems to be crashing because of some parsing error in a .cu file, though, so I'm not sure if
that's what's causing this.

BTW: I redacted a bunch of other warnings and error messages from the snippet above. Let me know if you'd like the full stack.

[tridao]

I tried again with nvcc 11.4 (& gcc 9.3) and it's still fine. So it could very well be gcc version.
Googling that error seems to suggest problem with gcc.
Can you try with a newer gcc version? I've been using the Nvidia's Pytorch docker image.

I'll try to think of some workaround in the mean time.

[tridao]

I've just pushed a work-around for gcc 7, can you try that?

[gahdritz]

Essentially the same error:

/mnt/ceph/users/dberenberg/gustaf_stuff/repos/flash-attention/csrc/flash_attn/src/fmha_fprop_fp16_kernel.sm80.cu:65:522: internal compiler error: in maybe_undo_parenthesized_ref, at cp/semantics.c:1740
     BOOL_SWITCH(launch_params.is_dropout, IsDropoutConst, [&] {

[tridao]

That's interesting. Let me set up an environment with gcc 7 to test things more thoroughly.

[gahdritz]

Have you had a chance to check this out?

[syorami]

Is there any progress? I'm having this same issue and I've tried gcc version of 8.5 but still doesn't work. My installation with gcc 9.5 encounters many problems. It would be great if there is any solution without installing gcc 9.5.

[tridao]

Is there an easy way to get earlier versions of gcc in docker?
I've been using Nvidia's docker image which has Ubuntu 20.04 and gcc 9.4.
If I can set up a docker image with earlier versions of gcc then maybe I can reproduce and fix it.

[syorami]

how about directly reinstall gcc using apt command? Or maybe I can upload my current gcc 7.5 installation so that you can modify the environment path and set the gcc version to 7.5.

[tridao]

Can you try compiling from the branch gccerror on your setup, and paste the compiling log?
This was my attempt at simplifying the macro, but I have not been able to test it (since I can't reproduce the compiler error).

I've tried an old docker image (nvcr.io/nvidia/pytorch:20.11-py3) with gcc 7.5 but I was not able to reproduce the compiler error (gcc 7.5 compiles fine).
Installing gcc 7 in the newest docker image (nvcr.io/nvidia/pytorch:22.11-py3) and then compiling with it also works fine.
If there's a docker image with the environment that can reproduce the error that would be most helpful.

[syorami]

Here is my install log install_gccerror.log. I hope this can help.

BTW, I finally installed gcc with 10.1 and the installation succeeds. It seems to depend on the gcc version for me.

[tridao]

From the log it looks like compilation was fine but linking errors out because it couldn't link against pytorch.

        /mnt/cache/share/platform/dep/binutils-2.27/bin/ld: cannot find -lc10
        /mnt/cache/share/platform/dep/binutils-2.27/bin/ld: cannot find -ltorch
        /mnt/cache/share/platform/dep/binutils-2.27/bin/ld: cannot find -ltorch_cpu
        /mnt/cache/share/platform/dep/binutils-2.27/bin/ld: cannot find -ltorch_python
        /mnt/cache/share/platform/dep/binutils-2.27/bin/ld: cannot find -lc10_cuda
        /mnt/cache/share/platform/dep/binutils-2.27/bin/ld: cannot find -ltorch_cuda_cu
        /mnt/cache/share/platform/dep/binutils-2.27/bin/ld: cannot find -ltorch_cuda_cpp

1. Does the main branch have the same linking error or is it a different error?
2. Can you check that pytorch was installed correctly in your conda env?
3. The linker points to this directory /mnt/petrelfs/zhoubeitong/.local.ray1.13.0/lib/python3.9/site-packages/torch/lib. Can you check that this directory has the files libc10.so, libtorch.so etc?

[syorami]

1. If I didn't mess up, this install_main.log is the installation log on the main branch.
2. Torch version 1.12.1 and nvcc version 11.6. As I can install with gcc 10.1 and train the model successfully, I think the torch version should be correct.
3. Yes I can find this two files under lib directory.

The conda environment I'm now using is actually a shared environment and I don't have permission to install into it. So that the torch I'm now using is only installed in my home directory and not in the conda path. I'm not sure if this would make a difference.