Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add -ccbin/--compiler-bindir flag for nvcc compiler #415

Closed
OpportunityLiu opened this issue May 16, 2019 · 25 comments

Comments

Projects
None yet
2 participants
@OpportunityLiu
Copy link
Contributor

commented May 16, 2019

See https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#file-and-path-specifications-compiler-bindir

-ccbin flag should be given as xmake configured cxx compiler

If not given

the default host compiler executable (gcc and g++ on Linux, clang and clang++ on Mac OS X, and cl.exe on Windows) found in the current execution search path will be used

See https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#supported-host-compilers

@OpportunityLiu OpportunityLiu changed the title Add -ccbin flag for nvcc compiler Add -ccbin/--compiler-bindir flag for nvcc compiler May 16, 2019

@waruqi

This comment has been minimized.

Copy link
Member

commented May 16, 2019

You can try xmake f --nvcc=/xxx/bin/nvcc --cxx=/xxx/bin/gcc

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2019

跟这有啥关系啊,就算是workaround也应该是
add_cuflags("-ccbin $(cxx)")
这种吧,我现在就是这么搞的

设了cxx也拦不住nvcc用啥啊

@waruqi

This comment has been minimized.

Copy link
Member

commented May 16, 2019

我以为你要改nvcc的配置,如果只是加flags,你现在的add_cuflags不是已经解决了么,还有什么问题呢?

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2019

意思就是我觉得这句应该是默认的配置,

xmake f --cxx=clang++

之后我的预期就是已经带这个flag了,而不需要在配置文件里再写什么

@waruqi

This comment has been minimized.

Copy link
Member

commented May 16, 2019

就是说 当前cxx的配置 默认通过-ccbin透传给nvcc? 但是-ccbin是设置bin目录,而如果cxx=gcc 并没有给定全路径,-ccbin应该不支持吧,而且cc/cxx分别指定了c/c++编译器,两者也许并不相同,如何同时透传给nvcc呢?

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2019

所以才需要更复杂的脚本来把cxx展开成全路径,上面那个addcuflags能用只是因为clang在path里,甚至可以认为是巧合(虽然一般就是这么巧)

只有编译cu才用nvcc,我也没见过什么cupp之类的扩展名,应该是只给cxx就好了,我明天再看看文档吧

@waruqi

This comment has been minimized.

Copy link
Member

commented May 16, 2019

如果只是 *.c + *.cu 的混合编译工程,是不设置cxx,只设置cc,如果是 *.c + *.cpp + *.cu的混编工程,会同时设置cc/cxx,那应该传哪个给 nvcc呢? cc? cxx?

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 17, 2019

cu 实际是cpp文件
图片
指定clang
图片
指定clang++
图片

可以看到是通过 -x c++ flag 强行调c++编译器的,所以只要给cxx就可以,只设了cc的时候给cc也没关系

@waruqi

This comment has been minimized.

Copy link
Member

commented May 17, 2019

ok, 我回头看下。

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 17, 2019

而且说是bin dir,只给dir就会这样
图片
所以还是要给到实际的可执行文件

@waruqi

This comment has been minimized.

Copy link
Member

commented May 17, 2019

这样其实比传递dir更好处理些

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 17, 2019

又踩到一个坑,在windows上只支持cl.exe,设别的它也会试图用cl的调用方式调用,即使clang-cl也会报

nvcc fatal   : Host compiler targets unsupported OS.

图片
图片
图片

@waruqi

This comment has been minimized.

Copy link
Member

commented May 17, 2019

那win上就没必要传ccbin了吧,仅对linux/macos 下设置下ccbin 绑定cxx

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 17, 2019

mac上标的也是只支持clang啊,可惜我没机器测

@waruqi waruqi added the improvement label May 17, 2019

@waruqi waruqi added this to the v2.2.6 milestone May 17, 2019

@waruqi

This comment has been minimized.

Copy link
Member

commented May 17, 2019

我在linux/macos上加上支持了,你更新下dev分支试试,默认会透传cxx的配置,你也可以手动通过xmake f --cu-cxx=gcc 来修改传入nvcc的cxx编译器

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 18, 2019

并没有自动用cxx的配置啊,我搞错了什么吗?

➜  PowerFlow git:(master) ✗ xmake f -c                                    
checking for the architecture ... x86_64
checking for the Cuda SDK directory ... /usr/local/cuda
➜  PowerFlow git:(master) ✗ xmake f --cc=clang --cxx=clang++ --ld=clang++ -v
checking for the architecture ... x86_64
checking for the Cuda SDK directory ... /usr/local/cuda
checking for the gcc ... /usr/bin/gcc
checking for the cuda host c++ compiler (cu-cxx) ... gcc
configure
{
    ld = clang++
    cxx = clang++
    mode = release
    ccache = true
    host = linux
    cu-cxx = /usr/bin/gcc
    plat = linux
    kind = static
    buildir = build
    cuda = /usr/local/cuda
    cc = clang
    arch = x86_64
}


然后就直接-ccbin /usr/bin/gcc

@waruqi

This comment has been minimized.

Copy link
Member

commented May 18, 2019

嗯 如果不手动改cxx的话 ,cxx 和ccbin 是按相同规则检测的,完全同步,如果手动改写cxx的话,需要手动配置xmake f --cu-cxx=gcc --cxx=gcc

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 19, 2019

好的

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 31, 2019

还需要个 -cu-ld 来指定 nvcc 当连接器用的时候给的 -ccbin 参数

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 31, 2019

未指定

➜  /usr/local/cuda/bin/nvcc -o build/linux/x86_64/release/main build/.objs/main/linux/x86_64/release/./main.cu.o -m64 -Xcompiler -m64 -L/usr/local/cuda/lib -Xlinker -rpath=/usr/local/cuda/lib -dryrun
#$ _SPACE_= 
#$ _CUDART_=cudart
#$ _HERE_=/usr/local/cuda/bin
#$ _THERE_=/usr/local/cuda/bin
#$ _TARGET_SIZE_=
#$ _TARGET_DIR_=
#$ _TARGET_DIR_=targets/x86_64-linux
#$ TOP=/usr/local/cuda/bin/..
#$ NVVMIR_LIBRARY_DIR=/usr/local/cuda/bin/../nvvm/libdevice
#$ LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:
#$ PATH=/usr/local/cuda/bin/../nvvm/bin:/usr/local/cuda/bin:/home/dps/.local/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
#$ INCLUDES="-I/usr/local/cuda/bin/../targets/x86_64-linux/include"  
#$ LIBRARIES=  "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib"
#$ CUDAFE_FLAGS=
#$ PTXAS_FLAGS=
#$ nvlink --arch=sm_30 --register-link-binaries="/tmp/tmpxft_00003ac2_00000000-2_main_dlink.reg.c"  -m64 -L"/usr/local/cuda/lib"   "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib" -cpu-arch=X86_64 "build/.objs/main/linux/x86_64/release/./main.cu.o"  -o "/tmp/tmpxft_00003ac2_00000000-4_main_dlink.sm_30.cubin"
#$ fatbinary --create="/tmp/tmpxft_00003ac2_00000000-3_main_dlink.fatbin" -64 -link "--image=profile=sm_30,file=/tmp/tmpxft_00003ac2_00000000-4_main_dlink.sm_30.cubin" --embedded-fatbin="/tmp/tmpxft_00003ac2_00000000-3_main_dlink.fatbin.c" 
#$ rm /tmp/tmpxft_00003ac2_00000000-3_main_dlink.fatbin
#$ gcc -c -x c++ -DFATBINFILE="\"/tmp/tmpxft_00003ac2_00000000-3_main_dlink.fatbin.c\"" -DREGISTERLINKBINARYFILE="\"/tmp/tmpxft_00003ac2_00000000-2_main_dlink.reg.c\"" -I. -D__NV_EXTRA_INITIALIZATION= -D__NV_EXTRA_FINALIZATION= -D__CUDA_INCLUDE_COMPILER_INTERNAL_HEADERS__  -m64 "-I/usr/local/cuda/bin/../targets/x86_64-linux/include"    -D__CUDACC_VER_MAJOR__=10 -D__CUDACC_VER_MINOR__=1 -D__CUDACC_VER_BUILD__=168 -m64 -o "/tmp/tmpxft_00003ac2_00000000-5_main_dlink.o" "/usr/local/cuda/bin/crt/link.stub" 
#$ g++ -m64 -m64 -o "build/linux/x86_64/release/main" -Wl,--start-group "/tmp/tmpxft_00003ac2_00000000-5_main_dlink.o" "build/.objs/main/linux/x86_64/release/./main.cu.o" -Xlinker -rpath=/usr/local/cuda/lib -L"/usr/local/cuda/lib"   "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib" -lcudadevrt  -lcudart_static  -lrt -lpthread  -ldl  -Wl,--end-group 

指定 clang

➜  /usr/local/cuda/bin/nvcc -o build/linux/x86_64/release/main build/.objs/main/linux/x86_64/release/./main.cu.o -m64 -Xcompiler -m64 -L/usr/local/cuda/lib -Xlinker -rpath=/usr/local/cuda/lib -ccbin clang -dryrun
#$ _SPACE_= 
#$ _CUDART_=cudart
#$ _HERE_=/usr/local/cuda/bin
#$ _THERE_=/usr/local/cuda/bin
#$ _TARGET_SIZE_=
#$ _TARGET_DIR_=
#$ _TARGET_DIR_=targets/x86_64-linux
#$ TOP=/usr/local/cuda/bin/..
#$ NVVMIR_LIBRARY_DIR=/usr/local/cuda/bin/../nvvm/libdevice
#$ LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:
#$ PATH=/usr/local/cuda/bin/../nvvm/bin:/usr/local/cuda/bin:/home/dps/.local/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
#$ INCLUDES="-I/usr/local/cuda/bin/../targets/x86_64-linux/include"  
#$ LIBRARIES=  "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib"
#$ CUDAFE_FLAGS=
#$ PTXAS_FLAGS=
#$ nvlink --arch=sm_30 --register-link-binaries="/tmp/tmpxft_00003ad5_00000000-2_main_dlink.reg.c"  -m64 -L"/usr/local/cuda/lib"   "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib" -cpu-arch=X86_64 "build/.objs/main/linux/x86_64/release/./main.cu.o"  -o "/tmp/tmpxft_00003ad5_00000000-4_main_dlink.sm_30.cubin"
#$ fatbinary --create="/tmp/tmpxft_00003ad5_00000000-3_main_dlink.fatbin" -64 -link "--image=profile=sm_30,file=/tmp/tmpxft_00003ad5_00000000-4_main_dlink.sm_30.cubin" --embedded-fatbin="/tmp/tmpxft_00003ad5_00000000-3_main_dlink.fatbin.c" 
#$ rm /tmp/tmpxft_00003ad5_00000000-3_main_dlink.fatbin
#$ clang -c -x c++ -DFATBINFILE="\"/tmp/tmpxft_00003ad5_00000000-3_main_dlink.fatbin.c\"" -DREGISTERLINKBINARYFILE="\"/tmp/tmpxft_00003ad5_00000000-2_main_dlink.reg.c\"" -I. -D__NV_EXTRA_INITIALIZATION= -D__NV_EXTRA_FINALIZATION= -D__CUDA_INCLUDE_COMPILER_INTERNAL_HEADERS__  -m64 "-I/usr/local/cuda/bin/../targets/x86_64-linux/include"    -D__CUDACC_VER_MAJOR__=10 -D__CUDACC_VER_MINOR__=1 -D__CUDACC_VER_BUILD__=168 -m64 -o "/tmp/tmpxft_00003ad5_00000000-5_main_dlink.o" "/usr/local/cuda/bin/crt/link.stub" 
#$ clang++ -m64 -m64 -o "build/linux/x86_64/release/main" -Wl,--start-group "/tmp/tmpxft_00003ad5_00000000-5_main_dlink.o" "build/.objs/main/linux/x86_64/release/./main.cu.o" -Xlinker -rpath=/usr/local/cuda/lib -L"/usr/local/cuda/lib"   "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib" -lcudadevrt  -lcudart_static  -lrt -lpthread  -ldl  -Wl,--end-group 

指定 clang++

➜  /usr/local/cuda/bin/nvcc -o build/linux/x86_64/release/main build/.objs/main/linux/x86_64/release/./main.cu.o -m64 -Xcompiler -m64 -L/usr/local/cuda/lib -Xlinker -rpath=/usr/local/cuda/lib -ccbin clang++ -dryrun
#$ _SPACE_= 
#$ _CUDART_=cudart
#$ _HERE_=/usr/local/cuda/bin
#$ _THERE_=/usr/local/cuda/bin
#$ _TARGET_SIZE_=
#$ _TARGET_DIR_=
#$ _TARGET_DIR_=targets/x86_64-linux
#$ TOP=/usr/local/cuda/bin/..
#$ NVVMIR_LIBRARY_DIR=/usr/local/cuda/bin/../nvvm/libdevice
#$ LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:
#$ PATH=/usr/local/cuda/bin/../nvvm/bin:/usr/local/cuda/bin:/home/dps/.local/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
#$ INCLUDES="-I/usr/local/cuda/bin/../targets/x86_64-linux/include"  
#$ LIBRARIES=  "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib"
#$ CUDAFE_FLAGS=
#$ PTXAS_FLAGS=
#$ nvlink --arch=sm_30 --register-link-binaries="/tmp/tmpxft_00003ae8_00000000-2_main_dlink.reg.c"  -m64 -L"/usr/local/cuda/lib"   "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib" -cpu-arch=X86_64 "build/.objs/main/linux/x86_64/release/./main.cu.o"  -o "/tmp/tmpxft_00003ae8_00000000-4_main_dlink.sm_30.cubin"
#$ fatbinary --create="/tmp/tmpxft_00003ae8_00000000-3_main_dlink.fatbin" -64 -link "--image=profile=sm_30,file=/tmp/tmpxft_00003ae8_00000000-4_main_dlink.sm_30.cubin" --embedded-fatbin="/tmp/tmpxft_00003ae8_00000000-3_main_dlink.fatbin.c" 
#$ rm /tmp/tmpxft_00003ae8_00000000-3_main_dlink.fatbin
#$ clang++ -c -x c++ -DFATBINFILE="\"/tmp/tmpxft_00003ae8_00000000-3_main_dlink.fatbin.c\"" -DREGISTERLINKBINARYFILE="\"/tmp/tmpxft_00003ae8_00000000-2_main_dlink.reg.c\"" -I. -D__NV_EXTRA_INITIALIZATION= -D__NV_EXTRA_FINALIZATION= -D__CUDA_INCLUDE_COMPILER_INTERNAL_HEADERS__  -m64 "-I/usr/local/cuda/bin/../targets/x86_64-linux/include"    -D__CUDACC_VER_MAJOR__=10 -D__CUDACC_VER_MINOR__=1 -D__CUDACC_VER_BUILD__=168 -m64 -o "/tmp/tmpxft_00003ae8_00000000-5_main_dlink.o" "/usr/local/cuda/bin/crt/link.stub" 
#$ clang++ -m64 -m64 -o "build/linux/x86_64/release/main" -Wl,--start-group "/tmp/tmpxft_00003ae8_00000000-5_main_dlink.o" "build/.objs/main/linux/x86_64/release/./main.cu.o" -Xlinker -rpath=/usr/local/cuda/lib -L"/usr/local/cuda/lib"   "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib" -lcudadevrt  -lcudart_static  -lrt -lpthread  -ldl  -Wl,--end-group 
@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 31, 2019

不过现在已经有一个 cu-ld 了?貌似是可以链接的时候不用nvcc?

@waruqi

This comment has been minimized.

Copy link
Member

commented May 31, 2019

默认设置的cu-ld=nvcc去链接,你也可以指定 cu-ld=gcc/g++ 去直接链接试试,应该也是可以的。。

@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 31, 2019

图片
是可以链接,但是看上面的 -dryrun的结果,会漏掉nvlink生成的步骤(灰色部分)

@waruqi

This comment has been minimized.

Copy link
Member

commented May 31, 2019

把 nvcc --ccbin gcc 全部设置到 --cu-ld就行了。

$ xmake f -c --cu-ld="/Developer/NVIDIA/CUDA-10.1/bin/nvcc -ccbin gcc"
$ xmake -r -v
[100%]: linking.release cuda_console
"/Developer/NVIDIA/CUDA-10.1/bin/nvcc -ccbin gcc" -o build/macosx/x86_64/release/cuda_console build/.objs/cuda_console/macosx/x86_64/release/src/main.cu.o -gencode arch=compute_30
,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_6
0,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_70,code=compute_70 -m64 -Xlinker -arch -Xlinker x86_64 -L/Developer/NVID
IA/CUDA-10.1/lib -Xlinker -rpath -Xlinker /Developer/NVIDIA/CUDA-10.1/lib
@OpportunityLiu

This comment has been minimized.

Copy link
Contributor Author

commented May 31, 2019

无所谓了,clang编译完用默认的gcc链接也不会怎么样,反正折腾了半天 nvlink 一碰 -flto 生成的 .o 就炸,还轮不到clang和gcc冲突

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.