Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I spent 4 days trying to compile the C++ library of this library, and this is the only solution that worked. #707

Open
EternalSaga opened this issue Jun 27, 2024 · 4 comments

Comments

@EternalSaga
Copy link

EternalSaga commented Jun 27, 2024

Here is the Dockerfile which can build the C++ library successfully, but it requires several workarounds for successful building.
Please make it easier for C++ inference. I'd appreciate any suggestions for streamlining the process.

FROM nvidia/cuda:12.2.2-cudnn8-devel-ubuntu22.04 AS build_stage
ENV TZ=Asia/Shanghai
ENV DEBIAN_FRONTEND=noninteractive
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt update && apt install -y lsb-release wget software-properties-common gnupg 
RUN sed -i 's/archive.ubuntu.com/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list
RUN apt update && apt install lsb-release wget software-properties-common gnupg -y && apt update && apt upgrade -y && apt install -y vim wget gdb pkg-config autoconf automake build-essential git libpoco-dev uuid-dev libncurses5-dev python3-dev python3-pip \
    libtool libc6 libc6-dev unzip libnuma1 libnuma-dev libturbojpeg0-dev apt-utils \
    && python3 -m pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --upgrade pip

WORKDIR /workspace
RUN git clone https://github.com/traveller59/spconv.git && git clone https://github.com/FindDefinition/cumm
# set environmnet variables for building cumm
ENV CUMM_CUDA_VERSION=12.2
ENV CUMM_DISABLE_JIT=1
ENV SPCONV_DISABLE_JIT=1
ENV CUMM_INCLUDE_PATH=/workspace/cumm/include
ENV CUMM_CUDA_ARCH_LIST="8.9"
# install pccm for building cumm wheel
RUN pip3 install wheel pccm -i https://pypi.tuna.tsinghua.edu.cn/simple
RUN cd /workspace/cumm && git checkout v0.5.3 && python3 setup.py bdist_wheel && pip3 install dist/cumm_cu122-0.5.3-cp310-cp310-linux_x86_64.whl
# remove cumm depandance for building spconv
COPY pyproject.toml /workspace/spconv/pyproject.toml
ENV PATH=/usr/local/cuda-12.2/bin${PATH:+:${PATH}}
ENV LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
RUN apt install cmake -y
# It must build spconv wheel first, then use cmake to build c++ shared library.
# for cmake, I have to add gcc flags to make it could find headers and cudart.so
RUN cd /workspace/spconv && python3 setup.py bdist_wheel && python3 -m spconv.gencode --include=./cpp/include --src=./cpp/src && \
    cd cpp/ && mkdir build && cd src && cmake -DCMAKE_CXX_FLAGS="-I/workspace/spconv/cpp/include -L/usr/local/cuda/lib64/" . -B ../build && \
    cmake --build ../build --config Release -j 8

FROM scratch AS export
COPY --from=build_stage /workspace/spconv/cpp .

Please notice that the pyproject.toml file should be prepared first which used to be copied into the building container.

Then run:

docker build -t spconv:x86 --output ./out .

You can get the libspconv.so in your output direcotry.

@barrydoooit
Copy link

barrydoooit commented Jun 28, 2024

Hi @EternalSaga,
Thanks for the script. I tried to put your commands into a bash script like:

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

#git clone https://github.com/FindDefinition/cumm.git $SCRIPT_DIR/cumm

export CUMM_CUDA_VERSION=11.4 # cuda version, required but only used for flag selection when build libspconv.
export CUMM_DISABLE_JIT=1
export SPCONV_DISABLE_JIT=1
export CUMM_INCLUDE_PATH=$SCRIPT_DIR/cumm/include # if you use cumm as a subdirectory, you need this to find cumm includes.
export CUMM_CUDA_ARCH_LIST="8.6" # cuda arch flags

pip3 install wheel pccm -i https://pypi.tuna.tsinghua.edu.cn/simple
cd $SCRIPT_DIR/cumm && git checkout v0.3.7 && python3 setup.py bdist_wheel # cumm has already been installed

export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

cd $SCRIPT_DIR/spconv && python3 setup.py bdist_wheel && python3 -m spconv.gencode --include=./cpp/include --src=./cpp/src && \
    cd cpp/  && cd src && cmake -DCMAKE_CXX_FLAGS="-I/builds/libspconv/spconv/cpp/include -L/usr/local/cuda/lib64/" . -B ../build && \
    cmake --build ../build --config Release -j 8

but got the following error about accessing the header files (though they do exist). Have you ever encountered such issues? I suppose there would be some additional settings in CMakeLists.

-- Configuring done
-- Generating done
-- Build files have been written to: /builds/libspconv/spconv/cpp/build
[ 1%] Building CXX object spconvlib/cumm/conv/main/ConvMainUnitTest/CMakeFiles/spconvlib_cumm_conv_main_ConvMainUnitTest.dir/ConvMainUnitTest_extract_mnk.cc.o
[ 2%] Building CXX object spconvlib/spconv/csrc/sparse/alloc/StaticAllocator/CMakeFiles/spconvlib_spconv_csrc_sparse_alloc_StaticAllocator.dir/StaticAllocator_StaticAllocator.cc.o
[ 2%] Building CXX object spconvlib/cumm/gemm/main/GemmMainUnitTest/CMakeFiles/spconvlib_cumm_gemm_main_GemmMainUnitTest.dir/GemmMainUnitTest_get_all_algo_desp.cc.o
[ 3%] Building CXX object spconvlib/spconv/csrc/sparse/alloc/ExternalAllocatorGuard/CMakeFiles/spconvlib_spconv_csrc_sparse_alloc_ExternalAllocatorGuard.dir/ExternalAllocatorGuard_ExternalAllocatorGuard.cc.o
[ 4%] Building CXX object spconvlib/spconv/csrc/sparse/alloc/ExternalAllocator/CMakeFiles/spconvlib_spconv_csrc_sparse_alloc_ExternalAllocator.dir/ExternalAllocator_zeros_guard.cc.o
[ 4%] Building CXX object spconvlib/spconv/csrc/sparse/convops/GemmTuneResult/CMakeFiles/spconvlib_spconv_csrc_sparse_convops_GemmTuneResult.dir/GemmTuneResult_is_valid.cc.o
[ 5%] Building CXX object spconvlib/cumm/common/CompileInfo/CMakeFiles/spconvlib_cumm_common_CompileInfo.dir/CompileInfo_get_compiled_cuda_arch.cc.o
[ 6%] Building CXX object spconvlib/spconv/csrc/sparse/convops/gemmops/GemmTunerSimple/CMakeFiles/spconvlib_spconv_csrc_sparse_convops_gemmops_GemmTunerSimple.dir/GemmTunerSimple_GemmTunerSimple.cc.o
/builds/libspconv/spconv/cpp/src/spconvlib/cumm/common/CompileInfo/CompileInfo_get_compiled_cuda_arch.cc:1:10: fatal error: spconvlib/cumm/common/CompileInfo.h: No such file or directory
1 | #include <spconvlib/cumm/common/CompileInfo.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/builds/libspconv/spconv/cpp/src/spconvlib/cumm/conv/main/ConvMainUnitTest/ConvMainUnitTest_extract_mnk.cc:1:10: fatal error: spconvlib/cumm/conv/main/ConvMainUnitTest.h: No such file or directory
1 | #include <spconvlib/cumm/conv/main/ConvMainUnitTest.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/builds/libspconv/spconv/cpp/src/spconvlib/cumm/gemm/main/GemmMainUnitTest/GemmMainUnitTest_get_all_algo_desp.cc:1:10: fatal error: spconvlib/cumm/gemm/main/GemmMainUnitTest.h: No such file or directory
1 | #include <spconvlib/cumm/gemm/main/GemmMainUnitTest.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/builds/libspconv/spconv/cpp/src/spconvlib/spconv/csrc/sparse/alloc/StaticAllocator/StaticAllocator_StaticAllocator.cc:1:10: fatal error: spconvlib/spconv/csrc/sparse/alloc/StaticAllocator.h: No such file or directory
1 | #include <spconvlib/spconv/csrc/sparse/alloc/StaticAllocator.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/builds/libspconv/spconv/cpp/src/spconvlib/spconv/csrc/sparse/alloc/ExternalAllocator/ExternalAllocator_zeros_guard.cc:1:10: fatal error: spconvlib/spconv/csrc/sparse/alloc/ExternalAllocator.h: No such file or directory
1 | #include <spconvlib/spconv/csrc/sparse/alloc/ExternalAllocator.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
/builds/libspconv/spconv/cpp/src/spconvlib/spconv/csrc/sparse/convops/GemmTuneResult/GemmTuneResult_is_valid.cc:1:10: fatal error: spconvlib/spconv/csrc/sparse/convops/GemmTuneResult.h: No such file or directory
1 | #include <spconvlib/spconv/csrc/sparse/convops/GemmTuneResult.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

@barrydoooit
Copy link

I found out that I forgot to prepare spconv/CMakeLists.txt. It works like a charm with this repo's example:

cmake_minimum_required(VERSION 3.20)

project(spconv LANGUAGES CXX CUDA)
include_directories(/builds/libspconv/include)

add_subdirectory(src)
# tell parent spconv2 include path.
set(SPCONV2_INCLUDE_PATH /builds/libspconv/spconv/include PARENT_SCOPE)

install (TARGETS spconv
         ARCHIVE DESTINATION lib
         LIBRARY DESTINATION lib
         RUNTIME DESTINATION bin)

@EternalSaga
Copy link
Author

EternalSaga commented Jun 29, 2024

@barrydoooit
Thank you for your suggestion. I didn't notice that there is a CMakeLists.txt in this library. I use the cmake cli arguments ("-DCMAKE_CXX_FLAGS="-I/workspace/spconv/cpp/include -L/usr/local/cuda/lib64/") as a workaround. If you use my solution, please make sure you use the correct include directory path and the cuda library path.

@EternalSaga
Copy link
Author

A little bit improvements with barrydoooit's suggestion.

FROM nvidia/cuda:12.2.2-cudnn8-devel-ubuntu22.04 AS basebuild
ENV TZ=Asia/Shanghai
ENV DEBIAN_FRONTEND=noninteractive
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt update && apt install -y lsb-release wget software-properties-common gnupg 
RUN sed -i 's/archive.ubuntu.com/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list
RUN apt update && apt install lsb-release wget software-properties-common gnupg -y && apt update && apt upgrade -y && apt install -y vim wget gdb pkg-config autoconf automake build-essential git libpoco-dev uuid-dev libncurses5-dev python3-dev python3-pip \
    libtool libc6 libc6-dev unzip libnuma1 libnuma-dev libturbojpeg0-dev apt-utils \
    && python3 -m pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --upgrade pip
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1A127079A92F09ED && apt-add-repository 'deb https://apt.kitware.com/ubuntu/ jammy main' && apt update && apt install -y cmake
WORKDIR /workspace


FROM basebuild as build_stage
RUN git clone https://github.com/traveller59/spconv.git && git clone https://github.com/FindDefinition/cumm
# set environmnet variables for building cumm
ENV CUMM_CUDA_VERSION=12.2
ENV CUMM_DISABLE_JIT=1
ENV SPCONV_DISABLE_JIT=1
ENV CUMM_INCLUDE_PATH=/workspace/cumm/include
ENV CUMM_CUDA_ARCH_LIST="8.9"
# install pccm for building cumm wheel
RUN pip3 install wheel pccm -i https://pypi.tuna.tsinghua.edu.cn/simple
RUN cd /workspace/cumm && git checkout v0.5.3 && python3 setup.py bdist_wheel && pip3 install dist/cumm_cu122-0.5.3-cp310-cp310-linux_x86_64.whl
# remove cumm depandance for building spconv
COPY pyproject.toml /workspace/spconv/pyproject.toml
ENV PATH=/usr/local/cuda-12.2/bin${PATH:+:${PATH}}
ENV LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
RUN cd /workspace/spconv && mkdir ./cpp

# from example/libspconv/spconv/CMakeLists.txt
COPY CMakeLists.txt /workspace/spconv/cpp/CMakeLists.txt
RUN cd /workspace/spconv && python3 setup.py bdist_wheel && python3 -m spconv.gencode --include=./cpp/include --src=./cpp/src
RUN cd /workspace/spconv/cpp/ && mkdir build && cmake . -B ./build && \
    cmake --build ./build --config Release --parallel $(nproc) --target install
RUN mkdir -p ./libspconv/lib && mkdir -p ./libspconv/include
# make sure the cumm headers have been moved out with spconv heads
RUN mv /usr/local/lib/libspconv.so ./libspconv/lib && mv /workspace/spconv/cpp/include ./libspconv/ && mv /workspace/cumm/include/tensorview ./libspconv/include/

FROM basebuild
RUN apt install -y gcc-12 g++-12
RUN wget https://apt.llvm.org/llvm.sh && chmod +x ./llvm.sh && ./llvm.sh 18 -s && rm ./llvm.sh 
COPY --from=build_stage /workspace/libspconv /usr/local/libspconv

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants