PyTorch Integration with Tensor Comprehensions #72
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu14.04 | ||
|
There was a problem hiding this comment. FMI, why do we rewrite a full new Dockerfile rather than build on top of what we have been using in CI? There was a problem hiding this comment. the one in CI doesn't have GPU but for building conda packages, I need to run python test which basically need GPU in all cases. Hence building the docker image which has GPUs. You can read how nvidia-docker works. I provided the readme instructions here for using and building this image. There was a problem hiding this comment. The docker image can't have GPUs, you can't download them :) All our trusty docker images used to inherit from this Still, when providing the Docker for conda_recipes you are using a very similar Docker base than what I wrote a few months back which got me confused. Does this mean we should uniformize what is in here and revert back to the old docker files like you did in this PR? I'll open a task specifically for this. There was a problem hiding this comment. The docker files will be moved from here to a separate repo https://github.com/prigoyal/tensorcomp-dockerfiles that I have setup for Jenkins setup. The question about whether the images should be tiered or not is something that I am discussing with @pietern and @ezyang already. Please open the issue in the above repo to centralize discussions properly.
Not quite sure, the docker images follow quite different structure now and provide more safeguarding.
why did we inherit from those earlier? did you use nvidia-smi or did you ever run gpu test in that docker image? :)
yeah, that's why we use
nope, setting up CircleCI to properly use the gpu images we build is cumbersome, so we are going to use AWS jenkins setup soon. There was a problem hiding this comment.
No since there is no cuda driver on the CPU-only CI machines. In particular you don't need a physical GPU to generate PTX and see whether compilation passes (e.g. NVRTC + CUB)... There was a problem hiding this comment. you can continue doing that, no need to start from NVIDIA images. the toolkit is installed in the images as you also pointed out. Nvidia-docker images are useless if you don't run CI gpu tests with it. |
||
|
|
||
| ENV DEBIAN_FRONTEND noninteractive | ||
|
|
||
| RUN apt-get update | ||
|
|
||
| RUN apt-get install -y --no-install-recommends make git ssh realpath wget unzip cmake3 vim | ||
| RUN apt-get install -y --no-install-recommends libgoogle-glog-dev libyaml-dev | ||
| RUN apt-get install -y --no-install-recommends libgtest-dev libz-dev libgmp3-dev | ||
| RUN apt-get install -y --no-install-recommends automake libtool valgrind subversion | ||
| RUN apt-get install -y --no-install-recommends ca-certificates software-properties-common | ||
|
|
||
| RUN cmake --version | ||
|
|
||
| # GCC 4.8.* | ||
| RUN add-apt-repository ppa:ubuntu-toolchain-r/test | ||
| RUN apt-get update | ||
| RUN apt-get install -y --no-install-recommends libcilkrts5 gcc-4.8 g++-4.8 | ||
|
There was a problem hiding this comment. could you explain why it is required to install There was a problem hiding this comment. It should but this is done as a safety measure. |
||
| RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50 | ||
| RUN update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 50 | ||
|
|
||
| ENV CC /usr/bin/gcc | ||
| ENV CXX /usr/bin/g++ | ||
|
|
||
| # LLVM+Clang | ||
| ENV LLVM_SOURCES /tmp/llvm_sources-tapir5.0 | ||
| WORKDIR $LLVM_SOURCES | ||
|
|
||
| ENV CLANG_PREFIX /usr/local/clang+llvm-tapir5.0 | ||
| ENV CMAKE_VERSION cmake | ||
|
|
||
| RUN git clone --recursive https://github.com/wsmoses/Tapir-LLVM llvm && \ | ||
| mkdir -p ${LLVM_SOURCES}/llvm_build && cd ${LLVM_SOURCES}/llvm_build && \ | ||
| ${CMAKE_VERSION} -DLLVM_ENABLE_EH=ON -DLLVM_ENABLE_OCAMLDOC=OFF -DLLVM_INSTALL_OCAMLDOC_HTML_DIR=/tmp -DLLVM_OCAML_INSTALL_PATH=/tmp -DCMAKE_INSTALL_PREFIX=${CLANG_PREFIX} -DLLVM_TARGETS_TO_BUILD=X86 -DCOMPILER_RT_BUILD_CILKTOOLS=OFF -DLLVM_ENABLE_CXX1Y=ON -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_BUILD_TESTS=OFF -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_BUILD_TYPE=Release -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_ENABLE_RTTI=ON ../llvm/ && \ | ||
| make -j"$(nproc)" -s && make install -j"$(nproc)" -s | ||
|
|
||
| RUN rm -Rf ${LLVM_SOURCES} | ||
|
|
||
| # Anaconda3 | ||
| ENV LD_LIBRARY_PATH /usr/local/cuda/lib64:/usr/local/cuda/targets/x86_64-linux/lib/stubs/:$LD_LIBRARY_PATH | ||
| ENV PATH /usr/local/bin:/usr/local/cuda/bin:$PATH | ||
|
|
||
| WORKDIR /conda-install | ||
| RUN echo 'export PATH=/opt/conda/bin:$PATH' > /etc/profile.d/conda.sh &&\ | ||
| wget --quiet https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh -O anaconda.sh && \ | ||
| chmod +x anaconda.sh && \ | ||
| ./anaconda.sh -b -p /opt/conda && \ | ||
| rm anaconda.sh | ||
|
|
||
| ENV PATH /opt/conda/bin:$PATH | ||
|
|
||
| RUN conda install numpy decorator six future cmake pyyaml | ||
|
|
||
| # Protobuf 3.4* | ||
| WORKDIR /proto-install | ||
| RUN wget --quiet https://github.com/google/protobuf/archive/v3.4.0.zip -O proto.zip && unzip -qq proto.zip -d / | ||
|
|
||
| RUN cd /protobuf-3.4.0 && ./autogen.sh && ./configure && make -j 8 | ||
| RUN cd /protobuf-3.4.0 && make install && ldconfig | ||
|
|
||
| RUN which conda | ||
| RUN conda --version | ||
| RUN which protoc | ||
| RUN protoc --version | ||
| RUN which python | ||
| RUN python --version | ||
|
|
||
| CMD ["bash"] | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
don't we anymore?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we do but this has been more properly highlighted in next steps when we are searching for libaten in pytorch install