Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't install Triton #22

Closed
StellaAthena opened this issue Jan 1, 2021 · 2 comments · Fixed by #24
Closed

Can't install Triton #22

StellaAthena opened this issue Jan 1, 2021 · 2 comments · Fixed by #24
Assignees
@lucidrains @sdtblck
Labels
bug Something isn't working

Comments

@StellaAthena
Copy link
Member

I booted up the server and figured I would try giving the code a whirl. I appear to be unable to successfully run pip install -r requirements.txt, with it throwing errors when it got to Triton. The first problem was that I didn't have llvm-9-dev installed, but apt-get install llvm-9-dev took care of that. The second problem was that I had to run pip install CMake. Now both packages are installed and yet pip is throwing it's most complicated error yet. Below is the full output

    Running setup.py install for triton ... error
    ERROR: Command errored out with exit status 1:
     command: /root/anaconda3/envs/ds/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-gwvnpboi/triton_22306e539ed64a7a9b6510483f530faa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-gwvnpboi/triton_22306e539ed64a7a9b6510483f530faa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ukz_au5s/install-record.txt --single-version-externally-managed --compile --install-headers /root/anaconda3/envs/ds/include/python3.7m/triton
         cwd: /tmp/pip-install-gwvnpboi/triton_22306e539ed64a7a9b6510483f530faa/
    Complete output (193 lines):
    /root/anaconda3/envs/ds/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'keyword'
      warnings.warn(msg)
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/triton
    copying triton/__init__.py -> build/lib.linux-x86_64-3.7/triton
    copying triton/kernel.py -> build/lib.linux-x86_64-3.7/triton
    package init file 'triton/_C/__init__.py' not found (or not a regular file)
    creating build/lib.linux-x86_64-3.7/triton/ops
    copying triton/ops/__init__.py -> build/lib.linux-x86_64-3.7/triton/ops
    copying triton/ops/batchnorm.py -> build/lib.linux-x86_64-3.7/triton/ops
    copying triton/ops/einsum.py -> build/lib.linux-x86_64-3.7/triton/ops
    creating build/lib.linux-x86_64-3.7/triton/_C
    creating build/lib.linux-x86_64-3.7/triton/_C/include
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen
    copying triton/_C/include/triton/codegen/pass.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen
    copying triton/_C/include/triton/codegen/target.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/analysis
    copying triton/_C/include/triton/codegen/analysis/align.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/analysis
    copying triton/_C/include/triton/codegen/analysis/allocation.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/analysis
    copying triton/_C/include/triton/codegen/analysis/axes.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/analysis
    copying triton/_C/include/triton/codegen/analysis/layout.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/analysis
    copying triton/_C/include/triton/codegen/analysis/liveness.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/analysis
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/selection
    copying triton/_C/include/triton/codegen/selection/generator.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/selection
    copying triton/_C/include/triton/codegen/selection/machine_layout.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/selection
    copying triton/_C/include/triton/codegen/selection/machine_value.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/selection
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/transform
    copying triton/_C/include/triton/codegen/transform/coalesce.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/transform
    copying triton/_C/include/triton/codegen/transform/cts.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/transform
    copying triton/_C/include/triton/codegen/transform/dce.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/transform
    copying triton/_C/include/triton/codegen/transform/disassociate.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/transform
    copying triton/_C/include/triton/codegen/transform/membar.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/transform
    copying triton/_C/include/triton/codegen/transform/peephole.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/transform
    copying triton/_C/include/triton/codegen/transform/reassociate.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/codegen/transform
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    copying triton/_C/include/triton/driver/backend.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    copying triton/_C/include/triton/driver/buffer.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    copying triton/_C/include/triton/driver/context.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    copying triton/_C/include/triton/driver/device.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    copying triton/_C/include/triton/driver/dispatch.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    copying triton/_C/include/triton/driver/error.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    copying triton/_C/include/triton/driver/handle.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    copying triton/_C/include/triton/driver/kernel.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    copying triton/_C/include/triton/driver/module.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    copying triton/_C/include/triton/driver/platform.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    copying triton/_C/include/triton/driver/stream.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/driver
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/external
    copying triton/_C/include/triton/external/half.hpp -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl_d3d10.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl_d3d11.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl_dx9_media_sharing.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl_dx9_media_sharing_intel.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl_egl.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl_ext.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl_ext_intel.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl_gl.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl_gl_ext.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl_platform.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl_va_api_media_sharing_intel.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/opencl.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl.hpp -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    copying triton/_C/include/triton/external/CL/cl2.hpp -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CL
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CUDA
    copying triton/_C/include/triton/external/CUDA/cuda.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CUDA
    copying triton/_C/include/triton/external/CUDA/nvml.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/external/CUDA
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/basic_block.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/builder.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/constant.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/context.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/context_impl.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/enums.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/function.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/instructions.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/metadata.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/module.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/print.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/type.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/utils.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/value.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    copying triton/_C/include/triton/ir/visitor.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/ir
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/ast.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/code_gen.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/cpp.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/encoding.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/error.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/evaluator.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/mem_pool.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/parser.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/scanner.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/scope.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/token.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/type.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    copying triton/_C/include/triton/lang/visitor.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/lang
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/runtime
    copying triton/_C/include/triton/runtime/arg.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/runtime
    copying triton/_C/include/triton/runtime/function.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/runtime
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/tools
    copying triton/_C/include/triton/tools/graph.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/tools
    copying triton/_C/include/triton/tools/thread_pool.h -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/tools
    copying triton/_C/include/triton/tools/bench.hpp -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/tools
    copying triton/_C/include/triton/tools/sha1.hpp -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/tools
    creating build/lib.linux-x86_64-3.7/triton/_C/include/triton/tools/sys
    copying triton/_C/include/triton/tools/sys/getenv.hpp -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/tools/sys
    copying triton/_C/include/triton/tools/sys/mkdir.hpp -> build/lib.linux-x86_64-3.7/triton/_C/include/triton/tools/sys
    running build_ext
    -- The C compiler identification is GNU 7.5.0
    -- The CXX compiler identification is GNU 7.5.0
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Check for working C compiler: /usr/bin/cc - skipped
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Check for working CXX compiler: /usr/bin/c++ - skipped
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Found LLVM: /usr/lib/llvm-9 (found version "9.0.0")
    -- Adding Python module
    -- Configuring done
    CMake Warning (dev) at CMakeLists.txt:45 (add_library):
      Policy CMP0038 is not set: Targets may not link directly to themselves.
      Run "cmake --help-policy CMP0038" for policy details.  Use the cmake_policy
      command to set the policy and suppress this warning.
    
      Target "triton" links to itself.
    This warning is for project developers.  Use -Wno-dev to suppress it.
    
    -- Generating done
    -- Build files have been written to: /tmp/pip-install-gwvnpboi/triton_22306e539ed64a7a9b6510483f530faa/build/temp.linux-x86_64-3.7
    Scanning dependencies of target triton
    [  1%] Building CXX object CMakeFiles/triton.dir/lib/codegen/analysis/allocation.cc.o
    [  3%] Building CXX object CMakeFiles/triton.dir/lib/codegen/analysis/align.cc.o
    [  5%] Building CXX object CMakeFiles/triton.dir/lib/codegen/analysis/axes.cc.o
    [  7%] Building CXX object CMakeFiles/triton.dir/lib/codegen/analysis/layout.cc.o
    [  9%] Building CXX object CMakeFiles/triton.dir/lib/codegen/analysis/liveness.cc.o
    [ 10%] Building CXX object CMakeFiles/triton.dir/lib/codegen/selection/generator.cc.o
    [ 12%] Building CXX object CMakeFiles/triton.dir/lib/codegen/selection/machine_layout.cc.o
    [ 14%] Building CXX object CMakeFiles/triton.dir/lib/codegen/selection/machine_value.cc.o
    [ 16%] Building CXX object CMakeFiles/triton.dir/lib/codegen/target.cc.o
    /tmp/pip-install-gwvnpboi/triton_22306e539ed64a7a9b6510483f530faa/src/lib/codegen/target.cc:5:10: fatal error: llvm/IR/IntrinsicsNVPTX.h: No such file or directory
     #include "llvm/IR/IntrinsicsNVPTX.h"
              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    CMakeFiles/triton.dir/build.make:185: recipe for target 'CMakeFiles/triton.dir/lib/codegen/target.cc.o' failed
    make[2]: *** [CMakeFiles/triton.dir/lib/codegen/target.cc.o] Error 1
    make[2]: *** Waiting for unfinished jobs....
    CMakeFiles/Makefile2:122: recipe for target 'CMakeFiles/triton.dir/all' failed
    make[1]: *** [CMakeFiles/triton.dir/all] Error 2
    Makefile:113: recipe for target 'all' failed
    make: *** [all] Error 2
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-gwvnpboi/triton_22306e539ed64a7a9b6510483f530faa/setup.py", line 129, in <module>
        'Programming Language :: Python :: 3.6',
      File "/root/anaconda3/envs/ds/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/root/anaconda3/envs/ds/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/root/anaconda3/envs/ds/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/root/anaconda3/envs/ds/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/root/anaconda3/envs/ds/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/root/anaconda3/envs/ds/lib/python3.7/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/root/anaconda3/envs/ds/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/root/anaconda3/envs/ds/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/root/anaconda3/envs/ds/lib/python3.7/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/root/anaconda3/envs/ds/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/root/anaconda3/envs/ds/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-gwvnpboi/triton_22306e539ed64a7a9b6510483f530faa/setup.py", line 55, in run
        self.build_extension(ext)
      File "/tmp/pip-install-gwvnpboi/triton_22306e539ed64a7a9b6510483f530faa/setup.py", line 95, in build_extension
        subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)
      File "/root/anaconda3/envs/ds/lib/python3.7/subprocess.py", line 363, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j4']' returned non-zero exit status 2.
    ----------------------------------------
ERROR: Command errored out with exit status 1: /root/anaconda3/envs/ds/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-gwvnpboi/triton_22306e539ed64a7a9b6510483f530faa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-gwvnpboi/triton_22306e539ed64a7a9b6510483f530faa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ukz_au5s/install-record.txt --single-version-externally-managed --compile --install-headers /root/anaconda3/envs/ds/include/python3.7m/triton Check the logs for full command output.```
@StellaAthena StellaAthena added the bug Something isn't working label Jan 1, 2021
@StellaAthena
Copy link
Member Author

StellaAthena commented Jan 1, 2021 •
edited

After conversing with @sdtblck, I’ve come to understand that this isn’t a problem with the base code and is instead a problem with @lucidrains’s sparse attention code.

In order to run the sparse attention code we appear to need to figure out how to get triton on the server.

StellaAthena pushed a commit that referenced this issue Jan 1, 2021
@lucidrains
fix small bug where sequence length is not passed into attention class (
7043aac 
#21)

* fix small bug where sequence length is not passed into attention class

* fix bug with mask and half values, as well as masking in dense attention

* make sure install deepspeed with pip sudo

This allows `gpt3small` to run but does not fix the problems with sparse attention. See #22
@StellaAthena StellaAthena mentioned this issue Jan 1, 2021
Merged
StellaAthena added a commit that referenced this issue Jan 1, 2021
@StellaAthena @lucidrains
fix small bug where sequence length is not passed into attention class (
3c7a44a 
#21) (#23)

* fix small bug where sequence length is not passed into attention class

* fix bug with mask and half values, as well as masking in dense attention

* make sure install deepspeed with pip sudo

This allows `gpt3small` to run but does not fix the problems with sparse attention. See #22

Co-authored-by: Phil Wang <lucidrains@gmail.com>
@StellaAthena
Copy link
Member Author

Removing Triton from requirements.txt and letting deepspeed install it solved this issue.

@StellaAthena StellaAthena linked a pull request Jan 3, 2021 that will close this issue
GPT-3 Small Works #24
Merged
leogao2 pushed a commit that referenced this issue Feb 13, 2021
@StellaAthena
Merge pull request #22 from EleutherAI/add_configs
796d8af 
add small -> 2-7B train scripts
leogao2 pushed a commit that referenced this issue Feb 13, 2021
@StellaAthena
Merge pull request #22 from EleutherAI/add_configs
4f8e185 
add small -> 2-7B train scripts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lucidrains lucidrains

@sdtblck sdtblck

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

GPT-3 Small Works EleutherAI/gpt-neox
3 participants
@lucidrains @StellaAthena @sdtblck