Unable to run TGI following the instructions on the readme

[xianbaoqian]

System Info

uname -a
Linux a3eb1d6a21b4 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

cargo --version
cargo 1.78.0 (54d8815d0 2024-03-26)

nvidia-smi
Wed Jun 12 07:26:30 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.67 Driver Version: 550.67 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 On | 00000000:81:00.0 Off | Off |
| 30% 28C P8 17W / 450W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

(base) root@C.11121520:~$ which conda                                                                            
/root/miniconda3/bin/conda                                                                                       
(base) root@C.11121520:~$ conda conda create -n text-generation-inference python=3.11                            
usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...                                                           
conda: error: argument COMMAND: invalid choice: 'conda' (choose from 'activate', 'deactivate', 'clean', 'compare'
, 'config', 'create', 'env', 'export', 'info', 'init', 'install', 'list', 'notices', 'package', 'remove', 'uninst
all', 'rename', 'run', 'search', 'update', 'upgrade', 'content-trust', 'doctor', 'repoquery', 'build', 'develop',
 'metapackage', 'render', 'index', 'debug', 'inspect', 'convert', 'skeleton')                                    
(base) root@C.11121520:~$ conda create -n text-generation-inference python=3.11                                  
Channels:                                                                                                        
 - defaults                                                                                                      
Platform: linux-64                                                                                               
Collecting package metadata (repodata.json): done                                                                
Solving environment: done                                                                                        
                                                                                                                 
## Package Plan ##                                                                                               
                                                                                                                 
  environment location: /root/miniconda3/envs/text-generation-inference                   

  added / updated specs:
    - python=3.11


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    openssl-3.0.13             |       h7f8727e_2         5.2 MB
    pip-24.0                   |  py311h06a4308_0         3.3 MB
    python-3.11.9              |       h955ad1f_0        32.9 MB
    setuptools-69.5.1          |  py311h06a4308_0         1.3 MB
    wheel-0.43.0               |  py311h06a4308_0         146 KB
    ------------------------------------------------------------
                                           Total:        42.9 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main 
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu 
  bzip2              pkgs/main/linux-64::bzip2-1.0.8-h5eee18b_6 
  ca-certificates    pkgs/main/linux-64::ca-certificates-2024.3.11-h06a4308_0 
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1 
  libffi             pkgs/main/linux-64::libffi-3.4.4-h6a678d5_1 
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1 
  libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1234567_1 
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1 
  libuuid            pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0 
  ncurses            pkgs/main/linux-64::ncurses-6.4-h6a678d5_0 
  openssl            pkgs/main/linux-64::openssl-3.0.13-h7f8727e_2 
  pip                pkgs/main/linux-64::pip-24.0-py311h06a4308_0 
  python             pkgs/main/linux-64::python-3.11.9-h955ad1f_0 
  readline           pkgs/main/linux-64::readline-8.2-h5eee18b_0 
  setuptools         pkgs/main/linux-64::setuptools-69.5.1-py311h06a4308_0 
  sqlite             pkgs/main/linux-64::sqlite-3.45.3-h5eee18b_0 
  tk                 pkgs/main/linux-64::tk-8.6.14-h39e8969_0 
  tzdata             pkgs/main/noarch::tzdata-2024a-h04d1e81_0 
  wheel              pkgs/main/linux-64::wheel-0.43.0-py311h06a4308_0 
  xz                 pkgs/main/linux-64::xz-5.4.6-h5eee18b_1 
  zlib               pkgs/main/linux-64::zlib-1.2.13-h5eee18b_1 


Proceed ([y]/n)? y


Downloading and Extracting Packages:

Preparing transaction: done 

Verifying transaction: done 
Executing transaction: done 
#                           
# To activate this environment, use
#
#     $ conda activate text-generation-inference
#
# To deactivate an active environment, use
#
#     $ conda deactivate


<!--- This is the problem -->

(base) root@C.11121520:~$ conda activate text-generation-inference
(text-generation-inference) root@C.11121520:~$ text-generation-launcher --model-id mistralai/Mistral-7B-Instruct-
v0.2
bash: text-generation-launcher: command not found

Expected behavior

expects TGI to be running correctly without issues since the above is copied from installation section in the readme.md

[LysandreJik]

Hey @xianbaoqian, could you share a bit about what is the exact problem you're seeing? I see two problems above, and I'm unsure of which one is affecting you, I see this one first:

(base) root@C.11121520:$ conda conda create -n text-generation-inference python=3.11
usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
conda: error: argument COMMAND: invalid choice: 'conda' (choose from 'activate', 'deactivate', 'clean', 'compare'
, 'config', 'create', 'env', 'export', 'info', 'init', 'install', 'list', 'notices', 'package', 'remove', 'uninst
all', 'rename', 'run', 'search', 'update', 'upgrade', 'content-trust', 'doctor', 'repoquery', 'build', 'develop',
'metapackage', 'render', 'index', 'debug', 'inspect', 'convert', 'skeleton')

Here it seems like you're calling conda twice.

I'm happy to help out, but it would be helpful if the formatting of the issue was easier to parse 🤗 thanks!

[xianbaoqian]

Hi @LysandreJik the problem is that after conda activate I still can't use text-generation-launcher as the binary file can't be found:

(base) root@C.11121520:$ conda activate text-generation-inference
(text-generation-inference) root@C.11121520:$ text-generation-launcher --model-id mistralai/Mistral-7B-Instruct-
v0.2
bash: text-generation-launcher: command not found

[xianbaoqian]

I updated the style (surrending it with a code block) to make it easier to read. The original format is indeed a bit misleading!

[LysandreJik]

Hey @xianbaoqian, thanks for clearing the style. There is definitely an issue in the code you run:

$ conda create -n text-generation-inference python=3.11                                  
$ conda activate text-generation-inference
$ text-generation-launcher --model-id mistralai/Mistral-7B-Instruct-v0.2
bash: text-generation-launcher: command not found

Here you're only creating a new conda environment and activating it. You named it text-generation-inference, but that doesn't mean that it's installed.

If you took the example from this part of the README, it seems that you forgot this line:

+ BUILD_EXTENSIONS=True make install # Install repository and HF/transformer fork with CUDA kernels
text-generation-launcher --model-id mistralai/Mistral-7B-Instruct-v0.2

Beware: it's a long installation. Using TGI with the docker image is recommended as much faster.

[rYoussefAli]

I had the same issue. Resolved by adding "$HOME/text-generation-inference/target/release-opt/" to PATH.
Change $HOME to wherever your text-generation-inference directory is.

On another note, cargo (or Rust) version 1.78.0 will not work as the Rust source files are written using inline const feature which is only available in cargo >= 1.79.0 .