[Misc] Define common requirements

[WoosukKwon]

This PR factors out the common dependencies to requirements-common.txt to better manage the dependencies across different backends.

[WoosukKwon]

Tbh, I'm not sure whether introducing requirements-common.txt is a good decision, since people can mistakenly skip pip install -r requirements-common.txt.

Alternatively, we can duplicate the contents of requirements-common.txt to all requirements-*.txt, while clearly separating the common requirements and backend-specific requirements. For example, requirements-coda.txt can be:

# Common dependencies (must be same for all `requirements-*.txt`)
cmake>=3.21
ninja  # For faster builds.
psutil
sentencepiece  # Required for LLaMA tokenizer.
numpy
requests
py-cpuinfo
transformers >= 4.39.1  # Required for StarCoder2 & Llava.
fastapi
uvicorn[standard]
pydantic >= 2.0  # Required for OpenAI server.
prometheus_client >= 0.18.0
outlines == 0.0.34
tiktoken == 0.6.0  # Required for DBRX tokenizer

# Dependencies for NVIDIA GPUs
ray >= 2.9
torch == 2.1.2
xformers == 0.0.23.post1  # Required for CUDA 12.1.
pynvml == 11.5.0
triton >= 2.1.0

while requirements-neuron.txt can be:

# Common dependencies (must be same for all `requirements-*.txt`)
cmake>=3.21
ninja  # For faster builds.
psutil
sentencepiece  # Required for LLaMA tokenizer.
numpy
requests
py-cpuinfo
transformers >= 4.39.1  # Required for StarCoder2 & Llava.
fastapi
uvicorn[standard]
pydantic >= 2.0  # Required for OpenAI server.
prometheus_client >= 0.18.0
outlines == 0.0.34
tiktoken == 0.6.0  # Required for DBRX tokenizer

# Dependencies for Neuron devices
transformers-neuronx >= 0.9.0
torch-neuronx >= 2.1.0
neuronx-cc

[njhill]

Tbh, I'm not sure whether introducing requirements-common.txt is a good decision, since people can mistakenly skip pip install -r requirements-common.txt.

@WoosukKwon you can ref one requirements file from another, so requirements-cuda.txt could be:

# Common dependencies
-r requirements-common.txt

# Dependencies for NVIDIA GPUs
ray >= 2.9
torch == 2.1.2
xformers == 0.0.23.post1  # Required for CUDA 12.1.
pynvml == 11.5.0
triton >= 2.1.0

[njhill]

Thanks @WoosukKwon, I really like the de-duplication!

[youkaichao]

@WoosukKwon #3859 is merged. If you merge main into this PR, it should resolve the vllm-nccl problem.