Can create LLM endpoints

[ruizehung-scale]

Summary

• Add missing a100 and t4 devices in values_sample.yaml that prevent llm-engine-cacher from starting successfully.
• Update http-forwarder container python command path for streaming endpoint
• Fix TGI repo name and forwarder config path
• Fix forwarder aws config mount path

Test Plan

data = {
    "name": "llama-7b",
    "model_name": "llama-7b",
    "source": "hugging_face",
    "inference_framework": "text_generation_inference",
    "inference_framework_image_tag": "0.9.1",
    "num_shards": 4,
    "endpoint_type": "streaming",
    "cpus": 32,
    "gpus": 4,
    "memory": "40Gi",
    "storage": "40Gi",
    "gpu_type": "nvidia-ampere-a10",
    "min_workers": 1,
    "max_workers": 12,
    "per_worker": 1,
    "labels": {"team": "infra", "product": "llm_model_zoo"},
    "metadata": {}
}
headers = {'Content-Type': 'application/json'}
response = requests.post("http://localhost:5000/v1/llm/model-endpoints", headers=headers, data=json.dumps(data), auth=('test_user_id', ''))
print(response.status_code)
print(response.json())
------------------------
200
{'endpoint_creation_task_id': 'e9a7a99b-6fe6-4e69-97e9-4b007ab2e7bb'}

data = {
    "prompts": ["hi"],
    "max_new_tokens": 10,
    "temperature": 0.1
}

headers = {'Content-Type': 'application/json'}
response = requests.post("http://localhost:5000/v1/llm/completions-sync?model_endpoint_name=llama-7b", headers=headers, data=json.dumps(data), auth=('test-user-id', ''))
print(response.status_code)
print(response.json())
------------------------
200
{'status': 'SUCCESS', 'outputs': [{'text': 'hi hi hi2 hi2 hi2 hi2', 'num_completion_tokens': 10}], 'traceback': None}

[ruizehung-scale]

@yunfeng-scale It turns out I need to update TGI repo name in order to skip image existence check in ECR repo given the logic here

llm-engine/server/llm_engine_server/domain/use_cases/model_bundle_use_cases.py

Line 386 in 1370d79

and self.docker_repository.is_repo_name(request.flavor.repository)

Is this change reasonable? Should we back propagate this back to hmi as well?

[yunfeng-scale]

we shouldn't hardcode this since it diverged internal / OSS code, can you add this as a parameter?

[yunfeng-scale]

yes this change makes sense

[song-william]

note: We may need to parameterize this entirely.

[song-william]

Note that this may need to be parameterized as well.

[song-william]

Discussed offline that we will parameterize these values in a later PR. Merging as is.