Skip to content

Commit

Permalink
LLM micro service extract input model name (#447)
Browse files Browse the repository at this point in the history
No input model from UI.
vLLM and OLLAMA get the model name from the envrionment
TGI mode model form ModelID

Signed-off-by: Wang, Xigui <xigui.wang@intel.com>
  • Loading branch information
xiguiw authored Aug 12, 2024
1 parent 5262d05 commit 761f7e0
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion comps/llms/text-generation/ollama/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
def llm_generate(input: LLMParamsDoc):
ollama = Ollama(
base_url=ollama_endpoint,
model=input.model,
model=input.model if input.model else model_name,
num_predict=input.max_new_tokens,
top_k=input.top_k,
top_p=input.top_p,
Expand Down Expand Up @@ -49,4 +49,5 @@ async def stream_generator():

if __name__ == "__main__":
ollama_endpoint = os.getenv("OLLAMA_ENDPOINT", "http://localhost:11434")
model_name = os.getenv("OLLAMA_MODEL", "meta-llama/Meta-Llama-3-8B-Instruct")
opea_microservices["opea_service@llm_ollama"].start()

0 comments on commit 761f7e0

Please sign in to comment.