ABOUT

This is a POC experimenting with CLI-based agent experience.

Running the POC

Note: If running the POC locally ollama can be used as backend. If like to run ollama on OpenShift check ollama-ubi repo for instructions.

• Create a .env file to pass environment variables for the script

# LLM Backends to enable (comma separated list)
LLM_BACKENDS=ollama,tgi
LLM_DEFAULT=ollama         # default LLM backend

# If using Ollama
# Some models: codellama, llama2, mistral:7b-instruct-q4_K_M, mistral
OLLAMA_API_URL=http://localhost:11434   # update to match your Ollama server
OLLAMA_MODEL=mistral:7b-instruct-q4_K_M # update to match your model

# if using Hugging Face TGI local endpont
TGI_API_URL=<url_for_your_tgi_inference_endpoint>
TGI_API_KEY=<optional_key>
TGI_MODEL=<model_name>

# if using OpenAI
OPENAI_API_URL=<local_url> # if using a local OpenAI drop-in replacement
OPENAI_API_KEY=sk-<your_key_here>
OPENAI_MODEL=<model_name>

# if using IBM WatsonX
WATSON_API_URL=<your_watsonx_url> # if using non-default URL
WATSON_API_KEY=<your_watsonx_key>
WATSON_PROJECT_ID=<your_watsonx_project_id> 
WATSON_MODEL=ibm/granite-13b-instruct-v1
#WATSON_MODEL=ibm/granite-13b-chat-v1

# If using BAM research lab
BAM_API_URL=<url_for_api_endpoing>
BAM_API_KEY=<api_key>
BAM_MODEL=<model_name>

• Create a Python 3.11+ virtual environment and install dependencies. NOTE: Must use Python 3.11 or higher.

python3.11 -m venv venv
pip install -r requirements.txt

• Execute the main agent script

python main.py

TO-DO

Current features and progress

• ✓ = completed
• ✗ = not available

Available?	Description
✓	Integration with TGI w/LLAMAv2
✓	Integration with Ollama (any Ollama model)
✓	Integration with WatsonX (any WatsonX model)
✗	Integration with OpenAI (removed)
✓	Class for multi-LLM backend
✓	Prompt templates directory
✓	Runtime switch of prompt template
✓	Debugging hooks with class (use logger class)
✗	Add memory for context continuity during chat
✗	Runtime switch of LLM-backend
✗	Move LLM logic from assist class to dedicated class
✗	Decouple CLI client from backend with client/server archicture
✗	Live connection/retrieval from Kuberenes/OpenShift cluster