[RFE] Additonal messages on prompt

[eloycoto]

Type: feature-request

Description:

When using complex scenarios, we need to teach the LLM how to generate the
response. This technique is a few-shot prompting, where a few messages are part
of the prompt before the human question.

Ollama implemented this using the message array, where the end user define it:

https://github.com/ollama/ollama/blob/dc6fe820512d1046f3a342e384baa64b8ce1758c/docs/api.md?plain=1#L451-L457

I think that in this case, it'll be cool to use something similar which can be appended here:

service/docs/openapi.json

Lines 509 to 595 in 5360e36

	"LLMRequest": {
	"properties": {
	"query": {
	"type": "string",
	"title": "Query"
	},
	"conversation_id": {
	"anyOf": [
	{
	"type": "string"
	},
	{
	"type": "null"
	}
	],
	"title": "Conversation Id"
	},
	"provider": {
	"anyOf": [
	{
	"type": "string"
	},
	{
	"type": "null"
	}
	],
	"title": "Provider"
	},
	"model": {
	"anyOf": [
	{
	"type": "string"
	},
	{
	"type": "null"
	}
	],
	"title": "Model"
	},
	"attachments": {
	"anyOf": [
	{
	"items": {
	"$ref": "#/components/schemas/Attachment"
	},
	"type": "array"
	},
	{
	"type": "null"
	}
	],
	"title": "Attachments"
	}
	},
	"additionalProperties": false,
	"type": "object",
	"required": [
	"query"
	],
	"title": "LLMRequest",
	"description": "Model representing a request for the LLM (Language Model) send into OLS service.\n\nAttributes:\n query: The query string.\n conversation_id: The optional conversation ID (UUID).\n provider: The optional provider.\n model: The optional model.\n attachments: The optional attachments.\n\nExample:\n ```python\n llm_request = LLMRequest(query=\"Tell me about Kubernetes\")\n ```",
	"examples": [
	{
	"attachments": [
	{
	"attachment_type": "log",
	"content": "this is attachment",
	"content_type": "text/plain"
	},
	{
	"attachment_type": "configuration",
	"content": "kind: Pod\n metadata:\n name: private-reg",
	"content_type": "application/yaml"
	},
	{
	"attachment_type": "configuration",
	"content": "foo: bar",
	"content_type": "application/yaml"
	}
	],
	"conversation_id": "123e4567-e89b-12d3-a456-426614174000",
	"model": "gpt-3.5-turbo",
	"provider": "openai",
	"query": "write a deployment yaml for the mongodb image"
	}
	]
	},

Another example can be found on PDL(prompt declaration language) project:
https://github.com/IBM/prompt-declaration-language/blob/572373a09e2d105cf6712859d4be5fb371ba1051/examples/tutorial/calling_llm_with_input_messages.pdl#L5-L10

This, as far as I know:

• Will not break any backward compatibility, it's a new parameter.
• It might override the system prompt given in the olsconfig.yaml.
• Can break the history message placeholder, it depends on where it's located.

Steps needed

• Change openAPI def
• Change history placeholder and validate the effectiveness
• What happens with the system prompt? Override or having an option?

Questions:

• What I should do to get it merged?
• Is this a valid approach?

[eloycoto]

Based on the private feedback.

This is a raw example using few-shot example:

curl http://ollama.acalustra.local:11434/v1/chat/completions -s \
    -H "Content-Type: application/json" \
    -d '{
        "model": "llama3.2:3b",
        "messages": [
            {
                "role": "system",
                "content": "You are an expert at classifying customer feedback. For each piece of feedback, you''ll determine: \n1. Sentiment (Positive/Negative/Neutral)\n2. Category (Product/Service/Support/Other)\n3. Priority (High/Medium/Low)\n4. Action Required (Yes/No)"
            },
            {
                "role": "user",
                "content": "The new dashboard is amazing! So much easier to find everything I need."
            },
            {
                "role": "assistant",
                "content": "{\"sentiment\": \"Positive\", \"category\": \"Product\", \"priority\": \"Low\", \"action_required\": \"No\", \"reasoning\": \"Positive feedback about product improvement, no issues to address\"}"
            },
            {
                "role": "user",
                "content": "I''ve been waiting 3 days for a response from support about my payment issue."
            },
            {
                "role": "assistant",
                "content": "{\"sentiment\": \"Negative\", \"category\": \"Support\", \"priority\": \"High\", \"action_required\": \"Yes\", \"reasoning\": \"Customer dissatisfaction with support response time on billing issue\"}"
            },
            {
                "role": "user",
                "content": "The mobile app sometimes freezes when uploading large files."
            },
            {
                "role": "assistant",
                "content": "{\"sentiment\": \"Negative\", \"category\": \"Product\", \"priority\": \"Medium\", \"action_required\": \"Yes\", \"reasoning\": \"Performance issue affecting functionality but not completely blocking usage\"}"
            },
            {
                "role": "user",
                "content": "Please analyze this feedback: ''Your customer service team was very professional but the software is too expensive for what it offers.''"
            }
        ]
    }' | jq .

When using this, the output is like this, the LLM "learn" how to respond correctly and with clear structured data:

-> ./tests/few-shot-examples.sh
{
  "id": "chatcmpl-857",
  "object": "chat.completion",
  "created": 1729764566,
  "model": "llama3.2:3b",
  "system_fingerprint": "fp_ollama",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\"sentiment\": \"Neutral\", \"category\": \"Product\", \"priority\": \"Low\", \"action_required\": \"Yes\", \"reasoning\": \"Feedback balances positive service experience with critical pricing concern\"}"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 307,
    "completion_tokens": 42,
    "total_tokens": 349
  }
}

When only the system prompt can be used, the output differs a bit:

-> curl http://ollama.acalustra.local:11434/v1/chat/completions -s \
    -H "Content-Type: application/json" \
    -d '{
        "model": "llama3.2:3b",
        "messages": [
            {
                "role": "system",
                "content": "You are an expert at classifying customer feedback. For each piece of feedback, you''ll determine: \n1. Sentiment (Positive/Negative/Neutral)\n2. Category (Product/Service/Support/Other)\n3. Priority (High/Medium/Low)\n4. Action Required (Yes/No)"
            },
            {
                "role": "user",
                "content": "Please analyze this feedback: ''Your customer service team was very professional but the software is too expensive for what it offers.''"
            }
        ]
    }' | jq .

{
  "id": "chatcmpl-37",
  "object": "chat.completion",
  "created": 1729764730,
  "model": "llama3.2:3b",
  "system_fingerprint": "fp_ollama",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Here's my analysis of the feedback:\n\n\n1. Sentiment: **Negative/Negative** (The sentiment is mixed, but overall is on the negative side)\n\nThe customer mentions that the customer service team was professional, which implies a positive aspect. However, they also express dissatisfaction with the software being too expensive, which tips the balance to a negative sentiment.\n\n\n2. Category: **Product (specifically pricing)**\n\nThe feedback specifically mentioned the software's pricing, indicating that it is a product-related issue.\n\n\n3. Priority: **Medium**\n\nWhile the customer expresses frustration with the price, they also mention a positive aspect about the customer service team. This suggests that while the issue is important to them, it is not a deal-breaker. Therefore, I would classify the priority as medium.\n\n\n4. Action Required: **Yes (but likely escalating to Pricing Decision Maker)**\n\nThe fact that the customer mentions the software's price is an issue they wish was different implies a need for consideration or rectification. Given that this is a product-related complaint, it may not directly impact their contact with the support team. However, it suggests that further discussion might be necessary to address the pricing concerns."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 111,
    "completion_tokens": 239,
    "total_tokens": 350
  }
}

Why we need this

When the LLM needs to generate something more than summarize or guide on the current docs, the LLM app needs more context, and large windown context. For our use case, we want to build a very complex json workflow, so we need to give a few examples on how to translate from user input to json.

Examples that we give:

Input: Output:

Input: Output:

Current real demo using custom backend that uses this tech:

https://drive.google.com/file/d/1hEQma6zFCgm0YIFHESjNjaRjj2KqFgkf/view?usp=drive_link

Does this break the current status?

Big no! The original request is like this:

curl -X 'POST' \
  'http://localhost:8080/v1/query' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
      "attachments": [],
      "conversation_id": "123e4567-e89b-12d3-a456-426614174000",
      "model": "llama3.2:3b",
      "provider": "ollama",
      "query": "Please analyze this feedback: ''Your customer service team was very professional but the software is too expensive for what it offers.''"
 }'

And with my proposed changes, will be something like this:

curl -X 'POST' \
  'http://localhost:8080/v1/query' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
      "attachments": [],
      "conversation_id": "123e4567-e89b-12d3-a456-426614174000",
      "model": "llama3.2:3b",
      "provider": "ollama",
      "messages": [
            {
                "role": "user",
                "content": "The new dashboard is amazing! So much easier to find everything I need."
            },
            {
                "role": "assistant",
                "content": "{\"sentiment\": \"Positive\", \"category\": \"Product\", \"priority\": \"Low\", \"action_required\": \"No\", \"reasoning\": \"Positive feedback about product improvement, no issues to address\"}"
            }
      ],
      "query": "Please analyze this feedback: ''Your customer service team was very professional but the software is too expensive for what it offers.''"
 }'

So current users will not have to change the code, and this change will allow new users which needs more "strict" output to build their AI apps on top of this backend.