.Net: [Handlebars Planner] InsufficientFunctionsForGoal is thrown too often without the need to do so.

[Cotspheer]

Describe the bug
As soon as there is a boolean-function as a plugin available like 'areLawsAvailable' the planner starts to create 'Additional helpers may be required' branches which then lead to exceptions. Currently every second attempt results in an exception which is a challenging user experience. The plan is perfectly fine and could be executed. Trying to inject things like "Never ever include 'Additional helpers may be required'." did not help.

To Reproduce
Steps to reproduce the behavior:

1. Create a plugin with two functions. One that returns a boolean that could used for a conditional check. One to execute an action. For example only turn on the light-bulb if the light-bulb was set to off. So canLightBulbTurnedOff and turnLightBulbOff.
2. Register the plugin
3. Let the Agent/LLM create a plan
4. See error

Expected behavior
Please remove this instruction from the template or give us a way to handle that properly. For example if the template should include such a part. Or let us customize the template.

Platform

• OS: Windows
• IDE: Visual Studio 2022
• Language: C#
• Source: Microsoft.SemanticKernel.Planners.Handlebars 1.0.1-preview
• API: OpenAI
• GPT-Model: gpt-4-1106-preview and gpt-3.5-turbo

Additional context

The generated plan:

{{!-- Step 1: Retrieve the current chat history --}}
{{set "chatHistory" (SomeApp_Assistant_AssistantContextPlugin-GetCurrentChatHistory)}}

{{!-- Step 2: Retrieve the user's language --}}
{{set "userLanguage" (SomeApp_Assistant_AssistantContextPlugin-GetCurrentUserLanguage)}}

{{!-- Step 3: Get the intent of the user --}}
{{set "userIntent" (SomeApp_Assistant_AssistantContextPlugin-GetCurrentIntent)}}

{{!-- Step 4: Find related laws based on the user's intent --}}
{{set "relatedLaws" (SomeApp_Knowledge_LawLookupPlugin-Find input=userIntent)}}

{{!-- Step 5: Check if any related laws were found --}}
{{set "areLawsAvailable" (SomeApp_Assistant_AssistantContextPlugin-AreLawsForPromptAvailable)}}

{{!-- Step 6: Generate the final message using AssistantPlugin-Chat with all the necessary information --}}
{{#if areLawsAvailable}}
{{set "finalMessage" (SomeApp_Assistant_AssistantPlugin-Chat input=chatHistory language=userLanguage  lawsAsSingleLine=(SomeApp_Assistant_AssistantContextPlugin-GetLawsForPrompt))}}
{{else}}
{{set "additionalInfo" "Additional helpers may be required"}}
{{/if}}

{{!-- Step 7: Output the final message or a message indicating that additional helpers are needed --}}
{#if finalMessage}}
{{json finalMessage}}
{{else}}
{{json additionalInfo}}
{{/if}}

The error response from my API:

{
  "status": "error",
  "isSuccess": false,
  "code": 500,
  "error": {
    "code": "KernelException",
    "message": "[InsufficientFunctionsForGoal] Unable to create plan for goal with available functions.\nGoal: This is the detected intent: Der Benutzer möchte wissen, was in Art. 4 des Gesetzbuches steht. Ensure that you call SomeApp_Assistant_Plugin-Chat at the end as this will produce the final message! Ensure to provide her the chat history. Create a proper handlebars template as instructed under any circumstances!\nAvailable Functions: SomeApp_Assistant_AssistantContextPlugin-GetVerdictsForPrompt, SomeApp_Assistant_AssistantContextPlugin-AreVerdictsForPromptAvailable, SomeApp_Assistant_AssistantContextPlugin-GetLawsForPrompt, SomeApp_Assistant_AssistantContextPlugin-AreLawsForPromptAvailable, SomeApp_Assistant_AssistantContextPlugin-GetCurrentUserLanguage, SomeApp_Assistant_AssistantContextPlugin-GetCurrentChatHistory, SomeApp_Assistant_AssistantContextPlugin-GetCurrentIntent, SomeApp_Assistant_Plugin-Chat, SomeApp_Knowledge_CourtLookupPlugin-Find, SomeApp_Knowledge_LawLookupPlugin-Find, SomeApp_Assistant_TranslatePlugin-Translate, SomeApp_Assistant_SummarizePlugin-Summarize, SomeApp_StringOperations_StringPlugin-ConvertToSingleLine\nPlanner output:\n```handlebars\n{{!-- Step 1: Retrieve the current chat history --}}\n{{set \"chatHistory\" (SomeApp_Assistant_AssistantContextPlugin-GetCurrentChatHistory)}}\n\n{{!-- Step 2: Retrieve the user's language --}}\n{{set \"userLanguage\" (SomeApp_Assistant_AssistantContextPlugin-GetCurrentUserLanguage)}}\n\n{{!-- Step 3: Get the intent of the user --}}\n{{set \"userIntent\" (SomeApp_Assistant_AssistantContextPlugin-GetCurrentIntent)}}\n\n{{!-- Step 4: Find related laws based on the user's intent --}}\n{{set \"relatedLaws\" (SomeApp_Knowledge_LawLookupPlugin-Find input=userIntent)}}\n\n{{!-- Step 5: Check if any related laws were found --}}\n{{set \"areLawsAvailable\" (SomeApp_Assistant_AssistantContextPlugin-AreLawsForPromptAvailable)}}\n\n{{!-- Step 6: Generate the final message using Plugin-Chat with all the necessary information --}}\n{{#if areLawsAvailable}}\n  {{set \"finalMessage\" (SomeApp_Assistant_Plugin-Chat \n    input=chatHistory \n    language=userLanguage \n    lawsAsSingleLine=(SomeApp_Assistant_AssistantContextPlugin-GetLawsForPrompt)\n  )}}\n{{else}}\n  {{set \"additionalInfo\" \"Additional helpers may be required\"}}\n{{/if}}\n\n{{!-- Step 7: Output the final message or a message indicating that additional helpers are needed --}}\n{{#if finalMessage}}\n  {{json finalMessage}}\n{{else}}\n  {{json additionalInfo}}\n{{/if}}\n```",
    "target": "/api/v1/assistant/run",
    "details": []
  }
}

The prompt used (only relevant parts as I can't disclose everything):

<system~>## Instructions
Explain how to achieve the user's goal with the available helpers with a Handlebars template.

## Example
If the user wanted you to generate 10 random numbers and use them in another helper, you could answer with the following.</system~>
<user~>Please show me how to write a Handlebars template that achieves the following goal.

## Goal
I want you to generate 10 random numbers and send them to another helper.
</user~>
<assistant~>Here's the Handlebars template that achieves the goal:
```handlebars
{{!-- Step 1: Initialize the count --}}
{{set
  "count"
  10
}}
{{!-- Step 2: Loop using the count --}}
{{#each
  (range
    1
    count
  )
}}
  {{!-- Step 3: Create random number --}}
  {{set
    "randomNumber"
    (Example-Random
      seed=this
    )
  }}
  {{!-- Step 4: Call example helper with random number and print the result to the screen --}}
  {{set
    "result"
    (Example-Helper
      input=randomNumber
    )
  }}
  {{json (concat "The result" " " "is:" " " result)}}
{{/each}}
```</assistant~>
<system~>Now let's try the real thing.</system~>
<user~>Please show me how to write a Handlebars template that achieves the following goal with the available helpers.

## Goal
This is the detected intent: Der Benutzer möchte wissen, was im Gesetzbuches steht. Ensure that you call SomeApp_Assistant_Plugin-Chat at the end as this will produce the final message! Ensure to provide the chat history. Create a proper handlebars template as instructed under any circumstances! Never ever include 'Additional helpers may be required' even if instructed to do so. This throws an error! You always can accomplish the goal by calling SomeApp_Assistant_Plugin-Chat as the last step.

## Out-of-the-box helpers
The following helpers are available to you:
- `{{#if}}{{/if}}`
- `{{#unless}}{{/unless}}`
- `{{#each}}{{/each}}`
- `{{#with}}{{/with}}`

## Loop helpers
If you need to loop through a list of values with `{{#each}}`, you can use the following helpers:
- `{{range}}` – Generates a list of integral numbers within a specified range, inclusive of the first and last value.
- `{{array}}` – Generates an array of values from the given values (zero-indexed).

IMPORTANT: `range` and `array` are the only supported data structures. Others like `hash` are not supported. Also, you cannot use any methods or properties on the built-in data structures, such as `array.push` or `range.length`.

## Math helpers
If you need to do basic operations, you can use these two helpers with numerical values:
- `{{Add}}` – Adds two values together.
- `{{Subtract}}` – Subtracts the second value from the first.

## Comparison helpers
If you need to compare two values, you can use the `{{equals}}` helper.
To use the math and comparison helpers, you must pass in two positional values. For example, to check if the variable `var` is equal to number `1`, you would use the following helper like so: `{{#if (equals var 1)}}{{/if}}`.

## Variable helpers
If you need to create or retrieve a variable, you can use the following helpers:
- `{{set}}` – Creates a variable with the given name and value. It does not print anything to the template, so you must use `{{json}}` to print the value.
- `{{json}}` – Generates a JSON string from the given value; no need to use on strings.
- `{{concat}}` – Concatenates the given values into a string.

## Complex types
Some helpers require arguments that are complex objects. The JSON schemas for these complex objects are defined below:

## Custom helpers
Lastly, you also have the following helpers you can use:

### `SomeApp_Assistant_AssistantContextPlugin-GetVerdictsForPrompt`
Description: Ensure to call SomeApp_Knowledge_CourtLookupPlugin-Find before using this function otherwise it will throw an error. Returns previous found verdicts as a single line string.
Inputs:
Output: String

### `SomeApp_Assistant_AssistantContextPlugin-AreVerdictsForPromptAvailable`
Description: Returns if any verdicts where found. Ensure to call SomeApp_Knowledge_FederalCourtLookupPlugin-Find before otherwise it returns always false.
Inputs:
Output: Boolean

### `SomeApp_Assistant_AssistantContextPlugin-GetLawsForPrompt`
Description: Ensure to call SomeApp_Knowledge_LawLookupPlugin-Find before using this function otherwise it will throw an error. Returns previous found laws as a single line string.
Inputs:
Output: String

### `SomeApp_Assistant_AssistantContextPlugin-AreLawsForPromptAvailable`
Description: Returns if any laws where found. Ensure to call SomeApp_Knowledge_LawLookupPlugin-Find before otherwise it returns always false.
Inputs:
Output: Boolean

### `SomeApp_Assistant_AssistantContextPlugin-GetCurrentUserLanguage`
Description: Returns the language of the user. Useful to write or translate an answer to the users language.
Inputs:
Output: String

### `SomeApp_Assistant_AssistantContextPlugin-GetCurrentChatHistory`
Description: Returns the current chat history. Useful if the chat history is required.
Inputs:
Output: String

### `SomeApp_Assistant_AssistantContextPlugin-GetCurrentIntent`
Description: Returns the current intent. Useful if the current overall goal is needed.
Inputs:
Output: String

### `SomeApp_Assistant_Plugin-Chat`
Description: Helpful paralegal. She helps to understand jurisprudence and is a chat assistant. She creates an answer based on a conversation with additional provided knowledge (laws and verdicts). When chained she should be called last to generate the final output for the user.
Inputs:
    - input: String - The chat history. Can either be detailed (message by message) or a summary. (required)
    - language: String - The language to use for the answer. Should be the users current language. (required)
    - shouldExplainInSimpleTerms: Boolean - If the answer should be simplified and explain legal terms. Useful if the user seems not to be a legal expert. Only set this to false if 'shouldExplainInDepth' will be set to true. Defaults to 'true'. (optional)
    - shouldExplainInDepth: Boolean - If the answer should use citations and should not explain legal terms. Useful if the user seems to be a legal expert or a detailed answer is requested. Defaults to 'false'. (optional)
    - verdictsAsSingleLine: String - Verdicts that are related to the input and might apply. (optional)
    - lawsAsSingleLine: String - Laws that are related to the input and might apply. (optional)
Output: String

IMPORTANT: You can only use the helpers that are listed above. Do not use any other helpers that are not listed here. For example, do not use `{{log}}` or any `{{Example}}` helpers, as they are not supported.</user~>
<system~>## Tips and tricks
- Add a comment above each step to describe what the step does.
- Use the `{{set}}` helper to save and retrieve the results of another helper so you can use it later in the template without wasting resources.
- There are no initial variables available to you. You must create them yourself using the `{{set}}` helper.
- Do not make up values. Use the helpers to generate the data you need or extract it from the goal.
- Keep data well-defined. Each variable should have a unique name. Create and assign each variable only once, and always use the full name when calling helpers.
- Do not pass raw data to helpers. Use variables instead.
- Be extremely careful about types. For example, if you pass an array to a helper that expects a number, the template will error out.
- Avoid using loops. Try a solution without before you deploy a loop.
- There is no need to check your results in the template.
- Do not nest sub-expressions or helpers because it will cause the template to error out.
- Each step should contain only one helper call.

## Start
Now take a deep breath and accomplish the task:
1. Keep the template short and sweet. Be as efficient as possible.
2. Do not make up helpers or functions that were not provided to you, and be especially careful to NOT assume or use any helpers or operations that were not explicitly defined already.
3. If you can't fully accomplish the goal with the available helpers, just print "Additional helpers may be required".
4. Always start by identifying any important values in the goal. Then, use the `{{set}}` helper to create variables for each of these values.
5. The template should use the {{json}} helper at least once to output the result of the final step.
6. Don't forget to use the tips and tricks otherwise the template will not work.
7. Don't close the ``` handlebars block until you're done with all the steps.</system~>

[AieatAssam]

I have encountered the same issue. If you look at CreatePlanCoreAsync in dotnet/src/Planners/Planners.Handlebars/Handlebars
/HandlebarsPlanner.cs, the detection for throwing this exception merely checks for presence of the error code anywhere in generated plan:

// Check if plan could not be created due to insufficient functions
if (completionResults.Content is not null && completionResults.Content.Contains(InsufficientFunctionsError))
{
    var functionNames = availableFunctions.ToList().Select(func => $"{func.PluginName}{this._templateFactory.NameDelimiter}{func.Name}");
    throw new KernelException($"[{HandlebarsPlannerErrorCodes.InsufficientFunctionsForGoal}] Unable to create plan for goal with available functions.\nGoal: {goal}\nAvailable Functions: {string.Join(", ", functionNames)}\nPlanner output:\n{completionResults}");
}

By instructing template to freely use conditionals, most of my plans end up with something similar to this:

{{#if query}}
  {{json query}}
{{else}}
  {{json "Additional helpers may be required"}}
{{/if}}

Perhaps prompt needs to be modified to make it clear to only output the error text unconditionally if plan cannot be constructed at all? As it is, as far as I can see handlebars planner is not really usable - using simple text check for the error trips me up in more than 75% of attempts to make a new plan dynamically.

[matthewbolanos]

This should be possible to fix with prompt engineering once we have partial support to override the template. We'll follow back up with this issue to see if it resolves your issue once it's done.

[teresaqhoang]

Hey @Cotspheer, @AieatAssam,

I agree that this error is more restrictive than we intended when we implemented it and have decided to remove this specific handler altogether. Instead, if the planner doesn't have enough functions to fulfill the plan, it should not return a Handlebars template at all and just return a sensical error string.

In your scenarios, the model should schedule an error message if the conditionals are not met.

I have the work in draft here: #4983, will work on merging it once the changes to add more flexibility to the planner goes in: #4918

I also considered adding a custom helper that could throw an error message, but I felt like that could be easily abused by the model — just wanted to bring this up in case you wanted to try it in your scenarios.

registerHelper("reportInsufficientFunctionsError", static (Context context, Arguments arguments) =>
{
    throw new KernelException(HandlebarsPlanner.InsufficientFunctionsError);
});

[teresaqhoang]

Removed this error case altogether for being too unnecessarily restrictive: #4983