Skip to content

Commit

Permalink
.Net: Removing Insufficient functions error (#4983)
Browse files Browse the repository at this point in the history
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

This PR removes the InsufficientFunctions error message. If the planner
doesn't have enough context to create a plan, it should forego a
template altogether and return an error string. This will be captured by
the `InvalidTemplate` check.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

Addresses #4442, which identified that this error handling was so
restrictive for most plan types.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
  • Loading branch information
teresaqhoang committed Feb 15, 2024
1 parent 27d5ffe commit 1dfb45f
Show file tree
Hide file tree
Showing 3 changed files with 3 additions and 16 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@
{{> AdditionalContext }}
{{> RetryLogic }}

{{> TipsAndInstructions }}
{{> TipsAndInstructions }}
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Follow these steps to create one Handlebars template to achieve the goal:
- Only use the helpers provided. Any helper not listed is considered hallucinated and must not be used.
- Do not invent or assume the existence of any functions not explicitly defined above.
3. What if I Need More Helpers?
- If the goal cannot be fully achieved with the provided helpers or you need a helper not defined, print the following message: "{{insufficientFunctionsErrorMessage}}".
- Stop here if the goal cannot be fully achieved with the provided helpers or you need a helper not defined, and just return a string with an appropriate error message.
4. Keep It Simple:{{#if allowLoops}}
- Avoid using loops or block expressions. They are allowed but not always necessary, so try to find a solution that does not use them.{{/if}}
- Your template should be intelligent and efficient, avoiding unnecessary complexity or redundant steps.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,6 @@ public Task<HandlebarsPlan> CreatePlanAsync(Kernel kernel, string goal, KernelAr

private readonly HandlebarsPromptTemplateFactory _templateFactory;

/// <summary>
/// Error message if kernel does not contain sufficient functions to create a plan.
/// </summary>
private const string InsufficientFunctionsError = "Additional helpers or information may be required";

private async Task<HandlebarsPlan> CreatePlanCoreAsync(Kernel kernel, string goal, KernelArguments? arguments, CancellationToken cancellationToken = default)
{
// Get CreatePlan prompt template
Expand All @@ -89,17 +84,10 @@ private async Task<HandlebarsPlan> CreatePlanCoreAsync(Kernel kernel, string goa
var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();
var completionResults = await chatCompletionService.GetChatMessageContentAsync(chatMessages, executionSettings: this._options.ExecutionSettings, cancellationToken: cancellationToken).ConfigureAwait(false);

// Check if plan could not be created due to insufficient functions
if (completionResults.Content is not null && completionResults.Content.IndexOf(InsufficientFunctionsError, StringComparison.OrdinalIgnoreCase) >= 0)
{
var functionNames = availableFunctions.ToList().Select(func => $"{func.PluginName}{this._templateFactory.NameDelimiter}{func.Name}");
throw new KernelException($"[{HandlebarsPlannerErrorCodes.InsufficientFunctionsForGoal}] Unable to create plan for goal with available functions.\nGoal: {goal}\nAvailable Functions: {string.Join(", ", functionNames)}\nPlanner output:\n{completionResults}");
}

Match match = Regex.Match(completionResults.Content, @"```\s*(handlebars)?\s*(.*)\s*```", RegexOptions.Singleline);
if (!match.Success)
{
throw new KernelException($"[{HandlebarsPlannerErrorCodes.InvalidTemplate}] Could not find the plan in the results\nPlanner output:\n{completionResults}");
throw new KernelException($"[{HandlebarsPlannerErrorCodes.InvalidTemplate}] Could not find the plan in the results. Additional helpers or input may be required.\n\nPlanner output:\n{completionResults}");
}

var planTemplate = match.Groups[2].Value.Trim();
Expand Down Expand Up @@ -236,7 +224,6 @@ private ChatHistory GetChatHistoryFromPrompt(string prompt)
{ "goal", goal },
{ "predefinedArguments", predefinedArgumentsWithTypes},
{ "nameDelimiter", this._templateFactory.NameDelimiter},
{ "insufficientFunctionsErrorMessage", InsufficientFunctionsError},
{ "allowLoops", this._options.AllowLoops },
{ "complexTypeDefinitions", complexParameterTypes.Count > 0 && complexParameterTypes.Any(p => p.IsComplex) ? complexParameterTypes.Where(p => p.IsComplex) : null},
{ "complexSchemaDefinitions", complexParameterSchemas.Count > 0 ? complexParameterSchemas : null},
Expand Down

0 comments on commit 1dfb45f

Please sign in to comment.