SequentialPlanner - Allow custom manual discovery and loading.

[lemillermicrosoft]

Motivation and Context

This change is part of a need to enable planners that can reference a large list of potential plugins that aren't loaded in a kernel instance. Additionally, consumers of planners need to be able to load plugins on demand when needed by a plan. The change includes some additional improvements to the sequential plan parser and prompt.

Part of #1728

Description

This pull request refactors and improves the sequential plan parser and config, simplifying the code, adding new features, and fixing some issues. The main changes are:

• Use the Azure Chat Completion Service instead of the Azure Text Completion Service for the sequential planner, and lower the relevancy threshold for creating a plan. This allows for more flexibility and creativity in generating plans, and improves the performance and accuracy of the planner.
• Refactor the plan parsing logic to simplify the code and improve readability. Remove unnecessary try-catch blocks, logging statements, and string splitting logic. Reorder the parameters of the ToPlanFromXml method to match the order of the XML attributes. Ignore text and comment nodes in the plan XML, as they are not valid steps and could cause confusion or errors. Remove the logic that adds empty nodes as steps, as they are not valid functions and could cause errors or confusion.
• Add two optional functions to the SequentialPlannerConfig class that allow the planner to use custom logic for getting available functions and finding functions by name. These functions can be useful for scenarios where the planner needs to access functions from different sources or with different criteria, such as filtering by skill category or function type. The default behavior of the planner remains unchanged if these functions are not specified.
• Simplify and standardize the syntax for calling functions in the plan XML, using fully qualified function names and consistent attribute names for inputs, outputs, and parameters. Update the skprompt.txt file to reflect the new syntax and add more instructions and examples for creating a plan. Update the sequential plan parser tests to use the new syntax and function names, and add a helper method to get the function from the kernel context.
• Fix some minor typos and errors in the code and the skprompt.txt file.

Details:

• dotnet/src/IntegrationTests/Planning/SequentialPlanner/SequentialPlannerTests.cs
  • Changed the KernelSyntaxExamples and SequentialPlannerTests to use the Azure Chat Completion Service instead of the Azure Text Completion Service, by replacing the deployment name, endpoint, and key environment variables.
  • Removed the EmailSkillFake import from the InitializeKernel method, as it is not used in any test case and is already imported in the TestHelpers class.
  • Updated the tests to reflect the changes in the plan parser and remove the redundant or invalid steps. For example, remove the text and empty nodes from the plan XML, and update the expected number and description of the steps.
• dotnet/src/Extensions/Planning.SequentialPlanner/SequentialPlanParser.cs
  • Simplified the ToPlanFromXml method by removing unnecessary try-catch blocks, logging statements, and string splitting logic.
  • Reordered the parameters of the ToPlanFromXml method to match the order of the XML attributes, as this makes the code more consistent and easier to read.
  • Ignored text and comment nodes in the plan XML, as they are not valid steps and could cause confusion or errors. Added TODO comments for possible future enhancements using text or comments as reasoning or desired functions.
  • Removed the logic that adds empty nodes as steps, as they are not valid functions and could cause errors or confusion. Added TODO comments for possible future enhancements using empty nodes as desired functions.
  • Added a GetFunction parameter to the ToPlanFromXml method that takes a delegate for finding a skill function by skill name and function name. This delegate can be specified in the SequentialPlannerConfig class or default to the existing logic of using the SKContext.
  • Added a new static method GetFunction to the SequentialPlanParser class that returns the default delegate for getting the ISKFunction instance, which calls context.Skills.TryGetFunction.
  • Fixed a minor typo in the GetFunctionName method, changing "functionName" to "functionNameAttribute". This makes the code more accurate and clear.
• dotnet/src/Extensions/Planning.SequentialPlanner/SequentialPlannerConfig.cs
  • Added a new property GetAvailableFunctionsAsync to the SequentialPlannerConfig class that takes a delegate of type Func<SequentialPlannerConfig, string, Task<IOrderedEnumerable>>. This delegate can be used to provide a custom logic for retrieving available functions from a semantic query, instead of using the default logic that calls context.GetAvailableFunctionsAsync. The default value of this property is null, which means the default logic will be used.
  • Modified the SKContextSequentialPlannerExtensions class to use the GetAvailableFunctionsAsync delegate if it is not null, otherwise fall back to the default logic.
• dotnet/src/Extensions/Planning.SequentialPlanner/skprompt.txt
  • Simplified and standardized the syntax for calling functions from <function.{FunctionName} ... /> to <function.{FullyQualifiedFunctionName} ... />, where {FullyQualifiedFunctionName} is the name of the function with its namespace or skill prefix, such as GLOBAL_FUNCTIONS.NovelOutline or AuthorAbility.Summarize.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows SK Contribution Guidelines (https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
• The code follows the .NET coding conventions (https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions) verified with dotnet format
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[dmytrostruk]

A couple of small formatting/convention comments, in general LGTM!

[teresaqhoang]

Overall, lgtm! Requesting some small changes for dev clarity