Skip to content

Commit

Permalink
.Net: Ensure Action is JSON format in Stepwise Planner (#1976)
Browse files Browse the repository at this point in the history
### Motivation and Context
Sometimes, the LLM populates the [ACTION] with non-JSON data. To
mitigate this from happening, this PR updates the labels for the
[ACTION] section of the prompt to more explicitly state that **_only_**
JSON should be written there.

### Description
- [ACTION] has been changed to [JSON ACTION] in the prompt to make it
clear that JSON should appear in this section.
- $JSON_BLOB has been changed to $JSON_ACTION to make it more clear that
the instructions above are for the [JSON ACTION] section

### Contribution Checklist
- [x] The code builds clean without any errors or warnings
- [x] The PR follows SK Contribution Guidelines
(https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [x] The code follows the .NET coding conventions
(https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions)
verified with `dotnet format`
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Gina Triolo <51341242+gitri-ms@users.noreply.github.com>
Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com>
  • Loading branch information
3 people committed Jul 17, 2023
1 parent e026da7 commit 8a33404
Show file tree
Hide file tree
Showing 3 changed files with 11 additions and 12 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,9 @@ public void WhenInputIsFinalAnswerReturnsFinalAnswer(string input, string expect
}

[Theory]
[InlineData("To answer the first part of the question, I need to search for Leo DiCaprio's girlfriend on the web. To answer the second part, I need to find her current age and use a calculator to raise it to the 0.43 power.\n[ACTION]\n{\n \"action\": \"Search\",\n \"action_variables\": {\"input\": \"Leo DiCaprio's girlfriend\"}\n}", "Search", "input", "Leo DiCaprio's girlfriend")]
[InlineData("To answer the first part of the question, I need to search the web for Leo DiCaprio's girlfriend. To answer the second part, I need to find her current age and use the calculator tool to raise it to the 0.43 power.\n[ACTION]\n```\n{\n \"action\": \"Search\",\n \"action_variables\": {\"input\": \"Leo DiCaprio's girlfriend\"}\n}\n```", "Search", "input", "Leo DiCaprio's girlfriend")]
[InlineData("The web search result is a snippet from a Wikipedia article that says Leo DiCaprio's girlfriend is Camila Morrone, an Argentine-American model and actress. I need to find out her current age, which might be in the same article or another source. I can use the WebSearch.Search function again to search for her name and age.\n\n[ACTION] {\n \"action\": \"WebSearch.Search\",\n \"action_variables\": {\"input\": \"Camila Morrone age\", \"count\": \"1\"}\n}", "WebSearch.Search", "input",
[InlineData("To answer the first part of the question, I need to search for Leo DiCaprio's girlfriend on the web. To answer the second part, I need to find her current age and use a calculator to raise it to the 0.43 power.\n[JSON ACTION]\n{\n \"action\": \"Search\",\n \"action_variables\": {\"input\": \"Leo DiCaprio's girlfriend\"}\n}", "Search", "input", "Leo DiCaprio's girlfriend")]
[InlineData("To answer the first part of the question, I need to search the web for Leo DiCaprio's girlfriend. To answer the second part, I need to find her current age and use the calculator tool to raise it to the 0.43 power.\n[JSON ACTION]\n```\n{\n \"action\": \"Search\",\n \"action_variables\": {\"input\": \"Leo DiCaprio's girlfriend\"}\n}\n```", "Search", "input", "Leo DiCaprio's girlfriend")]
[InlineData("The web search result is a snippet from a Wikipedia article that says Leo DiCaprio's girlfriend is Camila Morrone, an Argentine-American model and actress. I need to find out her current age, which might be in the same article or another source. I can use the WebSearch.Search function again to search for her name and age.\n\n[JSON ACTION] {\n \"action\": \"WebSearch.Search\",\n \"action_variables\": {\"input\": \"Camila Morrone age\", \"count\": \"1\"}\n}", "WebSearch.Search", "input",
"Camila Morrone age", "count", "1")]
public void ParseActionReturnsAction(string input, string expectedAction, params string[] expectedVariables)
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ To use the functions, specify a JSON blob representing an action. The JSON blob
Do not call functions directly; they must be invoked through an action.
The "action_variables" value should always include an "input" key, even if the input value is empty. Additional keys in the "action_variables" value should match the defined [PARAMETERS] of the named "action" in [AVAILABLE FUNCTIONS].
Dictionary values in "action_variables" must be strings and represent the actual values to be passed to the function.
Ensure that the $JSON_BLOB contains only a SINGLE action; do NOT return multiple actions.
Ensure that the $JSON_ACTION contains only a SINGLE action; do NOT return multiple actions.
IMPORTANT: Use only the available functions listed in the [AVAILABLE FUNCTIONS] section. Do not attempt to use any other functions that are not specified.

Here is an example of a valid $JSON_BLOB:
Here is an example of a valid $JSON_ACTION:
{
"action": "FUNCTION.NAME",
"action_variables": {"INPUT": "some input", "PARAMETER_NAME": "some value", "PARAMETER_NAME_2": "42"}
Expand All @@ -27,12 +27,11 @@ Here is an example of a valid $JSON_BLOB:
[END INSTRUCTION]

[THOUGHT PROCESS]
[QUESTION]
the input question I must answer
[QUESTION] the input question I must answer
[THOUGHT]
To solve this problem, I should carefully analyze the given question and identify the necessary steps. Any facts I discover earlier in my thought process should be repeated here to keep them readily available.
[ACTION]
$JSON_BLOB
[JSON ACTION]
$JSON_ACTION
[OBSERVATION]
The result of the action will be provided here.
... (These Thought/Action/Observation can repeat until the final answer is reached.)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -454,7 +454,7 @@ private static string ToFullyQualifiedName(FunctionView function)
/// <summary>
/// The Action tag
/// </summary>
private const string Action = "[ACTION]";
private const string Action = "[JSON ACTION]";

/// <summary>
/// The Thought tag
Expand All @@ -474,12 +474,12 @@ private static string ToFullyQualifiedName(FunctionView function)
/// <summary>
/// The regex for parsing the action response
/// </summary>
private static readonly Regex s_actionRegex = new(@"\[ACTION\][^{}]*({(?:[^{}]*{[^{}]*})*[^{}]*})", RegexOptions.Singleline);
private static readonly Regex s_actionRegex = new(@"\[JSON ACTION\][^{}]*({(?:[^{}]*{[^{}]*})*[^{}]*})", RegexOptions.Singleline);

/// <summary>
/// The regex for parsing the thought response
/// </summary>
private static readonly Regex s_thoughtRegex = new(@"(\[THOUGHT\])?(?<thought>.+?)(?=\[ACTION\]|$)", RegexOptions.Singleline);
private static readonly Regex s_thoughtRegex = new(@"(\[THOUGHT\])?(?<thought>.+?)(?=\[JSON ACTION\]|$)", RegexOptions.Singleline);

/// <summary>
/// The regex for parsing the final answer response
Expand Down

0 comments on commit 8a33404

Please sign in to comment.