Need support multi turn/multi pass for chat co-pilot

[alzhang-git]

We are building a co-pilot experience need multi turn, multi pass chat support. Example
User: I want to create a product
Chat: Here are 3 steps, step 1: give description, step 2: set price 3: set workload
Start with step 1, please set description
User: please suggest to me
Chat: here is suggestion,xxxxx
User: Looks good
Chat: Step 2 set price, here are options.....

Basically for each step, it need back and forth chat until that step confirms done then move to next step

Currently there is no native support by SK for this and we would like to request this feature we can leverage to build our co-pilot

[teresaqhoang]

Hey @alzhang-git, ack! This feedback loop has been an idea in our backlog, but we need to think through the dev design for this a bit. I'll get back to you by the end of the week.

[teresaqhoang]

@alzhang-git to help us understand your scenario more, are you trying to leverage the planner to generate these steps? Do you expect the steps to be skills/plug-ins available in the kernel? Or natural language?

[matthewbolanos]

@alzhang-git, we're introducing the iterative planner which we believe will help address this need. Do you mind taking a look at the PR and seeing if it satisfies what you need?

[teresaqhoang]

@alzhang-git we now have support for Stepwise Planner in SK! It enables chain of thought, where it enables a thought process of thought, action, observation. On each turn, it creates a static plan with a single step (thought) that runs the plan right away (action) then continues in a loop until it's able to complete the goal (observation).

Check out this YouTube video for an overview, and this example for a more technical walkthrough.

I'm closing out this discussion now, feel free to re-open if this doesn't fit your needs

[diptesh-bose]

Hi, is there a plan to release stepwiseplanner to the chat copilot app...

[teresaqhoang]

Hi @diptesh-bose, stepwise planner is supported in copilot chat now!

You can set the Planner:Type in appsettings to "Stepwise" to enable it. It works best with gpt-4 or gpt-35-turbo-0310.

Your appsettings.json should look something like:

 "AIService": {
    "Type": "AzureOpenAI",
    "Endpoint": "", // ignored when AIService is "OpenAI"
    // "Key": "",
    "Models": {
      "Completion": "gpt-35-turbo", // For OpenAI, change to 'gpt-3.5-turbo' (with a period).
      "Embedding": "text-embedding-ada-002",
      "Planner": "gpt-4" // For OpenAI, change to 'gpt-3.5-turbo' (with a period).
    }
  },
  //
  // Planner can determine which skill functions, if any, need to be used to fulfill a user's request.
  // https://learn.microsoft.com/en-us/semantic-kernel/concepts-sk/planner
  // - Set Planner:Type to "Action" to use the single-step ActionPlanner
  // - Set Planner:Type to "Sequential" to enable the multi-step SequentialPlanner
  //     Note: SequentialPlanner works best with `gpt-4`. See the "Enabling Sequential Planner" section in webapi/README.md for configuration instructions.
  // - Set Planner:Type to "Stepwise" to enable MRKL style planning
  // - Set Planner:RelevancyThreshold to a decimal between 0 and 1.0.
  //
  "Planner": {
    "Type": "Stepwise",
    // The minimum relevancy score for a function to be considered.
    // Set RelevancyThreshold to a value between 0 and 1 if using the SequentialPlanner or Stepwise planner with gpt-3.5-turbo.
    // Ignored when Planner:Type is "Action"
    "RelevancyThreshold": "0.25",
    // Whether to allow missing functions in the plan on creation then sanitize output. Functions are considered missing if they're not available in the planner's kernel's context.
    // If set to true, the plan will be created with missing functions as no-op steps that are filtered from the final proposed plan.
    // If this is set to false, the plan creation will fail if any functions are missing.
    "MissingFunctionsError": {
      "AllowSkips": "true",
      "MaxRetriesAllowed": "3" // Max retries allowed on MissingFunctionsError. If set to 0, no retries will be attempted.
    },
    // Whether to retry plan creation if LLM returned response with invalid plan.
    "AllowRetriesOnInvalidPlan": "true",
    "StepwisePlannerConfig": {
      "MaxTokens": "2048",
      "MaxIterations": "15",
      "MinIterationTimeMs": "1500"
    }
  },

[diptesh-bose]

Awesome. Thank you so much. Any recommendations on the relevancy threshold when using gpt 3.5. Currently set to 0.85... Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Teresa Hoang ***@***.***> Sent: Tuesday, August 15, 2023 6:44:04 PM To: microsoft/semantic-kernel ***@***.***> Cc: diptesh-bose ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/semantic-kernel] Need support multi turn/multi pass for chat co-pilot (Discussion #1356) Hi @diptesh-bose<https://github.com/diptesh-bose>, stepwise planner is supported in copilot chat now! You can set the Planner:Type in appsettings to "Stepwise" to enable it. It works best with gpt-4 or gpt-35-turbo-0310. Your appsettings.json should look something like: "AIService": { "Type": "AzureOpenAI", "Endpoint": "", // ignored when AIService is "OpenAI" // "Key": "", "Models": { "Completion": "gpt-35-turbo", // For OpenAI, change to 'gpt-3.5-turbo' (with a period). "Embedding": "text-embedding-ada-002", "Planner": "gpt-4" // For OpenAI, change to 'gpt-3.5-turbo' (with a period). } }, // // Planner can determine which skill functions, if any, need to be used to fulfill a user's request. // https://learn.microsoft.com/en-us/semantic-kernel/concepts-sk/planner // - Set Planner:Type to "Action" to use the single-step ActionPlanner // - Set Planner:Type to "Sequential" to enable the multi-step SequentialPlanner // Note: SequentialPlanner works best with `gpt-4`. See the "Enabling Sequential Planner" section in webapi/README.md for configuration instructions. // - Set Planner:Type to "Stepwise" to enable MRKL style planning // - Set Planner:RelevancyThreshold to a decimal between 0 and 1.0. // "Planner": { "Type": "Stepwise", // The minimum relevancy score for a function to be considered. // Set RelevancyThreshold to a value between 0 and 1 if using the SequentialPlanner or Stepwise planner with gpt-3.5-turbo. // Ignored when Planner:Type is "Action" "RelevancyThreshold": "0.25", // Whether to allow missing functions in the plan on creation then sanitize output. Functions are considered missing if they're not available in the planner's kernel's context. // If set to true, the plan will be created with missing functions as no-op steps that are filtered from the final proposed plan. // If this is set to false, the plan creation will fail if any functions are missing. "MissingFunctionsError": { "AllowSkips": "true", "MaxRetriesAllowed": "3" // Max retries allowed on MissingFunctionsError. If set to 0, no retries will be attempted. }, // Whether to retry plan creation if LLM returned response with invalid plan. "AllowRetriesOnInvalidPlan": "true", "StepwisePlannerConfig": { "MaxTokens": "2048", "MaxIterations": "15", "MinIterationTimeMs": "1500" } }, — Reply to this email directly, view it on GitHub<#1356 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BABWUOOVJHSWDCSEUYY24G3XVNYZZANCNFSM6AAAAAAZGIDUHY>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[teresaqhoang]

It depends on how many functions you have registered in the planner's kernel. If the count is in single digits, you're likely good to go with something lower like 0.25 or 0.35 since the planner doesn't have to filter much variety and should select the appropriate one(s), but if it's > 10, then I'd set it something around .65 or 0.75. Play around with it!

[diptesh-bose]

I think I will need to understand more about this...my functions that were working fine with Action and Sequential planners and not working with Stepwise planner...Seems like an unleashed / uncontrolled planner...I am sure I am not it the right way...I experiment and let you know .... Thanks Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Teresa Hoang ***@***.***> Sent: Tuesday, August 15, 2023 7:07:13 PM To: microsoft/semantic-kernel ***@***.***> Cc: diptesh-bose ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/semantic-kernel] Need support multi turn/multi pass for chat co-pilot (Discussion #1356) It depends on how many functions you have registered in the planner's kernel. If the count is in single digits, you're likely good to go with something lower like 0.25 or 0.35 since the planner doesn't have to filter much variety and should select the appropriate one(s), but if it's > 10, then I'd set it something around .65 or 0.75. Play around with it! — Reply to this email directly, view it on GitHub<#1356 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BABWUOPMZ5J5YOZMMINXSMDXVN3QTANCNFSM6AAAAAAZGIDUHY>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[teresaqhoang]

One known limitation across all planners is its inability to process large JSON responses well, so if you're calling an API that returns a really large JSON result, the Stepwise Planner can't handle it because there's not intermediary processing of the result like there is for Action or Sequential Planner.

If you open up the Prompt Dialog (by hitting the (i) icon on the chat message), you can see the steps that the StepwisePlanner took to help you debug.

[teresaqhoang]

Feel free to re-open another issue if you continue to run into issues. I won't be monitoring this thread anymore since it's closed. We also have Office hours today at 2PM PST if you want live help there: https://aka.ms/sk-community-office-hours