Skip to content

Latest commit

 

History

History
83 lines (64 loc) · 4.65 KB

human_interaction.md

File metadata and controls

83 lines (64 loc) · 4.65 KB
Raw

Human Interaction

Overview

The interpreter supports flexible natural language interaction. Users can change requirements, provide execution suggestions, and review results at multiple points. With human guidance, the interpreter can achieve higher success rates and better outcomes across various scenarios.

Example: Machine Learning

Task

We use the same machine learning scenario as an example. Here, we demonstrate how to confirm and modify the interpreter's plan based on our needs and guide it to redo specific tasks.

Code

python examples/di/machine_learning.py --auto_run False

Execution Results


During the interaction, we make multiple requests through natural language, such as:

  • When formulating the plan, we ask it to integrate and change tasks: “merge 4 and 5, also, change task 2, save the plot”
  • During execution, we ask it to update subsequent tasks: “confirm, but change task 3 only, use 30% as validation set”
  • During execution, we ask it to redo the current task: “redo, use a decision tree as the model instead”

The interpreter dynamically adjusts and executes to fulfill our requests.

Mechanism Explained

To use the human interaction mode, set auto_run=False when initializing the interpreter. The interpreter will request human review after proposing each plan and completing each task in the plan. Combining keywords, users can inform the interpreter how to modify the proposed plan or written code using natural language. The interpreter supports plan status management, preserving progress on unaffected tasks when adding, modifying, or deleting tasks. Please refer to the following usage example.

Example of an DataInterpreter-Generated Plan

[
  {
    "task_id": "1",
    "dependent_task_ids": [],
    "instruction": "Load the sklearn Wine recognition dataset."
  },
  {
    "task_id": "2",
    "dependent_task_ids": ["1"],
    "instruction": "Perform exploratory data analysis and include a plot of the dataset features."
  },
  {
    "task_id": "3",
    "dependent_task_ids": ["1"],
    "instruction": "Split the dataset into training and validation sets with a 20% validation split."
  },
  {
    "task_id": "4",
    "dependent_task_ids": ["3"],
    "instruction": "Train a model on the training set to predict wine class."
  },
  {
    "task_id": "5",
    "dependent_task_ids": ["4"],
    "instruction": "Evaluate the model on the validation set and show the validation accuracy."
  }
]

Interactables for Humans

Interaction Type Required Keywords Example Explanation
Confirm One of ["confirm", "continue", "c", "yes", "y"] confirm Confirm the current plan or task result and continue execution
Confirm and Update confirm + any other content confirm, and change task 3, use 30% as validation set Confirm the current task result and modify subsequent tasks
Redo redo redo, fix the error by using matplotlib Redo the current task, based on human guidance
redo, use decision tree as the model
Update - add a task, save the plot Update the plan based on human request
delete task 5
change task 4, use decision tree as the model
merge task 4 and 5

The interpreter determines the type of interaction based on human feedback text, with priority: Confirm and Update > Confirm > Redo > Update.