-
Notifications
You must be signed in to change notification settings - Fork 44.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(agent): Introduce Python code execution as prompt strategy #7142
Draft
majdyz
wants to merge
58
commits into
master
Choose a base branch
from
zamilmajdy/code-validation
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+2,447
β89
Draft
Changes from 26 commits
Commits
Show all changes
58 commits
Select commit
Hold shift + click to select a range
ed5f12c
Add code validation
majdyz ca7ca22
one_shot_flow.ipynb + edits to make it work
Pwuts ef1fe7c
Update notebook
majdyz 40426e4
Merge master
majdyz 22e2373
Add code flow as a loop
majdyz 0916df4
Fix async fiasco
majdyz 0eccbe1
Prompt change
majdyz f763452
More prompt engineering
majdyz ea134c7
Benchmark test
majdyz 7b5272f
Fix Await fiasco
majdyz 922e643
Fix Await fiasco
majdyz fb80240
Add return type
majdyz 834eb6c
Some quality polishing
majdyz 81ad3cb
Merge conflicts
majdyz 47eeaf0
Revert dumb changes
majdyz 3c4ff60
Add unit tests
majdyz 9f6e256
Debug Log changes
majdyz dfa7773
Remove unnecessary changes
majdyz 3a60504
isort
majdyz c8e16f3
Fix linting
majdyz ae43136
Fix linting
majdyz a825aa8
Merge branch 'master' into zamilmajdy/code-validation
majdyz fdd9f9b
Log fix
majdyz ae63aa8
Merge remote-tracking branch 'origin/zamilmajdy/code-validation' intoβ¦
majdyz 5c7c276
Merge branch 'master' into zamilmajdy/code-validation
Pwuts fcca4cc
clarify execute_code_flow
Pwuts 6e715b6
simplify function header generation
Pwuts b4cd735
fix name collision with `type` in `Command.return_type`
Pwuts 731d034
implement annotation expansion for non-builtin types
Pwuts 0578fb0
fix async issues with code flow execution
Pwuts c3acb99
clean up `forge.command.command`
Pwuts 6dd0975
clean up & improve `@command` decorator
Pwuts e264bf7
`forge.llm.providers.schema` + `code_flow_executor` lint-fix and cleanup
Pwuts 8144d26
fix type issues
Pwuts 111e858
feat(forge/llm): allow async completion parsers
Pwuts 3e8849b
fix linting and type issues
Pwuts 2c6e1eb
fix type issue in test_code_flow_strategy.py
Pwuts a9eb49d
Merge branch 'master' into zamilmajdy/code-validation
Pwuts 81bac30
fix type issues
Pwuts b59862c
Address comment
majdyz 3597f80
Merge branch 'master' of github.com:Significant-Gravitas/AutoGPT intoβ¦
majdyz e204491
Merge branch 'master' into zamilmajdy/code-validation
ntindle 901dade
Merge branch 'master' into zamilmajdy/code-validation
kcze 680fbf4
Merge branch 'master' into zamilmajdy/code-validation
Pwuts 9f80408
address feedback: pass commands getter to CodeFlowExecutionComponent(..)
Pwuts 37cc047
lint-fix + minor refactor
Pwuts 3e67512
Merge branch 'master' into zamilmajdy/code-validation
Pwuts 6d9f564
Merge branch 'master' into zamilmajdy/code-validation
Pwuts 38eafdb
Update `CodeFlowPromptStrategy` with upstream changes (#7223)
Pwuts 736ac77
Merge branch 'master' into zamilmajdy/code-validation
Pwuts 7f6b7d6
remove unused import in forge/llm/providers/openai.py
Pwuts 8b1d416
Merge branch 'master' into zamilmajdy/code-validation
Pwuts 2c4afd4
Migrate `autogpt/agents/prompt_strategies/code_flow.py` to Pydantic v2
Pwuts f03c654
Merge branch 'master' into zamilmajdy/code-validation
Pwuts e19636a
feat(agent/cli): Pretty-print code flow proposal
Pwuts 9dea6a2
Merge branch 'master' into zamilmajdy/code-validation
Pwuts da9360f
feat(agent/api): Pretty-print `execute_code_flow` proposal in Agent Pβ¦
Pwuts 7e0b115
feat(agent): Improve history format for code flow execution results
Pwuts File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,280 @@ | ||
import re | ||
from logging import Logger | ||
|
||
from forge.config.ai_directives import AIDirectives | ||
from forge.config.ai_profile import AIProfile | ||
from forge.json.parsing import extract_dict_from_json | ||
from forge.llm.prompting import ChatPrompt, LanguageModelClassification, PromptStrategy | ||
from forge.llm.providers import AssistantChatMessage, CompletionModelFunction | ||
from forge.llm.providers.schema import AssistantFunctionCall, ChatMessage | ||
from forge.models.config import SystemConfiguration | ||
from forge.models.json_schema import JSONSchema | ||
from forge.utils.exceptions import InvalidAgentResponseError | ||
from forge.utils.function.code_validation import CodeValidator | ||
from forge.utils.function.model import FunctionDef | ||
from pydantic import BaseModel, Field | ||
|
||
from autogpt.agents.prompt_strategies.one_shot import ( | ||
AssistantThoughts, | ||
OneShotAgentActionProposal, | ||
OneShotAgentPromptConfiguration, | ||
) | ||
|
||
_RESPONSE_INTERFACE_NAME = "AssistantResponse" | ||
|
||
|
||
class CodeFlowAgentActionProposal(BaseModel): | ||
thoughts: AssistantThoughts | ||
immediate_plan: str = Field( | ||
..., | ||
description="We will be running an iterative process to execute the plan, " | ||
"Write the partial / immediate plan to execute your plan as detailed and " | ||
"efficiently as possible without the help of the reasoning/intelligence. " | ||
"The plan should describe the output of the immediate plan, so that the next " | ||
"iteration can be executed by taking the output into account. " | ||
"Try to do as much as possible without making any assumption or uninformed " | ||
"guesses. Avoid large output at all costs!!!\n" | ||
"Format: Objective[Objective of this iteration, explain what's the use of this " | ||
"iteration for the next one] Plan[Plan that does not require any reasoning or " | ||
"intelligence] Output[Output of the plan / should be small, avoid whole file " | ||
"output]", | ||
) | ||
python_code: str = Field( | ||
..., | ||
description=( | ||
"Write the fully-functional Python code of the immediate plan. " | ||
"The output will be an `async def main() -> str` function of the immediate " | ||
"plan that return the string output, the output will be passed into the " | ||
"LLM context window so avoid returning the whole content!. " | ||
"Use ONLY the listed available functions and built-in Python features. " | ||
"Leverage the given magic functions to implement function calls for which " | ||
"the arguments can't be determined yet. " | ||
"Example:`async def main() -> str:\n" | ||
" return await provided_function('arg1', 'arg2').split('\\n')[0]`" | ||
), | ||
) | ||
|
||
|
||
FINAL_INSTRUCTION: str = ( | ||
"You have to give the answer in the from of JSON schema specified previously. " | ||
"For the `python_code` field, you have to write Python code to execute your plan " | ||
"as efficiently as possible. Your code will be executed directly without any " | ||
"editing, if it doesn't work you will be held responsible. " | ||
"Use ONLY the listed available functions and built-in Python features. " | ||
"Do not make uninformed assumptions " | ||
"(e.g. about the content or format of an unknown file). Leverage the given magic " | ||
"functions to implement function calls for which the arguments can't be determined " | ||
"yet. Reduce the amount of unnecessary data passed into these magic functions " | ||
"where possible, because magic costs money and magically processing large amounts " | ||
"of data is expensive. If you think are done with the task, you can simply call " | ||
"finish(reason='your reason') to end the task, " | ||
"a function that has one `finish` command, don't mix finish with other functions! " | ||
"If you still need to do other functions, " | ||
"let the next cycle execute the `finish` function. " | ||
"Avoid hard-coding input values as input, and avoid returning large outputs. " | ||
"The code that you have been executing in the past cycles can also be buggy, " | ||
"so if you see undesired output, you can always try to re-plan, and re-code. " | ||
) | ||
|
||
|
||
class CodeFlowAgentPromptStrategy(PromptStrategy): | ||
default_configuration: OneShotAgentPromptConfiguration = ( | ||
OneShotAgentPromptConfiguration() | ||
) | ||
|
||
def __init__( | ||
self, | ||
configuration: SystemConfiguration, | ||
logger: Logger, | ||
): | ||
self.config = configuration | ||
self.response_schema = JSONSchema.from_dict( | ||
CodeFlowAgentActionProposal.schema() | ||
) | ||
self.logger = logger | ||
self.commands: list[CompletionModelFunction] = [] | ||
|
||
@property | ||
def model_classification(self) -> LanguageModelClassification: | ||
return LanguageModelClassification.FAST_MODEL # FIXME: dynamic switching | ||
|
||
def build_prompt( | ||
self, | ||
*, | ||
messages: list[ChatMessage], | ||
task: str, | ||
ai_profile: AIProfile, | ||
ai_directives: AIDirectives, | ||
commands: list[CompletionModelFunction], | ||
**extras, | ||
) -> ChatPrompt: | ||
"""Constructs and returns a prompt with the following structure: | ||
1. System prompt | ||
3. `cycle_instruction` | ||
""" | ||
system_prompt, response_prefill = self.build_system_prompt( | ||
ai_profile=ai_profile, | ||
ai_directives=ai_directives, | ||
functions=commands, | ||
) | ||
|
||
self.commands = commands | ||
final_instruction_msg = ChatMessage.system(FINAL_INSTRUCTION) | ||
|
||
return ChatPrompt( | ||
messages=[ | ||
ChatMessage.system(system_prompt), | ||
ChatMessage.user(f'"""{task}"""'), | ||
*messages, | ||
final_instruction_msg, | ||
], | ||
prefill_response=response_prefill, | ||
) | ||
|
||
def build_system_prompt( | ||
self, | ||
ai_profile: AIProfile, | ||
ai_directives: AIDirectives, | ||
functions: list[CompletionModelFunction], | ||
) -> tuple[str, str]: | ||
""" | ||
Builds the system prompt. | ||
|
||
Returns: | ||
str: The system prompt body | ||
str: The desired start for the LLM's response; used to steer the output | ||
""" | ||
response_fmt_instruction, response_prefill = self.response_format_instruction() | ||
system_prompt_parts = ( | ||
self._generate_intro_prompt(ai_profile) | ||
+ [ | ||
"## Your Task\n" | ||
"The user will specify a task for you to execute, in triple quotes," | ||
" in the next message. Your job is to complete the task, " | ||
"and terminate when your task is done." | ||
] | ||
+ ["## Available Functions\n" + self._generate_function_headers(functions)] | ||
+ ["## RESPONSE FORMAT\n" + response_fmt_instruction] | ||
) | ||
|
||
# Join non-empty parts together into paragraph format | ||
return ( | ||
"\n\n".join(filter(None, system_prompt_parts)).strip("\n"), | ||
response_prefill, | ||
) | ||
|
||
def response_format_instruction(self) -> tuple[str, str]: | ||
response_schema = self.response_schema.copy(deep=True) | ||
|
||
# Unindent for performance | ||
response_format = re.sub( | ||
r"\n\s+", | ||
"\n", | ||
response_schema.to_typescript_object_interface(_RESPONSE_INTERFACE_NAME), | ||
) | ||
response_prefill = f'{{\n "{list(response_schema.properties.keys())[0]}":' | ||
|
||
return ( | ||
( | ||
f"YOU MUST ALWAYS RESPOND WITH A JSON OBJECT OF THE FOLLOWING TYPE:\n" | ||
f"{response_format}" | ||
), | ||
response_prefill, | ||
) | ||
|
||
def _generate_intro_prompt(self, ai_profile: AIProfile) -> list[str]: | ||
"""Generates the introduction part of the prompt. | ||
|
||
Returns: | ||
list[str]: A list of strings forming the introduction part of the prompt. | ||
""" | ||
return [ | ||
f"You are {ai_profile.ai_name}, {ai_profile.ai_role.rstrip('.')}.", | ||
# "Your decisions must always be made independently without seeking " | ||
# "user assistance. Play to your strengths as an LLM and pursue " | ||
# "simple strategies with no legal complications.", | ||
Comment on lines
+206
to
+208
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think remove if unused |
||
] | ||
|
||
def _generate_function_headers(self, funcs: list[CompletionModelFunction]) -> str: | ||
return "\n\n".join(f.fmt_header(force_async=True) for f in funcs) | ||
|
||
async def parse_response_content( | ||
self, | ||
response: AssistantChatMessage, | ||
) -> OneShotAgentActionProposal: | ||
if not response.content: | ||
raise InvalidAgentResponseError("Assistant response has no text content") | ||
|
||
self.logger.debug( | ||
"LLM response content:" | ||
+ ( | ||
f"\n{response.content}" | ||
if "\n" in response.content | ||
else f" '{response.content}'" | ||
) | ||
) | ||
assistant_reply_dict = extract_dict_from_json(response.content) | ||
|
||
parsed_response = CodeFlowAgentActionProposal.parse_obj(assistant_reply_dict) | ||
if not parsed_response.python_code: | ||
raise ValueError("python_code is empty") | ||
|
||
available_functions = { | ||
f.name: FunctionDef( | ||
name=f.name, | ||
arg_types=[(name, p.python_type) for name, p in f.parameters.items()], | ||
arg_descs={name: p.description for name, p in f.parameters.items()}, | ||
arg_defaults={ | ||
name: p.default or "None" | ||
for name, p in f.parameters.items() | ||
if p.default or not p.required | ||
}, | ||
return_type=f.return_type, | ||
return_desc="Output of the function", | ||
function_desc=f.description, | ||
is_async=True, | ||
) | ||
for f in self.commands | ||
} | ||
available_functions.update( | ||
{ | ||
"main": FunctionDef( | ||
name="main", | ||
arg_types=[], | ||
arg_descs={}, | ||
return_type="str", | ||
return_desc="Output of the function", | ||
function_desc="The main function to execute the plan", | ||
is_async=True, | ||
) | ||
} | ||
) | ||
code_validation = await CodeValidator( | ||
function_name="main", | ||
available_functions=available_functions, | ||
).validate_code(parsed_response.python_code) | ||
|
||
# TODO: prevent combining finish with other functions | ||
if re.search(r"finish\((.*?)\)", code_validation.functionCode): | ||
finish_reason = re.search( | ||
r"finish\((reason=)?(.*?)\)", code_validation.functionCode | ||
).group(2) | ||
result = OneShotAgentActionProposal( | ||
thoughts=parsed_response.thoughts, | ||
use_tool=AssistantFunctionCall( | ||
name="finish", | ||
arguments={"reason": finish_reason[1:-1]}, | ||
), | ||
) | ||
else: | ||
result = OneShotAgentActionProposal( | ||
thoughts=parsed_response.thoughts, | ||
use_tool=AssistantFunctionCall( | ||
name="execute_code_flow", | ||
arguments={ | ||
"python_code": code_validation.functionCode, | ||
"plan_text": parsed_response.immediate_plan, | ||
}, | ||
), | ||
) | ||
return result |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This approach requires the user to explicitly pass commands.
I think it's better to take commands from the agent as opposed to the current to the component. You could for example pass
Agent
(or some getter) to the component__init__
and then just accesscommands
when required.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't get it, so the component will depends on agent?
Now you need to build an agent to simply execute a component, and won't this also create a circular dependency?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are right, it's a bad idea to pass
Agent
, just pass a method/lambda instead:So you don't need to call
set_available_functions
in agent classes - it's the component responsibility and dev shouldn't worry about calling anything extra just so the component works.