0.31.0

AIs tend not to see the images, which makes the test inconsistent. We addressed this with several approaches:

• Reduced context size by optimizing prompts
• Added image referencing in the UI_STATE prompt

Breaking Change for Users Using userPromptTemplate

To make the project and prompts consistent, we have renamed {{COMMAND_TEMPLATES}} to {{ACTION_TEMPLATES}}. This will generate an error so you can identify if you are using the incorrect syntax.

This is the latest userPromptTemplate template.

<GOAL>{{USER_INPUT_GOAL}}</GOAL>

<STEP>
Current step: {{CURRENT_STEP}}
Max step: {{MAX_STEP}}

<PREVIOUS_STEPS>
{{STEPS}}
</PREVIOUS_STEPS>
</STEP>

<UI_STATE>
Please refer to the image above.
<ELEMENTS>
{{UI_ELEMENTS}}
</ELEMENTS>
<FOCUSED_TREE>
{{FOCUSED_TREE}}
</FOCUSED_TREE>
</UI_STATE>

<INSTRUCTIONS>
Based on the above, decide on the next action to achieve the goal. Please ensure not to repeat the same action. The action must be one of the following:
</INSTRUCTIONS>

<ACTION_OPTIONS>
{{ACTION_TEMPLATES}}
</ACTION_OPTIONS>

What's Changed

• Improve user prompts by adding default messages for empty focused tree and UI elements by @takahirom in #185
• Refactor agent command types to agent action types for consistency in naming by @takahirom in #186

Full Changelog: 0.30.0...0.31.0