Skip to content

[fix]: harden act variables prompt#1922

Merged
seanmcguire12 merged 1 commit intomainfrom
update-prompting-act-variables
Mar 31, 2026
Merged

[fix]: harden act variables prompt#1922
seanmcguire12 merged 1 commit intomainfrom
update-prompting-act-variables

Conversation

@seanmcguire12
Copy link
Copy Markdown
Member

@seanmcguire12 seanmcguire12 commented Mar 30, 2026

why

  • models would sometimes respond with the variable name without wrapping it in percentage signs.
  • when this happens, the regex would fail to replace the variable name with the actual variable value

what changed

  • adjusted the prompting for the condition when variables are provided by the user
    • the updated prompting is more specific around what the variables are used for, and the required return shape.

test plan

  • existing act evals

Summary by cubic

Hardened the act variables prompt so models always return argument placeholders as %variableName%, preventing failed variable substitution. Also clarified the element ID format with concrete examples.

  • Bug Fixes

    • Added explicit, repeated guidance to use %variableName% in the arguments array when variables are provided.
    • Improved element ID schema description with examples like '0-76' and '16-21'.
  • Refactors

    • Introduced buildActVariablesPrompt() and reused it in both act and step-two prompts to keep messaging consistent and avoid duplication.

Written for commit 8b592f2. Summary will update on new commits. Review in cubic

@changeset-bot
Copy link
Copy Markdown

changeset-bot bot commented Mar 30, 2026

⚠️ No Changeset found

Latest commit: 8b592f2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant App as User Application
    participant Inf as Inference Service (act/stepTwo)
    participant PB as Prompt Builder
    participant LLM as AI Model (External)
    participant RP as Regex Processor

    Note over App,RP: Execution flow for actions involving dynamic variables

    App->>Inf: act(instruction, variables)
    
    rect rgb(240, 240, 240)
        Note over Inf,PB: Prompt Construction Phase
        Inf->>PB: buildActPrompt / buildStepTwoPrompt
        PB->>PB: NEW: buildActVariablesPrompt(variables)
        Note right of PB: Maps keys to %key% format<br/>Injects strict usage instructions
    end
    
    PB-->>Inf: Combined Prompt String

    Inf->>LLM: Send Prompt + Zod Schema
    Note over LLM: Schema CHANGED: Element ID<br/>now includes '0-76' examples

    LLM-->>Inf: Structured JSON Response (elementId, arguments)
    
    rect rgb(230, 245, 230)
        Note over Inf,RP: Variable Substitution Phase
        Inf->>RP: processArguments(arguments, variables)
        
        alt Model followed instructions (%variable%)
            RP->>RP: Regex matches %variableName%
            RP->>RP: Replace placeholder with actual value
        else Model returned raw variable name
            Note right of RP: Regex fails to match.<br/>Action executed with raw string.
        end
        RP-->>Inf: Final Action Arguments
    end

    Inf->>App: Return Action Result
Loading

@seanmcguire12 seanmcguire12 added the act These changes pertain to the act function label Mar 31, 2026
@seanmcguire12 seanmcguire12 merged commit 555805a into main Mar 31, 2026
733 of 735 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

act These changes pertain to the act function

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants