SupportedUnderstudyAction Enum validation for 'method' on act/observe

[miguelg719]

why

act and observe don't fail on validation if the LLM generates an action with a non-supported method

what changed

• Added z.enum(supportedUnderstudyAction) to the act and observe inference schemas

test plan

• Evals (act/observe)

---

Summary by cubic

Enforces SupportedUnderstudyAction enum for the method field in act and observe so invalid actions are rejected during schema validation. Prevents unsupported methods from slipping through when the LLM generates actions and supports Linear STG-1212.

• Bug Fixes

  • Use z.enum(Object.values(SupportedUnderstudyAction)) for act/observe; unsupported methods fail fast across Zod v3/v4.
  • Added tests confirming enum validation works in both Zod v3 and v4.
  • Updated method description to reference Understudy interaction methods instead of Playwright.

• Refactors

  • Migrated enum compatibility tests from Playwright to Vitest and moved them to packages/core/tests.

^{Written for commit 4aa2573. Summary will update on new commits. Review in cubic}

[changeset-bot]

🦋 Changeset detected

Latest commit: 4aa2573

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@browserbasehq/stagehand	Patch
@browserbasehq/stagehand-evals	Patch
@browserbasehq/stagehand-server	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[cubic-dev-ai]

1 issue found across 2 files

Confidence score: 2/5

• SupportedUnderstudyAction is passed to z.enum in packages/core/lib/inference.ts, which will throw at runtime because z.enum expects a string literal tuple, making this a likely break in schema construction
• Given the 7/10 severity and direct runtime failure risk, this change carries high merge risk until the enum handling is corrected
• Pay close attention to packages/core/lib/inference.ts - fix the enum usage (e.g., z.nativeEnum or Object.values(...)) to avoid runtime schema errors.

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="packages/core/lib/inference.ts">

<violation number="1" location="packages/core/lib/inference.ts:258">
P1: `SupportedUnderstudyAction` is a TS enum, but `z.enum` expects a string literal tuple. This will fail schema construction at runtime; use `z.nativeEnum` (or pass `Object.values(...)`) for enums.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant User as User Code
    participant Core as Stagehand Core
    participant LLM as LLM Client
    participant Zod as Zod Validator

    User->>Core: act() / observe()
    
    Note over Core: CHANGED: Schema definition now uses<br/>z.enum(SupportedUnderstudyAction)<br/>instead of z.string()

    Core->>LLM: Generate completion (prompt + schema)
    LLM-->>Core: JSON Response (proposed action)

    Core->>Zod: Validate response against schema

    alt Valid Method (e.g. "click", "type")
        Zod-->>Core: Parsed Object
        Core->>Core: Proceed to execution
        Core-->>User: Success
    else NEW: Invalid Method (e.g. "hover", "unknown")
        Note over Zod: Validation fails immediately<br/>(Value not in Enum)
        Zod--xCore: ZodError
        Core--xUser: Throw Validation Error
    end

Loading

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds enum validation to the method field in both act and observe inference schemas to prevent LLM-generated actions with unsupported methods from passing validation.

Key Changes:

• Replaced .string() with .enum(Object.values(SupportedUnderstudyAction)) for the method field in both act and observe schemas in packages/core/lib/inference.ts:257-264 and packages/core/lib/inference.ts:398-405
• Used Object.values() approach for Zod v3/v4 compatibility (Zod v3 doesn't accept TypeScript enums directly in z.enum())
• Updated method field description from "Playwright interaction methods" to "Understudy interaction methods" for accuracy
• Added comprehensive test suite validating the approach works in both Zod v3 and v4

How it Works:
The PR validates that LLM-generated method values must match one of the 11 supported actions (click, fill, type, press, scrollTo, nextChunk, prevChunk, selectOptionFromDropdown, hover, doubleClick, dragAndDrop) before being passed to performUnderstudyMethod in packages/core/lib/v3/handlers/handlerUtils/actHandlerUtils.ts:84-110, which throws UnderstudyCommandException for unsupported methods. This change shifts validation earlier in the pipeline, providing faster feedback.

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The change adds validation without breaking existing functionality. The implementation correctly handles Zod v3/v4 compatibility using Object.values(), which is verified by comprehensive test coverage. The validation aligns perfectly with the actual supported methods in performUnderstudyMethod.
• No files require special attention

Important Files Changed

Filename	Overview
.changeset/floppy-zebras-lie.md	Adds standard changeset file documenting the patch-level change
packages/core/lib/inference.ts	Adds enum validation for method field in act/observe schemas, compatible with both Zod v3 and v4
packages/core/lib/v3/tests/zod-enum-compatibility.spec.ts	Comprehensive test suite verifying Zod v3/v4 compatibility with Object.values() approach

Sequence Diagram

sequenceDiagram
    participant LLM as LLM Client
    participant Inference as act/observe inference
    participant Zod as Zod Schema Validator
    participant Handler as ActHandler/ObserveHandler
    participant Understudy as performUnderstudyMethod
    
    LLM->>Inference: Generate action with method field
    Inference->>Zod: Validate against schema with enum
    alt Invalid method (e.g., "invalidMethod")
        Zod-->>Inference: ❌ Validation fails
        Inference-->>LLM: Reject early - unsupported method
    else Valid method (e.g., "click")
        Zod-->>Inference: ✅ Validation passes
        Inference->>Handler: Return validated action
        Handler->>Understudy: performUnderstudyMethod(method, xpath, args)
        Understudy->>Understudy: Check METHOD_HANDLER_MAP or switch
        Understudy-->>Handler: Execute method handler
        Handler-->>LLM: Action completed
    end

Loading

[greptile-apps]

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Should use z.nativeEnum(SupportedUnderstudyAction) instead of z.enum(SupportedUnderstudyAction). The z.enum() function expects a tuple of string literals like ["value1", "value2"], while z.nativeEnum() is designed for TypeScript enums.

Suggested change

	.enum(SupportedUnderstudyAction)
	.nativeEnum(SupportedUnderstudyAction)

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/core/lib/inference.ts
Line: 258:258

Comment:
Should use `z.nativeEnum(SupportedUnderstudyAction)` instead of `z.enum(SupportedUnderstudyAction)`. The `z.enum()` function expects a tuple of string literals like `["value1", "value2"]`, while `z.nativeEnum()` is designed for TypeScript enums.

```suggestion
            .nativeEnum(SupportedUnderstudyAction)
```

How can I resolve this? If you propose a fix, please make it concise.

[miguelg719]

Outdated for Zod 4. According to the documentation:
"This replaces the z.nativeEnum() API in Zod 3"

Docs:

• https://zod.dev/api#enums
• https://zod.dev/v4/changelog?id=znativeenum-deprecated

In Zod 4, z.enum() now accepts TypeScript enums directly:

  enum Fish {                                                                                                                    
    Salmon = 0,                                                                                                                  
    Tuna = 1                                                                                                                     
  }                                                                                                                              
                                                                                                                               
  const FishEnum = z.enum(Fish);  // ✅ Works in Zod 4  

[greptile-apps]

Same issue - should use z.nativeEnum(SupportedUnderstudyAction) instead of z.enum(SupportedUnderstudyAction).

Suggested change

	.enum(SupportedUnderstudyAction)
	.nativeEnum(SupportedUnderstudyAction)

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/core/lib/inference.ts
Line: 393:393

Comment:
Same issue - should use `z.nativeEnum(SupportedUnderstudyAction)` instead of `z.enum(SupportedUnderstudyAction)`.

```suggestion
      .nativeEnum(SupportedUnderstudyAction)
```

How can I resolve this? If you propose a fix, please make it concise.

[miguelg719]

Outdated for Zod 4. According to the documentation:
"This replaces the z.nativeEnum() API in Zod 3"

Docs:

• https://zod.dev/api#enums
• https://zod.dev/v4/changelog?id=znativeenum-deprecated

In Zod 4, z.enum() now accepts TypeScript enums directly:

  enum Fish {                                                                                                                    
    Salmon = 0,                                                                                                                  
    Tuna = 1                                                                                                                     
  }                                                                                                                              
                                                                                                                               
  const FishEnum = z.enum(Fish);  // ✅ Works in Zod 4  

[miguelg719]

Addresses follow up on #1608

[miguelg719]

@greptileai @cubic-dev-ai

[cubic-dev-ai]

@greptileai @cubic-dev-ai

@miguelg719 I have started the AI code review. It will take a few minutes to complete.

[greptile-apps]

I can see the AI code review has been initiated for this PR. The changes look focused on adding proper enum validation for the method field in act/observe operations to prevent invalid actions from being accepted.

The implementation appears to address an important validation gap where unsupported methods could slip through without proper validation. Using z.enum(Object.values(SupportedUnderstudyAction)) is the correct approach for ensuring type safety across different Zod versions.

Feel free to ask any questions about the changes or share the AI review results when they're ready!

[greptile-apps]

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[cubic-dev-ai]

No issues found across 3 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant Client
    participant Inference as Inference (act/observe)
    participant Zod as Zod Validator
    participant LLM as LLM Provider

    Note over Client, LLM: Flow applies to both act() and observe()

    Client->>Inference: Call act() or observe()
    
    Note over Inference: CHANGED: Schema Definition
    Inference->>Inference: Define 'method' using z.enum(Object.values(Enum))<br/>(Ensures Zod v3/v4 compatibility)

    Inference->>LLM: Prompt with updated schema description
    LLM-->>Inference: Return JSON response

    Inference->>Zod: Validate response against schema
    
    alt NEW: Invalid "method" (not in SupportedUnderstudyAction)
        Note right of LLM: e.g. { method: "non_existent_action" }
        Zod-->>Inference: Validation Error (Invalid Enum Value)
        Inference-->>Client: Throw Error (Fail Fast)
    else Valid "method"
        Note right of LLM: e.g. { method: "click" }
        Zod-->>Inference: Success (Parsed Action)
        Inference->>Inference: Execute Action / Return Observation
        Inference-->>Client: Result
    end

Loading