Skip to content

Conversation

@fg-nava
Copy link
Collaborator

@fg-nava fg-nava commented Aug 12, 2025

Ticket

Resolves: Enable Playwright artifact storage across environments

Changes

  • Artifacts to filesystem/GCS
    • Output dir for Playwright MCP artifacts: artifacts/ enabled via --output-dir
    • Added --save-session and --save-trace for full artifact storing
    • File: src/mastra/mcp.ts
  • Screenshots
    • Added the “Screenshot Protocol” instructions to system prompt
    • Take one full-page (fullPage: true) screenshot after completing each benefits application page, not per interaction
    • Example tool call is also set in the prompt since gemini-2.5-pro isn't so great at following instructions (sonnet is much better).
    • File: src/mastra/agents/web-automation-agent.ts

Context for reviewers

We need consistent, reviewable browser automation output (screenshots, traces, sessions)

Artifact Storage Locations

.mastra/output/artifacts/
├── page-{ISO-timestamp}.png                    # Screenshots (auto-generated timestamps)
├── session-{timestamp}/                        # Session artifacts (--save-session)
│   ├── session.md                             # Session summary and metadata
│   └── {number}.snapshot.yml                  # DOM snapshots at different points
└── traces-{timestamp}/                         # Playwright traces (--save-trace)
    ├── trace.trace                            # Main trace file for Playwright inspector
    ├── trace.network                          # Network activity logs
    └── resources/                             # Additional trace resources

@fg-nava fg-nava changed the title Fg/store playwright artifacts feat: setup playwright artifacts GCS storage Aug 12, 2025
@KayTV
Copy link
Collaborator

KayTV commented Aug 12, 2025

Can you update the PR with the description of what you changed?

@fg-nava fg-nava changed the title feat: setup playwright artifacts GCS storage feat: env-aware Playwright artifacts (local + GCS), screenshot protocol Aug 12, 2025
@fg-nava fg-nava changed the title feat: env-aware Playwright artifacts (local + GCS), screenshot protocol feat: enable playwright artifacts and screenshot protocol Aug 13, 2025
@fg-nava fg-nava requested a review from KayTV August 13, 2025 15:47
Copy link
Collaborator

@KayTV KayTV left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

**Screenshot Protocol:**
- Take a screenshot after completing all fields on a page
- Use fullPage: true to capture the complete viewport including off-screen content
- Do NOT take screenshots for individual form interactions
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a note I have is not taking screenshots of security risk ones like SSN. Most forms will hide this after they enter in their SSN with *** but there are less technical advance forms that do not have this feature

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I agree, the defensible approach is to not take a screenshot. However if the form is a single page this means we are not able to visually confirm.

There are targeted approaches to taking screenshots as well:

https://playwright.dev/docs/screenshots#element-screenshot

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving as is for now based on standup conversation, but underscoring this as a security consideration along with other artifacts (traces, snapshots).

@fg-nava fg-nava merged commit 1d8efaa into main Aug 13, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants