AutoSpec AI v2.1.2

Dependency update

• Updates @anthropic-ai/sdk from 0.125.0 to 0.126.0, including the latest client retry and abort-handling fixes.
• Rebuilds and verifies the checked-in Node 24 ESM action bundle against the updated SDK.

This release includes all fixes and dependency upgrades from v2.1.1. Hosted CI passed clean install, lint, typecheck, 88 tests, audit, build, dist smoke, and cross-platform reproducibility checks.

Use the immutable autospec-ai/playwright@v2.1.2 reference or the rolling autospec-ai/playwright@v2 alias.

AutoSpec AI v2.1.1

Fixes

• Makes the bundled-action smoke test independent of the caller's Git history and GitHub event payload.
• Makes third-party license generation deterministic across macOS and Linux filesystems.
• Restores a fully green hosted CI pipeline for the Node 24 ESM release.

Dependency updates

• Upgrades actions/checkout to 7.0.1 and actions/setup-node to 7.0.0 using immutable commit SHAs.
• Updates the Anthropic and OpenAI SDKs, glob/minimatch, Jest 30, ESLint 10, TypeScript ESLint, and supporting tooling.
• Migrates linting to ESLint's flat configuration format.
• Keeps TypeScript on 5.9 until the current TypeScript ESLint and ts-jest release lines support newer majors.

Use the immutable autospec-ai/playwright@v2.1.1 reference or the rolling autospec-ai/playwright@v2 alias.

AutoSpec AI v2.1.0

Highlights

• Upgrades the GitHub Action runtime to Node 24 with checked-in ESM bundles and third-party license notices.
• Adds strict input, plan, path, and generated-TypeScript validation before files are written.
• Moves trace artifact upload into an always-run post step so failed test runs can still preserve diagnostics.
• Hardens pull-request diff collection, POM generation, fixture extraction, and release-branch Git operations.
• Expands automated coverage to 88 tests and strengthens CI with dist smoke tests, dependency auditing, pinned actions, and Dependabot.

Use the immutable autospec-ai/playwright@v2.1.0 reference or the rolling autospec-ai/playwright@v2 major-version alias.

v2.0.1 — Rebuild dist & deduplicate AutoSpec PRs

Patch release on top of v2.0.0.

Fixes

• Rebuilt dist/index.js — the v2.0.0 bundle was stale relative to its source. The raw-source POM injection, POM output directory, and temperature: 0 changes announced in v2.0.0 are now actually present in the published bundle.
• Deduplicate AutoSpec PRs — when the workflow triggers multiple times for the same commit (e.g. push + pull_request events), it now returns the existing open AutoSpec PR for that SHA instead of opening a duplicate.
• Log base branch selection — surfaces which branch the AutoSpec PR will target, making misconfigurations easier to diagnose from the Action logs.

Recommendation

If you pinned autospec-ai/playwright@v2.0.0, upgrade to @v2.0.1 (or pin @v2 to track the v2 line).

v2.0.0 — Raw source injection, deterministic output, POM output directory

Breaking changes

This is a major rework of how AutoSpec handles project context. The regex-based POM scanner has been replaced with raw source code injection.

What changed

1. Raw POM source in prompts (replaces regex extraction)

The LLM now sees the actual source code of your page objects and utilities — not a regex-extracted summary. This eliminates hallucination at the root:

• No more inventing methods like elements.getCaseFlowChart() that don't exist
• No more wrong class names in imports (Investigations instead of InvestigationsPage)
• Works with ANY POM pattern — nested objects, factory functions, deeply nested elements — because the LLM reads the actual code

The scanner is now ~100 lines instead of ~300 lines of fragile regex.

2. Temperature 0 — deterministic output

Same PR now produces the same tests on every run. Previously temperature: 0.1/0.2 caused different test plans and code for identical diffs.

3. POM output directory

New pom_output_directory input. When configured, the LLM can create new page object files as separate files instead of inlining classes in test specs.

- uses: autospec-ai/action@v1
  with:
    pom_output_directory: 'e2e/pages'  # new POMs go here as separate files

When not set, the LLM is instructed to use existing POMs or raw page.locator() calls — no more inline class definitions.

4. Better test coverage detection

The coverage section now shows individual test names (e.g., "should login with valid credentials | should show error for invalid password") so the LLM can see exactly what's already tested and avoid duplicating coverage.

Also

• Added **/*.po.ts to default POM auto-detection patterns
• project_context_budget now controls how much source code is included (increase for large projects with many POMs)

Recommended configuration

- uses: autospec-ai/action@v2
  with:
    llm_api_key: ${{ secrets.OPENAI_API_KEY }}
    test_directory: 'e2e/tests'
    pom_patterns: '**/*.po.ts,**/pages/**/*.ts'
    pom_output_directory: 'e2e/pages'
    project_context_budget: '12000'

v1.4.2 — Exact POM imports & factory-function extraction

What changed

This release consolidates v1.4.0 and v1.4.1 into a clean release with no vendor-specific references in the test suite.

Exact import statements in LLM prompts

The prompt now shows copy-pasteable import statements so the LLM can't rename classes:

### InvestigationsPage
`import { InvestigationsPage } from '../../pages/investigations.page';`
Methods: goto(), waitForTable()
Locators: getSearchInput = this.page.getByTestId('search'), ...

Previously the LLM would abbreviate class names (e.g., Investigations instead of InvestigationsPage).

Arrow-function & factory-function POM extraction

The scanner now handles three additional element patterns common in real-world POM classes:

Arrow functions in objects — getButton: (): Locator => this.page.getByTestId('btn')

Nested objects — advancedSection: { getToggle: (): Locator => ... } → extracted as advancedSection.getToggle

Factory functions — getFilters: () => { const x = ...; return { x } } → extracted as getFilters().x

Trace injection safety

test.use({ trace: ... }) is no longer inserted inside class bodies when the LLM emits stray imports mid-file.

Conditional anti-hallucination prompts

When the scanner finds no artifacts (e.g., naming conventions don't match defaults), the prompt no longer says "do NOT invent page objects" with an empty list — which was confusing the LLM into creating POMs inline.

Documentation

README now documents Project Structure Discovery: pom_patterns, utility_patterns, project_context_budget, default patterns, and a diagnostic tip for Action logs.

v1.4.1 — Exact POM imports & factory-function extraction

What changed

Fixes the two issues Sravya reported from PR #3347:

1. Class name mismatch in imports

The LLM was renaming POM classes in imports — e.g., import { Investigations } instead of import { InvestigationsPage }. The prompt now shows exact, copy-pasteable import statements:

Before (v1.4.0):

### InvestigationsPage (from '../../pages/investigations/investigations.po')
Methods: goto(), ...

After (v1.4.1):

### InvestigationsPage
`import { InvestigationsPage } from '../../pages/investigations/investigations.po';`
Methods: goto(), ...

The IMPORTANT section now explicitly says: "Copy the import statements EXACTLY as shown — do NOT rename, abbreviate, or alias class names."

2. Factory-function element groups

The InvestigationsPage POM uses a pattern where element groups are factory functions:

elements = {
  getSummaryElements: () => {
    const getSelectStatus = (): Locator => this.page.getByTestId('status-btn');
    return { getSelectStatus };
  },
}

The scanner now extracts these with qualified names like getSummaryElements().getSelectStatus, so the LLM knows both the group accessor and the individual locators inside it.

Also extracts nested object locators: v3Elements.getDashboardGrid for v3Elements: { getDashboardGrid: (): Locator => ... }.

v1.4.0 — Arrow-function POM support & trace injection fix

What changed

This release addresses the issues reported by Sravya on PR #3347: hallucinated POM methods, broken inline class output, and hallucinated navigation routes.

1. Arrow-function locator extraction (POM scanner)

The scanner now recognizes locators defined as arrow functions inside object literals — the pattern used by the Sophos CustomDashboards POM and many other codebases:

elements = {
  getWidgets: (): Locator => this.page.locator('[id*="widget"]'),
  getEditButton: (): Locator => this.page.getByTestId('edit-btn'),
  nested: {
    getDashboardGrid: (): Locator => this.page.locator('scwx-grid'),
  },
};

Previously the scanner found the class name but returned 0 locators for this pattern, causing the LLM to hallucinate element methods like elements.getCaseFlowChart() that don't exist.

2. Trace injection no longer breaks class bodies

The test.use({ trace: ... }) post-processor now stops scanning for imports at the first non-import code line. Previously, when the LLM emitted stray import statements mid-file (e.g., inside an inline class), the post-processor would insert test.use() inside the class body, producing invalid TypeScript.

3. Route extraction from POM classes

The scanner now extracts navigation URLs from POM goto() methods:

• this.page.goto('/dashboards/custom') → route: /dashboards/custom
• this.page.goto(path ?? '/default') → route: /default

These routes are injected into the LLM prompt so it uses real navigation paths instead of guessing from component/file names (e.g., /case-flow-expand-mode which doesn't exist).

Also included (from v1.3.1)

• Anti-hallucination prompts are now conditional — only injected when the scanner actually finds artifacts
• README documents Project Structure Discovery configuration

Upgrading

No configuration changes required. If you have custom pom_patterns, the new arrow-function extraction works automatically.

For projects with non-standard POM naming, set pom_patterns in your workflow:

pom_patterns: '**/*.po.ts,**/pages/**/*.ts'

v1.3.1 — Conditional project discovery prompts

What changed

Bug fix: anti-hallucination prompts now adapt to scanner results

When the project scanner finds no page objects or utilities (e.g., your naming conventions don't match the default patterns), the LLM prompts no longer include "do NOT invent page objects" with an empty list. Previously, this confused the model into creating POM classes inline inside test specs.

Now the prompts adapt:

• Artifacts found → "MUST reuse existing page objects listed above. Do NOT invent new ones."
• Nothing found → Neutral best-practice guidance (use accessible selectors, data-testid, etc.)

Documentation: Project Structure Discovery

The README now documents:

• What the scanner discovers (page objects, utilities, test coverage)
• Default auto-detection patterns and how to override them
• Configuration reference for pom_patterns, utility_patterns, and project_context_budget
• Diagnostic tip: check Action logs for Discovered: X page objects... to verify detection

If you're seeing missing POMs or incorrect imports

Add these inputs to your workflow:

- uses: autospec-ai/action@v1
  with:
    test_directory: 'e2e/tests'                        # where your tests live
    pom_patterns: '**/*.po.ts,**/pageobjects/**/*.ts'  # match YOUR naming convention
    utility_patterns: '**/helpers/**/*.ts'

Check the Action logs for Discovered: X page objects, Y utility files — if the counts are 0, your patterns don't match the defaults.

v1.3.0 — Project structure discovery

What's New

AutoSpec AI now scans your project for existing page objects, locators, utility functions, and test coverage before generating tests. This context is injected into the LLM prompts so generated tests reuse your real POM classes and locators instead of hallucinating ones that don't exist.

Features

• Page Object Discovery — Finds existing POM/page object classes, extracts method signatures and locator definitions (assignment, getter, readonly patterns)
• Utility Discovery — Finds exported helper functions and constants for reuse in generated tests
• Test Coverage Analysis — Analyzes existing tests for routes, describe blocks, test names, and imported page objects to avoid generating redundant tests
• Relevance Scoring — Prioritizes context related to the current diff (imports, same directory, name references)
• Token Budget Management — Configurable budget (default 8000 tokens) to keep prompts within limits
• Anti-Hallucination Instructions — Explicit prompt instructions to reuse listed objects and not invent new ones

New Action Inputs

Input	Default	Description
pom_patterns	(auto-detect)	Comma-separated globs for page object files
utility_patterns	(auto-detect)	Comma-separated globs for helper/utility files
project_context_budget	8000	Max token budget for project context in prompts

Auto-Detection

When no custom patterns are provided, AutoSpec automatically detects common conventions:

• POM: **/*.page.ts, **/pages/**/*.ts, **/page-objects/**/*.ts, **/*.pom.ts
• Utilities: **/helpers/**/*.ts, **/utils/**/*.ts, **/fixtures/**/*.ts, **/*.helper.ts