Skip to content

Releases: Someone-anon-coder/Eir

v1.0.0 — 1.0.0 closure release

Choose a tag to compare

@Someone-anon-coder Someone-anon-coder released this 17 Jul 10:11
668637e

playwright-eir 1.0.0 — closure release

A closure release, not a feature release. 0.3.0's own known-limitations
list named twelve conscious-closure items plus an external verification
step and a completed security review as everything standing between it
and a 1.0.0 stability commitment. Every one reached a final disposition
this session — fixed (with tests), measured (with real data), or
stated as a deliberate, documented stance — nothing silently dropped,
nothing softened. Full detail: FULL_UNDERSTANDING.md
and docs/security-review-1.0.md.

Measured results (seed 42, 8-class mutation benchmark — unchanged since 0.3.0)

Mutation Class Heal Rate False-Heal Rate Suggestion Rate Miss Rate
id-rename 75.0% 0.0% 25.0% 0.0%
text-change 87.5% 0.0% 12.5% 0.0%
tag-swap 100.0% 0.0% 0.0% 0.0%
class-shuffle 25.0% 0.0% 75.0% 0.0%
sibling-reorder 0.0% 0.0% 0.0% 100.0%
wrapper-inject 100.0% 0.0% 0.0% 0.0%
near-duplicate-sibling-swap 25.0% 0.0% 75.0% 0.0%
compound-release 50.0% 0.0% 25.0% 25.0%

False-heal rate: 0.0% across every class.

Closure-list summary

Fixed:

  • EirLocator/EirPage now unwrap themselves at every real
    Locator-argument boundary (.and(), .or(), .dragTo(), .filter(),
    .locator(sel, {has}), addLocatorHandler/removeLocatorHandler — 8
    call sites, centralized in one helper). This was the single largest
    reason 0.3.0 wasn't 1.0.0.
  • expect(...).toHaveScreenshot() now works through both wrappers
    (_expectScreenshot/_frame/_selector forwarding).
  • ci-action's "has findings" check no longer misses a genuine heal
    whose suggestion generation itself failed.
  • Retried-test duplicate rows are collapsed in the rendered PR comment
    (the raw eir-report.json artifact still keeps every attempt).
  • One real security finding, fixed: a markdown/table-structure
    injection in the PR-comment renderer, from page-derived content
    (verified with a real hostile fixture before and after the fix).
    Found during a completed 7-area security review — every other area
    (secrets, browser-context code, filesystem writes, network calls, the
    ci-action threat model, supply chain) passed clean.

Verified live, not just unit-tested:

  • The GitHub Action's documented adoption path, from a real external
    fork — zero friction, despite the fork's own repo-level permissions
    defaulting to read-only.
  • The no-heals "comment updates to a clean state" path, exercised for
    the first time by a real PR losing its findings across two pushes.
  • The CI dogfood demo, generalized from a one-time branch name to a
    repeatable eir-dogfood/* prefix.

Measured:

  • suggestThreshold's default (0.3) got a real anchoring attempt — 66
    matched attempts across the full benchmark, confidence range
    0.5849–1.0000, nothing near the floor. Stays an honest estimate,
    confirmed rather than assumed.

Documented as deliberate 1.0.0 stances:

  • The fingerprint schema's class-token ceiling (measured 25% on
    class-shuffle) — schema v2 is the first Post-1.0 roadmap item.
  • Capture-point coverage boundary (6 named methods; everything else,
    including Frame, is an accepted untracked passthrough).
  • The small set of undocumented Playwright internals this package
    forwards, with CI-on-peer-range as the ongoing tripwire.
  • The Gemini fallback's measured ~23% free-tier reliability — shipped
    disabled by default on that evidence.
  • GitHub Marketplace publication of ci-action — intentionally parked.

Semver honesty

The public API surface — the three exports subpaths (.,
./globalTeardown, ./reporter), the EirConfig/EirMode shape, and
the .eir/routes/*.json/*.postconditions.json file formats — is now a
stability commitment. A breaking change to any of these is a major
version bump going forward, not folded into a minor or patch. Fingerprint
schema v2 (the roadmap item above) is explicitly named as the kind of
change that would trigger one.

Links

npm i -D playwright-eir

playwright-eir 0.3.0

Choose a tag to compare

@Someone-anon-coder Someone-anon-coder released this 16 Jul 14:07
38baa60

playwright-eir 0.3.0

Self-healing locator engine for Playwright. This release adds CI integration, an opt-in (and measured-not-helpful, disabled-by-default) Gemini fallback, and a full hardening/documentation pass — see the README for the complete picture.

Install

npm i -D playwright-eir
// before
import { test, expect } from "@playwright/test";
// after
import { test, expect } from "playwright-eir";

Results — measured, seed 42, pnpm bench:all

Mutation class Heal rate False-heal rate Suggestion rate Miss rate
id-rename 75.0% 0.0% 25.0% 0.0%
text-change 87.5% 0.0% 12.5% 0.0%
tag-swap 100.0% 0.0% 0.0% 0.0%
class-shuffle 25.0% 0.0% 75.0% 0.0%
sibling-reorder 0.0% 0.0% 0.0% 100.0%
wrapper-inject 100.0% 0.0% 0.0% 0.0%
near-duplicate-sibling-swap 25.0% 0.0% 75.0% 0.0%
compound-release 50.0% 0.0% 25.0% 25.0%

False-heal rate: 0.0% in every class, every measured run. class-shuffle and sibling-reorder are documented structural ceilings (fingerprint schema never captures an element's own class tokens; a position-anchored selector after a reorder doesn't throw, so triage never runs) — not tuning gaps. Full failure-mode analysis in the README; full tuning history in docs/tuning-log.md.

What's new since 0.2.0

Added

  • packages/ci-action — auto-updating PR comments with heal/suggestion diffs, confidence scores, linked screenshots.
  • Opt-in Gemini LLM fallback, structurally suggestion-capped (can never promote a row to healed). Measured across 74 real invocations: no accuracy benefit on any of the 8 mutation classes, and a real free-tier reliability ceiling (77% degraded to no-verdict on rate-limiting). Shipped disabled by default on that evidence — see docs/hybrid-comparison.md.
  • RetryOutcome/ReportRow now distinguish a heal genuinely verified against a recorded post-condition from one accepted because no baseline existed to check against.
  • Full README, a real-timed demo path (demo/README.md, ~20s clone-to-suggestion), and complete docs/ coverage.

Fixed

  • classifyFailureSpecies now recognizes a Playwright test-level timeout message, not only a bounded action-timeout one — closes a gap where a suite without actionTimeout configured would silently never engage Eir's triage on a real broken selector.
  • Benchmark evidence CLIs refuse to overwrite an existing report without --force.

Proven live, not just unit-tested

  • Mechanism A (post-condition verification) demonstrated catching a real, live false heal produced end-to-end by the actual (unmocked) matching engine — note005RealFalseHeal.test.ts.

Documented (known limitations, not fixed this release)

  • EirLocator passed where Playwright expects a real Locator (.and(), .or(), .dragTo(), .locator(sel, {has})) is a real, confirmed bug — characterized with a committed test, fix tracked for a future release.
  • An element's own class tokens are never captured in its fingerprint (the class-shuffle ceiling) — a fingerprint schema-v2 candidate.
  • Full list in the README's "Known limitations" section.

Full changelog: packages/eir/CHANGELOG.md