Production-grade, plug-and-play Playwright + TypeScript QE framework for UI, API, mobile, AI-powered, security, i18n, and file-operations testing — with built-in CI/CD, AI agents, LLM-as-Judge evaluation, MCP server, Jenkins orchestration, Jira sync, and Slack reporting.
| Advantage | Details |
|---|---|
| Zero-config start | Clone → npm ci → npm run test:smoke — works out of the box with OrangeHRM demo |
| Full Page Object Model | Every page extends BasePage; no inline selectors in spec files |
| AI-native testing | LLM-powered assertions, semantic similarity, AI locator fallback, hallucination detection |
| LLM-as-Judge | Multi-dimensional rubric scoring (relevance, accuracy, coherence, safety, faithfulness) via AWS Bedrock with anti-narcissistic-bias dual-model design |
| MCP integration | Model Context Protocol server (stdio + SSE) lets AI agents navigate, screenshot, run tests, and list specs |
| 7 CLI agents | Code review, test healer, suggestion generator, Jenkins trigger, Jira sync, Slack bot, run-and-report — all from the terminal |
| Security testing | SQLi/XSS fuzzer, security header validator, OWASP ZAP proxy integration |
| Multi-environment | `TEST_ENV=dev |
| Mobile testing | Dedicated mobile test suite with device emulation (Pixel 5, iPhone 13, iPad), touch/swipe helpers, responsive assertions |
| Multi-browser | Chromium, Firefox, WebKit, mobile Chrome, mobile Safari via ALL_BROWSERS=true |
| Mutation testing | Stryker Mutator with configurable thresholds and HTML/JSON reports |
| Full CI/CD | 10 GitHub Actions workflows: CI, nightly, smoke, PR gate, security scan, mutation, AI tests, API tests, report publishing, Copilot review |
| Jenkins pipeline | Jenkinsfile with EC2 provisioning scripts, single-test trigger, remote job management |
| Jira integration | Auto-create/update/close bugs from test results, attach JUnit XML, link test executions to stories |
| Slack notifications | Real-time build results, failure alerts, and report links posted to Slack channels |
| Allure + HTML reports | Dual reporting with Allure (GitHub Pages auto-publish) and Playwright HTML |
| Data factories | Faker-powered factories with auto-teardown registry for clean test isolation |
| File operations | Download verification, Excel read/write, Word document generation, PDF parsing with content assertions |
| i18n testing | Locale switcher, RTL detection, string validation across en, fr, ar, ja |
| Auth matrix | Multi-role testing (Admin, Manager, Employee) with cached storage state |
| Scaffolding CLI | scaffold:page, scaffold:test, scaffold:api generators for consistent code structure |
| Code quality | ESLint + Prettier + Husky pre-commit/pre-push hooks + lint-staged |
| Type safety | Strict TypeScript with Zod schema validation for API responses |
┌──────────────────────────────────────────────────────────────────┐
│ BestTester │
├─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬──────────┤
│UI Tests │API Tests│ Mobile │AI Tests │Security │ i18n │ File-Ops │
│(PW) │(Axios) │(Emulate)│(Bedrock)│(ZAP) │(Locale) │(xlsx/pdf)│
├──────────┴──────────┴──────────┴──────────┴──────────┴───────────┤
│ Page Object Model (src/pages/) │ API Layer (src/api/) │
│ Mobile Pages (src/mobile/) │ Security (src/security/) │
│ AI Helpers (src/ai/) │ Data Factories (src/data/) │
│ Fixtures (src/fixtures/) │ i18n (src/i18n/) │
├──────────────────────────────────────────────────────────────────┤
│ AI: LLM Client │ AI Assert │ Semantic Assert │ LLM Judge │
│ AI Locator │ Embeddings (Titan) │ MCP Server (stdio/SSE) │
├──────────────────────────────────────────────────────────────────┤
│ Agents: Code Review │ Test Healer │ Suggestions │ Jenkins │
│ Jira Sync │ Slack Bot │ Run-and-Report │
├──────────────────────────────────────────────────────────────────┤
│ CI/CD: GitHub Actions (10 workflows) │ Jenkins (EC2) │
│ Reports: Allure │ Playwright HTML │ Slack │ JUnit XML │ Jira │
└──────────────────────────────────────────────────────────────────┘
# 1. Clone
git clone https://github.com/your-org/BestTester.git && cd BestTester
# 2. Install
npm ci
# 3. Configure
cp .env.example .env # edit with your credentials — NEVER commit .env
# 4. Install browsers
npm run setup
# 5. Run smoke tests
npm run test:smokeBestTester/
├── .github/workflows/ # 10 CI/CD pipelines (ci, nightly, smoke, pr-gate, security, mutation, ai, api, report, copilot-review)
├── agents/ # 7 AI CLI agents (review, heal, suggest, jenkins, jira-sync, slack-bot, run-and-report)
├── config/ # Playwright config, global-setup, per-environment .env files (dev/staging/prod)
├── mcp/ # MCP server (stdio + SSE) and AI test generator client
├── mutation/ # Stryker mutation testing config and HTML/JSON reports
├── scripts/ # Shell & TS utilities (Jenkins provisioning, Slack, scaffolding, EC2)
├── src/
│ ├── ai/ # LLM client (Bedrock), AI assertions, semantic assertions, AI locator
│ │ └── judge/ # LLM-as-Judge: multi-rubric evaluation, safety checks, pairwise comparison
│ ├── api/ # API client layer (BaseAPI + auth, employee, leave clients)
│ ├── auth/ # Multi-role auth (Admin/Manager/Employee), storage state caching
│ ├── components/ # Reusable UI components (form, modal, navbar, table)
│ ├── constants/ # Shared constants
│ ├── data/ # Test data factories (Faker) + teardown registry
│ ├── fixtures/ # Custom Playwright fixtures (base, api, auth, mobile)
│ ├── i18n/ # Locale switcher, RTL detection, string validator
│ ├── mobile/ # Mobile page objects (BaseMobilePage + login, products, cart)
│ ├── pages/ # Page Object Model (login, dashboard, employee, leave, reports)
│ ├── security/ # SQLi/XSS fuzzer, security header validator, OWASP ZAP client
│ ├── types/ # TypeScript interfaces (API responses, config, employee, Jira)
│ └── utils/ # Logger (Winston), Slack, Jenkins, 1Password, Jira, file-handler, download-verifier, Excel, webhook
├── tests/
│ ├── ai/ # AI/LLM response validation and content tests
│ ├── api/smoke/ # API smoke tests
│ ├── api/regression/ # API regression tests with Zod schema validation
│ ├── auth-matrix/ # Multi-role access control tests
│ ├── file-ops/ # File download, Excel, Word, PDF tests
│ ├── i18n/ # Internationalization smoke tests
│ ├── mobile/ # Mobile device emulation tests (login, products, responsive)
│ ├── security/ # Security header and penetration tests
│ ├── ui/smoke/ # UI smoke suite (@smoke)
│ ├── ui/regression/ # UI regression suite (@regression)
│ ├── ui/e2e/ # End-to-end journeys (@e2e)
│ └── ui/visual/ # Visual regression tests (@visual)
├── Jenkinsfile # Main Jenkins pipeline
├── Jenkinsfile.single-test # Single-test Jenkins pipeline
└── README.md
# All tests
npm test
# By type
npm run test:ui # UI tests
npm run test:api # API tests
npm run test:ai # AI/LLM tests
npm run test:mobile # Mobile device emulation tests
npm run test:file-ops # File operation tests
npm run test:security # Security tests
npm run test:i18n # Internationalization tests
npm run test:auth-matrix # Multi-role auth tests
# By tag
npm run test:smoke # @smoke
npm run test:regression # @regression
npm run test:e2e # @e2e
npm run test:visual # @visual
# Debug modes
npm run test:headed # Headed browser
npm run test:debug # Playwright inspector
# Mutation testing
npm run test:mutation # Stryker mutation analysisTEST_ENV=dev npm test
TEST_ENV=staging npm test
TEST_ENV=prod npm testALL_BROWSERS=true npm test # Chromium + Firefox + WebKit + mobile Chrome + mobile Safarinpm run test:mobile # Pixel 5 + iPhone 13
npm run test:mobile -- --project mobile-chrome # Android only
npm run test:mobile -- --project mobile-safari # iOS only
MOBILE_TABLET=true npm run test:mobile # Include iPad
npm run test:mobile -- --headed # Watch executionBuilt on AWS Bedrock Converse API with Amazon Nova Pro as the default model. Supports chat, summarization, and embeddings (Titan Embed v2).
- Semantic similarity — cosine similarity on Titan embeddings with configurable threshold
- Hallucination detection — keyword-based grounding checks
- AI locator fallback — sends DOM snapshot to LLM to resolve broken selectors
Multi-dimensional evaluation system with 6 built-in rubrics:
| Rubric | Use Case | Pass Threshold |
|---|---|---|
| Standard | General response quality | 3.5 |
| RAG Faithfulness | Grounding in source documents | 4.0 |
| Customer Support | Empathy, accuracy, actionability | 3.5 |
| Code Generation | Correctness, security, structure | 4.0 |
| Summarization | Coverage, conciseness, faithfulness | 3.5 |
| Toxicity & Safety | Hate speech, PII, prompt injection | 4.5 |
Scores across 6 dimensions: relevance, accuracy, coherence, safety, faithfulness, instruction adherence.
Anti-narcissistic bias: Primary model (Nova Pro) and judge model (Claude Haiku) use different architectures to prevent self-evaluation bias. Borderline scores (2–3) trigger automatic strict re-evaluation.
import { semanticExpect } from '@ai/semantic-assert';
await semanticExpect(response).toBeRelevantTo(prompt);
await semanticExpect(response).toBeGroundedIn(sourceContext);
await semanticExpect(response).toBeSafe();
await semanticExpect(response).toNotHallucinate(knownFacts);
await semanticExpect(response).toBeBetterThan(alternativeResponse);
await semanticExpect(response).toHaveScore({ min: 4.0, rubric: RUBRICS.CODE_GENERATION });Model Context Protocol server for AI-driven test orchestration. Supports both stdio and SSE transports.
npm run mcp:server # stdio mode
npm run mcp:server:sse # SSE mode on port 3001
npm run mcp:generate # AI test generator clientAvailable MCP tools:
| Tool | Description |
|---|---|
navigate |
Open URL, return title + screenshot |
screenshot |
Full-page screenshot of any URL |
get_page_text |
Extract visible text from a page |
run_test |
Execute a specific test file |
list_tests |
List all available test specs |
npm run agent:review -- --file tests/ui/login.spec.ts
npm run agent:review -- --file tests/ui/login.spec.ts --slack # post to SlackAuto-detects broken locators in failed tests and suggests AI-powered fixes on a new git branch.
npm run agent:heal -- --test-result reports/playwright-report/results.jsonAnalyzes page objects + existing test coverage and generates 5–10 new test case suggestions.
npm run agent:suggest -- --page src/pages/employee.page.tsnpm run agent:jenkins -- --job playwright-tests --params ENV=staging
npm run agent:jenkins -- --job playwright-tests --build 42
npm run agent:jenkins -- --job playwright-tests --build 42 --logsAuto-creates/updates/closes Jira bugs from test results. Attaches JUnit XML and links test executions.
npm run agent:jira -- --build latest
npm run agent:jira -- --junit reports/playwright-report/junit.xml --build 42
npm run agent:jira -- --junit reports/playwright-report/junit.xml --dry-runFull workflow: trigger Jenkins → stream console → download artifacts → sync to Jira → notify Slack.
npm run agent:run-and-report
npm run agent:run-and-report -- --suite smoke --jira-key SCRUM-5
npm run agent:run-and-report -- --suite regression --jira-key SCRUM-5 --dry-runnpx ts-node agents/slack-bot-agent.ts --report reports/playwright-report/results.jsonDedicated mobile test suite using Playwright device emulation against SauceDemo — a free, publicly available e-commerce demo app.
Device profiles:
| Project | Device | Viewport | Touch |
|---|---|---|---|
mobile-chrome |
Pixel 5 | 393×851 | ✓ |
mobile-safari |
iPhone 13 | 390×844 | ✓ |
tablet |
iPad (gen 7) | 810×1080 | ✓ |
Mobile page objects extend BaseMobilePage which provides:
swipe(direction, distance)— simulate touch swipe gesturestap(selector)— touch tap interactionisResponsive()— assert viewport is mobile-sized (<768px)getViewportSize()— retrieve current device dimensions
Test coverage:
- Login flows (valid, invalid, locked user)
- Product browsing and cart management
- Responsive layout assertions
- Burger menu navigation
- Touch scroll interactions
import { Page } from '@playwright/test';
import { BaseMobilePage } from './base-mobile.page';
export class MobileCheckoutPage extends BaseMobilePage {
private readonly firstNameInput = this.page.locator('[data-test="firstName"]');
constructor(page: Page) { super(page); }
async fillName(name: string): Promise<void> { await this.firstNameInput.fill(name); }
}Then register in src/fixtures/mobile.fixture.ts.
- SQLi/XSS fuzzer — automated form fuzzing with common injection payloads, validates no 500 errors or reflected XSS
- Security header validator — asserts CSP, HSTS (≥1 year), X-Frame-Options, X-Content-Type-Options, Referrer-Policy
- OWASP ZAP integration — proxy mode for passive/active scanning with spider and alert retrieval
| Workflow | Trigger | What it does |
|---|---|---|
ci.yml |
Push / PR | Lint → smoke tests → Playwright report → Slack |
nightly.yml |
Cron 2 AM UTC | Full regression → Allure report → Slack → auto-create GitHub issue on failure |
smoke.yml |
Manual / reusable | Smoke suite only |
pr-gate.yml |
Pull request | Quality gate checks |
api-tests.yml |
Manual / schedule | API smoke + regression |
ai-tests.yml |
Manual / schedule | AI/LLM validation tests |
security-scan.yml |
Manual / schedule | Security test suite |
mutation.yml |
Manual | Stryker mutation testing |
report.yml |
Post-nightly | Publish Allure to GitHub Pages |
copilot-review.yml |
PR | AI-assisted code review |
Jenkinsfile— full pipeline with parameterized suite selectionJenkinsfile.single-test— run a single test file remotely- EC2 provisioning scripts for Jenkins server setup
- Remote job creation and configuration utilities
npm run report:allure # Generate + open Allure report
npm run report:playwright # Open Playwright HTML reportReports are generated in 4 formats: HTML, JSON, JUnit XML, and Allure. Allure is auto-published to GitHub Pages on nightly runs.
npm run scaffold:page -- --name MyFeature # → src/pages/my-feature.page.ts + test stub
npm run scaffold:test -- --name my-feature # → tests/ui/regression/my-feature.spec.ts
npm run scaffold:api -- --name my-resource # → src/api/my-resource.api.tsimport { Page } from '@playwright/test';
import { BasePage } from './base.page';
export class MyFeaturePage extends BasePage {
private readonly submitButton = this.page.getByRole('button', { name: 'Submit' });
constructor(page: Page) { super(page); }
async goto(): Promise<void> { await this.navigate('/my-feature'); }
async submit(): Promise<void> { await this.submitButton.click(); }
}Then register in src/fixtures/base.fixture.ts.
| Category | Technologies |
|---|---|
| Test framework | Playwright, TypeScript |
| Mobile testing | Playwright device emulation (Pixel 5, iPhone 13, iPad) |
| AI/LLM | AWS Bedrock (Nova Pro, Claude Haiku, Titan Embeddings), OpenAI |
| MCP | @modelcontextprotocol/sdk (stdio + SSE) |
| API testing | Axios, Zod schema validation |
| Security | OWASP ZAP, custom SQLi/XSS fuzzer |
| Data | Faker.js, ExcelJS, pdf-parse, PDFKit, docx |
| CI/CD | GitHub Actions, Jenkins |
| Reporting | Allure, Playwright HTML, JUnit XML |
| Integrations | Jira (REST API), Slack (Webhooks + Web API), GitHub API, 1Password CLI |
| Code quality | ESLint, Prettier, Husky, lint-staged |
| Mutation testing | Stryker Mutator |
| Logging | Winston |
| Version control | simple-git |
| Secret management | 1Password CLI (op) |
Securely manage secrets via the 1Password CLI instead of storing them in .env.
- Install the 1Password CLI (
op): https://developer.1password.com/docs/cli/get-started - Create a Service Account and paste the token into
OP_SERVICE_ACCOUNT_TOKENin.env - Use the
OnePasswordutility in global-setup, fixtures, or scripts:
import { OnePassword } from '@utils/one-password';
const op = new OnePassword('BestTester'); // vault name
// Inject all fields from a vault item into process.env
op.injectEnv('BestTester Secrets');
// Or map specific fields to env vars
op.injectEnv('BestTester Secrets', {
SLACK_BOT_TOKEN: 'Slack Bot Token',
JIRA_API_TOKEN: 'Jira API Token',
AWS_SECRET_ACCESS_KEY: 'AWS Secret Key',
});
// Create a new secret
op.createItem('API Credentials', { apiKey: 'sk-abc123' });
// Update an existing field
op.setField('BestTester Secrets', 'GitHub Token', 'ghp_new...');| Method | Description |
|---|---|
getField(item, field) |
Read a single secret value |
getItem(item) |
Read all fields from an item |
listItems() |
List all vault items |
createItem(title, fields) |
Create a new item |
setField(item, field, value) |
Update a field on an existing item |
deleteItem(item) |
Delete an item |
injectEnv(item, mapping?) |
Bulk-inject fields into process.env |
whoAmI() |
Verify CLI authentication |
See CONTRIBUTING.md for full guidelines.
- Fork and create a feature branch:
git checkout -b feat/my-feature - Follow ESLint + Prettier rules (enforced by Husky pre-commit hooks)
- All tests must use Page Object Model — no inline selectors in spec files
- No
page.waitForTimeout()— use Playwright auto-waiting - Every test file must have a JSDoc block with
@file,@description,@tags - Run
npm run lint && npm run typecheck && npm run test:smokebefore opening a PR - Open a PR against
develop
playwright, typescript, test-automation, e2e-testing, api-testing, ai-testing, llm-testing, aws-bedrock, page-object-model, playwright-typescript, playwright-framework, playwright-boilerplate, playwright-template, test-framework, qa-automation, qe-framework, selenium-alternative, ui-testing, visual-regression, security-testing, owasp-zap, mutation-testing, stryker, allure-report, ci-cd, github-actions, jenkins, jira-integration, slack-bot, mcp, model-context-protocol, llm-as-judge, ai-assertions, semantic-testing, hallucination-detection, cross-browser-testing, i18n-testing, mobile-testing, device-emulation, responsive-testing, touch-testing, data-factory, test-data-management, download-verification, excel-testing, pdf-testing, playwright-best-practices, playwright-page-object, playwright-fixtures, playwright-ci-cd, playwright-ai, bedrock-testing, nova-pro, claude-haiku, titan-embeddings
Apache License 2.0 — see LICENSE for details.
You are free to use, modify, and distribute this framework in personal and commercial projects. Contributions are welcome and will be licensed under the same terms.