Skip to content

Add solitary/sociable/integration test levels to agent prompts and quality gate#2545

Merged
hiroshinishio merged 1 commit intomainfrom
wes
Apr 17, 2026
Merged

Add solitary/sociable/integration test levels to agent prompts and quality gate#2545
hiroshinishio merged 1 commit intomainfrom
wes

Conversation

@hiroshinishio
Copy link
Copy Markdown
Collaborator

@hiroshinishio hiroshinishio commented Apr 17, 2026

Summary

  • Introduce 3 test levels (solitary/sociable/integration) in agent prompts, replacing the old "unit test only" bias
  • Add integration test quality gate (db_operations_use_real_test_db, api_calls_tested_end_to_end, env_var_guards_for_secrets) to checklist
  • Add env var guard patterns: free services skip locally but run in CI, paid APIs always skip but tests exist as documentation
  • Remove duplicated rules from pr.xml (now single source of truth in coding_standards.xml)
  • Update PR titles from "Add unit tests" to "Add unit and integration tests"
  • Prefer npm "test" script over "test:unit" so integration tests (with env var guards) are included

Social Media Post (GitAuto)

Upgrade test generation to produce solitary, sociable, and integration tests

  • Agent now writes tests at 3 levels instead of only mocked unit tests
  • Integration tests use real test infrastructure when available
  • Env var guards let integration tests skip locally, run in CI where secrets exist
  • New quality gate catches missing integration tests during automated reviews

Social Media Post (Wes)

Customer told us they wanted more integration tests from our agent. Checked the PRs. Every single one was pure mocks, even when their repo had test DB helpers sitting right there. The prompt said "use test infrastructure if available" but the model ignored it every time. Why? One buried line loses against the rest of the prompt biasing toward mocks. Prompt alone is not enough. Added a quality gate that programmatically flags missing integration tests.

…ality gate

- Define 3 test levels in coding_standards.xml: solitary (mocked), sociable (real collaborators), integration (real external services)
- Add integration test quality gate in checklist.py (db_operations_use_real_test_db, api_calls_tested_end_to_end, env_var_guards_for_secrets)
- Add env var guard patterns for free services (skip locally, run in CI) and paid APIs (always skip)
- Remove duplicated rules from pr.xml that already exist in coding_standards.xml
- Update PR title from "Add unit tests" to "Add unit and integration tests"
- Prefer "test" over "test:unit" npm script to run all test levels
- Update "unit test" references globally in prompts, constants, and test fixtures
@hiroshinishio hiroshinishio self-assigned this Apr 17, 2026
@hiroshinishio hiroshinishio merged commit 6456338 into main Apr 17, 2026
1 check passed
@hiroshinishio hiroshinishio deleted the wes branch April 17, 2026 06:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant