Skip to content

[PM-39748] llm: Add skill to fix flaky tests - #2839

Merged
fedemkr merged 3 commits into
mainfrom
PM-39748/llm-skill-fix-flaky-tests
Jul 16, 2026
Merged

[PM-39748] llm: Add skill to fix flaky tests#2839
fedemkr merged 3 commits into
mainfrom
PM-39748/llm-skill-fix-flaky-tests

Conversation

@fedemkr

@fedemkr fedemkr commented Jun 29, 2026

Copy link
Copy Markdown
Member

🎟️ Tracking

PM-39748

📔 Objective

Adds a new .claude/skills/fixing-flaky-tests/SKILL.md skill for diagnosing and fixing intermittently failing tests in the Bitwarden iOS codebase.
The skill guides Claude through a structured workflow: locate the test → diagnose the root cause → fix → stress-test 100 times → run the full unit suite → run build-test-verify

What the skill does

When a developer reports a flaky test by name, Claude will:

  • Find the test file and read both the test and the SUT it exercises
  • Classify the root cause (race condition, shared mutable state, time/date dependence, order dependence, resource contention)
  • Apply a fix scoped to the test file where possible; if the SUT needs to change, present a plan and ask before touching production code
  • Verify stability by running the fixed test 100 times using xcodebuild test-without-building -test-iterations 100
  • Run the full -Unit test plan once to check for regressions
  • Invoke the build-test-verify skill as the final gate

The skill references project-specific test helpers (waitFor, waitForAsync, withContinuationTimeout from SwiftTestingHelpers.swift / BaseBitwardenTestCase) and the TimeProvider protocol for time-dependent SUT fixes.

@fedemkr
fedemkr requested review from a team as code owners June 29, 2026 14:10
@github-actions github-actions Bot added the t:llm Change Type - LLM related change (e.g. CLAUDE.md files) label Jun 29, 2026
@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Claude Code is reviewing this pull request...

If this comment does not update with results, check the Actions log.

@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.97%. Comparing base (3b93b7b) to head (c297e7d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2839      +/-   ##
==========================================
- Coverage   81.25%   78.97%   -2.29%     
==========================================
  Files        1028     1151     +123     
  Lines       66164    73459    +7295     
==========================================
+ Hits        53760    58012    +4252     
- Misses      12404    15447    +3043     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@fedemkr fedemkr added the ai-review Request a Claude code review label Jun 29, 2026
Comment thread .claude/skills/fixing-flaky-tests/SKILL.md
fedemkr added 2 commits July 16, 2026 13:52
Correct the test plan mapping table (ViewInspector/Networking targets are
not in BitwardenKit-Unit), fix try await for Swift Testing waitForAsync,
and clarify order-dependence behavior between Default and Unit plans.
@fedemkr
fedemkr enabled auto-merge (squash) July 16, 2026 19:39
@fedemkr
fedemkr merged commit dae7094 into main Jul 16, 2026
19 of 20 checks passed
@fedemkr
fedemkr deleted the PM-39748/llm-skill-fix-flaky-tests branch July 16, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-review Request a Claude code review t:llm Change Type - LLM related change (e.g. CLAUDE.md files)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants