Skip to content

Fix run skipped/ignored tests with --run-ignored#674

Open
OmChillure wants to merge 3 commits intokarva-dev:mainfrom
OmChillure:fix-run-skipped-test-with-run-ignored
Open

Fix run skipped/ignored tests with --run-ignored#674
OmChillure wants to merge 3 commits intokarva-dev:mainfrom
OmChillure:fix-run-skipped-test-with-run-ignored

Conversation

@OmChillure
Copy link
Copy Markdown

Fixes #548

Summary

This PR adds run-ignored support through a dedicated CLI option and fixes skip-handling behavior while preserving existing filter semantics.

You can now run:

karva test --run-ignored only
karva test --run-ignored all

What Changed

Added a new test CLI option: --run-ignored WHICH

Supported modes:
default
only
all

Wired run-ignored mode through CLI, runner orchestration, worker settings, and semantic test execution.

Updated skip handling so:

  • default: skipped tests stay skipped
  • only: only skipped tests are executed
  • all: skipped and non-skipped tests are both executed
  • Added skip-marker presence handling in tag evaluation to support only mode correctly.

Added Integration Coverage For

  1. run-ignored only mode
  2. run-ignored all mode
  3. no-skipped-tests behavior under only mode
  4. conditional skip behavior with skipif(False, ...)
  5. module wiring for the new run_ignored integration test suite

Behavior

  1. Without --run-ignored: skipped tests remain skipped.
  2. With --run-ignored only: only skipped tests execute.
  3. With --run-ignored all: skipped and non-skipped tests execute together.

Validation

  1. Ran targeted integration tests for run_ignored scenarios.
  2. Verified expected outcomes for both only and all modes.
  3. Confirmed compile and test consistency for updated semantic runner flow.

Screenshots

Before
Screenshot from 2026-04-12 18-41-32

--run-ignored only
Screenshot from 2026-04-12 18-41-41

--run-ignored all
Screenshot from 2026-04-12 18-41-49

Notes
This PR introduces run-ignored behavior via a dedicated CLI option, not via filter expressions.

Copy link
Copy Markdown
Member

@MatthewMckee4 MatthewMckee4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

}
}
RunIgnoredMode::Only => {
if !tags.has_skip_tag() {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we not check should_skip here?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used has_skip_tag rather than should_skip so that tests with a skip decorator are included even when the skip condition evaluates to False (e.g. @karva.tags.skip(False,=reason=...)). should_skip would exclude those tests because the condition is inactive, but --run-ignored only targets any test decorated with skip, regardless of the condition.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this should be the case. If a test is not skipped normally (if it has a conditional skip tag) then it runs when we run skipped tests, that doesn't seem right.

Another way I think of this is that if you run tests normally, then you run tests with --run-ignored only, all tests should have run once and only once.

Copy link
Copy Markdown
Author

@OmChillure OmChillure Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh yeah , I reconsidered the thinking using has_skip_tag would cause tests with a False condition to run in both normal and --run-ignored passes, breaking the once-and-only-once . will update to 'should_skip'

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed now

PS : I have used .0 extracts the bool from (bool, Option<String>); we don't need the reason here since we are overriding the skip. thats fine right?

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Apr 12, 2026

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 1 untouched benchmark


Comparing OmChillure:fix-run-skipped-test-with-run-ignored (b43db8c) with main (094c588)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run skipped/ignored tests with --run-ignored

2 participants