RFC: Implement Automated Reproducer Verification ("Fail-on-Base") for Bug Fix Integration Tests

[hanthor]

We have extensice tests but need to test our test before merging PRs. This is what other projects do like systemd. I propose trying this out. Maybe not require contributors to do it but if we can automate it enough we can make this work on the maintainer end

The merge-and-immediate-revert loop on #4539 / #4685 highlighted a gap in our CI: we verify that new integration tests pass on the patched PR build, but we don't verify that they actually catch the bug on the broken baseline. This allows false positives and flaky tests to land undetected.

Proposal: Fail-on-Base, Pass-on-Patch

When a PR introduces a bug fix and a new integration test, the CI pipeline should execute two phases:

1. Phase 1 (Fail-on-Base): Run the new Gherkin/Dogtail test against the current production image (bluefin:latest). The test must fail. If it passes, the reproducer is invalid or flaky.
2. Phase 2 (Pass-on-Patch): Run the same test against the newly built PR image. The test must pass.
   Because Bluefin is cloud-native, we don't need to rebuild old code—we can just pull latest from GHCR to act as our control group.

Conceptual Workflow

name: ci/reproducer-check
on:
  pull_request:
    paths: ['tests/features/**']

jobs:
  verify-reproducer:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Phase 1: Must fail on baseline
      - name: Test Stable Base
        id: run_base
        continue-on-error: true
        run: ./scripts/run-vm-test.sh --image ghcr.io/ublue-os/bluefin:latest --test tests/features/new-fix.feature

      - name: Assert Base Failure
        if: steps.run_base.outcome == 'success'
        run: echo "❌ Error: Test passed on baseline. It doesn't reproduce the bug." && exit 1

      # Phase 2: Must pass on patch
      - name: Build PR Image
        run: ./scripts/build-local-image.sh --tag bluefin-pr:${{ github.sha }}

      - name: Test PR Patch
        run: ./scripts/run-vm-test.sh --image bluefin-pr:${{ github.sha }} --test tests/features/new-fix.feature

Why do this?

• Catches Fake Passes: Stops "always-green" tests from bloating the test suite without actually asserting the failure state.
• Prevents Revert Loops: Ironclad protection against complex session, startup, and desktop race conditions.
• Prior Art: This is heavily utilized in systemd-testsuite to validate that regression tests are accurate before a patch is merged downstream.

Discussion Points

1. Flakiness vs Determinism: For race conditions (like the xdg-desktop-portal startup ordering), a GUI test might pass on the base image due to CPU timing. Should we require lower-level systemd/DBus assertions (busctl, systemctl --user show) to make the failure state 100% deterministic?
2. Triggering: Run this automatically via file paths (tests/features/**), or gate it behind a PR label like requires-reproducer?

Thoughts, @castrojo and team? Let's prototype this for our service integration tests.

[castrojo]

I'm focused on the other stuff first but will come back to this, let's collect opinions!