WheelsTest: spec files that skip describe() or error during load fail silently in the runner

[bpamiri]

Symptom

When a spec file under tests/specs/ either:

1. has it() calls at the top of run() without a describe() wrapper, OR
2. errors on load (e.g., the spec references application.wo.X from a global include that doesn't exist yet)

…the test runner reports nothing about it. The summary line shows:

0 passed

(for case 1, when running only the malformed spec via --spec=path/to/X)

or:

48 passed, 0 failed, -1 error(s) (9.13s)

(for case 2, when running alongside passing specs — note the -1 error count).

No filename, no error message, no indication that a spec file failed to load.

Reproduction

Case 1 — it() at top of run(), no describe()

component extends="wheels.WheelsTest" {
    function run() {
        it("does a thing", () => {
            expect(true).toBeTrue();
        });
    }
}

Running wheels test --spec=path/to/that/file reports 0 passed. Wrapping the it() in describe("X", () => { ... }) makes it run normally. The runner gives no hint that the bare it() form is invalid.

Case 2 — Spec errors during load

component extends="wheels.WheelsTest" {
    function beforeAll() {
        variables.helper = application.wo.functionDoesNotExist;
    }

    function run() {
        describe("foo", () => {
            it("does a thing", () => {
                expect(true).toBeTrue();
            });
        });
    }
}

If the spec file throws during load (here application.wo.functionDoesNotExist is undefined), the runner reports -1 error(s) in the summary count rather than printing the actual exception and the offending file name.

Why it matters

Both shapes are easy to land accidentally during iteration (especially the application.wo.X case during TDD when you're writing the spec before the helper). The silent failure costs minutes of "wait, why isn't my new test running?" debugging per occurrence, and the -1 error(s) count is itself a clue that something is wrong but unhelpful as a starting point for diagnosis.

Suggestion

For case 1: at minimum, log a warning when a spec file's run() contains it() calls that didn't end up registered. Ideally, register them under an auto-generated top-level describe (e.g. <filename>) so they DO run, since that's likely what the author intended.

For case 2: when a spec file throws during load, print the file path + the exception message + a short stack frame, and count it as an error (positive, with file context) rather than a -1.

Discovered during

Phase 1 of an internal RBAC portal build on Wheels 4.0.2 / Lucee 7 — full context in https://github.com/paiindustries/titan/pull/3337 (private, but happy to extract specific snippets if useful).

[wheels-bot]

Wheels Bot — Triage

Classified as bug.

Suspected layer

cli — see .ai/wheels/testing/ for the runner/BDD reference (the layer list doesn't have a dedicated testing entry; the user-visible summary line and exit aggregation live in cli/lucli/services/TestRunner.cfc, while the BDD describe/it DSL itself lives in vendor/wheels/WheelsTest.cfc and vendor/wheels/wheelstest/).

Fix sketch

Two adjacent defects:

• Case 1 (bare it() at top of run() is silently dropped): the BDD DSL in vendor/wheels/WheelsTest.cfc registers it() callbacks against a current-describe context. When run() invokes it() without an enclosing describe(...), the call has no parent to attach to and gets discarded. Fix shape: either (a) throw a clear Wheels.Test.OrphanIt exception naming the file when it() is called outside a describe, or (b) auto-wrap orphan it() calls under a synthetic top-level describe named after the spec file. The suggestion box in the issue argues for (b); the safer mechanical fix is (a). Pick one — this is a small design call propose-fix should make explicit.

• Case 2 (-1 error(s) for load-time exceptions): the summary aggregation in cli/lucli/services/TestRunner.cfc (and/or the harness in vendor/wheels/tests/TestRunner.cfc for the framework suite) is computing errors = total - pass - fail after a load-time throw left total lower than pass + fail. Fix shape: catch per-spec load exceptions, record {file, message, stackFrame} against an explicit errors list, and surface filename + message in the summary instead of relying on subtraction.

Confidence: medium

Two distinct defects in one report and Case 1 has a design call (warn vs. auto-wrap); spec runner code paths are exercised on every engine so any DSL change wants a cross-engine sanity check before landing.

<!-- wheels-bot:triage:2829 -->
<!-- wheels-bot:triage-class:bug -->
<!-- wheels-bot:triage-confidence:medium -->

[wheels-bot]

Wheels Bot — Fix proposed

Draft PR: #2831

Spec: vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc — failing → passing (via fixtures OrphanItFixture.cfc and BeforeAllErrorFixture.cfc under vendor/wheels/tests/resources/wheelstest/)
Implementation: vendor/wheels/wheelstest/system/runners/BDDRunner.cfc — bundle stats start before target.run(), the outer try now wraps the suite-descriptor pass and the bundle execution together, and the catch records a positive error count.

A human review is required before merge. Reviewer A and Reviewer B will weigh in shortly. Doc updates (MDX guides, .ai/wheels/, CLAUDE.md) are handled separately by bot-update-docs.yml.