Property-based testing

[mofsesam]

Property-based testing like with https://hypothesis.readthedocs.io/en/latest/ can be very powerful, but good properties to test can be hard to come up with. AI assistance on finding good properties to test and writhing these tests would be welcome addition to the spec-kit process.

[localden]

@mofsesam - can you elaborate on what you envision the end-to-end developer experience to be here?

[mofsesam]

This relates to #169
If passing tests define when you have a working solution, than good property based tests will boost your confidence that you also test for unknown-unknowns.

For "end-to-end developer experience to be here", I think tests should run through all the steps of spec-kit.
If intent is changed in Specify,
Plan for testing needs to change, new Tasks must be made with new tests, Implemented code must pass old/modified and new tests.

[adam-paterson]

This could tie in with these discussions:

• Proposal: Add benchmarking via TerminalBench to spec-kit #159
• Proposal: GEPA prompt optimization and template generation in spec-kit #160

This will help us guardrail against:

• Temperature, sampling, and upstream model drift make identical prompts yield different outputs.
• Many tasks don’t have a single “correct” answer; proxy metrics can miss quality regressions.
• Real-world inputs evolve; a prompt that passes today may fail tomorrow.

flowchart TD
    A["Prompt template changed"] --> B["Property-based tests (Hypothesis)"]
    B --> C["Run TerminalBench benchmarks"]
    C --> D{"Degradation below threshold?"}
    D -- "Yes" --> E["GEPA Prompt Optimiser"]
    E --> B
    D -- "No" --> F["Release"]

Loading