Reduce Reliance on Test-Driven Development

[GoranHalvarsson]

Hey friends,

I’d like to raise a concern about the current use of test-driven development in the spec kit. Right now, the kit enforces a pattern where tests are expected to fail first and then be corrected. This process can make development feel unnecessarily slow and cumbersome, especially when working with an LLM that can overdo steps(And they all do that). I believe a more streamlined approach would be beneficial to avoid unnecessary complexity.

*I let an AI write the text for me, because I had tons of swear words ;-)

Finally I love Specify. You guys have done a beautiful job

[nord73]

I hear the concern about TDD feeling cumbersome, especially with LLMs piling on steps. Still, fail-first TDD is clutch for catching bugs early and keeping complex projects manageable—without it, I’ve seen codebases get messy fast. Streamlining’s tempting, but what specific validation methods do you think could match TDD’s reliability without the slowdown?

[essenbee2]

As far as I can see, the user is in control of the contents of constitution.md and the other artefacts. When you review them prior to hitting /implement, edit them to suit your needs.

[Ian1971]

I'm not convinced that TDD delivers the same value when used by an LLM that it does when used by a human. The real benefit of TDD is that it makes the design of your code better because it has to be testable. But when I use an LLM to do TDD I rarely see it taking that in to account. It creates the tests first, and then satisfies them, but it never decides to make the mocking less complicated by refactoring. However, the LLM is generally good at getting a class that follow SOLID principals off the bat (if you tell it to). In that case adding the tests after seems just as valid, as long as they are comprehensive.

[albahrani]

Of cause the LLM does not consider it evil changing test code just make the incorrect implementation passing the test (at least if you do not tell him it is evil and you want to be asked before evil things are done).
And also an LLM does write tests for the features firsts, then implements the code and improves the tests until the tests run (no more runtime errors), then reports back it is done because all tests are "running" with half of the tests still failing.
These are just two of the scenarios I had during my work with spec-kit where I got really frustrated. Then I thought about kids sometime searching the fastest, obviously not intended, path out of chores and acknowledged that it was my wrong for having some impicit excpectations. By the way: I often tell the LLM it has to get my approval about test source changes, only to have the LLM ask me if it should just stub the implementation to make the test work ;-) .

However, especially having coding LLMs it is important to have good tests to avoid the LLM causing unwanted side effects. Especially usefull the more taks or features are already implemented and working. TDD is one way to make the LLM realize these tests as soon as possible. The consequence is you have to review any changes made to the tests afterwards to figure out if they are required changes or unwanted/illegal shortcuts.

Personally ,I will stick with TDD until I have found a better aproach. The important thing is that the approach needs to be well-defined, documented and within the knowledge (cut-off date) of the LLM I use, so I can simply reference it.