AI-generated shovels or second-order slop? | Protocols Made Fun

[giscus[bot]]

AI-generated shovels or second-order slop? | Protocols Made Fun

Author: Igor Konnov

https://protocols-made-fun.com/llms/testing/2026/02/12/second-order-slop.html

[ericsson49]

LLMs are great assistants in shovel building :). Modern LLMs (Claude 4.5, ChatGPT 5.x) are so good that I stopped calling their output slop.

The output can be sloppy and it's overly verbose, which is a problem. However, if chunks are of manageable size then reviewing and rewriting is not a big deal now.
And feels like more productive :). Both because LLMs are very verbose, but also because they are much faster at re-implementing stuff, which has been already built.

My philosophy is that custom shovels matter. I'd rather use a more traditional "meta-programming" instead :). LLMs add a new dimension in a form of inexact matching/generation, which is often handy.

I'm still experimenting (with LLM-assisted shovel building), but already very impressed :).
E.g. I've asked it to implement a sub-typing constraint simplifier. With guidance, iteration, manual reviewing and fixing, the result is of better quality then I'd do alone (I won't allocate so much time on documentation and unit tests). And took much less time (an hour vs weeks), however, it's because I've re-implementing it.
Wrt new stuff, it likely won't be very impressive. However, LLMs produce a very reasonable starting point quickly.

I think that for custom shovels, code quality is not critical, mostly. If it sucks, it tends to reveal itself during later stages (testing, type-checking, etc). E.g. if my custom type inference engine made a mistake, then resulting code won't pass type-checking.

The shovel quality can be critical. E.g. a custom model checker will be highly likely unsound. It can find bugs, but cannot really guarantee absence of them. I'd go for certified stuff then (checkable by external tools).