Discussion How do we work with AI?

[rozyczko]

We started this discussion recently, so let's formalize it with everyone's opinion.
We DO need a consensus, since we review each other's code and it will become more and more difficult when the bulk of it comes from the non-deterministic LLM engines.

Let's assume we have three ways of writing code with LLMs:

1. Vibecoding for the win
   Humans create the list of requirements.
   LLMs create the design, the code, the tests and the docs.
   Humans only interact on the receiving end by scrutinizing the ready product for functionality and functional tests for scientific correctness.
   Unit tests and code itself are assumed to be internally consistent.

2. Mixture of LLMs and handcrafted code (PR pipeline)
   Humans create the list of requirements
   LLMs create the design.
   Humans review the design and sign it off
   LLMs write the code, unit tests and docs
   (other) LLMs review the code, unit tests and docs
   Humans review LLM's reviewed/corrected code and docs
   Humans implement functional tests for scientific correctness

3. Handcrafted code. As done back in the stone age of 2023.

These can be summarised in a simple table.

Stage	Vibecoding	Mixture	Handcrafting
Requirements	human	human	human
Plan	LLM	LLM	human
Plan review	LLM	both	human
Implementation	LLM	LLM	human
Tests	LLM	split by test type	human
Docs	LLM	LLM	human
Code review	LLM	both	human
Sign-off	human	human	human

The questions to be answered:

• Which of those stages is the most important? Implementation? Sign-off?
• Do we need the unit tests to be reviewed? After all, they just mirror the code itself
• Can we trust one, high-level LLM for all the functions or should we split the review LLMs from implementation LLMs?
• How do we deal with functional tests for non-scientific features?

Maybe we should be assigning risks for each row, in terms of the LLM reliance?

[henrikjacobsenfys]

I think 2 is the most reasonable way forward. I've been doing this for spectroscopy-lib and dynamics-app (except I haven't reached the human review stage) and it is so much faster and better than handcrafting that I don't think we can justify not doing it. Based on my progress on dynamics-lib, LLMs have in a week produced what would take me more than a year.

Perhaps this is what you mean by sign-off, but just wanted to clarify: I think it's important that the human who used the LLM should take full responsibility for the code they produced. That should also encourage us to scrutinize our own LLM-generatede code.

As for the questions:

1. I'm not sure I understand the question. Important in what sense?
2. I'd say yes, a human should have read all the code. It's tedious, but it builds trust. (I don't know when/how I'll get to that, spectroscopy-lib has 1700 tests and counting)
3. Good idea to use another LLM for review. We can also consider LLM/human collaboration for review
4. I'm unsure about this one as well, could you give an example?

Some thoughts from my last few weeks of vibes: it requires quite a bit of expert guidance to produce good results, so we are not (yet) obsolete. The templates and asking LLMs to use similar structures as existing libraries/apps helps a lot. For spectroscopy-lib, I have the luxury that I'm essentially reproducing Horace functionality, so I can test the code extensively against well-tested human-written code in a different language. Such tests will become ever more important, also so that our users will trust our code.

An adjacent but important thought: right now we each own one product. This needs to change as soon as possible, because right now, reviewing someone else's code can feel bad, because it means taking away time from "our own" work. In contrast, every time someone has written code for corelib that I use, or when Andrew has done work on dynamics-lib/ helped me with templates, I've been very motivated to review it, since I need it and it helps me.

[rozyczko]

(I'm not actively proposing this "black box" solution; I'm merely being curious about how those arguments can be addressed)

[henrikjacobsenfys]

True that we didn't know the future of compilers, but they didn't start out being sold at way below their price, and anyone could in principle make their own. I believe it's even a standard exercise in some programming courses (not certain about that though). On the other hand, we don't have the capacity to run/develop LLMs ourselves. We also KNOW that the current LLM pricing model is not sustainable, and that the real cost is much much higher than what we pay. So it's not a "will LLMs continue to exist?" but "will we continue to be able to afford LLMs while keeping our jobs?".

I see the LLMs as a train we're hitching a ride on, but which can at any point stop, crash, or kick us off of. In that analogy, we still need to be able to navigate on foot when that happens. Until then, let's jump on the train and get as far as we can. Of course, that's a position that can and will be reassessed as time passes.

It's important to understand the code if we still want to (be able to) write code ourselves as well. That opens a more philosophical question: what does it mean to be a software developer? If we no longer look at and write code, we could also switch to Rust for a major performance boost. It'd probably take less than a month for an LLM to convert all our libraries to Rust and pass all the current tests.

[AndrewSazonov]

If LLM prices increase dramatically, we can always fall back to more manual development. That doesn't mean we should stick to a less efficient workflow today just because that might happen in the future.

Regarding Rust, I actually think this is a good example of the shift. If LLMs become good enough to migrate a large codebase while preserving its behaviour, then our main job is no longer translating ideas into syntax. Instead, it becomes defining requirements, validating scientific correctness, making architectural decisions, and ensuring maintainability. The programming language itself becomes much less important than the specification and verification process.

I also think understanding the architecture, APIs, data flow, and design decisions is much more valuable than remembering implementation details. In that sense, the role of a software developer shifts from writing code to engineering systems. Code is still important, but it becomes one of several implementation artifacts rather than the primary product.

[damskii9992]

If LLM prices increase dramatically, we can always fall back to more manual development.

That is definitely not true. If we implement the full black-box methodology, returning to manual development will be nearly impossible, depending on how long we do it for. It's akin to taking over a decade-long academic software project and suddenly being told to maintain/improve it. If you haven't written/read any of the code yourself, it will take you an extremely long time to read and understand the code to the point where you can make improvements.
In most realistic cases I've seen, people simply choose to start all over and trash all the old code in such cases. And with AI's ability to churn out code, and its tendency to make code overly complex, I can only imagine it is gonna be even worse in such a case.

And also don't forget that you get rusty in skills that you don't use. If you haven't coded for several years, you will have to re-learn python before even attempting to read and understand code. And new developers won't even know Python. Imagine having to learn coding first . . .

[AndrewSazonov]

Well, I think it depends a lot on how you are going to do it. If LLM-based development is done under clear constraints, with tests, and in a language you are comfortable with, then I don’t think switching back to working directly with the code should be such a big problem. At least not much more difficult than joining a new team that has already been developing a project for several years.

If the project is large, you are not the only developer, and you also contribute to several other projects, you cannot keep all the codebases in your head anyway. You already need to understand the relevant parts when you start working on a particular problem.

But I agree that there are cases where this can become a serious problem. If you ask an LLM to write something in a language you don’t know at all, and later you need to maintain and develop it yourself, then yes, this is a problem. The same is true if the LLM produces overcomplicated and unreadable code. This is why I think constraints, tests, code quality requirements, and keeping the implementation in technologies the team understands are still important even with a black-box approach.

[rozyczko]

FWIW: My personal approach is a complex version of 2 - Mixture of LLM and human.

I am trying to address the non-deterministic nature of LLM results by having them converge on both the design document and code implemented.

I start with writing requirements and asking Fable to create a detailed design document (as an md file)
Then, I ask another LLM (DeepSeek, Qwen, Grok, GPT...) to review it and find problems.
Next step - ask Fable again to scrutinize the review and update the main md file.
Redo the review step until both LLMs agree that all the issues are either solved or non-existent.

Only then will I start implementation - based on the design doc I write the general structure of the classes/modules/methods, functionality and layout. Then, ask Fable to fill this structure with docstrings, comments, typehints and address edge cases etc.
So the implementation step is also mixed.

The docs/unit tests are entirely LLM written.

The implementation review follows the design doc review, with multiple stages until convergence.

(Only then, I hand it over to Christian, who, after 120 comments and a complete rewrite agrees to merge it... 😆 )

The following chart shows what would the fully black-box project pipeline be:

flowchart TD
    A[HUMAN: Write requirements] --> B[Fable: draft detailed design `doc.md`]
    B --> C[Reviewer LLM: find problems<br/>DeepSeek / Qwen / Grok / GPT]
    C --> D[Fable: scrutinize review,<br/>update design doc]
    D --> E{Both LLMs agree:<br/>all issues solved<br/>or non-existent?}
    E -- no --> C
    E -- yes --> F[Fable: write structure —<br/>classes, modules, methods,<br/>functionality, layout]
    F --> G[Fable: fill structure —<br/>docstrings, comments,<br/>typehints, edge cases]
    G --> H[LLM: write unit tests + docs]
    H --> I[Reviewer LLM: review implementation]
    I --> J[Fable: scrutinize review,<br/>update code]
    J --> K{Convergence?}
    K -- no --> I
    K -- yes --> L[HUMAN: accept and consider done]

Loading

[AndrewSazonov]

I agree that a human must take responsibility for the final result. However, I do not think this has to mean that human reviews all the final code and tests line by line.

The scale and speed of development have changed. In this discussion, we already have an example where LLMs produced in one week what would previously have taken more than a year. A complete human review may still work for a small project, but it will not scale to large projects or when one person needs to contribute to several projects growing in parallel. I can no longer keep track of all the diffraction-related code I wrote myself, and the project is still far from its planned scope. Human reading speed will become the main bottleneck, while reading everything still does not guarantee that every problem will be found.

I am in favour of the following model:

• A human writes the requirements, constraints, API specification, and acceptance criteria.
• Two independent LLMs iterate on the design and implementation plan until the important issues are resolved.
• A separate LLM writes tests and verification examples based on the specification and reference results, rather than copying the implementation.
• One LLM implements the plan, and another reviews the implementation, again through several iterations.
• The human checks scientific correctness, the public API, important risks, and the final user experience, while directly reviewing critical parts only when needed.

This is not about blindly trusting a black box. It means moving human control to a level that can scale: specifications, architecture, independent verification, and final sign-off. Reading selected parts of the code is still useful for learning, debugging, and checking critical areas, but it should not be the only mandatory quality check for every LLM-generated line of code.

We should also not keep an inefficient process today simply because LLMs may become unavailable or too expensive in the future. We do not organise our work around the possibility that computers or compilers may disappear. LLMs are becoming a normal part of software development, and our role is gradually changing from writing and reading every line of code to writing precise specifications and verifying that the result satisfies them.

[rozyczko]

I am in favour of the following model:

This is almost exactly what I diagrammed above, except for the initial stage of the implementation.
However, if we assume code review convergence, this could potentially also be removed from the human list of responsibilities.
I do it because I prefer a certain class/module structure, which LLM will not necessarily give me.
That said, those preferences can be added explicitly to the main .claude harness, rendering the manual part redundant.

moving human control to a level that can scale: specifications, architecture, independent verification, and final sign-off

Precisely that. Moving the abstraction level from writing the code to writing the specs.

[rozyczko]

After thinking more about potential ramification, I don't think we can justify choosing a single methodology.

For our technique-specific libs/apps the fully AI-autonomous solution might be useful in terms of huge productivity boost and well defined requirement list, which leads itself to a more or less "black box" solution.

For modules which we all use and rely on, like core, utils, crystallography or EasyApp - understanding the code might be very useful, since we are constantly tinkering with it, improving, refactoring, modifying etc. This would mean we might need to rely on well-devised partial-AI pipeline AT THE MOST.

[seventil]

• Complexity growth
  In my experience LLMs have a tendency to treat variants in their context as requirements. Contrary to that, a human could potentially rearrange code to accomodate for new features to ensure that codebase remains well organized.
  That means, without handholding LLMs seem to complicate the codebase, requiring more and more context to maintain, which leads to reduced readability for humans and increased prices for LLMs in the future.

• Idea pitfall and iterative improvement
  With handmade approach you automatically do the iterations on code specification and interfaces, but with LLMs it's too easy to have your head wrapped around a certain way of thinking/designing. But iterating on the design, on the APIs, on the requirements and specs is what makes the code well-structured, well-organized and therefore maintainable.
  AI is extremely good at parsing large amounts of data, so using it to assist with design, implementation or review iterations is a nobrainer, but I gravitate towards using it to show blindspots, rather than handing off the iteration process entirely.

• Black-box approach
  Specs can exist on different levels of abstraction, so when we discuss whether or not LLMs allow us to shift from writing code to writing specs, we need to remember that to a degree, code is the specs. Some specs are important, others are variant implementation detail. We should agree on where do the "black-box" borders begin/end. We can review, rearrange existing code due to our ability to understand what is an actual requirement and what is a variant / derivative outcome. This ability is only 'innate' to humans, because we do not formalize it anywhere (except ADR, but that's too high level). If we come up with a way to formally define black-box / interface specs, then we can agree on not carrying if underlying black-box code implemented by LLM is ugly or not.

• Subject to review
  With handmade approach, specs are written down as code, so during review every row is subject to criticism. To review the code written with LLM-assistance, I would like to what exactly was human designed and what were the higher level specifications. So how do we write code specification? MD files? If they're written/modified by LLM, as I mentioned, they tend to write variant implementation details/outcomes there and then treat it as requirements.

We cannot assume, that we will have the same level of access to LLMs in the future (or at least to good models at good prices). So we need to be cautious with its outputs: we need to handhold LLMs during design and review iterations, somehow find a way to stay openminded and outside of LLMs suggestions during these iterations and maybe formalize/review our code specification?

[damskii9992]

I think having only 1 way of mixing LLM with human work is a harsh over-simplification. I am personally not in favour of your proposed mixed way of coding, too much LLM for me, but I still want to use LLMs where I believe it is beneficial.

How I think LLM's should be used in code:

• To do a first round of PR reviews (or first and second, or more)
• • I still believe any developer should ALWAYS read their own code too, after the LLMs and before passing it to review by another human
• To write up DRAFTS of tests/docs/small isolated code snippets
• • Everything should be looked through by the developer, LLMs make mistakes, many of them. And many of them are "hidden" mistakes that won't get discovered by other LLMs.
• To write up drafts for notebooks, small scripts or other ISOLATED pieces of software.

I think this is the only way to use LLMs to write code which can still be maintained by humans. And until LLMs become deterministic (they never will be) I think we NEED to write code which CAN be maintained by humans.

[rozyczko]

From https://blog.rust-lang.org/inside-rust/2026/08/05/rust-langrust-is-adopting-an-llm-policy/ as posted in the scipp slack and forwarded by Torben:

It's fine to use LLMs to answer questions, analyze, distill, refine, check, suggest, review. But not to create.

Which on the surface sounds fine, but what does create really mean here?
If I write code, have LLM review it and it then suggests large scope changes/rewrite- is the result LLM created if I click "OK"?

If I create a top level module/class/method structure and ask LLM to fill it with code - is this creation or merely suggestion?

[damskii9992]

This link is actually a very good read, and we should all read it in our team, I think. Especially their actual rules.

No one except the author is required to read LLM output unless they choose to: LLM output isn't allowed in public docs, PR descriptions, or Github comments unless it's clearly marked; reviewers aren't required to look at LLM PRs if they don't want to.

I like this one.

LLM reviews cannot substitute for human review or self-review.

And this one.

And to your comment about what it means to "create":

Pre-arranged, non-critical, high-quality, well-tested, and well-reviewed code changes that are originally created by an LLM are allowed, with disclosure.

LLMs must not generate soundness-critical changes unless the author is already a domain expert, and even then it's strongly discouraged.

[damskii9992]

The full list of what is allowed, is here:
https://forge.rust-lang.org/policies/llm-usage.html#-allowed