Use of AI in Research processes #10
Replies: 4 comments 4 replies
|
Experiment Log — Research Writing Pipeline: Claims + Narrative Director + Writer What was tested A three-stage AI-assisted pipeline for writing iSRL research reports. The pipeline works as follows: A claims list is a flat document that records every factual statement the report will make, verified against the actual data before any writing begins. If the report will say "896 SKUs were extracted," that number appears in the claims list with the source file and the calculation that produced it. Nothing enters the report that is not in the claims list first. It is a verification instrument — its job is to ensure that what gets written was checked before it was written, not after. A papertable is the equivalent instrument for external citations — a list of every published source the report will reference, confirmed accessible and correctly attributed before writing begins. Empirical findings from the lab's own data cite the claims list. Anything the report asserts that comes from outside the lab — a prior study, a regulatory document, a published framework — cites the papertable. The Narrative Director is an AI agent that reads the verified claims and decides how to tell the story. It produces a framing memo: a short document that locks the argument structure — what the report establishes, in what order, and what it does not cover — before any prose is written. A human reviews and approves this memo. Nothing proceeds to writing without that approval. The Writer is an AI agent that takes the approved framing memo and the claims list and writes the full draft. It is constrained to the approved argument — no new claims, no new citations, no filling gaps with inference. Where something is missing, it flags the gap rather than inventing around it. This pipeline was tested on the ingredient data collection methodology report, using the sampling claims file and the OpenFoodFacts claims file as inputs. The hypothesis was that this structure — verify first, frame second, write third — would produce traceable, auditable research prose without sacrificing readability. What the pipeline got right Traceability holds. Every factual statement in the draft maps to a verified claim. The number is right because it was checked before the Writer touched it, not because the Writer happened to get it right. The class of error where an AI writer adds a plausible-sounding statistic that was never verified does not appear in output produced this way. For a lab publishing open research, that property matters. The framing memo checkpoint works as a human review point. Catching a wrong argument before writing starts is genuinely cheaper than restructuring after a full draft exists. The pipeline enforces that sequence correctly. Where the pipeline failed and why The direct claims-to-prose path does not produce readable research. A claims list records what is true. A research paper communicates what it means and why a reader should engage with it. These are different documents. The pipeline treated the claims list as both, and the Writer produced output that was accurate, complete, and had no reason for a reader to care about it — because the claims list carries no background, no current context, no big picture framing, and no implications. Those are not claims. They are the architecture the claims sit inside. The specific failures, and why the AI behaved that way in each case: 1. No background or motivation section. 2. No implication section. 3. Technical terms used without explanation. 4. The Narrative Director produced data dumps despite explicit instructions. 5. The AI invented definitions for undefined terms rather than flagging them. 6. Context had to be supplied fresh every session. The core tension this pipeline surfaces Traceability and human readability pull in different directions if the pipeline treats them as the same problem. A claims list is a verification instrument. It ensures that what gets written was checked before it was written. It is not a writing input. Giving a writer a claims list and saying "write from this" is like giving an architect a structural inspection report and saying "design the house from this." The inspection report tells you what holds. It does not tell you what to build or who will live in it. A research paper needs both layers: the verified substrate that the claims list provides, and the human context layer — background, big picture, implications, reader orientation. The pipeline as designed conflated the two. The claims list was treated as both the verification instrument and the narrative input. It can only be the former. What a working version of this pipeline needs A stage between pre-flight and the Narrative Director that is not a verification stage and not a writing stage — a context assembly stage. It takes the claims list, the shared project context document, prior published reports in the series, and any background the PI supplies, and produces a structured brief: what the reader needs to know before the methodology, what the methodology establishes, and what follows from it for the reader's domain. The Director works from that brief, not from the claims list alone. The claims list remains what it is — a verification instrument the Auditor checks against — not the input the story is built from. Until that stage exists, the pipeline produces auditable prose. It does not produce research that earns its reader. Session files: |
|
An interesting experiment by Professor Matthew D. Schwartz of Harvard as shared by Chandrasekar Subramanian, Phd https://www.anthropic.com/research/vibe-physics Can AI do theoretical physics? In this guest post, professor of physics Matthew Schwartz decided to find out by supervising Claude through a real research calculation, start to finish, without ever touching a file himself. His account of what happened is shared in the blog post. |
|
Experiment Log — Document Revision Analysis: What Makes a Research Log Succeed on First Draft What was observed Two research synthesis logs were written in the same session on 26 March 2026. One — This log asks: what accounts for the difference? The answer is not accuracy. The sentences that were removed across all three rounds were, in the main, true. The problem was not that they were wrong. The problem is that "is this true?" is a necessary condition for a sentence to belong in a research document. It is not sufficient. The insufficient test A document that applies only the truth test will accumulate noise. Every sentence that survives the test looks like it belongs — it is accurate, it came from the data, it is correctly attributed. But noise that passes the truth test is harder to catch than noise that fails it, because nothing in the document signals that it doesn't belong. The curation-synthesis accumulated this kind of noise across all three revision rounds:
The sufficient test The fortification log applied a different test to every sentence: why is this relevant, and how does it contribute to the governing question? The governing question was stated in the document's first sentence: should there be a fortification checkbox in the IFID form, and if so, what should it hold? Every sentence in that document can be tested against it. The test is not "is this true?" but "does this contribute to the answer, or to documenting the evidence quality behind the answer, or to naming what the question cannot answer?" The difference this makes is concrete. Here is the same fact written two ways: Fact: Variant 4 notation — Written because it's true:
Written for the governing question (what must the form's fields hold?):
The first version is accurate and says nothing. The second version takes the same fact and answers: here is what this means for the form. The regulatory question moves to Open Questions not because it is less true but because it does not contribute to the design answer. The search requirement does. Fact: Class-only colour declarations appeared in the dataset. Written because it's true:
Written for the governing question: Fact: FD&C-style names appear on labels. Written because it's true:
Written for the governing question:
The first version spends its first sentence on an absence, then reaches the finding in the second. Written for the question, the absence is irrelevant — the finding is where FD&C names appear and what that requires from the form. One sentence, complete. Epistemic category is a structural property, not a label The four categories a sentence can belong to are not stylistic choices. Each one makes a different claim on the reader — a different demand on how they should receive and use what they're reading. A verified fact from data asks the reader to accept it. The evidence chain is closed. 88 products declare at least one E-number. That number came from a count on a specific file. The reader can verify it. They do not need to do anything except receive it. An inference from verified fact asks the reader to follow reasoning. The 12 entries classified as synthetic but carrying biological sources in the sources field are not wrong — they reflect a schema constraint, not a curation error. That is an interpretation of verified data, not the data itself. The reader is being asked to track a chain of reasoning. If that chain is sound, they follow it. If not, they push back. An open question asks the reader to hold uncertainty. Whether Variant 4 notation reflects FSSAI prescription or manufacturer convention has not been verified. The reader is not being asked to accept it or follow reasoning from it. They are being told: here is something we do not yet know, and it matters enough to name. A design question asks the reader to make a decision. Whether the form should expose a per-product source qualifier — and what values it should accept — is not answerable from the data. The data surfaces the need. The decision requires domain input the document does not have. These are not interchangeable because the reader cannot hold all four stances simultaneously. When the category is wrong, the reader takes the wrong action. FSSAI Variant 4 in findings. The sentence — "whether this form reflects FSSAI-prescribed declaration style for certain categories — rather than manufacturer convention — has not been verified" — is an open question. It contains the phrase "has not been verified." The category is written into the sentence itself. But it was placed in the findings section, which signals to the reader: this is a verified fact from data. The reader encounters a sentence that says it is unverified, inside a section that says it is verified. One of those signals wins. In a document the reader trusts, the section wins. They carry the claim forward with more confidence than it deserves. Schema assertions in implications. "The current schema has no field for it" — placed in the implications section, which the reader reads as: here is an inference from the findings, grounded in verified data. But it is neither. It is an assertion about a system the document has no visibility into. It does not derive from the data. No chain of reasoning produces it. The correct category is open question at best, and not-a-claim-we-can-make at worst. In implications it reads as a confirmed design gap. The reader may act on it. They would be acting on something that was never verified. Class-only declarations as taxonomy finding. Placed in §2c findings, these strings — The reason category misplacement is hard to catch is that the truth test passes. Every one of these sentences is accurate. The FSSAI observation is correctly described as unverified. The schema observation may be true. The class-only strings did appear in the data. Nothing in the sentence itself reveals the mismatch. The mismatch is between what the sentence is and where it sits — and catching it requires asking not "is this true" but "what is this sentence asking the reader to do, and is that the right thing to ask them to do here?" Why the fortification log didn't have this problem The fortification log was built around its question from the start. The question acted as a filter before writing, not a test applied during revision. Unverified FSSAI notes were flagged explicitly in a table and deferred to Track B — not because anyone ran a revision pass, but because the question made it immediately visible that unverified regulatory claims do not belong in findings. The "Answered / Not answered" section exists because the question demands it: some things the data answered, some it didn't, and the document must be honest about which is which. The curation-synthesis had no governing question. Without one, the filter was "is this true?" — and that filter let through everything. The revision history is the record of applying the sufficient test after the fact, sentence by sentence. The finding A sentence that is true but does not contribute to the governing question is not a finding that belongs in the document. It is a fact that belongs somewhere else — or nowhere. The discipline is not accuracy. It is relevance: why is this here, what does it contribute to the answer, and would the document be worse without it? A document without a governing question cannot apply this test. It has no answer to "contributes to what?" Every true sentence looks like it belongs. The result is not a wrong document — it is a document that cannot be read, because the reader cannot tell what matters. What Lalitha A R did, what Claude did Lalitha A R:
Claude (Anthropic, Sonnet 4.6):
The finding is Lalitha's. The instantiation of it — the examples, the categories, the argument structure — is Claude's drafting from Lalitha's correction record. Session files: |
|
Adding some more observations that came as someone asked it (in the context of industrial research) Thank you I haven't tested Claude Science extensively. Incase of general Claude, here are some things that work really well,
If I remember more, I will update here |
Uh oh!
There was an error while loading. Please reload this page.
I use AI in my research workflow and wanted to share what has worked, what has not, and open this up for others to share their own processes.
The core problem with AI-assisted literature review
Finding 10 relevant papers in minutes looks efficient until you are manually verifying each one later and realising half of them do not say what the summary claimed. The speed is borrowed time, not saved time.
So we have been experimenting with a claims-first structure: build the claims list, verify citations and the exact lines that support each claim, and identify the type of support — before writing a single argument. The full structure is in STANDARDS.md: https://github.com/isrl-research/sandbox-research/blob/main/STANDARDS.md
The papertable structure was inspired by SciSpace, but the key difference is that verification happens pre-writing rather than during revision. When you discover a major supporting point is wrong after you have built an argument around it, you are not fixing a citation — you are rewriting the logic. Doing it before means you only write once.
Tools and what each one is actually for
I used to use Perplexity specifically for finding obscure PDFs buried in websites — it was genuinely good at that. Recent unannounced changes have made it less reliable and I have limited my use. If anyone has found a solid replacement for that specific task, I would genuinely like to know.
Gemini's job in my workflow is to poke holes. I give it a framework or argument and its only task is to find edge cases where it breaks. This does two things: it strengthens the framework where the edge case is valid, and it helps me build an exhaustive limitations section where it is not. It has no role in writing — only in stress-testing.
Once claims and papertable are ready and all bib citations are confirmed, I write a rough outline manually. It is not meant to be grammatically correct — it is just the narrative flow I want to follow, in my own words, so I know exactly what I am trying to convey before anyone else touches it.
At that point I use Claude as a co-author who writes well but has no domain knowledge. The constraints are explicit:
Claude Code for research — what actually works
A few things that have been genuinely useful:
Running tests and finding statistics quickly from a dataset without having to write the script from scratch each time.
Running scripts for specific research questions — but with explicit guardrails built into the prompt: what is the ground source of truth for this script, what counts as an anomaly, when should it flag one, and how should it report it. Leaving these undefined means the script makes its own decisions about what matters, which defeats the purpose.
A logger habit: once a task is done and I move to a new prompt, I ask it to log what it just did — the ask, the process, the dead ends, and how each was resolved. This means the session is documented even when you are moving fast and not stopping to write things down yourself.
Claim verification scripts: if I ask it to find implications or claims from a dataset, it also generates a Python script that reads directly from the source and verifies each claim. This prevents the specific failure mode where it saw one number and reported another, or correctly read a number but misinterpreted what it meant.
Anyone who wants to share their own process is welcome here. A few ground rules for this thread:
Critique is always about the tool or the method, never directed at the person using it. For example: "Perplexity's citation accuracy has been unreliable in my experience" is about the tool. "You shouldn't use Perplexity" directed at someone who does is not the kind of conversation this thread is for.
Respect other workflows even when you would not use them yourself. Everyone is working with different constraints, different domains, and different stages of figuring this out. Share what works for you, ask genuine questions, and leave room for the answer to be different from yours.
All reactions