SAST 2026: Topics to talk about on testing at an AI Native startup

[eddbuildsthings]

Hey!

I am due to give a talk at SAST 2026 on how we test at Legora, along with our SDLC & quality processes.

We're an AI startup in Stockholm and I was hired as the only QA within the org (before all our titles changed to Member of Technical Staff!). I've spent this year building out traditional testing techniques and seeing where they buckle, along with learning about how we can make the software safe for users when agents are everywhere in our working processes.

I would love to get some ideas on what topics people would like to hear about AI native quality!

❤️

[anupamck]

Dear @eddbuildsthings – thank you for initiating this interesting discussion!

You've stated that we'll have agents everywhere in our working processes. I would go further and say that we'll have agentic users and co-workers. I would love to hear a talk on how we should think about quality assurance if agents were the users. How should we configure our quality gates differently, if an agent were the developer.

Another talk I would love to listen to on the topic is how we can design different layers of evals in a manner similar to conventional software tests, where some tests are closer to the source of the problem and faster to execute, whereas others evaluate the AI-system as a whole, but are slower and more resource heavy to execute.

A persistant challenge with AI-native system is non-determinism and inconsistency, which makes a single pass or fail result unreliable. We would need to repeat an evaluation several times and calculate a pass %. A talk that will go into the details of how to do this (like @sharovatov's post here) would be very interesting.

Could you talk a little more about the work you do at Legora? Are these the types of challenges you have tackled there? I would love for your talk to explain a concept with a practical grounding on your experience.