Can we govern AI without measuring it?

[WesselBraakman]

Describe the enquiry

Initial idea

Over the past two years I've been working together with colleagues in Norsk AI-Etikkforening (NAIE) on NoBBQ, a localized benchmark for measuring social bias in large language models.

When we started, the idea was actually quite simple: could we localize BBQ to Norwegian?
As we worked on it, that question slowly changed into something much bigger.

The more time I spent looking at AI evaluation, the more I started wondering whether we're measuring the right things.

Hard vs soft skills for AI systems

Don't get me wrong. There are already a lot of excellent benchmarks out there, and many of them are run continuously as new models are released. We have great ways of measuring reasoning, coding ability, mathematics, factual knowledge, speed and all sorts of other capabilities.

But what about the "softer" side of AI?
Things like bias. Fairness. How models behave towards different groups of people.

Those are all things we care about when we talk about AI governance. Yet many of the benchmarks in this space are developed as part of a research project, used to answer a specific research question, published in a paper, and then naturally receive less attention while models continue to evolve.

Coming from a quality assurance background, that feels a little strange to me.
We would never regression test an application once and assume it will behave the same forever.

Foundation models don't stand still either..
. They receive updates
. New versions are released
. Their behavior changes over time
. New capabilities are introduced

Shouldn't at least some governance-related benchmarks be treated more like regression test suites that we rerun over time?

To me, it comes down to a simple idea: we cannot govern what we cannot measure.

If we want meaningful AI governance, we need governance-related measurements that evolve alongside the systems we're trying to govern.

Localization

One of the first things we discovered was that localization is much harder than translation.

Most benchmark development has understandably happened in English and often reflects an American social context. But Norwegians don't interact with AI as Americans do. The same is true for users in other parts of the world. Even within the English language, users in Australia are not always helped by answers that implicitly assume an American legal, cultural or societal context.

Simply translating a benchmark isn't enough. Occupations, institutions, stereotypes, names and everyday situations don't always carry the same meaning between countries. Preserving the purpose of a benchmark often requires rewriting or even replacing questions entirely.

Reusability & expanding

That's the direction NoBBQ has gradually taken.

Rather than building something that is only useful once, we're trying to build a benchmark that can evolve alongside the models it evaluates.

NoBBQ started as a Norwegian project, but we're now expanding the methodology to the Netherlands, and we already have concrete plans for Denmark and Ireland.

Our goal isn't to build four benchmarks. It's to develop a repeatable methodology that others can use to build and maintain their own localized benchmarks. Over time, that also opens up the possibility of comparing results across countries while respecting each country's own language and cultural context.

Question & discussion

I'm curious how others look at this.

Should governance-related AI benchmarks primarily remain research outputs?
Or should we start treating them as continuously evolving regression suites for measuring how AI systems change over time?

AI compliance pyramid

See below an "AI Compliance" pyramid I sketched up not too long ago. The purple layer is what I think is currently missing.

AI fairness and bias certifications already exist, but what exactly are they measuring?
If we don't have continuously maintained, localized benchmarks, what evidence are those certifications ultimately based on?

Collaboration

I'd love to hear other perspectives on this.

Am I looking at this the wrong way? Are there existing governance-related benchmarks that are already continuously maintained? And if you've worked on localization or AI evaluation yourself, what challenges did you run into?