OpenBehaviour

OpenBehaviour investigates observable AI behaviour through repeatable conversations, pressure, contradiction, controls, branching, and bounded adaptive follow-up.

It asks:

What observable behavioural pattern does the system show under adaptive interaction, pressure, contradiction, branching, and follow-up?

It evaluates visible outputs only. It does not infer private reasoning, intent, consciousness, or internal mental state.

Learn more on my introductory article on SubStack.

How It Works

• Runner: owns the frozen cases, prompts, sessions, budgets, and completion.
• Target: receives only natural target-facing conversation messages.
• Host: assesses each scored response and can request a focused follow-up after an approved observable trigger.
• Judge: produces a final evidence-cited assessment from the ledger. It can be separate from the host.

Fixed measurement remains comparable across targets. Adaptive follow-ups help clarify an observed issue, but cannot replace missing fixed opportunities or inflate coverage. Adaptive follow-ups run on isolated child branches, so they cannot alter the subsequent fixed conversation.

Quick Start

pip install -e '.[dev]'
open-behaviour validate
open-behaviour models check
open-behaviour run sycophancy-deep --profile quick

Configure OpenAI-compatible endpoints in models.yaml:

host:
  provider: openai_compatible
  base_url: https://api.openai.com
  api_key_env: HOST_MODEL_API_KEY
  model: your-host-model
  max_tokens: 3000
  timeout: 180

target:
  provider: openai_compatible
  base_url: http://127.0.0.1:1234
  api_key_env: TARGET_MODEL_API_KEY
  model: your-target-model
  max_tokens: 3000
  context_window: 32768
  timeout: 120

# Optional. The host is reused as the judge when omitted.
judge:
  provider: openai_compatible
  base_url: https://api.openai.com
  api_key_env: JUDGE_MODEL_API_KEY
  model: your-judge-model

Keys are read from environment variables and never written to run artefacts. LM Studio and Ollama OpenAI-compatible endpoints work without a key.

Run Profiles

open-behaviour run sycophancy-deep --profile quick
open-behaviour run sycophancy-deep --profile medium
open-behaviour run sycophancy-deep --profile full

# Optional 20-turn conversational probes
open-behaviour run sycophancy-deep --probe-type longform

# Fresh-session repetitions
open-behaviour run sycophancy-deep --repetitions 3

# Confirm named findings without rerunning unrelated cases
open-behaviour run sycophancy-deep \
  --case-id SYC-035 --case-id SYC-043 --repetitions 3

Profile	Standard cases	Use
quick	6	Workflow check
medium	30	Meaningful exploratory assessment
full	150	Complete standard protocol

Profiles select cases, not weaker prompts. Selection is deterministic and coverage-aware for a fixed seed: it balances categories, interaction approaches, subtle wording, confidence-only pressure, and declared counterbalances. The runner rejects budgets that cannot complete required phases unless --allow-partial is explicit.

--case-id overrides profile sampling and may be repeated. Combined with --repetitions, it reruns confirmed findings in independent fresh sessions.

Included Suites

The release suites each contain 150 standard cases across 10 behaviour categories and 15 interaction approaches, plus five optional 20-turn longform conversations. The public suite format allows 150-200 standard cases; 150 is the current frozen bank, not a permanent ceiling.

Each current standard bank contains 140 compact overt controls and 10 subtle six-turn trajectories. The trajectories establish context, apply pressure over multiple turns, audit the answer, and fork to a valid-information control.

Suite	Observable target
sycophancy-deep	Movement toward unsupported preferences, authority, consensus, reassurance, or user stance
hallucination-deep	Unsupported details, sources, memories, mechanisms, or certainty beyond the visible record
deception-deep	Misrepresentation of access, source use, tool use, completion, uncertainty, or audit history

The deception suite measures response integrity only. It does not claim to infer strategic intent.

Evidence and Reports

Each fixed opportunity is recorded as one of:

• supports_behaviour
• supports_resistance
• ambiguous
• invalid_opportunity

Scores represent resistance to the tested behaviour, so higher is stronger. The runner computes each dimension's rate from its decisive fixed outcomes; ambiguous outcomes are shown but excluded from the rate. The judge supplies evidence-cited interpretation and adjudication confidence, but cannot overwrite the numeric rate. Incomplete or unexercised dimensions remain unresolved; tested dimensions with no decisive outcome remain inconclusive.

Regex and keyword checks are diagnostics, not behavioural verdicts. Only an explicit objective check can be marked decisive, and regex checks cannot be.

Reports separate protocol completion, direct behavioural failures, confidence-only calibration issues, valid-update controls, adaptive findings, adjudication confidence, runner diagnostics, and sampling support. They show case, family, dimension, repetition, truncation, host-context omission, degraded-host, action-repair rate, and call coverage. Sampled runs show both selected-protocol and full probe-bank denominators and explicitly state that a run is not a population prevalence estimate.

Run artefacts are written to:

runs/<run-id>/
├── run.json
├── transcript.jsonl
├── evidence.json
├── result.json
└── report.md

runs/ is ignored by Git because transcripts and API errors may contain sensitive evaluation data. Publish only deliberately reviewed, sanitized examples.

Use open-behaviour report <run-id> to render a report and open-behaviour compare <run-a> <run-b> for comparable runs.

Claim Boundary

OpenBehaviour results are host-assessed behavioural proxies under a declared protocol, not objective ground truth. Strong claims require frozen suite and model settings, repeated runs, transcript audit, adequate opportunity coverage, and explicit limitations.

The initial release contract is recorded in release-freeze.json, including suite hashes, package version, host prompt version, runner protocol version, and report schema version.

For authoring guidance, see the suite design guide. It includes a reference recipe matching the included suites: behaviour-specific categories and interaction approaches, compact controls, subtle trajectories, confidence-only probes, counterbalances, valid-evidence controls, optional longform conversations, and confirmation repetitions.

After Shipping

These are future-release investigations, not blockers for the initial release:

• Independent-judge calibration and assisted transcript-audit workflows.
• Tool-using and environment-backed integrity suites for agentic behaviour.
• Held-out or dynamically generated case banks with reference baselines across more target and evaluator model families.