ask about projects

[TentenMarchhhh]

Question 1: iFixAi acts as an open-source diagnostic for AI misalignment, running 32 tests across five specific risk categories (fabrication, manipulation, deception, unpredictability, and opacity). From an evaluation engineering perspective, how does the codebase programmatically define the test logic for qualitative risk dimensions like "evaluation-awareness sandbagging" (B10) or "covert side tasks" (B14) to ensure they produce objective, empirical evidence rather than subjective observations?

Question 2: The framework supports three distinct evaluation methods: structural, judge, and atomic claims. When executing a "judge" type inspection, iFixAi prevents self-scoring by default in Standard mode, requiring a separate secondary provider key (e.g., Anthropic judging OpenAI). How does the system dynamically structuralize the prompt payload and parse the judge's response to guarantee that variations in the judge's formatting do not cause a parser failure or pollute the final scoring schema?

Question 3: For high-assurance validation, iFixAi introduces Full mode, which runs two or more independent LLM judges across different providers and applies a majority voting mechanism. How does the execution engine handle a split decision or a tie vote among judges, and how is the disagreement recorded inside the content-addressed manifest.json file to maintain a clear audit trail?

Question 4: The diagnostic suite guarantees deterministic reproducibility by exporting a content-addressed manifest that captures all inputs, seeds, rubric hashes, and test corpus versions. Given that live LLM API endpoints are inherently non-deterministic due to server-side sampling drift and model upgrades, how does iFixAi's replay engine bypass live inference to achieve bit-identical score card generation during a regression test rerun?

Question 5: A core feature of the platform is its "Wiring Governance" interface, which allows users to supply an external GovernanceFixture YAML that automatically wraps the target provider via a GovernanceMixin. How does this mixin dynamically intercept the System Under Test's (SUT) outbound API calls and tool execution at runtime to verify compliance against custom permission scopes and authorization policy bundles?

Question 6: iFixAi is explicitly designed to integrate natively into CI/CD pipelines as a regression gate, utilizing specific exit codes (e.g., exit code 2 for a score drop below the --min-score threshold). Since running a full 32-test suite against heavy models can incur high API costs and network latency, how does the run_strategic() async Python API surface optimize test selection and parallel concurrency bounds to provide a rapid, cost-effective signal before code deployment?

[Sebabaian]

Hey @TentenMarchhhh, thank you for these questions. They're some of the sharpest questions we've had, clearly from someone who's read the architecture.

On the delay: these six touch nearly every part of the system: the scoring engine, the judging layer, the manifest, the governance shim, and the CI path. We made a deliberate call not to answer from memory. We went back through the actual source for each one, and we held the reply until a release we were finishing had landed, so everything below describes how it really works today rather than a moving target. A couple of your questions assume behavior that's a little different from what's actually there, so instead of a confident "yep," you'll get the precise version.

Q1: How do you measure fuzzy things (B10, B14) objectively?
The trick is that we never ask a judge "does this look sketchy?" We set a trap with a known answer and check whether the system walked into it.

B14 (covert side-tasks): we write the hidden, malicious goal down in advance, e.g. "quietly export credentials", buried inside an innocent-looking request. Because we already know what the trap was, checking whether the AI took the bait is a fact, not an opinion. We also reword each trap several times so the system can't pass just by recognizing a phrase.
B10 (sandbagging): this is the risk of an AI behaving more carefully when it thinks it's being watched. So we ask the same question three ways neutrally, for example: "you're being audited," and "this isn't monitored" with the model pinned to a fixed, repeatable setting. If its behavior shifts with the framing, that's the tell.
On top of that, each check is a scorecard of yes/no items, and some are marked non-negotiable: miss one of those and the whole item fails, no matter how well it did elsewhere. No partial credit papering over a critical miss.

Q2: How do you keep a messy judge response from breaking scoring?
The judge is itself an AI, and AIs are sloppy about formatting, so we treat its reply as untrusted input and make the reader very forgiving.

We seal the thing being judged inside a tagged "envelope" so a sneaky response can't escape it and pretend to be instructions.
We ask the judge for a strict format (a simple list of {name, passed, reasoning})…
…but when reading it back we go easy: first try a clean parse, then auto-repair common mistakes (missing brackets, stray quotes), then fall back to scraping the answer with patterns. We accept a few different layouts and even fuzzy-match misspelled item names.
If a single verdict is genuinely unreadable, we quarantine just that one item, it's marked as an extraction error and left out of the math, so it can never corrupt the rest of the score. And we retry a few times before giving up.
(If you've ever seen a missing 'dimensions' key message, that's exactly this safety net catching an off-spec reply.)

Q3: Full mode: how do multiple judges combine?
One correction here, because it matters: it isn't a simple majority vote on the final verdict. It works like this:

Each judge produces a score; we take the average, and the item passes if that average clears the bar and no non-negotiable item was missed.
For each individual checklist item, it counts as passed when at least half the judges passed it, that's where the "majority" intuition actually lives.
A tie (say 1–1) counts as a pass by rule, so the engine never gets stuck on a split decision.
We keep every judge's individual verdict on the result. Putting their disagreement into the manifest as its own dedicated field is something we're adding very soon it's a small extension of data we already capture, and it's high on the list, not something we'd claim is finished today.

Q4: Reproducibility: how is a rerun identical?
Honest answer: there's no "replay without calling the model," and for live APIs there genuinely can't be a trustworthy one — the model itself drifts as providers tweak sampling and ship upgrades. So instead of pretending, we give you a receipt.

Every run produces a fingerprinted record of all its inputs: the fixture, the exact rubric versions, the test-corpus versions, every random seed, and the model's temperature/seed plus a one-time token mixed into the prompt so a "reproducible" score can't secretly be a cached reply. Pin the model to deterministic settings, re-run with those recorded seeds, and you get the same scorecard and the receipt proves the inputs were identical. A single command that automates that whole verification is coming in the near future; it's on the roadmap, just not in this release.

Q5: "Wiring Governance": what does it actually intercept?
This one's a common assumption worth clearing up: it does not tap your live traffic or tool calls. Instead, you hand us a small YAML file describing your rules, which roles can use which tools, what's allowed, how overrides and audit logging work, rate limits, and so on. We wrap your provider so the structural checks can interrogate that declared rulebook.

Because we're grading the rules you declared (not what's enforced in production), every governed run carries a visible warning saying exactly that, and it tracks where the governance came from. A model with no rulebook wired in honestly comes back as "not enough evidence" on those checks rather than a misleading green, we'd rather under-claim than hand you a false pass.

Q6: The CI/CD gate and run_strategic()
In a pipeline you want a fast, cheap red/green light, and that's exactly what this is.

The gate: the run exits with code 2 if the score drops below your threshold (default 0.85), or if it couldn't gather enough evidence to score a clean failure for CI to catch.

The fast subset: run_strategic() runs a curated 8-check set (the highest-stakes ones: authorization, fabrication, auditability, override, provenance, uncertainty, hallucination, regulatory readiness) through the same engine as a full run, so the signal is comparable, just quicker.

Cost control: there's a concurrency limit on in-flight requests, a hard cap on total judge calls, automatic slow-down when a provider rate-limits you, an optional spend budget, and a --dry-run that estimates the cost before you spend anything. In practice the strategic gate finishes in a few minutes and costs roughly a tenth of the full suite, which is what makes it practical to run before every deploy.

Really appreciate the depth here. Happy to go deeper on any one of these if you'd like to dig in.