Skip to content

FAQ

Jump to bottom
Roberton003 edited this page Jul 22, 2026 · 2 revisions

Frequently Asked Questions

General

Q: Can I use this without an AI agent?

A: Yes. The JSONL log is portable and queryable. You can write decisions directly or integrate into any system that can call HTTP or execute Python.

Q: What if I don't have all the information when recording?

A: That's fine. Record what you know:

  • problem and chosen_solution are required
  • rejected_alternatives, predictions, domain are optional
  • Add predictions later via record-prediction

Q: How long are decisions kept?

A: Forever. The JSONL log has no retention policy. Manage storage as you would any other data file.

Technical

Q: Does this need a database?

A: No. It's a single JSONL file (append-only). Zero external dependencies, zero schema migrations.

Q: Can I change a decision after it's recorded?

A: No. The log is immutable. If you want to supersede a decision, record a new one with status="SUPERSEDED" and reference the old decision ID in adr_ref.

Q: What if the log file gets corrupted?

A: JSONL is line-delimited. Each decision/prediction/outcome is a separate line. A corrupted line is skipped; valid lines are still readable. Consider periodic backups.

Q: Can multiple agents write to the same log?

A: Yes, as long as they append (not rewrite). JSONL is append-safe. Lock-free concurrent writes work because each entry is independent.

Q: How do I export decisions?

A: The log is plain JSON. Parse it directly:

import json
with open("~/.local/share/mcp-decisions/decisions_log.json") as f:
    for line in f:
        entry = json.loads(line)
        print(entry)

Predictions & Outcomes

Q: What accuracy score should I use?

A: 0-100 scale, comparing predicted to actual:

  • 90-100: Prediction was spot-on (SUCCESS)
  • 50-89: Prediction was in the ballpark (PARTIAL_SUCCESS)
  • 0-49: Prediction missed significantly (FAILED)

Example: Predicted "p99 < 200ms", actual "p99 = 180ms" → 95 accuracy.

Q: Can I have multiple predictions per decision?

A: Yes. Each decision can link to many predictions (latency, cost, scalability, etc.).

Q: What if I realized the prediction was wrong after recording the outcome?

A: The outcome is immutable once logged. If needed, record a new outcome with corrected measurement for a different session/date, or file an issue if this is a common need.

Q: Do I have to record outcomes?

A: No, but you should. Unvalidated predictions don't help you learn. The Outcome Gate nudges you to close the loop.

Outcome Gate

Q: What's the Outcome Gate?

A: A nudge in tool responses that lists any predictions from this session still lacking outcomes. It's in-band (embedded in the response) so you see it exactly when making decisions.

Q: Can I disable the Outcome Gate?

A: It's always on. If it's noise, either:

  • Close the loops (record outcomes)
  • Use a separate log file for experiments (set MCP_DECISIONS_LOG_PATH)

Q: Why does it only show predictions from this session?

A: To avoid alert fatigue. Old unvalidated predictions are your problem; new ones are actionable right now.

Performance Registry

Q: How is "successful" defined?

A: A decision is successful if:

  • It has >= 1 outcome with validation_status=SUCCESS
  • AND it has NO outcomes with validation_status=FAILED

Mixed SUCCESS + PARTIAL_SUCCESS still counts as successful.

Q: What do confidence tiers mean?

A: Based on sample size:

  • LOW: <3 outcomes recorded (not enough data)
  • MEDIUM: 3-9 outcomes (useful signal)
  • HIGH: >=10 outcomes (strong signal)

Q: Can I use the registry for recommending technologies?

A: Yes! Check the registry before proposing a tech. If DuckDB has 91% average accuracy across 12 decisions, it's safer than a tech with 3 samples and 52% accuracy.

Didn't find your answer? Open an issue or start a discussion.

Clone this wiki locally