Confidence Methodology

Confidence Methodology

The 3.5 Confidence Release is the methodological backbone of PolicyWatcher.

Its purpose is to make the platform more inspectable, cautious and accountable. The key rule is simple:

If a source cannot be verified, PolicyWatcher should not present its analysis as public evidence.

Confidence Principles

1. Evidence before interpretation
   AI-assisted analysis is only useful when grounded in retrieved policy text.

2. No fabricated data
   Demo records, seeded placeholders and incomplete retrievals are blocked from production public views.

3. Traceability over convenience
   Each policy should expose source URL, retrieval method, last check, last successful check, current hash and evidence status.

4. Suspension is better than false confidence
   A blocked or ambiguous source becomes temporarily suspended until reviewed or remediated.

5. Re-baseline without false changes
   When a seeded policy receives its first verified text, it becomes the baseline without generating a fake PolicyChange event.

Data Statuses

Policy records can move through states such as:

• Configured
• Available
• Partial
• Needs Review
• Unavailable
• Reviewed

Only source-backed, publicEvidence-gated records should power public timelines, scores, matrix values and change pages.

Public Evidence Gate

Public APIs and UI views must reject:

• seed-only records;
• incomplete captures;
• unsupported source states;
• unavailable sources;
• private admin-only history;
• policy changes created without a publicEvidence-backed snapshot.

AI Boundaries

AI analysis is used for structured policy interpretation, but it is not treated as a legal conclusion. The platform should clearly distinguish:

• retrieved text;
• extracted evidence;
• AI-generated interpretation;
• human review;
• unresolved source anomalies.

Operational Alerts

When a source becomes suspended because of anomalies in the latest fetch/update cycle, the system can notify the administrator by email with metadata only: source, status, reason, strategy, HTTP status and Dataset QA link.