Policy Ingestion Pipeline

Policy Ingestion Pipeline

The ingestion pipeline retrieves configured policy sources and decides whether the result is acceptable evidence.

Retrieval Cascade

PolicyWatcher uses a five-strategy cascade:

1. Direct HTTP/1.1 fetch
2. Explicit HTTP/2 fetch
3. VPS rendered fetch
4. Wayback Machine fallback
5. Common Crawl fallback

Each strategy records:

• status;
• HTTP status where available;
• rejection or failure reason;
• escalation path;
• final accepted source if any.

Cascade Logic

flowchart TD
    start["Configured policy URL"] --> direct["Direct fetch"]
    direct -->|accepted| gate["Evidence validation"]
    direct -->|failed/rejected| h2["HTTP/2 fetch"]
    h2 -->|accepted| gate
    h2 -->|failed/rejected| rendered["VPS rendered fetch"]
    rendered -->|accepted| gate
    rendered -->|failed/rejected| wayback["Wayback fallback"]
    wayback -->|accepted with timestamp| gate
    wayback -->|failed/stale| cc["Common Crawl fallback"]
    cc -->|accepted with evidence| gate
    cc -->|failed| suspended["Temporarily suspended source"]
    gate --> baseline["Verified baseline or change detection"]

Loading

Evidence Validation

Fetched content is accepted only when it satisfies source-quality checks such as:

• substantive policy-like text;
• sufficient content length;
• valid final URL;
• no homepage/path drift;
• no unsupported cross-host drift;
• no bot-wall or placeholder text;
• no over-cap partial capture treated as complete.

Re-Baseline Rule

If a policy is still seeded and receives its first verified source text, the platform stores it as the first verified baseline. It does not create a fake change from placeholder text to real text.

Suspended Sources

When every retrieval strategy fails or returns insufficient evidence, the source is marked as unavailable or needing review. Public views should not expose the policy text, score, timeline event, KPI value or AI interpretation until the source is verified again.

WAF and Anti-Bot Policy

PolicyWatcher should not use stealth masking or residential proxy rotation to bypass provider access controls. If a provider blocks automated access, that condition is recorded as an operational source issue rather than hidden.