Per Document Generator

D - Per-Document Generator — Algorithm and Flow Documentation

This document describes the design, execution pipeline, and algorithmic heuristics used by the AI Classifier Tool (AICT) Generator defined in Per-Document-Generator.py.

Core Pipeline Architecture

The generator operates as a multi-stage deterministic sorting and metadata extraction pipeline:

flowchart TD
    START[Raw PDF Document & Metadata] --> GRAZE{1. Grazing Pre-pass}
    GRAZE -->|Yes: Study, Plan, Appendix| PT[Vein: PT - Project Technical Detail]
    GRAZE -->|No: Administrative File| COARSE{2. Family / Vein Sort}
    
    COARSE --> CP[Vein: CP - Certificate Proper]
    COARSE --> PE[Vein: PE - Public Engagement]
    COARSE --> AT[Vein: AT - Act Transition]
    COARSE --> CM[Vein: CM - Communication]
    COARSE --> OA[Vein: OA - Official EA Artifact]
    
    PT & CP & PE & AT & CM & OA --> ACT_REC{3. Operative Act Recovery}
    ACT_REC --> OUTPUT[Generate Verified JSON Metadata Output]

Loading

---

Processing Stages

1. The Grazing Pre-pass (Isolate Bulk Content)

• Objective: Exclude high-volume technical and public submission attachments (~76% of all files) from deep classification.
• Rule: If a document type or name contains standard bulk keywords (e.g., appendix, chapter, technical memo, baseline report, study, management plan), set Project Content = YES and tag it as PT (Project Technical Detail).
• Performance Impact: Reduces fine classification evaluation scope by 74%, dramatically preserving processing resources.

2. Functional Family and Vein Sorting

For non-bulk administrative records, the generator categorizes files into functional tracks ("Veins"):

Vein Code	Vein Name	Precedence	Triggers & Identifiers
CP	Certificate Proper	1	Contains certificate in title and does not contain amend, transfer, application.
AT	Act Transition	2	Contains request to start new, section 10 order, issued process order with transition indicators.
PE	Public Engagement	3	Title contains comment, open house, notice of intent, PCP, or advertisement.
OA	Official EA Artifacts	4	Title/Type contains order, amendment, exemption, extension, assessment report.
CM	Communication	5	Contains letter or meeting notes (lowest-precedence).

3. Operative Act Recovery

• Rule: Resolves whether a document operates under the 2002 Act or the 2018 Act.
• Threshold Date: Uses the formal enactment and implementation of the 2018 EAA (e.g., 2021-04-16 Process Order date).
• Result: Publication dates before the threshold are assigned to the 2002 Act, while newer dates default to the 2018 Act.

---

Outputs & Naming Conventions

The generator modifies the physical filename suffix to represent the coarse classification state without destructive renaming:

• Pattern: <original_name> -[VV] -P.json (where VV is the two-letter Vein code, and P is Pending Verification status).
• Schema structure matches Per-Document-Data-Format.md.