Skip to content

P2: specify validation metrics reporting format in Annex A Week 7 deliverables #165

@yohei1126

Description

@yohei1126

Context

The submission commits to Precision@50 but does not pre-agree the format or definition of other evaluation metrics (Recall, AUROC, computational cost units, granularity, explainability) with Cap Vista.

Required

Add a "Validation & Reporting" subsection to Annex A Week 7 deliverables with a pre-agreed metrics table:

Metric Target Unit Definition
Precision@50 ≥ 0.60 Fraction Top-50 ranked candidates that are OFAC-listed
Recall@200 ≥ 0.40 Fraction Fraction of all OFAC-listed vessels in dataset captured in top-200
AUROC ≥ 0.80 0–1 Area under ROC curve across all scored vessels
Computational cost ≤ SGD 250/month SGD Cloud + storage for continuous 15-min re-scoring at PoC scale
Granularity Per-MMSI + per-feature Every prediction tied to a MMSI; top-3 SHAP signals per vessel
Explainability SHAP + ATT + path Qualitative Top-3 SHAP attributions, DiD ATT ± 95% CI, ownership graph path per watchlist entry

Acceptance criteria

  • Annex A Week 7 deliverables section includes the metrics table above (or equivalent agreed with Cap Vista)

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    Status

    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions