Skip to content

Correct latest-label tiebreak, harden NDJSON/CLI, honest summary wording#1

Merged
wietzesuijker merged 2 commits into
mainfrom
sota/audit-fixes-2026-06-13
Jun 13, 2026
Merged

Correct latest-label tiebreak, harden NDJSON/CLI, honest summary wording#1
wietzesuijker merged 2 commits into
mainfrom
sota/audit-fixes-2026-06-13

Conversation

@wietzesuijker

Copy link
Copy Markdown
Member

An audit pass over the export/flatten core. All behaviour-preserving except the latest-label fix (a real edge-case bug) and the CLI summary wording. Tests go 18 → 27; ruff + mypy strict clean.

Correctness

  • Latest-label selection on missing timestamps. _latest_label used max(labels, key=created_at), where a label without created_at sorts as "" and always loses to a timestamped one — so a reviewer's label exported without a timestamp could be silently overridden by the annotator's. Now: if every label has a created_at, newest wins as before; if any is missing, fall back to export order (last = newest). Rare in practice (Labelbox usually populates created_at), but it's the exact failure the function exists to prevent.

Hardening

  • read_export_file reports the line number on a malformed NDJSON line instead of a bare JSONDecodeError.
  • The CLI validates --status against WORKFLOW_STATUSES and fails at the boundary with a clear message rather than forwarding a typo to the API.

Honest output

  • The CLI printed "<n> reached unlabelled", but that count includes never-reached ToLabel rows (rows no annotator ever opened). It now prints "<n> without labels"; the field comment is corrected to match.

Tests

Lock the documented FeatureRow contract: relationship annotations, the classification/geometry "unknown" fallbacks, and the missing-API-key error — all previously unexercised.

Open questions (your call — not changed here)

  • Summary.n_reached_unlabelled is misnamed (it counts all rows without labels, not just reached ones). Renaming to n_without_labels is the honest fix but a breaking API change for a published package — deferred to a major version.
  • Missing-timestamp tiebreak now falls back to export order. If you'd rather treat a missing created_at differently (e.g. drop such labels, or surface a warning), say so — it's a semantic choice.

… honest summary

Audit fixes on the export/flatten core (all with tests; 18 -> 27).

- _latest_label: when any label lacks a created_at, the old max() let a
  timestamped label beat a missing-timestamp one lexicographically — it could
  return the annotator's label over the reviewer's. Use created_at when every
  label has one; otherwise fall back to export order (last = newest).
- read_export_file: a malformed NDJSON line now raises with its line number
  instead of a bare JSONDecodeError traceback.
- CLI --status: validate against WORKFLOW_STATUSES and fail at the boundary with
  a helpful message instead of passing a typo to the Labelbox API.
- Summary: the CLI printed "<n> reached unlabelled", but the count includes
  never-reached ToLabel rows; print "<n> without labels" and fix the field
  comment. Field name unchanged (rename would break the public API).
- Lock the documented FeatureRow contract with tests: relationship annotations,
  the classification/geometry "unknown" fallbacks, and the missing-API-key error.
@wietzesuijker wietzesuijker merged commit a1b0b9e into main Jun 13, 2026
6 checks passed
@wietzesuijker wietzesuijker deleted the sota/audit-fixes-2026-06-13 branch June 13, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant