Define the Kolkhoz-to-zavod CSV handover contract

[pudo]

Define the wide CSV format through which Kolkhoz hands extracted political office-holder observations to zavod.

Kolkhoz produces source-shaped observations rather than FollowTheMoney entities. Zavod applies its existing validation, normalization, identifier, and FtM-generation tooling.

One Kolkhoz crawl project may produce multiple CSV files. Each file represents a source group that can become a separate zavod dataset.

Row grain

One row represents one person holding one position, as extracted from one source snapshot.

Kolkhoz does not deduplicate observations across sources. A person with multiple positions produces multiple rows. The same observation supported by multiple pages also produces multiple rows.

Initial columns

source_group
source_id
source_url
snapshot_id
snapshot_retrieved_at
snapshot_freshness
organisation_name
page_title
person_name
person_dob
person_bio
person_country
position_name
position_description
position_jurisdiction
position_start_date
position_end_date
evidence_quote

snapshot_freshness is fresh or stale. snapshot_retrieved_at describes the snapshot used for extraction, not a later failed refresh attempt.

Contract

• Preserve source wording rather than performing FtM normalization.
• Treat dates as source strings.
• Use blank values for fields not stated or not extracted.
• Preserve the Pravda snapshot identifier for data lineage.
• Produce deterministic UTF-8 CSV files with a fixed column order.
• Version the export schema.
• Keep extraction diagnostics and model metadata in Kolkhoz unless zavod needs them for validation.
• Permit one crawl project to emit several independently named source-group CSVs.

Questions

• Is source_group both the output filename and the zavod dataset boundary?
• Which fields are mandatory for a row to be exported?
• Does evidence require only a quote or also a page locator?
• Should country fields contain raw source text, normalized territory codes, or both?
• How are source-group metadata and schema version delivered?
• Where are generated CSVs stored, and how does zavod discover them?
• Does zavod consume every extraction or only observations accepted by a Kolkhoz quality gate?

[monneyboi]

Implemented a first cut of the wide CSV export. Columns now:

dataset
source_url
snapshot_id
snapshot_retrieved_at
organisation_name
person_name
person_dob
person_bio
person_country
position_name
position_description
position_jurisdiction
position_start_date
position_end_date
evidence_quotes

source_group renamed to dataset. Currently name of input CSV & the export filename and zavod dataset boundary. One CSV in & one out seems simple.evidence_quotes is plural because a row can carry several, poliloom taught that the model will randomly merge multiple lines if you force it to output only one. So if you want to stay true to the source document, allowing the model to output multiple quotes will allow it to be verbatim. However, outputting one CSV row per observation demands a list of free-text strings fit into one cell, but no separator is safe when the strings may contain that separator verbatim. How would you like to receive this data?

Left out:

• snapshot_freshness — Extractions are keyed to snapshot ID. If a newer snapshot does not succeed, we're not re-running a extraction. I'm not sure it makes sense to send the data of a old extraction if a newer snapshot did not succeed, as the caller should already have that data. Please correct me if I'm wrong.
• source_id — source_url is already the de facto identity; no need for an opaque id yet.
• page_title — Would it be okay to leave this out?

Answers to the open questions:

• source_group = filename = dataset boundary → yes, we call it dataset.
• Mandatory fields: we currently export every observation;
• Evidence page locators. That would come for free when we implement the locator stuff for data extraction for pages where we can.
• Country: currently raw source text only, no normalization. I think that makes sense in regard to staying consistent with the other normalization living in zavod.
• Schema version + per-dataset metadata: not solved — the main thing to align on next, I think warrants a separate issue.
• Storage/discovery: one <dataset>.csv per dataset into a directory; no manifest yet, could be part of the schema + dataset metadata issue mentioned above.
• Quality gate: currently we export every extraction; no gate yet.