dp-data 0.4.6

This release corrects Australia's shared knowledge checklist and preserves the distinction between its initial combined “None / Don't know” option and the substantive exit answer “Nothing will change.”

• Give the initial combined option zero credit in both retained batteries. Score the exit answer against the unchanged answer key. The twelve-item baseline mean falls 3.75 percentage points; the ten-item exit mean rises 3.20 points among the same 347 participants.
• Retain original answers and the combined source reason, with a nullable integer n_none_or_dk count separate from literal DK. Version the affected definitions and declare their source dependencies.
• Rebuild dependent respondent, peer, group and poll measures centrally. Attendance, group assignments, attitudes, demographics and other polls are unchanged. All earlier approved corrections remain recorded.
• Restore 52 omitted issue links in the poll metadata and decision register. Preserve unavailable-source and instrument limitations in the existing poll documentation.

Downstream projects must update their data pin and regenerate knowledge estimates explicitly. Attitude model inputs remain unchanged; the dp-distortions migration remains an unmerged PR.

dp-data 0.4.5

This release applies common questionnaire-completion, knowledge-response, and attitude conventions across the public poll builds.

• Empty departure questionnaires imply nonattendance; previous classifications and their evidence remain available. Analytical participation requires each collected pre-arrival, arrival, and exit questionnaire, without treating uncollected stages or missing later follow-up as absences.
• Observed knowledge blanks and DK answers receive zero credit while retaining their distinct source reasons. Absent forms and invalid codes remain missing. Typed categories are correct, incorrect, and dk.
• Primary attitude responses use scale endpoints from zero to one and preserve missing values. Plain indices and explicitly named midpoint-imputed alternatives have separate definitions.
• California gains 39 arrival/exit policy ratings. Reviewed respondent covariates fill previously missing canonical fields without changing any existing nonmissing value or introducing new normalization.
• Unusual San Mateo and Zeguo zero-score and blank batteries remain in the main data, with local publication comparisons and typed robustness flags.
• The concise poll decision register links to the complete preserved evidence and source-specific limits. Missing source material and accepted ambiguities remain documented; this release does not assert complete coverage of every poll or every collected wave.

Downstream projects should adopt these inputs explicitly and compare unchanged model specifications before revising substantive interpretations. The dp-distortions migration remains an unmerged PR at the user's request.

v0.4.4

Restore 36 original source fields across Michigan, BTP 2007, NIC, Denmark, and Tomorrow’s Europe, including previously omitted questionnaire responses. Preserve Michigan’s historical question crosswalk with a PDF companion. Actual contact fields remain excluded with documented reasons.

Add Michigan’s nine-item arrival knowledge measure alongside its existing pre-arrival and exit measures. The 309 observed arrival questionnaires permit matched comparisons across all three stages. Ambiguous text receives no credit; a wholly absent questionnaire remains missing.

Previously published numerical scores, attitudes, attendance, respondent samples, and weights are unchanged. Fifty of 57 generated files remain byte-identical; the other changes contain the new Michigan measurements and updated source or timing provenance.

Validated locally, independently reviewed, and rebuilt from the source archive without Git history or vault files. All 57 rebuilt output files match the tested candidate byte for byte. Detailed evidence and unresolved source questions remain in docs/poll-issues.md.

v0.4.3

Restore scientific source material and improve questionnaire-presence metadata while preserving numerical estimates.

• Restore 120 original fields across eight polls, including literal written responses, with source and dictionary checks.
• Preserve original Michigan and Northern Ireland data files and Michigan, Europolis, and Climate reference materials in their poll folders. All 68 Word documents have registered PDF companions.
• Add 21 verified Climate attitude items and distinguish observed questionnaires from missing forms consistently, including Tanzania, Denmark, and Vermont.
• Correct five invalid New Haven item codes and preserve documented don't-know responses. Remove redundant occupation sidecars after exact comparison with the canonical source.

All 1,027,858 existing numerical respondent measures and 63,234 survey weights remain unchanged. The release rebuilds all 57 output files byte-for-byte from a clean source archive without Git history or vault files. Local checks, independent review, and GitHub validation passed. Remaining source and coding questions are recorded in docs/poll-issues.md.

v0.4.2: Reviewed questionnaires and source evidence

• Restore Northern Ireland's literal argument-coder labels. CSV type inference
  had collapsed 176 comma-separated code sets; source whitespace is also retained.
  Respondent identities, missing slots and all knowledge outputs are unchanged.

• Preserve Denmark's four original questionnaires and readable PDF companions;
  record the later follow-up's verified telephone mode without changing dates,
  scores or cohorts. Preserve its wave crosswalk, two California index drafts
  and Northern Ireland's version 16 coding guide in their poll folders.

• Allow the full CI build and numerical audit up to 60 minutes. The previous
  30-minute limit canceled a run after tests and lint passed, before its final
  metadata verification could finish.

• Use reviewed whole-questionnaire evidence in knowledge scoring. Preserve
  observed blank quizzes as zero; unavailable questionnaires have missing scores.

• Retain verified attendance before inferring nonattendance from absent exits,
  including Australia's nine attendees without exit questionnaires and two
  UK Health attendees without completed post-event questionnaires.

• Preserve Australia's original combined DP codebook and its readable PDF,
  distinguishing it from the separate constitutional-referendum codebook.

• Export 829 source-level attitude definitions and 1,039,910 response rows for
  Denmark, Vermont, Marousi and America in One Room 2024. Retain all source
  people, original identifiers, labels, units and documented interview phases.

• Exclude 372 reviewed attitude nonanswers and 40 invalid Vermont responses
  from numeric values. Preserve 271 Marousi responses with unverified scales
  as unclassified raw responses, with missing numeric values.

• Preserve Marousi's original identifiers separately for telephone, arrival
  and exit while retaining the authored row bridge and its 16 unresolved
  departure-ID conflicts.

• Apply reviewed full-form masks across ten polls and calculate group knowledge
  means and learning opportunity from observed peers. Preserve BTP General
  Election's 299-person group cohort before its 248-person analytical selection.

• Declare complete questionnaire-presence dependencies and use registered source
  column order. Individual recodes support reordered inputs and local subsets
  without changing reviewed values or weakening production identity checks.

• Retain 2,299,630 additional raw questionnaire responses used by the presence
  rules; every existing raw response and reviewed measure value is unchanged.

• Apply 152 codebook-backed CPL nonanswer declarations through a shared
  field-specific registry. Preserve substantive Other codes and dollar amounts;
  raw values and reviewed scores remain unchanged.

Validation: full local tests and lint; independent source, numerical and document reviews; all 35 respondent, analysis and aggregate output files rebuilt byte-for-byte without Git history or vault. Exact-commit CI is required before publication.

Australia combined none/DK and San Mateo empty exit retain existing scoring pending consultation. Missing sources and measurement decisions are documented in docs/poll-issues.md.

v0.4.1: Preserve recovered poll sources

This release preserves recovered poll-level source materials and historical coding evidence without changing existing knowledge or attitude estimates.

• Adds 37 registered files: 13 original historical scripts, additional Denmark/Europolis/UK–EU/BTP Health source versions and their dictionaries, original cross-poll inventory documents, and readable PDF companions.
• Retains Denmark's dated later follow-up as t3 in the wave catalog. Its knowledge answers are not scored until the changed question wording and answer keys are reviewed.
• Preserves three additional supplied BTP Health weight columns with original source identities, missing values, and zero weights. How to use weights remains undecided.
• Improves wide-workbook PDF previews so study names and headers remain readable across pages.

All 50 other generated output files are byte-identical to v0.4.0. Source-version differences and remaining instrument/linkage gaps are documented in docs/poll-issues.md; this release does not claim that all poll corrections are finished.

Validation: local metadata validation, full R/Python tests, and lint passed; an independent review verified 37 source checksums and original archive copies, reproduced the weight and wave tables, and passed 1,060 focused assertions. GitHub checks passed for both the reviewed candidate and merged commit e92b9aa.

dp-data v0.4.0

This release rebuilds the corrected historical aggregates from public poll-level sources and preserves the available survey phases in typed Parquet.

Corrections distinguish absent questionnaires from blank answers within observed forms, retain positive attendance evidence, and treat unknown discussion groups as missing membership. The release includes the approved New Haven, BTP National and UK–EU fixes, a shared questionnaire-presence reconciliation, and both authored Zeguo township-image definitions with distinct names.

Original Bulgaria Crime survey files and coding drafts, Australia attendance evidence, Europolis's master questionnaire, and New Haven recruitment identifiers are retained in their poll folders with provenance and readable document companions.

Validation: full local tests and lint passed; independent review verified the corrections and source preservation; full CI passed on release commit 0970d70. The issue register records the remaining source and instrument limitations. Bulgaria 2007 respondent data are still unavailable, so this release does not claim a completed respondent-level audit of that poll. Weight use remains undecided.

v0.3.0 — Historical polardata rebuilt from poll sources

Recreate historical polardata from public poll-level sources, without the local vault or frozen aggregate inputs.

• Reconstruct all 848 respondent-field targets across 21 historical polls.
• Export the complete 6,084-row, 364-column historical schema and 129-row attitude-index catalog under output/polardata/.
• Keep canonical people unique, preserve the 217 duplicated Primaries people in the historical wide export, and publish derived measures separately.
• Preserve historical coding choices, source versions, sample stages and intermediate precision. Detailed poll-specific findings and required instrument checks are recorded in docs/poll-issues.md for later corrections.
• Explain 288 generalized-variance differences in 24 groups through exact source-matrix checks and numerical diagnostics. Export row numbers are regenerated; every other value and missingness comparison passes.
• Preserve all 15 existing core, linkage and partial UK Health outputs byte-for-byte. Downstream repositories retain their existing input pins.

The release archive contains the source inputs, identity/order ledgers, code and outputs. make check rebuilds and validates the pipeline. Third-party materials retain their existing rights; this release does not grant new rights to them.

Validation: local and independent isolated make check each passed 2,141 assertions with zero failures, warnings, skips or lint findings. Hosted CI passed for the reviewed candidate and publication commit. A clean source-archive rebuild, with the vault, benchmarks and prior outputs absent, reproduced all seven aggregate products byte-for-byte; the restored benchmark comparison also passed.

Publication commit: a408904b20d65c2b8a1ccd556cdb8c9dff100aa9. Reviewed candidate: 4fa913adce4a567c61e0291f6496406cff97fdde (identical tree).

v0.2.3 — Shared OOS source inputs

Centralize the unchanged source inputs for the dp-distortions out-of-sample study.

• Catalog all 24 public inputs and supporting documents, including original URLs, access dates and SHA-256 hashes.
• Add 20 files under data/<study>/ and reuse four existing copies. Preserve original bytes, including line endings.
• Keep study-specific recoding and estimation in dp-distortions. No historical recodes or existing analysis outputs change.
• Record the historical benchmark consumers dp-distortions and dp-deliberately.

The source archive contains the inputs; no local vault or frozen Git history is needed to obtain them. metadata/oos_sources.csv provides their paths. Third-party materials retain their own licensing terms; blank license fields are not new license grants.

Validation: complete local and independent make check, hosted CI, and source-archive byte-count/SHA-256 verification. Candidate: 3b7a03215720dab27563cc4c785aa2d746f66d46.

v0.2.2: Public Northern Ireland analysis inputs

dp-nireland can now reproduce its analysis and paper using only public dp-data inputs: the numeric survey, group roster, and new argument coder-label extract. No vault or historical Git extraction is required.

The new data/northern-ireland-2007/argument-codes.parquet retains 65,760 coder-slot records for 274 respondents, including missing labels. It contains pseudonymous respondent IDs and the original coder/adjudicator labels; response text is excluded. R/argument_codes.R records the extraction, and the artifact catalog pins the original source and its SHA-256. Normalization, adjudication, and scoring remain in dp-nireland.

The existing survey, roster, historical benchmarks, and other outputs are unchanged. dp-nireland preserves its historical first-roster-record handling pending the separate NI-01 correction. All 19 pinned downstream numerical outputs remain byte-identical on the baseline Mac environment. Linux reproduces 17 exactly; the two estimate CSVs differ only at numerical precision (maximum observed difference 6.59e-11, in degrees of freedom). The downstream check retains original references and hashes, allowing absolute differences up to 1e-10 only for six inferential statistics; labels, counts, missingness and other outputs must match exactly.

Validation: upstream build, historical parity, tests and lint; downstream make check including the paper build against an extracted upstream archive with no vault or .git directory; independent review and archive checks. Upstream: 833 passing expectations. Downstream: 56 passing expectations. Both have zero failures, skips, or lint findings.

• Upstream commit: bce1e0e3aea578b71090dde5348ca66ccb778048 (CI)
• Downstream migration: soodoku/dp-nireland@4aaae4163c7f64d0adc36c77e93766dadb408033 (CI)

Full polardata reconstruction remains at eight of 21 historical polls; this release does not change that scope or apply substantive coding corrections.