v0.6.0

canpumf 0.6.0

Historical Labour Force Survey and a 1976-onward LFS timeline

The LFS microdata now span 50 years. The monthly public-use files for 1976–2005, which Statistics Canada does not post for download, are loaded from Borealis and can be stacked with the current LFS in one harmonised table.

• New series "LFS_HIST" loads the monthly LFS PUMFs for January 1976 to December 2005 from Borealis (ODESI), in the legacy pre-2017 layout: get_pumf("LFS_HIST", "1995-06") loads one month, and get_pumf("LFS_HIST", "1995") loads all twelve months of a year. Like "LFS", every loaded month is appended to one shared database (LFS_HIST/LFS_HIST.duckdb), so the whole period can be queried as one table. refresh = "auto" loads every month not yet present. French labels come from the French ODESI deposits.
• ODESI labelled the same codes differently from era to era ("Unemployed, temporary layoff" / "Unemploy,temp layoff"). Loaded month by month, that would give one factor level per spelling. LFS_HIST therefore ships one harmonised bilingual dictionary, built from the SAS setup files of all 718 monthly deposits (tools/build_lfs_hist_reference.R). Each code gets its most recent wording, and codes whose meaning changed are curated by hand. Each month is still checked against its own setup file when it is loaded.
• New get_lfs_timeline() returns one lazy table spanning "LFS_HIST" and "LFS", 1976 onward, with a curated common schema. Variables with identical codes carry the current labels. LFSSTAT, MARSTAT, CMA, SCHOOLN, AGYOWNK, the industry groups (NAICS_18), education (1990 onward) and sex/gender are recoded to common categories. Hours and wages are in plain units, and the weight is FINALWT. The two databases stay separate and are attached read-only, so the timeline never takes or waits for a write lock. The mappings ship in inst/extdata/lfs_timeline/ (tools/build_lfs_timeline_reference.R). refresh = "auto" first brings both series up to date, so an analysis script that calls get_lfs_timeline(refresh = "auto") always picks up newly released LFS months.
• The LFS vignette has a new section on the long timeline: loading "LFS_HIST", 50 years of unemployment and participation rates by gender, wages by CMA, and caveats for working across eras.
• The LFS database engine is now a general framework for longitudinal series (time slices sharing nearly the same variables, appended to one database per series). LFS and LFS_HIST are its two instances. label_pumf_columns(), pumf_metadata(), list_pumf_cache(), remove_pumf_cache() and open_pumf_documentation() handle both series. LFS behaviour is unchanged.
• The import was validated against Statistics Canada table 14-10-0287. For all 360 months the weighted population matches the published unadjusted estimates exactly, by province, gender and age group. Employment and unemployment match to rounding, except for January 1985 to November 1986: there the published series appears to carry a revision the public-use files lack, and microdata unemployment runs about 1.2% above the published figure (#23). The same issue documents the problems found in the Borealis/ODESI metadata and how LFS_HIST works around them.

New features

• PUMF data can now be loaded from the Borealis Dataverse, which hosts the ODESI collection of Statistics Canada PUMFs. Statistics Canada stays the primary source. The 1971, 1976, 1981 and 1986 Census PUMFs, which StatCan distributes only by EFT, are now downloaded from Borealis automatically — get_pumf("Census", "1971") works out of the box. A deposited EFT bundle is still preferred when present; add "eft" or "borealis" to the version to choose explicitly. The Borealis copies carry English labels only. For 1971 they are also more accurate than the EFT text files, whose negative household incomes use a sign overpunch the fixed-width reader does not decode.

• Any other Borealis PUMF dataset can be loaded with get_pumf(series, version, borealis = "doi:..."). New list_borealis_pumf_catalogue() browses the collection (cached, and persisted to the cache directory) and list_borealis_pumf_files() lists a dataset's files. The catalogue's statcan column marks datasets that Statistics Canada also posts for direct download, together with the matching StatCan title. Prefer StatCan's copy for these, since the Borealis re-deposits can carry transcription errors. get_pumf(borealis =) warns when asked for such a dataset. Setting BOREALIS_DATAVERSE_KEY gives access to restricted files.

• Registry entries accept a borealis field (list(doi =, files =)), also via pumf_registry_entry().

• Truncated labels are now repaired from the survey's user guide. Statistics Canada's shipped command files routinely carry damaged value and variable labels — hard cuts at 60 characters, dropped leading text, dropped interior text. The damage is upstream of the flavour-specific renderers (the SAS, SPSS and Stata copies of a file agree byte for byte), and it is quiet: codes and frequencies are correct, only the human-readable label is wrong. A ninth metadata parser, parse_pdf_freq_codebook(), reads the data-dictionary appendix of a PUMF user guide (GSS cycles, SGVP, PALS, SFS, Time Use), which carries the full text.

• Trusting a PDF scrape over a machine-readable command file would normally be a bad trade. What makes it a good one here is that this appendix prints the frequency of every code, so the parse is reconciled against the actual data file before any of it is believed, and a label is replaced only where the guide's text demonstrably extends the command file's — a strict superstring, or a subsequence sharing a long anchor ("Single-ded house" → "Single-detached house"). For GSS Cycle 16 this repairs 2,865 labels across the four modules with no variable contradicting the data.

• A repair additionally requires evidence that the label it replaces was damaged. Some guides print the full question wording where the command file carries a hand-written short label ("How many hours do you work per week" against "How many hours do you (does ....) usually work per week?"); these are different fields, not a truncation, and the abbreviation is often a subsequence of the question, so the text-shape test alone would rewrite good labels into questions. A hard cut leaves a spike of labels at a fixed ceiling — GSS Cycle 16 has 1,665 of 1,860 variable labels at 59–60 characters against 42 in the six lengths below — while a hand-abbreviated set thins out towards its longest entry. Labels are only repaired where that ceiling exists and the label sits at it; everything else is flagged.

• Dropped leading text leaves a short label rather than one at a ceiling, so it carries its own signature: what survives is a strict suffix of the guide's text ("relative in a family farm or business?", "les réserves indiennes)"). Two things that are not damage produce the same shape — a guide that prefixes an editorial note onto a label the command file has in full (SGVP 2007's "Grouped variable: Age group"), and a guide leaking its own field header into the text ("Longueur : 2 Age du répondant …"). Both are rejected by looking at the dropped text rather than at what survived: StatCan writes these as "Key: value", while text lost to truncation is running prose. That distinction is what lets the mid-list case be repaired — GSS Cycle 16's value label "Co-worker of respondent and Other relatives)", which is really the tail of "Other (Do not include organizations here) (Includes …)" and reads as a category about co-workers, starts with a capital exactly as an intact label would. Together these take SGVP from 35 repairs to 1 and PALS 2006 from 245 to 100, while GSS Cycle 16 keeps 2,860 of 2,865.

• The guide is also checked as a document before any of it is used, against two independent channels: whether its printed counts reproduce a tabulation of the data, and whether its printed field positions reproduce the command file's layout. A guide failing both is the wrong document and is discarded whole. A release shipping both its original and revised user guide (PALS 2006) is disambiguated the same way — the two describe different field positions, and only the layout tells them apart. Where the positions agree but the counts do not, the guide's frequencies were simply tabulated on another base (PALS 2006 computes its over the disability sub-population) and are treated as absent rather than as contradiction.

• Nothing is repaired silently. New pumf_label_repairs(tbl) returns every divergence found — repaired, filled, or merely flagged — with both texts, the reason, and the variable's validation status; new pumf_freq_validation(tbl) reports per variable whether the guide's frequencies reconciled. Substantive disagreements (where the guide and the command file say different things rather than one being a truncation of the other) are always flagged rather than applied. Set options(canpumf.pdf_crosscheck = FALSE) to skip the step.

• The guide's frequency tables are read off the number column nearest the FREQ header word rather than by walking right from it. The columns are right-aligned but pdftools reconstructs them with a few characters of drift, so walking out stopped at a thousands comma (reading 9,520 as 9) and never reached a value printed further right than its neighbours (reading a bare 0 as absent). A code row's label is also now optional, so a 0–10 scale that labels only its endpoints keeps the counts for the nine bare codes in between. Together these take the GSS Cycle 26 guide from 4 contradicted variables to none across all 610.

• Codes the guide documents but the command file never declared are reported, never injected — that stays a registry codes_supplement decision. The report covers only variables the command file treats as categorical: where it declares...

v0.5.2

CRAN release of refreshed canpumf

v0.3.0

Last legacy version before breaking changes

v0.2.0

Using EFT download of old census PUMF