FreeREG Core Records

FreeREG Core Records

This domain covers the heart of FreeREG: how a parish register transcription gets from a volunteer's CSV file into the searchable database, how it's organised geographically, and how researchers find it.

Scoped to FreeREG on master (as of 882fb030e, 1 Oct 2026). Several models here (Place, SearchRecord, SearchQuery, PlaceCache) are shared with FreeCEN and carry template_set branches. This domain covers only the FreeREG paths and mentions the others where they get in the way.

The upstream half of the story (who is allowed to transcribe, how images are allocated, how a CSV gets uploaded) is in Transcriber-and-Coordinator-Workflow. This domain starts at the point where Csvfile-System hands a file to the processor.

Pages

• Gazetteer-Place-Church-Register: the Place → Church → Register location hierarchy, PlaceCache, and what renaming/merging/relocating actually touches
• Batch-Records-Freereg1CsvFile-and-Entry: the persistent batch (Freereg1CsvFile) and record (Freereg1CsvEntry) models, locking, online editing, removal
• CSV-Processing-Pipeline: the build:freereg_new_update rake processor, the FreeREG CSV file format (headers, +LDS, #,DEF), and how re-uploads are reconciled
• Search-Record-Indexing: how an entry becomes a SearchRecord: name roles, soundex, emendation, UCF, search dates, location strings
• Search-Query-System: SearchQuery: validation, query construction, index hints, the secondary-date search, embargo filtering, UCF results
• Embargoes-and-Gaps: EmbargoRule/EmbargoRecord (hiding recent records) and Gap (documenting missing years)
• FreeREG-Content-and-Statistics: the public "Database Contents" pages, the denormalised counters behind them, site statistics and Open Data
• FreeREG-Operations-and-Rake-Tasks: the rake tasks and file-based queues that keep all of the above running
• Emendation-System: forename abbreviation rules (Wm → William) and how to re-apply them
• County-Management: County records, county coordinators, and the county dashboard

Shared infrastructure that FreeREG depends on:

• Authentication-System: Devise User ↔ UseridDetail, login, acting roles, require_login
• Error-Management: Errbit/Airbrake, error pages, rescue_from conventions, log prefixes
• Project-Configuration: every config file and mongo_config.yml key, and where it's used

Data model

flowchart TD
    PL["Place<br/>chapman_code + place_name<br/>location (2dsphere)"] -->|has_many| CH[Church]
    CH -->|has_many| REG["Register<br/>register_type: PR / BT / AT / ..."]
    REG -->|has_many| F["Freereg1CsvFile<br/>one batch = one userid + file_name<br/>+ place/church/register/record_type"]
    REG -->|has_many| ER[EmbargoRule]
    REG -->|has_many| GAP[Gap]
    REG -->|has_many| SRC["Source<br/>(image server — see Image-Server-System)"]
    F -->|has_many| E["Freereg1CsvEntry<br/>one CSV line"]
    F -->|has_many| BE["BatchError<br/>lines that failed validation"]
    E -->|embeds_many| MW[MultipleWitness]
    E -->|embeds_many| EMR[EmbargoRecord]
    E -->|has_one| SR["SearchRecord<br/>denormalised, what search reads"]
    SR -->|embeds_many| SN[SearchName]
    SR -.->|belongs_to| PL
    PF["PhysicalFile<br/>the file on disk, per userid"] -.->|same userid + file_name| F
    UD[UseridDetail] -->|has_many| F

Loading

Two things in that diagram aren't obvious from the model names:

• A CSV file isn't a batch. One uploaded file can produce several Freereg1CsvFile documents: one per distinct (county, place, church, register_type) found in its lines. They all share userid + file_name, and most file-level operations (lock, force_unlock, change_owner_of_file, merge_batches) loop over every batch with that pair. See CSV-Processing-Pipeline.
• SearchRecord is a separate, denormalised copy, not a view over entries. It carries its own place_id, chapman_code, location_names, names and dates. Anything that changes the location or the contents of an entry must also rewrite its search record, and several code paths do that with their own bulk updates. See Gazetteer-Place-Church-Register.

Lifecycle of a record

sequenceDiagram
    participant T as Transcriber
    participant W as Web (CsvfilesController)
    participant Q as PhysicalFile queue
    participant P as rake build:freereg_new_update
    participant DB as MongoDB
    participant R as Researcher

    T->>W: upload CSV
    W->>Q: PhysicalFile waiting_to_be_processed = true
    W->>P: Kernel.spawn rake (fire-and-forget)
    P->>DB: Freereg1CsvFile(s) + Freereg1CsvEntry per line
    P->>DB: SearchRecord per entry (if enough names)
    P->>DB: Register/Church/Place counters recalculated
    P-->>T: email: success / failure log
    R->>W: search form (SearchQuery)
    W->>DB: find on search_records with index hint
    W-->>R: results (embargoed filtered out)

Loading

Vocabulary you'll meet in the code

• Chapman code: three-letter county code (SOM, KEN, WRY…). Stored as chapman_code on Place/SearchRecord, but as county on Freereg1CsvFile and Freereg1CsvEntry (the CSV header calls it county). lib/chapman_code.rb holds the mapping.
• Record type: ba (baptism), ma (marriage), bu (burial): lib/record_type.rb.
• Register type: the kind of source document: PR parish register, BT bishop's transcript, AT archdeacon's transcript, TR, EX, MI, DW, PH, OD, OR, UK (unknown) and ' ' (unspecified; a literal single space for historical reasons). See lib/register_type.rb.
• Batch: a Freereg1CsvFile. Coordinators and the UI say "batch" and "file" interchangeably, but in the code they're different things (see above).
• UCF: "uncertain character format": transcriber notation for letters that can't be read, e.g. Sm_th, Jo[hn], Wil*. See Search-Record-Indexing.
• Emendation: forename expansion rules (e.g. Wm → William) applied at indexing time.
• DEF / flexible format: a CSV whose column order is declared in its own header rather than fixed. See CSV-Processing-Pipeline.

Cross-cutting patterns worth knowing

• Counts are denormalised and recalculated on write. records, datemin, datemax, daterange, transcribers, last_amended live on Freereg1CsvFile, Register, Church and Place, and are rebuilt bottom-up (calculate_register_numbers → calculate_church_numbers → calculate_place_numbers) after processing or an online edit. They are stored as strings (field :records, type: String), so every consumer calls .to_i. If the numbers on a contents page look wrong, rerun freereg:calculate_freereg_content before you go looking for a bug. See FreeREG-Content-and-Statistics.
• Long work goes to Kernel.spawn + rake, coordinated by files in tmp/. There's no job queue (see Architecture). CSV processing, embargo application and batch deletion each use a lock file or an append-only list file under tmp/ or a path from mongo_config.yml. See FreeREG-Operations-and-Rake-Tasks.
• Methods return [success, message] tuples rather than raising (merge_places, relocate_church, remove_batch, change_name…). Controllers redirect_back with the message on false. Several of these return early after partial writes, and none run in a MongoDB transaction.
• Hand-written bulk updates bypass callbacks. Propagation code uses SearchRecord.collection.find(...).update_many(...) with explicit .hint(...) index names. Renaming or dropping an index can make these raise, and they skip Mongoid validations and callbacks by design.