FreeREG Operations and Rake Tasks

FreeREG Operations and Rake Tasks

Part of FreeREG-Core-Records.

FreeREG has no job queue (Architecture). Background work is done by rake tasks, either spawned from a web request with Kernel.spawn or run from cron via the shell scripts in lib/tasks/scripts/. They coordinate through files: lock files in tmp/ and append-only list files whose paths come from config/mongo_config.yml. This page lists the ones that matter for FreeREG and the files they share.

lib/tasks/ has ~200 rake files. Most are one-off migrations, repairs or reports (correct_*, check_*, extract_*, update_*), and many are FreeCEN-only. The ones below are the ones in regular use.

Scheduled (cron) jobs

The production crontab itself isn't in the repo, but the scripts it calls are:

Script	Runs	Purpose
update_freereg2_production.sh	build:freereg_new_update[create_search_records,waiting,no_force,a-9]	Safety net: drains the processor queue in case a spawned processor died
	extract_county_stats	county statistics report
	freereg:calculate_freereg_content	rebuild all Register/Church/Place counters (FreeREG-Content-and-Statistics)
	foo:check_and_refresh_places_cache	rebuild any county PlaceCache that's empty but has places with data
	delete_file_no_sleep[0]	work through the delete list: destroy entries + search records of removed batches
	freereg:bounced_emails_set_invalid[fix]	mark bounced email addresses on UseridDetail
calculate_site_statistics.sh	freereg:calculate_site_statistics	daily SiteStatistic
calculate_search_statistics.sh	freereg:calculate_search_queries	daily SearchStatistic
delete_old_search_queries.sh	build:delete_search_queries[0]	prune search_queries older than days_to_retain_search_queries
add_embargo_records.sh	foo:add_embargo_record[0]	apply embargo rules for registers in the embargo list (Embargoes-and-Gaps)
update_unique_names.sh	reports:extract_collection_unique_names[0]	rebuild PlaceUniqueName/ChurchUniqueName/RegisterUniqueName
freereg_monthly_cron.sh	freereg:clean_ucf_place_list (lib/tasks/ucf/)	repair every place's ucf_list (Search-Record-Indexing)
check_freereg_processor_queue.sh	check_length_processor_queue[10]	email an alert when ≥ 10 files are waiting
send_upload_report_freereg.sh	roles_upload_report	upload activity report
delete_old_messages.sh	foo:delete_or_archive_old_messages_feedbacks_and_contacts	see Communication-and-Engagement

Shared files

File	Written by	Read (and truncated) by	Contents
tmp/processor_initiation_lock_file.txt	Csvfile#process_the_batch before spawning	processor, removed at end	"a processor has been requested"
tmp/processing_rake_lock_file.txt	first build:freereg_new_update (waiting mode)	same task; flock held while draining	single-drainer lock (CSV-Processing-Pipeline)
config.delete_list	Freereg1CsvFile#remove_batch	delete_file_no_sleep	batch_id,userid,file_name per removed batch
config.register_embargo_list	EmbargoRule create/update/destroy callbacks	foo:add_embargo_record	register_id,timestamp
tmp/<rule_id>_rake_lock_file.txt	foo:process_embargo_records	same	per-rule embargo run lock
tmp/embargo_lock_file.txt	foo:add_embargo_record	same	scheduled embargo run lock
tmp/cleanup_lock_file.txt	freereg:clean_up_processed_batch	same	reprocessing clean-up lock
<datafiles>/<userid>/<file>	upload, download, online edit	processor	the CSV files themselves (config.datafiles)
<datafiles>/<userid>/.attic/	save_to_attic	nobody (manual)	previous versions of files
log/update_freereg_messages_*.log	processor	people	processor project log
log/delete_list_processing.txt	delete_file_no_sleep	people	what was deleted

Every list file is read and then truncated. If a task crashes after truncation and before finishing, the unprocessed lines are lost. The batch documents are already gone at that point (removal uses delete), so the leftover entries become true orphans. rake extract_orphans / delete_orphans and foo:check_and_delete_orphan_records[limit,sleep,fix] find and remove them.

Troubleshooting runbook

"My file has been waiting for hours"

1. PhysicalFile.waiting.count: how long is the queue? (check_length_processor_queue logs this to log/processor_queue_length.log.)
2. Is a processor running? ps aux | grep freereg_new_update. Remember it sleeps 100 s between files and 300 s between queue passes by design.
3. Is tmp/processing_rake_lock_file.txt present with no process running? Remove it and run rake build:freereg_new_update[create_search_records,waiting,no,a-9] by hand. The cron run of update_freereg2_production.sh does the same.
4. Check log/update_freereg_messages_*.log for the last file started. A file that crashes the processor gets its failure email and is removed from the queue (clean_up_physical_files_after_failure), so it won't block the files behind it.

"I removed a batch but its records still show" Search hides them as soon as it finds them (the broken chain check in SearchQuery#persist_results), but they're only really gone after delete_file_no_sleep runs. Check the delete list file for the batch id.

"The contents page numbers are wrong" rake freereg:calculate_freereg_content, or reprocess the batch. For a single place in the console: place.churches.each { |c| c.registers.each(&:calculate_register_numbers); c.calculate_church_numbers }; place.calculate_place_numbers.

"A place is missing from the search form's place list" It needs data_present: true and disabled: "false". Then PlaceCache.refresh(chapman_code).

"Search error: hint provided does not correspond to an existing index" An index named in SearchRecord::REG_*_INDEXES doesn't exist in this database. Create it from doc/design/indexes/search_records.reg (locally: bin/dev-create-search-indexes, see Development-Setup). The user sees this as "your search is taking too long".

"Embargo released but the record is still hidden" (or the reverse) Check the entry's last embargo_records element and its search record's embargoed/release_year. Reprocess with "Process" on the rule, or foo:add_embargo_record. Also check the rule's period_type (see Embargoes-and-Gaps).

Reprocessing

• One file: rake build:freereg_new_update[create_search_records,individual,no,<userid>/<file_name>]. Add force_rebuild as the third argument to skip lock checks and digest reconciliation (every line is recreated, so search record ids change).
• A county: rake freereg:reprocess_batches_for_a_county[CHAPMAN], followed by freereg:clean_up_processed_batch[CHAPMAN] (lib/tasks/reprocess_batches_for_a_county.rake). freereg:reprocess_marriage_search_records_scan is a resumable variant for marriages only, with a checkpoint file in tmp/.
• Search records only (entries unchanged, e.g. after changing the translator or emendation rules): rake foo:update_search_records[limit,record_type,version,force,order] (lib/update_search_records.rb). The search_record_version/software_version stamped on batches and search records (from SoftwareVersion) records which code generation built each record.
• Emendations: load_emendations (seeds the rule set defined in lib/tasks/load_emendations.rake), apply_emendations, correct_all_emendations. The rule cache is per process, so restart the web server after changing rules.
• Location string format: rake backfill_location_names_delimiter[chapman_codes,limit,fix] converts legacy "Place (Church)" strings to "Place ||| Church" (dry run unless the last argument is fix).