Reindex stale search records

[jorgemanrubia]

Summary

• Adds a runner script to backfill search records for cards/comments that became invisible after the search table rebuild on 2025-11-13
• Migration 20251113190256 dropped the old search_index_* tables and created empty search_records_* tables without data migration, so all cards not created/updated since then are missing from the search index
• Unlike search:reindex, this script only upserts missing records — it doesn't clear and rebuild everything

bin/rails runner scripts/reindex_stale_search_records.rb              # default cutoff: 2025-11-13
bin/rails runner scripts/reindex_stale_search_records.rb 2026-01-01   # custom cutoff

Bug: https://3.basecamp.com/2914079/buckets/27/card_tables/cards/9782824728

[Copilot]

Pull request overview

Adds a one-off runner script intended to backfill missing Search::Record entries for cards/comments that became unsearchable after the search table rebuild on 2025-11-13.

Tip

If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

Changes:

• Add scripts/reindex_stale_search_records.rb to reindex cards and comments before a cutoff date.
• Provide CLI usage for default vs custom cutoff dates.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The cutoff logic is exclusive ("updated_at < ?" with a Date parsed to midnight). For a default cutoff of 2025-11-13 this will skip records updated on 2025-11-13, even though they were created/updated before the search table rebuild on that date and can still be missing from the index. Consider treating the cutoff as an inclusive day (e.g., compare against cutoff + 1.day) or accept/parse a full timestamp and compare against that timestamp.

[Copilot]

Selecting cards by updated_at risks missing pre-rebuild cards whose updated_at has been bumped by comment activity (Comment belongs_to :card, touch: true), since a touch triggers Card's after_touch but not Searchable's after_update_commit reindexing. Those cards can still be missing search records while having updated_at >= cutoff. Using created_at as the primary filter (or explicitly detecting missing Search::Record entries) would avoid leaving some stale cards un-reindexed.

[Copilot]

This repository already has a script/ directory (including script/maintenance/ and script/migrations/) for operational/one-off scripts. Introducing a separate top-level scripts/ directory makes it easy to miss scripts and splits conventions (shebang + require_relative "../config/environment" vs rails runner). Consider moving this into the existing script/maintenance/ area (or aligning with that structure) to keep script discovery consistent.

[Copilot]

Date.parse will raise on invalid input (and also loses the time-of-day/time-zone information that matters for a migration timestamp). It would be safer to validate the argument and print a short usage message (or default cleanly) when parsing fails, and consider using Time.zone.parse if you want the cutoff to be time-zone aware.

Suggested change

	cutoff = Date.parse(ARGV[0] || "2025-11-13")
	default_cutoff = Time.zone.parse("2025-11-13")
	cutoff_arg = ARGV[0]
	cutoff =
	if cutoff_arg.nil?
	default_cutoff
	else
	begin
	parsed_cutoff = Time.zone.parse(cutoff_arg)
	rescue ArgumentError, TypeError
	parsed_cutoff = nil
	end
	if parsed_cutoff.nil?
	warn "Invalid cutoff: #{cutoff_arg.inspect}"
	warn "Usage: bin/rails runner scripts/reindex_stale_search_records.rb [YYYY-MM-DD or timestamp]"
	exit 1
	end
	parsed_cutoff
	end

[Copilot]

Same as cards: using an exclusive < cutoff comparison with a Date parsed to midnight will skip comments updated on the cutoff date. If the intent is to backfill everything created/updated on or before 2025-11-13, the query should treat the cutoff as inclusive (or accept a full timestamp).

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

indexed_card_ids is not scoped to the current account. Since search_records_* tables store multiple accounts per shard, this will pluck IDs for all accounts on the shard, which is both extremely expensive and can cause incorrect “missing” detection. Filter by account_id: account.id when collecting existing records (and similarly for comments below).

[Copilot]

indexed_comment_ids is not scoped to the current account, so it will pluck comment IDs for every account on the same search shard. This can blow up memory/SQL size and distort the missing set. Filter by account_id: account.id here as well.

[Copilot]

Plucking all indexed IDs into Ruby arrays (pluck(:searchable_id)) can generate huge arrays and very large WHERE NOT IN (...) clauses for big accounts. A more scalable approach is to use a subquery (e.g., where.not(id: shard.where(...).select(:searchable_id))) so the DB can evaluate it without materializing all IDs in memory.

[Copilot]

The usage comment points to scripts/reindex_stale_search_records.rb, but this file lives under script/maintenance/. This is likely to cause the wrong script to be run (and there are currently two similarly named scripts). Update the usage line to the correct path, and consider consolidating to a single script location to avoid operator confusion.

[Copilot]

Same scalability concern for comments: pluck(:searchable_id) materializes all indexed comment IDs into Ruby and can lead to huge memory use / SQL NOT IN lists. Prefer a DB subquery for the exclusion set to keep the work in SQL.