Back-fill the Lucene index when registered on a populated GigaMap

[fh-ms]

Problem

A Lucene full-text index registered on a GigaMap that already contained entities only indexed entities added after registration — pre-existing entities were silently invisible to full-text search. The bitmap and JVector indices already back-fill from existing entities on registration; Lucene did not, which is both an inconsistency and a silent correctness gap (queries miss data).

This also affected the annotation-driven path (@FullText / LuceneAnnotationHandler / IndexerGenerator.generateIndices), which is user-callable on a populated map.

Change

• Add a default no-op lifecycle hook internalOnRegistered() to IndexGroup.Internal, invoked by GigaIndices.register(...) immediately after a group is added. It is not called on deserialization (groups are reconstructed via their binary handlers), so there is no re-indexing / double-indexing on restart.
• LuceneIndex overrides the hook to index all existing entities via iterateIndexed, using the real GigaMap-assigned entity ids (so later update/remove, which look up by id, keep working). Documents are added incrementally and committed once at the end, honoring the autoCommit contract and skipping an empty commit on an empty map.
• Bitmap and JVector groups inherit the no-op, so registering the (empty) VectorIndices container does not trigger a wasteful full-map scan; its existing per-index back-fill is unchanged.
• Extracted a shared toDocument(entityId, entity) helper, now the single document-construction point used by internalAdd, internalAddAll, internalUpdateIndices, and the back-fill.

Registration order no longer matters: an index can be added before or after data is loaded.

Why a hook (not a generic back-fill in register)

A generic iterateIndexed in register would force a full-map scan (loading every Lazy segment) just to register the empty VectorIndices container. Doing it in createIndexGroup would scan the map even for a discarded duplicate group (the already-registered check runs after creation). The hook lets each group opt in; only Lucene needs it.

Tests

New LuceneBackfillTest:

• existing entities are indexed on registration
• register on empty map, then add still indexes (no regression / no empty commit)
• back-filled entities carry correct ids for update/remove
• persistence round-trip: back-fill persists and is not double-indexed on reload
• manual-commit (autoCommit=false) context

All module suites pass: gigamap (1034), lucene (58), jvector (189).

Docs

Updated the Lucene index and annotation pages to state that registering on a populated GigaMap back-fills existing entities, with a note about the one-time full-entity scan on very large maps.

[Copilot]

Pull request overview

This PR fixes a correctness gap where a Lucene full-text index registered on an already-populated GigaMap would only index entities added after registration, by adding a post-registration lifecycle hook and using it to back-fill existing entities into the Lucene index. It also adds tests and documentation updates to cover and explain the new behavior.

Changes:

• Add IndexGroup.Internal#internalOnRegistered() (default no-op) and invoke it after successful group registration.
• Implement Lucene back-fill on registration via iterateIndexed, and refactor document creation into a shared toDocument(...) helper.
• Add LuceneBackfillTest and update Lucene indexing docs to state that registration on populated maps back-fills existing entities.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
gigamap/lucene/src/test/java/org/eclipse/store/gigamap/lucene/LuceneBackfillTest.java	New tests validating Lucene back-fill behavior, id correctness for update/remove, and persistence/manual-commit scenarios.
gigamap/lucene/src/main/java/org/eclipse/store/gigamap/lucene/LuceneIndex.java	Adds Lucene back-fill on registration and centralizes document construction in toDocument(...).
gigamap/gigamap/src/main/java/org/eclipse/store/gigamap/types/IndexGroup.java	Introduces the internalOnRegistered() lifecycle hook (default no-op) for index groups.
gigamap/gigamap/src/main/java/org/eclipse/store/gigamap/types/GigaIndices.java	Calls internalOnRegistered() immediately after registering a new index group.
docs/modules/gigamap/pages/indexing/lucene/index.adoc	Documents that registering Lucene on a populated map back-fills existing entities and may require a one-time scan.
docs/modules/gigamap/pages/indexing/lucene/defining.adoc	Documents that annotation-generated Lucene indices are also back-filled when generated on populated maps.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.