Gigamap: when is DocumentPopulator called?

[bachlge]

I am wondering, what triggers the DocumentPopulator to be executed.

I created some tests and these are my observations:

Test 1:

1. Create an Entity
2. Add some 1:N Details (Tags) to the Entity
3. Add Entity to the GigaMap —> DocumentPopulator is called
4. Call EntityManager.store(DB.data().entities());

EclipseStore has Entity and Details
Index contains Field from Entity and Field from Details

Test 2:

1. Create an Entity
2. Add Entity to the GigaMap —> DocumentPopulator is called
3. Add some 1:N Details to the Entity
4. Call EntityManager.store(entity.getDetails());
5. Call EntityManager.store(DB.data().entities());

EclipseStore has Entity and Details
Index contains Field from Entity BUT NOT and Field from Details

Test 3:

1. Update Entity Text Field
2. Call EntityManager.store(entity);
3. Call EntityManager.store(DB.data().entities());

EclipseStore has Entity and Details
Index DOES NOT contain the updated field

I Cannot figure out how to keep Lucene in sync with EclipseStore.

public final GigaMap<DummyEntity> entities = GigaMap.<DummyEntity>Builder().build();

Path path = Paths.get("data", "lucene", this.getClass().getSimpleName());
DummyDocumentPopulator documentPopulator = new DummyDocumentPopulator();
LuceneContext<DummyEntity> luceneContext = LuceneContext.New(path, documentPopulator);

LuceneIndex<DummyEntity> luceneIndex = entities.index().register(LuceneIndex.Category(luceneContext));

[hg-ms]

Did you try to use gigamap.update() to update the indices before storing?

// use the update method, to automatically propagate the changes to the index
gigaMap.update(person, p -> {
    p.setLastName("Smith"),
    p.setAddress(newAddress)
});

gigamap.store();

please see https://docs.eclipsestore.io/manual/gigamap/crud.html

[bachlge]

Wanting to use gigamap.update() forced me to create a (non-Lucene)index on my Data Object to overcome an IllegalStateException, but then did the job as you suggested.

Still open question: is update() also the recommended way to add the 1:N Details?
As a result it worked.
However, looking into the log, both

1. DB.data().dummies().add(dummy2); and
2. DB.data().dummies().entities.update(dummy2, dd -> { dd.tags.add(new DummyTag("one")); dd.tags.add(new DummyTag("two")); dd.tags.add(new DummyTag("three")); });

do call the DocumentPopulator immediately.

I'd have expected that the store()-operation would do this.
The experienced behaviour does not read transactional like save it when I tell you to save.

Please correct me, if I read it wrongly and it is transactional and does update the same document.

Interesting side note - the Tags (a Set) are stored "upside down" in EclipseStore for the Entities where update() had been applied and again in a different sequence in the object handed over to the DocumentPopulator 😏

Dummy one 2026-06-17T09:57:50.038738 | one, two, three
Dummy two 2026-06-17T09:57:50.038738 | three, two, one
modified Dummy three 2026-06-17T09:57:50.038738 | three, two, one

Dummy2
Dummy2 - add to GigaMap
Document populator called for Dummy two 2026-06-17T09:20:21.142295 with tags []
Dummy2 - Tags
Document populator called for Dummy two 2026-06-17T09:20:21.142295 with tags [three, one, two]
Dummy2 - store

        	// Test Set 2: Create Dummy -> add Dummy -> add Tags -> store
        	LOGGER.info("Dummy2");
        	DummyEntity dummy2 = new DummyEntity(2L, "Dummy two " + date.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME));
        	LOGGER.info("Dummy2 - add to GigaMap");
    		DB.data().dummies().add(dummy2);
        	LOGGER.info("Dummy2 - Tags");
        	DB.data().dummies().entities.update(dummy2, dd -> {
				dd.tags.add(new DummyTag("one"));
				dd.tags.add(new DummyTag("two"));
				dd.tags.add(new DummyTag("three"));
			});
        	LOGGER.info("Dummy2 - store");
    		DB.data().dummies().entities.store();

public class DummyDocumentPopulator extends DocumentPopulator<DummyEntity> {

	private final static Logger LOGGER = LoggerFactory.getLogger(DocumentPopulator.class);

	@Override
	public void populate(Document document, DummyEntity dummy) {

		LOGGER.info("Document populator called for " + dummy + " with tags " + dummy.getTags());

		document.add(createTextField("name", dummy.getName()));
		for (DummyTag tag : dummy.getTags()) {
			document.add(createTextField("tag", tag.getName()));
		}
	}
}

[fh-ms]

Thanks for the detailed write-up @bachlge - you've actually surfaced several real problems, not just a usage misunderstanding. I dug into the code and you're right on most of it. Here's what's going on and what we're tracking.

Why your three tests behave differently

The root cause is that EclipseStore has no dirty tracking, and a GigaMap index (Lucene included) is only ever refreshed through the GigaMap mutation methods - add / update / apply / replace / remove. The DocumentPopulator runs inside those calls, not at store() time:

• internalAdd / internalUpdateIndices call documentPopulator().populate(...) synchronously, and with the default autoCommit=true the Lucene writer is flushed and committed right then.
• store() only persists the index's storage blob; it never re-runs the populator.

So:

• Test 1 works because the Details existed at add() time, when the document was populated.
• Test 2 leaves Detail fields out because the document was populated at add() (Details still empty); mutating + store() afterwards never re-populates it.
• Test 3 never re-indexes because a plain field mutation + store() doesn't go through any mutation method.

gigaMap.update(entity, e -> …) fixes it because that's the one sanctioned path that re-runs the populator.

The parts that are genuine problems (now tracked)

You shouldn't have had to add a non-Lucene index just to get update() to work, and "save it when I tell you to save" is a reasonable expectation that today's behavior doesn't fully meet. We've filed these as #713–#716:

1. update()/apply() require a bitmap index. They resolve the entity id via a bitmap-only lookup that throws IllegalStateException if no bitmap index is registered - so a Lucene-only (or jVector-only) map has no legal way to re-index, which is exactly why you had to add the extra index. This was not user error; the API left no other option.
2. Silent desync on mutate-then-store(). For bitmap indexes a desync often surfaces as an exception; for Lucene it silently returns stale results, which is the worst failure mode. At minimum this needs to be documented prominently; ideally detected.
3. Not transactional for external directories. With the default autoCommit=true and an MMap(path) directory, the Lucene commit hits disk immediately at update() time, decoupled from store(). (With the default embedded directory it's only persisted on store(), so disk stays consistent at store boundaries - but the in-memory index still changes immediately.) We're looking at coupling the commit to store() so you can get genuine store-boundary transactionality.
4. Misleading Javadoc. update()'s Javadoc says index changes are "automatically persisted when store() is called," which isn't accurate for an external-directory Lucene index under the default config.

The one I'd set aside

The Set element ordering you noticed is just HashSet iteration nondeterminism in the populator - field order within a Lucene document doesn't affect indexing or search results, so that one's cosmetic rather than a correctness issue.

Thanks again for the thorough report - it directly led to four concrete issues. We'll follow up on those.