Changes to text index not saved correctly

[reboot]

I had a problem that sometimes when I open a database and search for something using a text index in a repository, the entry was not found also it exists. I investigated the problem and found out that the corresponding entries in the text index did not exist. After that I did a rebuild of the index after opening the database and before I did the search and the entry was found. But when I ran the test program again without reindexing the entry was not found again. So it seems that the changes to the index were not saved in the database.

I did some research and I think the problem is in the NitriteTextIndexingService. The loop that updates the database ids for a word looks like this:

for (String word : words) {
    ConcurrentSkipListSet<NitriteId> nitriteIds = indexMap.get(word);

    synchronized (indexLock) {
        if (nitriteIds == null) {
            nitriteIds = new ConcurrentSkipListSet<>();
            indexMap.put(word, nitriteIds);
        }
    }

    nitriteIds.add(id);
}

It gets the nitriteIds-Set from the store and adds new entries if it already exists. If MVStore persists the data page for some reason that contains the set this seems to be no problem. But if there are no other changes to the page then it does not detect the change in the set, and the modified set is not persisted in the database so the changes are lost. I did change the loop like this:

for (String word : words) {
    ConcurrentSkipListSet<NitriteId> nitriteIds = indexMap.get(word);

    synchronized (indexLock) {
        if (nitriteIds == null) {
            nitriteIds = new ConcurrentSkipListSet<>();
        }
    }

    nitriteIds.add(id);
    indexMap.put(word, nitriteIds);
}

and that seemed to remove the problem. The index was correctly persisted after reindexing and I could find the database entry even without reindexing the database after opening it. I also noticed that sometimes if i have some application running for a longer time that adds new entries from time to time, these entries were not found. I had to restarted the application with reindexing to find it. This could be related that the data was not persisted and removed from memory and then the old state was reloaded on the next access.

I'm not sure if this is the right way to fix it, but I wanted to report the problem at least.

[anidotnet]

Have you called commit to save the database while closing the application? Or have you somehow disabled auto commit?

[anidotnet]

On further investigation I see, you are right. IndexingService is using the same logic you used.

// create the id list associated with the value
ConcurrentSkipListSet<NitriteId> nitriteIdList = indexMap.get((Comparable) fieldValue);
if (nitriteIdList == null) {
    nitriteIdList = new ConcurrentSkipListSet<>();
}

if (index.getIndexType() == IndexType.Unique
        && nitriteIdList.size() == 1) {
    // if key is already exists for unique type, throw error
    throw new UniqueConstraintException(errorMessage(
            "unique key constraint violation for " + field,
            UCE_BUILD_INDEX_CONSTRAINT_VIOLATED));
}

// add the id to the list
nitriteIdList.add(entry.getKey());
indexMap.put((Comparable) fieldValue, nitriteIdList);

I'll give a quick fix for text indexing service.

[reboot]

Yes I have called commit and tried with and without autocommit.

I think the problem is, that MVStore.storeNowTry() checks all MVMaps if they have been modified:

for (MVMap<?, ?> m : list) {
    ...
    long v = m.getVersion();
    ...
    if (v >= 0 && v >= lastStoredVersion) {
        MVMap<?, ?> r = m.openVersion(storeVersion);
        if (r.getRoot().getPos() == 0) {
            changed.add(r);
        }
    }
}

and the later calls p.writeUnsavedRecursive(c, buff); on the root page of these MVMaps. If you just modify the Set that is stored in the values array of the Page object this will never set the version on the Page. This is done by the MVMap#put method. So when the database is closed some pages still have the default version of -1 and are not saved.

[anidotnet]

Right you are. That is the root cause of not saving the changes on set. Any way this fix will be available in 3.1.0-SNAPSHOT once the ci build is complete.