searchWithHighlight crashes after persistence restore

[nekdolan]

Describe the bug
When db is restored using data persistence plugin the search with higlight functionality no longer works

To Reproduce
Steps to reproduce the behavior:

• Add presistence plugin
• Add match highlight plugin
• Insert documents
• call persist
• call restore
• call searchWithHighlight

Expected behavior
not crash with error

Screenshots

const db = await create({
        schema: {
            // id: 'string',
            text: 'string',
            name: 'string',
            type: 'string',
            year: 'number',
            source: 'string',
            author: 'string'
        },
        components: {
            // Register the hook
            afterInsert: [highlightAfterInsert],
            tokenizer: {
                stopWords: englishStopwords,
                stemming: true,
                stemmer
            },
        }
    })

TypeError: Cannot read properties of undefined (reading '21650872-55')          1:49:20 PM
    at file:///Users/trinimac/WebstormProjects/lazarites/node_modules/@orama/plugin-match-highlight/dist/index.js:52:78
    at Array.map (<anonymous>)
    at searchWithHighlight (file:///Users/trinimac/WebstormProjects/lazarites/node_modules/@orama/plugin-match-highlight/dist/index.js:51:30)
    at async cachedEventHandler.swr (file:///Users/trinimac/WebstormProjects/lazarites/.nuxt/dev/index.mjs:5363:21)
    at async file:///Users/trinimac/WebstormProjects/lazarites/.nuxt/dev/index.mjs:1584:20
    at async _resolve (file:///Users/trinimac/WebstormProjects/lazarites/.nuxt/dev/index.mjs:1445:23)

Desktop (please complete the following information):

• NodeJS Nuxt 3 server

[allevo]

Hi, thank you for opening this issue!
Which version of Orama are you using for dumping and searching?
Which version of Orama persistence & highlight plugins are you using?
I suggest you to use the same version everywhere

[micheleriva]

Hi @nekdolan any update on the above questions from @allevo above?

[benjaminleonard]

I am seeing this too, not with Orama persistence but I have my own code for retrieving it from json.

"@orama/plugin-match-highlight": "^1.2.1",
"@orama/orama": "^1.2.1",

[benjaminleonard]

I believe the issue might be in the save() function. If I don't use that I can see "positions" in the output, but it's not in what is returned from save()

Looks like it's missing from here:

orama/packages/orama/src/methods/serialization.ts

Lines 20 to 28 in fc6e863

	export async function save(orama: Orama): Promise<RawData> {
	return {
	internalDocumentIDStore: orama.internalDocumentIDStore.save(orama.internalDocumentIDStore),
	index: await orama.index.save(orama.data.index),
	docs: await orama.documentsStore.save(orama.data.docs),
	sorting: await orama.sorter.save(orama.data.sorting),
	language: orama.tokenizer.language,
	}
	}

[benjaminleonard]

Perhaps something like this within @orama/plugin-match-highlight:

export type RawDataWithPositions = RawData & {
  positions: Record<string, Record<string, Record<string, Position[]>>>
}

export async function saveWithHighlight(
  orama: OramaWithHighlight,
): Promise<RawDataWithPositions> {
  const data = await save(orama)

  return {
    ...data,
    positions: orama.data.positions,
  }
}

Edit: I figured that would fix the issue but I am still getting the error even when positions is present

[benjaminleonard]

I think I have it working, but there seems to be an issue in the load() function. It's not adding the positions object. My example works with:

export async function loadWithHighlight(
  orama: OramaWithHighlight,
  raw: RawDataWithPositions,
): Promise<void> {
  await loadOramaDB(orama, raw)
  orama.data.positions = raw.positions
}