FTS fails (any everything crashes with it) if value of the field is not present

[gotenxds]

[X ] Bug report

Hya again ^^

When adding an fts field if the field is optional in the collection, fts will fail on insert, this will cause the insert to fail and stop at the last item inserted.

for example, let's say I have an Item collection and I want to be able to search in the field group but this field is optional, i.e sometimes it has a value, sometimes null

When fts tokenizes the field it will try to split it by whitespaces

LokiDB/packages/full-text-search/src/analyzer/tokenizer.ts

Line 12 in 8b99821

return value.split(/[\s]+/);

and fails, as null.split is not a function

Edit:
After testing for 1 more minute 🤦
I found the offending line:

LokiDB/packages/full-text-search/src/full_text_search.ts

Line 49 in 8b99821

if (doc[fieldName] !== undefined) {

Changing this to

if (doc[fieldName] !== undefined && doc[fieldName] !== null) { ...

fixes the issue

Expected behavior

If a field value is not present, that is null or undefined, skip.
Maybe even check for more types and throw/convert to a string?
Number => toString ? throw invalid type error? expected string got X

Minimal reproduction of the problem with instructions

Something like this should work:

//Initialize DB
    FullTextSearch.register()
    IndexedStorage.register()
    const lokiDb = new Loki('notAMimic')

// initialize collection
   db.addCollection<T>(collectionName, {
   fullTextSearch: [
        { field: 'name' },
        { field: 'group' },
        { field: 'category' },
      ],
  })

// add data
collection.value.insert([
{ name: 'Sword of madness', group: 'sword', category: 'Martial' },
{ name: 'Club', group:'club', category: 'simple' },
{ name: 'Shield of awesome', group: null, category: 'to lazy to come up with something' } // Cry here
])

[Viatorus]

Have a look. #181