Can't get persistence working in node.

[benoneal]

I'm trying to set up loki in node. I've followed through the various documentation around this. The docs are sometimes inconsistent with regards to things like adapters, and some things are never explained.

For example, when constructing loki, the docs always pass in a string, like const db = new loki('loki.json') or something. Apparently this is "where data is persisted"? So some observations I've run into while trying to get this working:

const db = new loki('../data/db.json') > Error: no such file or directory c:/data/db.json
const db = new loki(path.resolve(__dirname, '../../data/db.json')) > Error: no such file or directory c:/code/app/data/db.json

So it needs to be an absolute path. The docs never mention this, and indeed all examples appear to be a relative path. Also the file needs to exist. The docs never mention this.

When I create a file in that location, even an empty file, loki will "work". I can add collections, insert items, and find and return them. Here's an example from my testing:

import path from 'path'
import loki from 'lokijs'

const fileStorage = path.resolve(__dirname, '../../data/db.json')
const db = new loki(fileStorage)

const testCol = db.addCollection('test', {unique: ['slug'], indices: ['slug']})
testCol.insert({slug: 'test1', test: 'This works'})

console.log(testCol.find({slug: 'test1'}))
// returns [{slug: 'test1', test: 'This works'}]

However, nothing is persisted to db.json. No matter how long I leave the process running, db.json remains completely empty. Reading further into the docs, I find reference to some options that need to be supplied.

const db = new loki(fileStorage, {autosave: true, autosaveInterval: 5000})

I have no idea why loki would throw an error if you don't supply a valid path to an existing file, if it's not going to use it anyway without explicit options... that seems weird. What's weirder, is that even supplying these options doesn't appear to do anything. No matter how long the process is left running, nor how many things items I insert into a collection, db.json is never changed. Nothing is being persisted.

So I try something else. What if db.json was pre-seeded with some data? After all, in production, this will be the norm after the first boot. But nowhere in the docs does it mention anything about the structure of the saved file. I had to copy an example from the LEAN ejs stack.

Now I have a manually seeded db.json:

{
  "filename": "db.json",
  "collections": [
    {
      "name": "test",
      "data": [
        {
          "$loki": 1,
          "slug": "test1",
          "test": "This doesn't work :("
        }
      ],
      "idIndex": [
        1
      ],
      "binaryIndices": {},
      "constraints": null,
      "uniqueNames": [
        "test1"
      ],
      "transforms": {},
      "objType": "test",
      "dirty": true,
      "cachedIndex": null,
      "cachedBinaryIndex": null,
      "cachedData": null,
      "transactional": false,
      "cloneObjects": false,
      "cloneMethod": "parse-stringify",
      "asyncListeners": false,
      "disableChangesApi": true,
      "autoupdate": false,
      "ttl": {
        "age": null,
        "ttlInterval": null,
        "daemon": null
      },
      "maxId": 1,
      "DynamicViews": [],
      "events": {
        "insert": [null],
        "update": [null],
        "pre-insert": [],
        "pre-update": [],
        "close": [],
        "flushbuffer": [],
        "error": [],
        "delete": [null],
        "warning": [null]
      },
      "changes": []
    }
  ],
  "databaseVersion": 1.1,
  "engineVersion": 1.1,
  "autosave": true,
  "autosaveInterval": 5000,
  "autosaveHandle": null,
  "options": {
    "autoload": true,
    "autosave": true
  },
  "persistenceMethod": "fs",
  "persistenceAdapter": null,
  "verbose": false,
  "events": {
    "init": [null],
    "loaded": [],
    "flushChanges": [],
    "close": [],
    "changes": [],
    "warning": []
  },
  "ENV": "NODEJS"
}

So modifying my original code, I should, in theory, be able to read directly from the pre-existing data:

import path from 'path'
import loki from 'lokijs'

const fileStorage = path.resolve(__dirname, '../../data/db.json')
const db = new loki(fileStorage, {autosave: true, autosaveInterval: 5000})

const testCol = db.getCollection('test')
console.log(testCol.find({slug: 'test1'}))

// TypeError: Cannot read property 'find' of null

Wat.

After more digging, I find another option autoload, that must surely be the thing I need?

const db = new loki(fileStorage, {autosave: true, autosaveInterval: 5000, autoload: true})

Alas, it is not. Even with this option, it still throws TypeError: Cannot read property 'find' of null.

About now I'm starting to abandon hope of ever using Lokijs. It demands a valid path to an existing file, then completely ignores the file contents and never changes it. It will happily create and read from data in memory, but then all is lost when the process ends.

The docs mention something about adapters. They do state that Lokijs will detect when it is running in node, and use its LokiFsAdapter by default. Maybe that's the problem? They talk about using the alternate LokiFsStructuredAdapter as a way to manage larger data stores (which I anticipate I'll need), so I try switching over to that adapter in a last-ditch effort to get this working:

import path from 'path'
import loki from 'lokijs'
import lfsa from '../../node_modules/lokijs/src/loki-fs-structured-adapter' // this is a bit cumbersome...

const fileStorage = path.resolve(__dirname, '../../data/db.json')
const db = new loki(fileStorage, {autosave: true, autosaveInterval: 5000, adapter: new lfsa()})

// ...lokijs\src\loki-fs-structured-adapter.js:116 
// self.dbref = JSON.parse(line);
// SyntaxError: Unexpected end of JSON input

Oh look a new error. Worth mentioning, this documentation on persistence adapters says to pass the adapter in directly (var db = new loki('sandbox.db', { adapter : lfsa})), which gives a whole different error (TypeError: this.persistenceAdapter.loadDatabase is not a function) which is why I passed it in as an instance above.

This seems like a pretty monumental problem. Surely it's big enough that other people would be running into it. But there are no other issues even remotely relating to this problem! I have literally no idea what is the problem, and I feel like I've exhausted my options to solve this myself.

Environment:

• Windows 10
• Node.js v7.7.2
• Lokijs v1.4.3

I would love to use Lokijs, because it's the cleanest and most direct API to my data. Obviously, without being able to load existing data, or persist new data, it's a non-starter for me. I'd really appreciate any help or insight into this problem, so I can avoid falling back on postgresql or some other convoluted impedence mismatch nightmare.

[Mika83AC]

Hello @benoneal,

just now I start using LokiJS and had the same issues than you: no persistence.

I did what you mentioned here in your post (adding {autosave: true, autosaveInterval: 5000, autoload: true} to the new loki() function and everything is working immediately.

So I would guess, that you have a permission issue on your computer which prevents LokiJS from doing what you expect it to to. Maybe you should check this and try a temp directory path where everything is allowed, just to be sure.

Here is my extremely simple starter implementation of LokiJS which persists and autoloads. Maybe it can help you in any way:

const loki = require('lokijs');
const dbFile = new loki('db/data.json', {autosave: true, autosaveInterval: 5000, autoload: true});

const db = {
   set: function set(tablename, value) {
      let table = dbFile.getCollection(tablename);
      if(!table) table = dbFile.addCollection(tablename);

      table.insert(value);
   },
   query: function get(tablename, searchObj) {
       const table = dbFile.getCollection(tablename);
       if(!table) return [];

       return table.find(searchObj);
   }
}

module.exports = db;

Regards,
Michael

[Mika83AC]

@techfort The docs should be fine-tuned in the section about persistence as I think this issues here hits many people trying out LokiJS and maybe let them put it aside because it's not working like described which gives a bad "first impression" ;)

[kurdin]

@Mika83AC in your example above you don't use persistence adapter at all. So your collection won't be saved to disk.

I had the same problem as @benoneal I could not make lokijs works with loki-fs-structured-adapter to save collection to the disk. The solution was not to use autoload option at all and use databaseInitialize with callback then run Db.loadDatabase.

In case anyone need working code, this is how to make node.js and lokijs with loki-fs-structured-adapter work together as module.

const loki = require('lokijs');
const path = require('path');
const LokiFSStructuredAdapter = require('lokijs/src/loki-fs-structured-adapter');

let dbFile = path.resolve(__dirname, './test-data.json');

let Db = new loki(dbFile, 
  { 
    verbose: true,
    autosave: true, 
    // autoload: true,
    autosaveInterval: 1000,
    adapter: new LokiFSStructuredAdapter(),
    // autoloadCallback: databaseInitialize,
    autosaveCallback: () => { 
      console.log('autosaved db'); 
    }
  }
);

function databaseInitialize(cb) {
  let entries = Db.getCollection('test');

  if (entries === null) {
    console.log('test collection is empty, ADDING new collection test');
    entries = Db.addCollection('test', { unique: ['id'] });
    entries.insert({id: 1, test: 'test'});
    entries.insert({id: 2, test: 'test2'});
    entries.insert({id: 3, test: 'test3'});
  }
  cb();
}

const loadDb = (cb) => {
  databaseInitialize(callback => {
    Db.loadDatabase({}, function (err) {
      if (err) console.error('db load errors', err);
      cb({
        Db: Db,
        test: Db.getCollection('test')      
      });
    }); 
  });
};

// module.export = loadDb; 
// then use loadDb in other file like index.js

loadDb(({Db, test})=> {
  console.log('db loaded', Db);
  console.log('entries in db collection test', test.find());
});

[obeliskos]

We now have some quickstart examples in the examples folder which cover basic persistence setup. Web users can find equivalent web quickstart scripts in LokiSandbox now.

These (node) examples from the examples folder should be a good start :
quickstart2
quickstart3

[kurdin]

@obeliskos
Those example are good for quick testing of Loki on node, but in reality you would want to export the module with db init and callback it's when init is done in case you want to use Loki with express or any other web node's server's framework.

[joshuaquek]

I built a library around LokiJs that might address this issue.

https://www.npmjs.com/package/lokijs-promise

Decided to build this as quite a lot of the issues that I found in the Issues section revolved around loading the persistent store (eg. store.json / store.db etc) and then ensuring that the store was loaded before doing inserts etc, which then automatically writes to the persistent store (eg. store.json / store.db etc)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[thw0rted]

A note that could help future visitors here: I was trying to write a script that runs and then exits on completion, after saving changes to the database. If I set this up as

// ... load DB and make changes
db.on("close", () => process.exit(0));
db.close();

the changes wouldn't save, but they do save correctly if I instead write db.close(() => process.exit(0)). Looking at the source, I don't understand why this should be the case, but it seems to happen consistently. Be warned, I guess?

[StudentESE]

@Mika83AC

const loki = require('lokijs');
const path = require('path');
const LokiFSStructuredAdapter = require('lokijs/src/loki-fs-structured-adapter');

let dbFile = path.resolve(__dirname, './test-data.json');

let Db = new loki(dbFile, 
  { 
    verbose: true,
    autosave: true, 
    // autoload: true,
    autosaveInterval: 1000,
    adapter: new LokiFSStructuredAdapter(),
    // autoloadCallback: databaseInitialize,
    autosaveCallback: () => { 
      console.log('autosaved db'); 
    }
  }
);

loadDb(({Db, test})=> {
  console.log('db loaded', Db);
  console.log('entries in db collection test', test.find());
});

console.log(Db); shows autosave: false

persistenceMethod: 'adapter',
  persistenceAdapter: LokiFsStructuredAdapter {
    mode: 'reference',
    dbref: {
      filename: 'db.json',
      collections: [Array],
      databaseVersion: 1.5,
      engineVersion: 1.5,
      autosave: false,
      autosaveInterval: 5000,
      autosaveHandle: null,
      throttledSaves: true,
      options: [Object],
      persistenceAdapter: null,
      verbose: false,
      events: [Object],
      ENV: 'NA'
    },

How do I get it enabled ?
is there a path I can change the value of options (switching autosave on/off) while running?
like ...
db.persistenceAdaper['LokiFsStructuredAdapter'].dbref['autosave] = true;
also manually save the collection doesn't trigger the autosaveCallback ... but showing the content of the memory db says it gots new entries.

I like to autosave to json. Please help :-)

[bthornemail]

@Mika83AC

const loki = require('lokijs');
const path = require('path');
const LokiFSStructuredAdapter = require('lokijs/src/loki-fs-structured-adapter');

let dbFile = path.resolve(__dirname, './test-data.json');

let Db = new loki(dbFile, 
  { 
    verbose: true,
    autosave: true, 
    // autoload: true,
    autosaveInterval: 1000,
    adapter: new LokiFSStructuredAdapter(),
    // autoloadCallback: databaseInitialize,
    autosaveCallback: () => { 
      console.log('autosaved db'); 
    }
  }
);

loadDb(({Db, test})=> {
  console.log('db loaded', Db);
  console.log('entries in db collection test', test.find());
});

console.log(Db); shows autosave: false

persistenceMethod: 'adapter',
  persistenceAdapter: LokiFsStructuredAdapter {
    mode: 'reference',
    dbref: {
      filename: 'db.json',
      collections: [Array],
      databaseVersion: 1.5,
      engineVersion: 1.5,
      autosave: false,
      autosaveInterval: 5000,
      autosaveHandle: null,
      throttledSaves: true,
      options: [Object],
      persistenceAdapter: null,
      verbose: false,
      events: [Object],
      ENV: 'NA'
    },

How do I get it enabled ? is there a path I can change the value of options (switching autosave on/off) while running? like ... db.persistenceAdaper['LokiFsStructuredAdapter'].dbref['autosave] = true; also manually save the collection doesn't trigger the autosaveCallback ... but showing the content of the memory db says it gots new entries.

I like to autosave to json. Please help :-)

I

@Mika83AC

const loki = require('lokijs');
const path = require('path');
const LokiFSStructuredAdapter = require('lokijs/src/loki-fs-structured-adapter');

let dbFile = path.resolve(__dirname, './test-data.json');

let Db = new loki(dbFile, 
  { 
    verbose: true,
    autosave: true, 
    // autoload: true,
    autosaveInterval: 1000,
    adapter: new LokiFSStructuredAdapter(),
    // autoloadCallback: databaseInitialize,
    autosaveCallback: () => { 
      console.log('autosaved db'); 
    }
  }
);

loadDb(({Db, test})=> {
  console.log('db loaded', Db);
  console.log('entries in db collection test', test.find());
});

console.log(Db); shows autosave: false

persistenceMethod: 'adapter',
  persistenceAdapter: LokiFsStructuredAdapter {
    mode: 'reference',
    dbref: {
      filename: 'db.json',
      collections: [Array],
      databaseVersion: 1.5,
      engineVersion: 1.5,
      autosave: false,
      autosaveInterval: 5000,
      autosaveHandle: null,
      throttledSaves: true,
      options: [Object],
      persistenceAdapter: null,
      verbose: false,
      events: [Object],
      ENV: 'NA'
    },

How do I get it enabled ? is there a path I can change the value of options (switching autosave on/off) while running? like ... db.persistenceAdaper['LokiFsStructuredAdapter'].dbref['autosave] = true; also manually save the collection doesn't trigger the autosaveCallback ... but showing the content of the memory db says it gots new entries.

I like to autosave to json. Please help :-)

You have to wait for the database to load. use the callback