Memleak

[pietercolpaert]

To test this library in order to use it for a bigger project, I've tried to ingest DBPedia. It crashes however at ~1% of the triples saying that the process is out of memory. It only happens when I try to write to a leveldb. If I comment the write, the memory usage stays stable.

some code

var db = levelgraph(levelup(dbname));
var dbputstream = db.putStream();
var filename = "pathtodbpedia.nt";
fs.createReadStream(filename).on("data", function (data) {
  N3Parser().parse(data, function (error, triple) {
    dbputstream.write(triple);
  }
});

Any ideas on how to limit memory usage?

[mcollina]

This is very interesting.
Loading a huge number of triples (how many it is the 1%?) is something I should dig into.

However, you are doing it wrongly: you are not providing a streaming interface with proper backpressure and LevelGraph couldn't keep up with the parsing speed. You should try writing a trasform step and use pipe, like so:

var db = levelgraph(levelup(dbname));
var through2 = require('through2')
var filename = "pathtodbpedia.nt";
fs.createReadStream(filename).pipe(through2({ objectMode: true }, function (data, enc, callback) {

  var that = this
  N3Parser().parse(data, function (error, triple) {
    that.push(triple)
  })

  callback() // as parse should be sync
}).pipe(db.putStream());

What package is N3Parser from?

(I've not tested it, if you can provide a working gist and a link to the dataset, I'll try)

BTW, LevelGraph is slower on writes (15k triples/s) than on reads (30-40k triples/s).

I'm pulling in @maxogden as it as done huge work on importing bulk data into LevelUp.
Also @RubenVerborgh for the need of a streaming N3-parsing API :).

[RubenVerborgh]

@mcollina It's the node-n3 parser, the non-streaming version.
It's interesting how you avoid the backpressure, does that work well? parse works sync.
This might make the stream implementation of node-n3 (rdfjs/N3.js#6) unnecessary.

On the other hand, @pietercolpaert, maybe the stream implementation can help out here then!

[pietercolpaert]

(note: @RubenVerborgh is a colleague)
How does your code avoid the backpressure?

I've implemented your method with through2. The memory still increases rapidly though...

Let's wait for rdfjs/N3.js#6?

[mcollina]

(note: @RubenVerborgh https://github.com/rubenverborgh is a colleague)
How does your code avoid the backpressure?

It is handled by through2, automatically

I've implemented your method with through2. The memory still increases
rapidly though..

Is it still crashing or not?
LevelDB uses HUGE amount of memory in writing.
For storing dbpedia, i think it can eat some GB of RAM and then settle
there, while it does his job.

I think you might want to try level-hyper, it uses a leveldb variant that
might handle your case better.

[mcollina]

I think this might be what is really happening: Level/levelup#171.
So, trying level-hyper might be the only real solution.

(Reopening, as it's easy to search for discussions on this topic).

[mcollina]

The thing is already packaged for usage in LevelGraph-N3: https://github.com/mcollina/levelgraph-n3#importing-n3-files.

[mcollina]

@pietercolpaert @RubenVerborgh any news on this? Have you tried the new stream-based importer?

[mcollina]

hey @pietercolpaert @RubenVerborgh, any news on this?

[pietercolpaert]

The memory leak is still there with level-hyper as well I'm afraid.

Test yourself

Download http://downloads.dbpedia.org/3.9/en/instance_types_en.nt.bz2

Load it in a levelgraph db using something like:

var fs = require("fs"),
    n3 = require("n3"),
    levelup = require('level'),
    levelhyper = require('level-hyper'),
    levelgraph = require("levelgraph");
var db = levelgraph(levelhyper("data"));
var readstream = fs.createReadStream("/path/to/instance_types_en.nt", { encoding: 'utf8' });
var count = 0;
var transform = new n3.Transform();
transform.pipe(db.putStream());
readstream.pipe(transform).on("data",function (triple) {
  count++;
  if (count%10000 === 0) {
    var str = count + " triples loaded";
    console.error( str );
  }
}).on("end",function () {
  console.error(count + " triples loaded");
});

Or am I doing something wrong here?

[mcollina]

First, using stream.on("data", function() {}) disable all the backpressure goodness. So, don't do that. If you want to track the data being written, we can add something to LevelGraph write stream.

Second, it might really be that Level needs more memory to store all that data, and you might need to increase it... a lot. Look at: https://github.com/joyent/node/wiki/FAQ#what-is-the-memory-limit-on-a-node-process.
Usually Level memory usage explode during writes, and then compact later.
In order to be sure, I'll try with a 3 or 4 GB memory limit.

You are insterting roughly 15.894.068 lines, which we can assume are triples.
That means 16 * 6 triples on levelgraph: roughly 1 billion entries.
Some tweaking might be necessary.

I'm hoping in some help from @rvagg, @maxogden and others in order to track it down.

Third, there might really be a memory leak. Have a look at https://hacks.mozilla.org/2012/11/tracking-down-memory-leaks-in-node-js-a-node-js-holiday-season/ to any help tracking it down.

I'll try giving a go at your script and see if I can get better.

[mcollina]

On the contrary of what I though, the following is not leaking on my Mac:

var fs = require("fs"),
  through2 = require("through2"),
  n3 = require("n3"),
  levelhyper = require('level-hyper'),
  levelgraph = require("levelgraph");

var db = levelgraph(levelhyper("data"));
var readstream = fs.createReadStream("./instance_types_en.nt", { encoding: 'utf8' });
var writestream = db.putStream();
var count = 0;
var transform = new n3.Transform();

readstream.pipe(transform).pipe(through2({ objectMode: true }, function (chunk, enc, callback) {
  count++;
  if (count % 10000 === 0) {
    var str = count + " triples loaded";
    console.error( str );
  }
  this.push(chunk)
  callback()
})).pipe(writestream);

writestream.on("end",function () {
  console.error(count + " triples loaded");
});

It is extremely slow, but it is not leaking.