skipblank memory usage and main loop blocking

[springmeyer]

I'm starting to be more convinced that the skipblank branch uses memory too aggressively. Partially related, it also blocks the main loop, leading to the eio threadpool being underutilized.

E.g. I am unable to detect any memory leaks at the v8/node/mapnik level (e.g. or deeper problems). Even if v8 garbage collection is forced during skipping (using while (!mapnik.gc()) {} it is still very easy to run out of memory when many tiles are being skipped.

This is a ticket to discuss possible ways to work around this.

Ideas:

1. move parts of code to c++ (threadpool)

2. move existing c++ calls to async (isSolid, getPixel, others ??)

3. more agressive use of process.nextTick (to help reduce blocking)

4. use child process nodes to handle key tasks

[kkaefer]

Can you try the schemes branch with bin/copy --scheme=pyramid? It uses a different skipping mechanism. I'd be interested in whether the main loop blocking occurs here as well.

[springmeyer]

closing, this is working awesomely now.