db input example that works #99

Closed
mgrosso opened this Issue May 18, 2012 · 2 comments

Comments

Projects
None yet
3 participants

mgrosso commented May 18, 2012

The example code for using database input and output in the wiki is a bit terse. Making it work took me some trial and error and some time spent perusing the implementation code.

Here is what I had to do to get a job working that used mongo for input and output. This pattern should work
for any other database.

Rather than improving the documentation for the existing behavior, it would be nice if simply defining cursor() and/or save() on a Job or its arguments without also overriding the corresponding input() or output() would trigger the behavior in this example. If a pull request would be welcome let me know.

nodeio  = require 'node.io'
db      = require("mongojs").connect("test",["domains"] )

#inheritance is broken for coffee in node.io right now so pass an Object ...
@job = new nodeio.Job({
  _cursor_live: true
  cursor: () ->
    db.domains.find().sort({_id:1})
  save: (row) ->
    db.domains.save row
  input: (start, num, f ) ->
    if(start == 0 )
      @cursor().forEach (err,val) =>
        if err
          console.log 'mongo cursor error', err
          f false
        else
          if val
            f [ val ]
          else
            @_cursor_live = false
            f false
      undefined
    else if ! @_cursor_live
      false
    else
      undefined
  run: (n) ->
    n['foo'] = 'frobbed'
    @emit n
  output: (rows) ->
    for row in rows
      console.log 'output', row
      @save row
})
Contributor

chriso commented May 23, 2012

The cursor approach to selecting rows seems to be mongo-specific. All other DB drivers I'm aware of (postgres, mysql, sqlite, etc.) take a completely different approach.

I agree it can be difficult hooking up certain drivers to the input/output methods but keeping it generic and decoupled is my intention for now, especially considering you might hook up files, streams, in-memory queues, etc.

chriso closed this May 23, 2012

Contributor

jacobSingh commented Sep 2, 2012

I don't read coffee all that well, but I'm curious what all the cursor stuff is about.

Doesn't
c.find().skip(start).limit(num).toArray(function(err, results) {
if (results.length == 0) {
cb(false);
}
cb(results);
});

Work okay for input?

I do see one issue, which is that the last batch (if you're doing concurrent procs) doesn't count right, but that's a node.io issue I think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment