Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

too fast gets corrupted jobs? #6

Closed
rschiavi opened this Issue · 5 comments

3 participants

@rschiavi

Write a put test that puts say 1000 jobs into a queue.

If you run this, you get assertion errors on undefined jobs (although using beanstalk telnet to checkout the queue all the jobs seem okay). If you put a setTimeout(function(){ watch()}, 500); around the call to the next watch(), I have yet to see an error. Without the timeout, get a corrupted job almost immediately

var ns = require('nodestalker');
var assert = require('assert');

var client = ns.Client();
var processed = 0;
var error = 0;

console.log("tube: " + process.argv[2]);
function watch(){
client.watch(process.argv[2]).onSuccess(function(){
client.reserve().onSuccess(function(data){
console.log(data);
console.log("processed: " + processed++);
assert.ok(data.id);
client.deleteJob(data.id).onSuccess(function(data){
console.log("deleted: " + data.id + " " + data);
watch();
});
});
});
}

watch();

@pascalopitz
Owner

You're probably overloading a buffer of some sorts. I ASSUME that you need to give the process a chance to write into the scoket every now and then ... but since you have identified the solution already, why exactly is that a problem?

@rschiavi

well, i think because my solution actually failed. i only tried a timeout of up to 1 second, but even that one eventually failed.

ended up switching to the other nodejs beanstalk library and it works, so assumed it was a bug here (ie, running a full throttle test puting and reserving using the other library works. with this one, it always ended up with corrupted jobs in the queue)

@rschiavi rschiavi closed this
@pascalopitz pascalopitz reopened this
@pascalopitz
Owner

right, so a timeout doesn't help at all?

@nickdaugherty

I've been running into a similar problem when processing jobs very quickly. Occasionally (can't produce on demand) my node process will exit with this error from the beanstalk stream:

undefined:1

^^
SyntaxError: Unexpected token ILLEGAL
at Object.parse (native)
at /my/node/app.js:169:22
at BeanstalkCommand. (/usr/local/lib/node/.npm/nodestalker/0.1.1/package/lib/beanstalk_client.js:63:3)
at BeanstalkCommand.emit (events.js:64:17)
at BeanstalkCommand.responseHandler (/usr/local/lib/node/.npm/nodestalker/0.1.1/package/lib/beanstalk_client.js:73:8)
at BeanstalkCommand.emit (events.js:67:17)
at Socket. (/usr/local/lib/node/.npm/nodestalker/0.1.1/package/lib/beanstalk_client.js:138:7)
at Socket.emit (events.js:64:17)
at Socket._onReadable (net.js:673:14)
at IOWatcher.onReadable as callback

@pascalopitz
Owner

Haven't had more input, so I'll close this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.