too fast gets corrupted jobs? #6

Closed
rschiavi opened this Issue Sep 20, 2011 · 5 comments

Comments

Projects
None yet
3 participants

Write a put test that puts say 1000 jobs into a queue.

If you run this, you get assertion errors on undefined jobs (although using beanstalk telnet to checkout the queue all the jobs seem okay). If you put a setTimeout(function(){ watch()}, 500); around the call to the next watch(), I have yet to see an error. Without the timeout, get a corrupted job almost immediately

var ns = require('nodestalker');
var assert = require('assert');

var client = ns.Client();
var processed = 0;
var error = 0;

console.log("tube: " + process.argv[2]);
function watch(){
client.watch(process.argv[2]).onSuccess(function(){
client.reserve().onSuccess(function(data){
console.log(data);
console.log("processed: " + processed++);
assert.ok(data.id);
client.deleteJob(data.id).onSuccess(function(data){
console.log("deleted: " + data.id + " " + data);
watch();
});
});
});
}

watch();

Owner

pascalopitz commented Sep 20, 2011

You're probably overloading a buffer of some sorts. I ASSUME that you need to give the process a chance to write into the scoket every now and then ... but since you have identified the solution already, why exactly is that a problem?

well, i think because my solution actually failed. i only tried a timeout of up to 1 second, but even that one eventually failed.

ended up switching to the other nodejs beanstalk library and it works, so assumed it was a bug here (ie, running a full throttle test puting and reserving using the other library works. with this one, it always ended up with corrupted jobs in the queue)

rschiavi closed this Sep 20, 2011

pascalopitz reopened this Sep 20, 2011

Owner

pascalopitz commented Sep 20, 2011

right, so a timeout doesn't help at all?

I've been running into a similar problem when processing jobs very quickly. Occasionally (can't produce on demand) my node process will exit with this error from the beanstalk stream:

undefined:1

^^
SyntaxError: Unexpected token ILLEGAL
at Object.parse (native)
at /my/node/app.js:169:22
at BeanstalkCommand. (/usr/local/lib/node/.npm/nodestalker/0.1.1/package/lib/beanstalk_client.js:63:3)
at BeanstalkCommand.emit (events.js:64:17)
at BeanstalkCommand.responseHandler (/usr/local/lib/node/.npm/nodestalker/0.1.1/package/lib/beanstalk_client.js:73:8)
at BeanstalkCommand.emit (events.js:67:17)
at Socket. (/usr/local/lib/node/.npm/nodestalker/0.1.1/package/lib/beanstalk_client.js:138:7)
at Socket.emit (events.js:64:17)
at Socket._onReadable (net.js:673:14)
at IOWatcher.onReadable as callback

Owner

pascalopitz commented Feb 25, 2015

Haven't had more input, so I'll close this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment