kr / beanstalkd
- Source
- Commits
- Network (15)
- Issues (13)
- Downloads (22)
- Wiki (3)
- Graphs
-
Branch:
master
-
Can be used, e.g., for deduplication of work.
Comments
-
This falls outside the main scope of beanstalkd's purpose, but I think it opens up a lot of interesting possibilities.
Such tubes can be marked as special by prefixing the name with a special char. I think "~" would work fine as that char.
Comments
-
2 votes
0 comments Created 6 months ago by pdaClient disconnect shouldn't auto-release reserved job onto front of queue.featurexCurrently when a client disconnects while it has a reserved job, that job is instantly released onto the front of the queue, without respect of its remaining TTR.
This means that:
- A worker executing long jobs cannot disconnect/reconnect to beanstalkd during execution. Perhaps that would make it too hard to track which client owns which job reservation though...
- A job that causes a worker to crash constantly hogs the front of the queue, blocking subsequent jobs from coming through.
Perhaps auto-release due to disconnect should at least release onto the end of the queue, at a lower priority, or with a delay..?
Comments
-
It would be nice to have a client that just forwards jobs from one beanstalkd instance to another. See http://groups.google.com/group/beanstalk-talk/t/984bfc8037847e53 for some motivation for this setup.
Comments
-
Probably just means adding a CRC to stored records.
Comments
-
When submitting jobs around 2MB, beanstalkd gets really slow. See here for the original report:
http://groups.google.com/group/beanstalk-talk/t/f27bd53d62d23f0c
Comments
-
Given a job id, I would like to be able to reprioritize it. Currently I can only achieve this by calling peek, delete, then recreating the job.
One possibility would be to allow release to be called without first having reserved the job. Currently the interface for delete and release is inconsistent:
put 0 0 60 3 foo INSERTED 1 delete 1 DELETED put 0 0 60 3 bar INSERTED 2 release 2 100 0 NOT_FOUND delete 2 DELETEDComments
-
0 votes
4 comments Created 2 months ago by pkieltykaBroken pipe error and heavy CPU use with misbehaving clientsbugx| See below... This happens when clients send reserve | commands constantly on an empty queue.
I'm running the master beanstalkd on Snow Leopard and it's been working well, except I've come across the error:
./beanstalkd: prot.c:672 in check_err: writev(): Broken pipe ./beanstalkd: prot.c:672 in check_err: writev(): Broken pipe ..
Without the clients doing anything else to the queue, this causes beanstalkd to run at 97% CPU usage. I'm not sure what caused this specifically but it occurs when I have one client adding things onto the queue and another removing them (about 10 items per second).
Cheers
Peter
Comments
Another thing is I noticed that beanstalkd's memory usage increases slowly and never releases it's allocated memory. Perhaps I'm doing something wrong or its my environment, but I'm interested to know the expected behaviour. By the way, I am using a persistent queue via the binlog.
Something is wrong if it takes 97% cpu with so little activity. I will try to reproduce it.
I looked into this some more and it results from my client code doing crazy things, it was sending "reserve" commands in an infinite loop without any jobs on the queue. I've fixed my code since then and it's been working great.
Perhaps though, you could ignore subsequent reserve commands for a particular connection .. or something.
Thx.
-
0 votes
0 comments Created 2 months ago by krfeaturexaccept LF in addition to CR LF to terminate commandskrxComments
-
1 vote
2 comments Created 2 months ago by krfeaturexadd underscore "_" to list of valid tube name characterskrx -
Our jankity shell scripts don't work with all versions of netcat. It's just too hard to use reliably.
I'm leaning toward a really simple, custom python framework. But if there are any really good protocol testing tools out there, I'd love to know.
Comments
-
2 votes
3 comments Created 29 days ago by thijsterlouwSequence numbers / Synchronous clients?featurexI just read the protocol discription for Beanstalkd. The project looks great and the protocol is really simple. Should be very straightforward to implement in a language such as Erlang. The only problem I could see is the lack of sequence numbers. This basically would force synchronous usage for the clients of Beanstalkd ? Do you have any plans for a binary protocol with sequence numbers?
Comments
thijsterlouw
Mon Dec 07 08:18:12 -0800 2009
| link
If the responses always arrive in the same order of the requests (fifo), then sending/receiving async using a queue should also be a viable option?
Nonetheless, a binary protocol like memcached would also be great :)
There was discussion of a binary protocol back when beanstalkd was still young, but nothing came of it. I think it would be easier to revisit now that the text protocol changes so rarely.
And yes, you can use the existing protocol asynchronously, but you have to keep track of the order of commands you sent and responses you expect. In particular, you can send many commands before reading the replies (aka pipelining).
There is an Erlang client library at http://github.com/tim/erlang-beanstalk ; perhaps that will be illustrative. At some point I'd like to make a proof-of-concept in node.js as well.
thijsterlouw
Thu Dec 10 17:34:26 -0800 2009
| link
thanks for the reply. I looked at that erlang-beanstalk client, but it's implementation is too simple for our purposes: Tim uses a (blocking) gen_tcp:send/recv combination, which means that each tcp connection (server process) can only handle one request. When you have many hundreds of requests coming quickly, this design will easily lead to timeouts in the gen_server:call(). Thus it is unsuitable for high request rates. Using async with seq. numbers is the easiest approach, but we will probably use pipelining instead for our client.
Yes please! ;-)
+1 I saw that in the roadmap, and know exactly what I'd use it for.
What would be the difference between this and the existing job ids? If we're suggesting features I'd love deleting or re-prioritising jobs by id - related?
You have no control of existing job ids. The idea is to be able to assign job ids on the client. This way, multiple (distributed) clients could interact (reschedule, cancel, pop, etc) the same job.
You can already delete any job by id. The only exception: you cannot delete a job some other client currently has reserved.
Re-prioritising jobs is a nice idea. Feel free to open a ticket for it.
kr, you can delete by ID, but it assumes I know the id. I want the following scenario: client A queues new job (name: job1), client B, which knows nothing about client A, reaches in and modifies the payload / time to run / etc for job1. To accomplish this, the name has to depend on some external knowledge (not beanstalkd assigning job ids). For example.. let the job id be an email of the user, or a qualified url, etc.
Got it. That's exactly what this ticket is for.
I should try to be more clear about who I'm replying to. Sorry about that.
Ok now I understand the motivation for this feature, and it's another +1 from me. Definitely opens up useful possibilities. I'll open another ticket for reprioritizing jobs.
+1