See http://groups.google.com/group/beanstalk-talk/t/e589c79f86f4d0ac for the bug report.

Avoid stupid off-by-one errors at one-second granularity.

These accidentally got disabled in the switch to autoconf.

One of them breaks on Mac OS.

See http://groups.google.com/group/beanstalk-talk/t/8210428dd5f2c98f for details.

Currently, if a client reserves a job and then crashes, the job will be put in the ready queue with no indication that it had ever been reserved. This is not good; clients should be able to distinguish such a job from one that has never been reserved.

So we should add a "reserves" counter and/or increment the "timeouts" counter in this situation and/or introduce a separate counter just for this situation.

See http://groups.google.com/group/beanstalk-talk/t/951b6c35752257a4 for more info.

Currently the peek commands do a job_copy() before calling reply_job() presumably to avoid a race between the job being sent back to the client, and it being DELETEd by another client.

reset_conn() should free the job copy however it currently requires the job.id to be 0 before it will call job_free(). This works fine for the h_conn_timeout() case as in http://github.com/kr/beanstalkd/commit/d77814816f54c48b7bcc85c36bb681fd277d3e65 but it overloads the meaning of job.id which is an essential part of the peek responses.

The result is that we can't set a copied job.id to 0 for peek responses which means it won't get freed by reset_conn(), leading to a fast memory leak if any of the peek commands are used.

I'll submit a patch which adds a new job state to mark a job as copied (JOB_STATE_COPY) but I'm not sure if its the best way to solve this.

http://gist.github.com/139279 will hammer peek-buried sufficiently that you'll see the beanstalkd process size grow very rapidly.

Currently it can only create one extra binlog file on disk. After that it will deny operations that want to reserve space. (Search for "overextended" in binlog.c.) There just needs to be a linked list of future binlog files; currently there is a pointer to (up to) one future file. In practice this only happens when the queue holds more than about 10 MB of jobs.

Should have optional throttling. There should be three modes: always fsync (for best reliability), fsync at most once every N seconds, or never fsync (best performance).

Could be controlled by two options: "-s 0" for always fsync, "-s N" for sync at most once every N seconds, and "-S" for never fsync.

beanstalkd only handles SIGINT and does then a clean shutdown. The following patch adds this behavior for SIGTERM. Tested against version 1.3

Patch:

--- beanstalkd.c.orig   2009-08-15 13:48:46.000000000 +0200
+++ beanstalkd.c        2009-08-15 13:49:11.000000000 +0200
@@ -120,6 +120,10 @@
     sa.sa_handler = exit_cleanly;
     r = sigaction(SIGINT, &sa, 0);
     if (r == -1) twarn("sigaction(SIGINT)"), exit(111);
+
+    sa.sa_handler = exit_cleanly;
+    r = sigaction(SIGTERM, &sa, 0);
+    if (r == -1) twarn("sigaction(SIGTERM)"), exit(111);
 }

 /* This is a workaround for a mystifying workaround in libevent's epoll

I get a build error on Mac OS X Snow Leopard where it says it can't find fdatasync.

Output of ./configure:

https://gist.github.com/0290aa815842b888eed4

Error upon running make:

cc1: warnings being treated as errors
binlog.c: In function ‘binlog_write_job’:
binlog.c:491: warning: implicit declaration of function ‘fdatasync’ 
make[1]: *** [binlog.o] Error 1
make: *** [all] Error 2

Using libevent-1.4.12-stable ("make verify" passed).

reeler% git clone git://github.com/kr/beanstalkd.git
reeler% cd beanstalkd
reeler% ./autogen.sh
configure.in:10: installing ./install-sh'<br/> configure.in:10: installing./missing'
reeler% ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... nawk
checking whether make sets $(MAKE)... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for style of include used by make... GNU
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... none
checking for supported compiler flags... -Wall -Werror
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking dependency style of gcc... (cached) none
checking for a BSD-compatible install... /usr/bin/install -c
checking if compiler supports -R... yes
checking libevent install prefix... /usr/local
checking for posix_fallocate... no
checking for fdatasync... no
checking for bind in -lsocket... no
checking for inet_aton in -lnsl... no
checking for event_get_version in -levent... yes
checking for event_reinit in -levent... yes
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for uint16_t... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating beanstalkd.spec
config.status: creating config.h
config.status: executing depfiles commands
reeler% make
make all-am
gcc -DHAVE_CONFIG_H -I. -g -O2 -Wall -Werror -I/usr/local/include -c -o beanstalkd.o beanstalkd.c
gcc -DHAVE_CONFIG_H -I. -g -O2 -Wall -Werror -I/usr/local/include -c -o binlog.o binlog.c
gcc -DHAVE_CONFIG_H -I. -g -O2 -Wall -Werror -I/usr/local/include -c -o conn.o conn.c
gcc -DHAVE_CONFIG_H -I. -g -O2 -Wall -Werror -I/usr/local/include -c -o job.o job.c
gcc -DHAVE_CONFIG_H -I. -g -O2 -Wall -Werror -I/usr/local/include -c -o ms.o ms.c
gcc -DHAVE_CONFIG_H -I. -g -O2 -Wall -Werror -I/usr/local/include -c -o net.o net.c
In file included from net.h:22,

             from net.c:22:

/usr/include/netinet/tcp.h:40: error: expected '=', ',', ';', 'asm' or 'attribute' before 'tcp_seq' /usr/include/netinet/tcp.h:50: error: expected specifier-qualifier-list before 'u_short' /usr/include/netinet/tcp.h:175: error: expected specifier-qualifier-list before 'u_int8_t' *** Error code 1

Stop in /usr/home/peter/beanstalkd.
*** Error code 1

Stop in /usr/home/peter/beanstalkd.
reeler%

Hi,

I'm using beanstalkd 1.4.3 with PHP, and I found a little problem. The "peek-buried" command seems to be broken
Here are my commands :

watch test
WATCHING 2

stats-tube test
OK 248
name: test
current-jobs-urgent: 0
current-jobs-ready: 0
current-jobs-reserved: 0
current-jobs-delayed: 0
current-jobs-buried: 4
total-jobs: 4
current-using: 0
current-watching: 1
current-waiting: 0
cmd-pause-tube: 0
pause: 0
pause-time-left: 0

list-tubes-watched
OK 21
default
test

peek-buried
NOT_FOUND

So, I have some buried jobs in my queue, but peek-buried don't find them...

Joël

Can be used, e.g., for deduplication of work.

This falls outside the main scope of beanstalkd's purpose, but I think it opens up a lot of interesting possibilities.

Such tubes can be marked as special by prefixing the name with a special char. I think "~" would work fine as that char.

Currently when a client disconnects while it has a reserved job, that job is instantly released onto the front of the queue, without respect of its remaining TTR.

This means that:

• A worker executing long jobs cannot disconnect/reconnect to beanstalkd during execution. Perhaps that would make it too hard to track which client owns which job reservation though...
• A job that causes a worker to crash constantly hogs the front of the queue, blocking subsequent jobs from coming through.

Perhaps auto-release due to disconnect should at least release onto the end of the queue, at a lower priority, or with a delay..?

It would be nice to have a client that just forwards jobs from one beanstalkd instance to another. See http://groups.google.com/group/beanstalk-talk/t/984bfc8037847e53 for some motivation for this setup.

Probably just means adding a CRC to stored records.

When submitting jobs around 2MB, beanstalkd gets really slow. See here for the original report:

http://groups.google.com/group/beanstalk-talk/t/f27bd53d62d23f0c

Given a job id, I would like to be able to reprioritize it. Currently I can only achieve this by calling peek, delete, then recreating the job.

One possibility would be to allow release to be called without first having reserved the job. Currently the interface for delete and release is inconsistent:

put 0 0 60 3
foo
INSERTED 1
delete 1
DELETED
put 0 0 60 3
bar
INSERTED 2
release 2 100 0
NOT_FOUND
delete 2
DELETED

| See below... This happens when clients send reserve | commands constantly on an empty queue.

I'm running the master beanstalkd on Snow Leopard and it's been working well, except I've come across the error:

./beanstalkd: prot.c:672 in check_err: writev(): Broken pipe ./beanstalkd: prot.c:672 in check_err: writev(): Broken pipe ..

Without the clients doing anything else to the queue, this causes beanstalkd to run at 97% CPU usage. I'm not sure what caused this specifically but it occurs when I have one client adding things onto the queue and another removing them (about 10 items per second).

Cheers

Peter

http://groups.google.com/group/beanstalk-talk/t/4ed1d368d7b3a5a8

Our jankity shell scripts don't work with all versions of netcat. It's just too hard to use reliably.

I'm leaning toward a really simple, custom python framework. But if there are any really good protocol testing tools out there, I'd love to know.

I just read the protocol discription for Beanstalkd. The project looks great and the protocol is really simple. Should be very straightforward to implement in a language such as Erlang. The only problem I could see is the lack of sequence numbers. This basically would force synchronous usage for the clients of Beanstalkd ? Do you have any plans for a binary protocol with sequence numbers?