1.5/1.6 connection handling issues #129

Closed
keen99 opened this Issue Jun 22, 2012 · 8 comments

Comments

Projects
None yet
3 participants

keen99 commented Jun 22, 2012

After updating to 1.5 (from 1.4.6), we're seeing intermittent but heavy connection failures. centos5.7 -ish world. We initially saw this as pheanstalk errors in our application servers, but my stats gathering scripts also see the same problem with nc.

I compiled 1.6 and have the same issues.

Here's a script to reproduce it:


#!/bin/sh
beanstalkd -v
for ((a=0; a < 2000 ; a++))
 do
  if printf "stats\r\n" | nc localhost 11300|grep -q curr
   then
    let ++i
#    echo yes $i
  else
   let ++j
#   echo no $j
 fi
done
{
echo Total:     $a
echo Success:   $i
echo Fail:      $j
} | column -t

easod_rpmbuild:easod:i-82fca2fb:rpmbuild:[draistrick@ec2-107-21-182-212 ~]$ sh beantest.sh beanstalkd 1.4.6 Total: 2000 Success: 2000 Fail:

easod_rpmbuild:easod:i-82fca2fb:rpmbuild:[draistrick@ec2-107-21-182-212 ~]$ sh beantest.sh
beanstalkd 1.5
nc: Write error: Connection reset by peer
Total: 2000
Success: 758
Fail: 1242

easod_rpmbuild:easod:i-82fca2fb:rpmbuild:[draistrick@ec2-107-21-182-212 ~]$ !sh
sh beantest.sh
beanstalkd 1.6
Total: 2000
Success: 644
Fail: 1356

We've just started looking into this - if we find anything specific I'll update here.

Owner

kr commented Aug 31, 2012

@keen99 could you please run this build of beanstalkd and let
me know if you see any problems?

https://s3.amazonaws.com/krheroku/beanstalkd

$ ./beanstalkd -v
beanstalkd 1.6+4+g236c669

If it works well I'll make a release.

keen99 commented Aug 31, 2012

the binary was dynamic linked, but not against my running environment....

easod_rpmbuild:easod:i-82fca2fb:rpmbuild:[draistrick@ec2-107-21-182-212 ~]$ ldd beanstalkd
./beanstalkd: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by ./beanstalkd)
libc.so.6 => /lib64/libc.so.6 (0x0000003cecc00000)
/lib64/ld-linux-x86-64.so.2 (0x0000003cec800000)

so I grabbed latest from kr/beanstalkd/master and rolled it into our rpm for testing. (no patches, just init scripts)

easod_rpmbuild:easod:i-82fca2fb:rpmbuild:[root@ec2-107-21-182-212 draistrick]$ sh beantest.sh
beanstalkd +mod
Total: 2000
Success: 890
Fail: 1110

easod_rpmbuild:easod:i-82fca2fb:rpmbuild:[root@ec2-107-21-182-212 rpm]$ ps wxa|grep beansta
3559 pts/1 S 0:00 /usr/bin/beanstalkd -l 0.0.0.0 -p 11300 -u beanstalkd

asod_rpmbuild:easod:i-82fca2fb:rpmbuild:[root@ec2-107-21-182-212 rpm]$ ldd /usr/bin/beanstalkd
libc.so.6 => /lib64/libc.so.6 (0x0000003cecc00000)
/lib64/ld-linux-x86-64.so.2 (0x0000003cec800000)

What else can I provide to help with this? I can probably provide access to an identical build machine (which matches our running architecture, roughly centos5.7). Obviously I haven't been able to get my own dev team on this - you know how it goes.

thanks.

Contributor

etanol commented Aug 31, 2012

@keen99: It appears that be binary you obtained has been linked against a later libc version. Your CentOS version has glibc 2.5.

Compiling beanstalkd is really straightforward since 1.5. In CentOS you only need the make, gcc and glibc-devel packages, then you just have to run make from the source directory and you'll get the binary.

keen99 commented Aug 31, 2012

@etanol: did you read the rest of my response? The one where I recompiled against my environment based on a tarball of latest and repeated the failure case??

@kr kr closed this in a9d4367 Aug 31, 2012

Owner

kr commented Aug 31, 2012

@keen99 I was able to reproduce this issue by running the script
you posted, thanks! The commit listed just above fixes it for me.
You can test it out right away by downloading the code at a9d4367
and running make CFLAGS='-g -O0'.

keen99 commented Aug 31, 2012

initial test looks good, running a larger scope just to confirm

keen99 commented Aug 31, 2012

looks good with everything I can throw at it short of actually putting it in production, thanks!

keen99 commented Sep 4, 2012

here's the run I left running EOD friday:

Total: 2000000
Success: 2000000
Fail:

I'd call that fixed. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment