New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Siege 4.0.1/4.0.2 hangs forever on 'Lifting the server siege...' with concurrency > 1 #66

Open
jbarratt opened this Issue May 23, 2016 · 88 comments

Comments

Projects
None yet
@jbarratt
Copy link

jbarratt commented May 23, 2016

This happens repeatedly when I run with these arguments.

I have reproduced this one two platforms:
OSX, siege 4.0.1 installed from brew
Linux, siege 4.0.2 built from git, running on Debian Jesse in docker

siege -c 2 -t 5S -H "Host:www.mydomain.com" http://myelb.us-west-2.elb.amazonaws.com

If I strace the process it's hanging on a futex:

futex(0x7f3711c4d9d0, FUTEX_WAIT, 8511, NULL...

With the exact same arguments I can trigger the behavior when -c is any value over 1. -c 1 completes happily.

@Juberstine

This comment has been minimized.

Copy link

Juberstine commented May 23, 2016

Same issue as well.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented May 23, 2016

Do you have the whole trace so I can see exactly where it's hung?

On Mon, May 23, 2016 at 2:37 PM, Juberstine notifications@github.com
wrote:

Same issue as well.


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#66 (comment)

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented May 23, 2016

Actually, that's not going to help. All the parent is telling us is that
it's waiting for threads to complete their task.

While it's hung, could you run:
ps -efL | grep siege

Then strace each of those process IDs. Unfortunately, I've never been able
to duplicate this problem.

strace -p

Regards,
J.

On Mon, May 23, 2016 at 4:11 PM, Jeff Fulmer jeff@joedog.org wrote:

Do you have the whole trace so I can see exactly where it's hung?

On Mon, May 23, 2016 at 2:37 PM, Juberstine notifications@github.com
wrote:

Same issue as well.


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#66 (comment)

@jbarratt

This comment has been minimized.

Copy link

jbarratt commented May 23, 2016

Hm, interestingly, I tried to run the whole process under strace -ff -v -s 2048 -o siegetrace and it actually works properly, which makes it seem like a race condition kind of thing.

When I do ps -efL | grep siege I get 2 processes back, both are doing similar things:

ubuntu@ip-10-0-0-213:~$ sudo strace -p 11813
Process 11813 attached
futex(0x623840, FUTEX_WAIT_PRIVATE, 2, NULL^CProcess 11813 detached
 <detached ...>
ubuntu@ip-10-0-0-213:~$ sudo strace -p 11812
Process 11812 attached
futex(0x7fb89591a9d0, FUTEX_WAIT, 11813, NULL^CProcess 11812 detached
 <detached ...>

Not sure if it's relevant, but I did lsof -p on the processes and I do see they have lots of TCP connections in CLOSE_WAIT still:

siege   11812 ubuntu    3u  IPv4 1670306      0t0    TCP ip-10-0-0-213.us-west-2.compute.internal:53632->sea15s01-in-f132.1e100.net:https (CLOSE_WAIT)
siege   11812 ubuntu    4u  IPv4 1670402      0t0    TCP ip-10-0-0-213.us-west-2.compute.internal:46196->72.21.91.66:https (CLOSE_WAIT)
siege   11812 ubuntu    5u  IPv4 1670308      0t0    TCP ip-10-0-0-213.us-west-2.compute.internal:53634->sea15s01-in-f132.1e100.net:https (CLOSE_WAIT)
siege   11812 ubuntu    6u  IPv4 1670406      0t0    TCP ip-10-0-0-213.us-west-2.compute.internal:46200->72.21.91.66:https (CLOSE_WAIT)
siege   11812 ubuntu    7u  IPv4 1670410      0t0    TCP ip-10-0-0-213.us-west-2.compute.internal:51486->instagram-p3-shv-01-iad3.fbcdn.net:https (CLOSE_WAIT)
siege   11812 ubuntu    8u  IPv4 1670412      0t0    TCP ip-10-0-0-213.us-west-2.compute.internal:51488->instagram-p3-shv-01-iad3.fbcdn.net:https (CLOSE_WAIT)
siege   11812 ubuntu    9u  IPv4 1670442      0t0    TCP ip-10-0-0-213.us-west-2.compute.internal:48366->ec2-52-10-140-172.us-west-2.compute.amazonaws.com:http (CLOSE_WAIT)
siege   11812 ubuntu   10u  IPv4 1670446      0t0    TCP ip-10-0-0-213.us-west-2.compute.internal:48370->ec2-52-10-140-172.us-west-2.compute.amazonaws.com:http (CLOSE_WAIT)

I also noticed quite a few error messages during the connection itself:
[error] HTTPS requires libssl: Unable to reach pbs.twimg.com with this protocol: Transport endpoint is already connected HTTP/1.1 200 0.07 secs: 542180 bytes ==> GET /wp-content/uploads/2015/10/IMG_4685.jpg [error] HTTPS requires libssl: Unable to reach pbs.twimg.com with this protocol: Transport endpoint is already connected [error] socket: read error Connection reset by peer sock.c:539: Connection reset by peer [error] socket: read error Connection reset by peer sock.c:539: Connection reset by peer [error] HTTPS requires libssl: Unable to reach scontent.cdninstagram.com with this protocol: Transport endpoint is already connected [error] HTTPS requires libssl: Unable to reach scontent.cdninstagram.com with this protocol: Transport endpoint is already connected [error] socket: read error Connection reset by peer sock.c:539: Connection reset by peer

@jbarratt

This comment has been minimized.

Copy link

jbarratt commented May 23, 2016

I don't want to share the exact command in public, but I sent it through via email to the joedog support page, if it helps your testing locally.

@bash99

This comment has been minimized.

Copy link

bash99 commented Jul 2, 2016

siege 3.1.4 also has this problem.
compiled on centos 7,

@KukyNekoi

This comment has been minimized.

Copy link

KukyNekoi commented Jul 5, 2016

I'm experiencing the same problem with siege 4.0.1, compiled on centos 7 too

@olifante

This comment has been minimized.

Copy link

olifante commented Jul 15, 2016

I have the same problem on OSX 10.11 El Capitan, with siege 4.0.1 installed from brew. Whatever the test I run, it hangs after printing "Lifting the server siege...". It doesn't respond to CTRL-C and hangs the terminal, forcing me to close the tab.

Unfortunately strace is only available for Linux. @jbarratt since you had success running siege under strace, do you perhaps know how to use dtrace to do the same on OSX?

@olifante

This comment has been minimized.

Copy link

olifante commented Jul 15, 2016

@jbarratt correction: after reading your comment more carefully, I tried running siege with "-c 1" and it works. That's enough for my purposes.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Jul 15, 2016

When siege hangs, are you running it with a timed run, i.e., -t 30s ? Does
it also hang when it's invoked with a set number of repetitions, i.e., -r
30

I suspect this has something to do with a system library blocking signals
but I can't reproduce the problem on any of the Linux systems to which I
have access.

On Fri, Jul 15, 2016 at 9:00 AM, Tiago Castro Henriques <
notifications@github.com> wrote:

@jbarratt https://github.com/jbarratt correction: after reading your
comment more carefully, I tried running siege with "-c 1" and it works.
That's enough for my purposes.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#66 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFMT5lO52Kq5FFzQWYsfXlqH_6f6cNrqks5qV4RtgaJpZM4IkvJu
.

@vusy-nathan

This comment has been minimized.

Copy link

vusy-nathan commented Jul 27, 2016

Hi
I have been looking for a better tool than apache benchmark and i came around siege. i am looking a tool where i can graph the results from experience in testing scalability of websites in my wireless network. any help will be appreciated

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Jul 27, 2016

You can graph your siege results with the bombard wrapper:

http://download.joedog.org/bombard/

(The perl dependencies are in that subdir called "deps")

On Wed, Jul 27, 2016 at 6:13 AM, vusy-nathan notifications@github.com
wrote:

Hi
I have been looking for a better tool than apache benchmark and i came
around siege. i am looking a tool where i can graph the results from
experience in testing scalability of websites in my wireless network. any
help will be appreciated


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#66 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFMT5gStGy-K_BfrJ6Xd4WhngpvyUuwRks5qZy9EgaJpZM4IkvJu
.

@davidak

This comment has been minimized.

Copy link

davidak commented Jul 28, 2016

@JoeDog i use -t30s and can't reproduce it with -r 30.

the strange thing is that siege -c 255 -b -t30s URL works but with -c 240 after that produces this issue.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Jul 28, 2016

so -r NUM is based on a count.

-t NUM is relies on signals to tell the threads to stop. So it must have
something to do with signal handling on some platforms. I wish I could
re-create this so I could get to the bottom of it.

Jeff

On Thu, Jul 28, 2016 at 10:11 AM, davidak notifications@github.com wrote:

@JoeDog https://github.com/JoeDog i use -t30s and can't reproduce it
with -r 30.

the strange thing is that siege -c 255 -b -t30s URL works but with -c 240
after that produces this issue.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#66 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFMT5thDyIYnCFP3SJlpjCrx7x2JHS-uks5qaLh7gaJpZM4IkvJu
.

@dongli

This comment has been minimized.

Copy link

dongli commented Aug 2, 2016

Same problem with siege 4.0.2 on Mac OS X 10.11.6.

@masual

This comment has been minimized.

Copy link

masual commented Aug 5, 2016

Same here, hanging when using time based limit, with repetition limit no problems so far. SIEGE 4.0.2 on Arch.

@houjincheng1992

This comment has been minimized.

Copy link

houjincheng1992 commented Aug 12, 2016

When I type the command like "siege -c 4000 -t 5s http://...", it hangs after printing "Lifting the server siege...". I install it through brew on OS X EI Captian 10.11.6. Any help will be appreciated.

@olegzzz

This comment has been minimized.

Copy link

olegzzz commented Aug 17, 2016

Seems to be working more reliable in OSX sh-3.2 and bash-3.2, but not in OSX zsh 5.0.8 (x86_64-apple-darwin15.0)

@ebbletarts

This comment has been minimized.

Copy link

ebbletarts commented Aug 18, 2016

Seeing this as well with:

siege -b -c 999 -t 30m

I'm attempting to compare two different sets of hardware, so I'm expecting to run into some timeouts and other nasty stuff as they get overloaded, but siege seems to hang if there is too much funny business. from lsof -p, I'm seeing a lot of connections in CLOSE_WAIT and during the siege, there were many "Connection timed out" messages. Examples:

siege   11378  <USER>  532u  IPv4 1604378      0t0      TCP <CLIENT>:39876-><SERVER>:https (CLOSE_WAIT)
siege   11378  <USER>  533u  IPv4 1599385      0t0      TCP <CLIENT>:39196-><SERVER>:https (CLOSE_WAIT)
siege   11378  <USER>  534u  IPv4 1603288      0t0      TCP <CLIENT>:40612-><SERVER>:https (CLOSE_WAIT)
siege   11378  <USER>  535u  IPv4 1605945      0t0      TCP <CLIENT>:39618-><SERVER>:https (CLOSE_WAIT)
siege   11378  <USER>  536u  IPv4 1607024      0t0      TCP <CLIENT>:40106-><SERVER>:https (CLOSE_WAIT)
[error] socket: -129038592 connection timed out.: Connection timed out
[error] socket: 1211696896 connection timed out.: Connection timed out
[error] socket: -1236875520 connection timed out.: Connection timed out
[error] socket: 382916352 connection timed out.: Connection timed out
[error] socket: 1782400768 connection timed out.: Connection timed out
[error] socket: -1262053632 connection timed out.: Connection timed out
[error] socket: 2120206080 connection timed out.: Connection timed out
[error] socket: -63994112 connection timed out.: Connection timed out

Both sites are vanilla installs of Wordpress 4.6 running under a cpanel account inside of a qemu/kvm vm.

While writing this message, I also just ran into a similar issue when I switched to using -r and misspelled the URL I was attempting to hit:

[error] Host not found: <URL>
[error] Host not found: <URL>
[error] Host not found: <URL>
[error] Host not found: <URL>
[error] Host not found: <URL>
[error] Host not found: <URL>
[error] Host not found: <URL>
[error] Host not found: <URL>

Although in that case, after hanging for thirty seconds or so it hit a seg fault and dumped core:

Lifting the server siege...Segmentation fault (core dumped)

I'm running siege on archlinux, kernel 4.7.0, seige v. 4.0.2.

After reading through the manual some more, I switched to HTTP/1.0 to see if that would make a difference, but it did not:

siege -b -c 999 -t 10m

[error] Failed to make an SSL connection: 5: Connection reset by peer
[error] socket: 868943616 connection timed out.: Connection timed out
[error] socket: 161859328 connection timed out.: Connection timed out
[error] socket: 348595968 connection timed out.: Connection timed out
[error] socket: -1052985600 connection timed out.: Connection timed out
[error] socket: 1252910848 connection timed out.: Connection timed out
[error] Failed to make an SSL connection: 5: Connection reset by peer

Lifting the server siege...

Is it possible to set a timeout for unresponsive threads so that siege will just kill unresponsive threads and have a count of timed out threads show up in the final report? Although I guess if you can't reproduce it, then I'm not sure what to do.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Aug 18, 2016

Can you run a debugger on that core dump and send me the backtrace?

On Thu, Aug 18, 2016 at 12:39 PM, handjohn notifications@github.com wrote:

Seeing this as well with:

siege -b -c 999 -t 30m

I'm attempting to compare two different sets of hardware, so I'm expecting
to run into some timeouts and other nasty stuff as they get overloaded, but
siege seems to hang if there is too much funny business. from lsof -p, I'm
seeing a lot of connections in CLOSE_WAIT and during the siege, there were
many "Connection timed out" messages. Examples:

siege 11378 532u IPv4 1604378 0t0 TCP :39876->:https (CLOSE_WAIT)
siege 11378 533u IPv4 1599385 0t0 TCP :39196->:https (CLOSE_WAIT)
siege 11378 534u IPv4 1603288 0t0 TCP :40612->:https (CLOSE_WAIT)
siege 11378 535u IPv4 1605945 0t0 TCP :39618->:https (CLOSE_WAIT)
siege 11378 536u IPv4 1607024 0t0 TCP :40106->:https (CLOSE_WAIT)

[error] socket: -129038592 connection timed out.: Connection timed out
[error] socket: 1211696896 connection timed out.: Connection timed out
[error] socket: -1236875520 connection timed out.: Connection timed out
[error] socket: 382916352 connection timed out.: Connection timed out
[error] socket: 1782400768 connection timed out.: Connection timed out
[error] socket: -1262053632 connection timed out.: Connection timed out
[error] socket: 2120206080 connection timed out.: Connection timed out
[error] socket: -63994112 connection timed out.: Connection timed out

Both sites are vanilla installs of Wordpress 4.6 running under a cpanel
account inside of a qemu/kvm vm.

While writing this message, I also just ran into a similar issue when I
switched to using -r and misspelled the URL I was attempting to hit:

[error] Host not found:
[error] Host not found:
[error] Host not found:
[error] Host not found:
[error] Host not found:
[error] Host not found:
[error] Host not found:
[error] Host not found:

Although in that case, after hanging for thirty seconds or so it hit a seg
fault and dumped core:

Lifting the server siege...Segmentation fault (core dumped)

I'm running siege on archlinux, kernel 4.7.0, seige v. 4.0.2.

After reading through the manual some more, I switched to HTTP/1.0 to see
if that would make a difference, but it did not:

siege -b -c 999 -t 10m

[error] Failed to make an SSL connection: 5: Connection reset by peer
[error] socket: 868943616 connection timed out.: Connection timed out
[error] socket: 161859328 connection timed out.: Connection timed out
[error] socket: 348595968 connection timed out.: Connection timed out
[error] socket: -1052985600 connection timed out.: Connection timed out
[error] socket: 1252910848 connection timed out.: Connection timed out
[error] Failed to make an SSL connection: 5: Connection reset by peer

Lifting the server siege...

Is it possible to set a timeout for unresponsive threads so that siege
will just kill unresponsive threads and have a count of timed out threads
show up in the final report? Although I guess if you can't reproduce it,
then I'm not sure what to do.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#66 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFMT5s8-GOnW9uGSrJS7ra2bQ2o56eHoks5qhIq1gaJpZM4IkvJu
.

@ebbletarts

This comment has been minimized.

Copy link

ebbletarts commented Aug 18, 2016

sure, here:

(gdb) core core.siege.1000.38615dd4b4694491928c1146051d2c62.19198.1471536627000000000000
BFD: Warning: /var/lib/systemd/coredump/core.siege.1000.38615dd4b4694491928c1146051d2c62.19198.1471536627000000000000 is truncated: expected core file size >= 7801901056, found: 2147483648.
[New LWP 20198]
[New LWP 19198]
[New LWP 19287]
[New LWP 19438]
[New LWP 19941]
[New LWP 19986]
[New LWP 19699]
[New LWP 19872]
Failed to read a valid object file image from memory.
Core was generated by `siege -b -c 999 -r 10 <URL>'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f4c543c6420 in ?? ()
[Current thread is 1 (LWP 20198)]
(gdb) bt
#0  0x00007f4c543c6420 in ?? ()
#1  0x000000000040b8b9 in ?? ()
#2  0x00000000007e3110 in ?? ()
#3  0x0000000000000000 in ?? ()
(gdb) 

I'm not sure which symbols I'd need to install, I should have them for perl (assuming that's the LWP perl module)

@ebbletarts

This comment has been minimized.

Copy link

ebbletarts commented Aug 18, 2016

Also, when it's just hanging, it's just waiting on a futex:

$ sudo strace -p 20607
strace: Process 20607 attached
futex(0x7fb9985799d0, FUTEX_WAIT, 20609, NULL

If I strace the siege command, it dies with a fatal error:

$ strace -o siege.log -f -s 2048 -tt siege -b -c 999 -t 10m <URL>
** SIEGE 4.0.2
** Preparing 999 concurrent users for battle.
The server is now under siege...[fatal] pthread wait

If you'd like the siege log, I can send it to you but I'd prefer to send it privately rather than post it here. I'm not used to using github so if there's some messaging system or some way for me to see your email address, let me know and I can send it along.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Aug 18, 2016

I might have enough to research this. Did you compile this copy or did you
download a binary?

On Thu, Aug 18, 2016 at 4:28 PM, ebbletarts notifications@github.com
wrote:

Also, when it's just hanging, it's just waiting on a futex:

$ sudo strace -p 20607
strace: Process 20607 attached
futex(0x7fb9985799d0, FUTEX_WAIT, 20609, NULL

If I strace the siege command, it dies with a fatal error:

$ strace -o siege.log -f -s 2048 -tt siege -b -c 999 -t 10m
** SIEGE 4.0.2
** Preparing 999 concurrent users for battle.
The server is now under siege...[fatal] pthread wait

If you'd like the siege log, I can send it to you but I'd prefer to send
it privately rather than post it here. I'm not used to using github so if
there's some messaging system or some way for me to see your email address,
let me know and I can send it along.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#66 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFMT5u40aOYzcsYKhfPtGzdRfihi7hkfks5qhMBkgaJpZM4IkvJu
.

@ebbletarts

This comment has been minimized.

Copy link

ebbletarts commented Aug 18, 2016

I installed it from the ArchLinux Community repo via pacman:

https://www.archlinux.org/packages/community/x86_64/siege/

From looking at the pkgbuild, it may be getting compiled:

./configure --prefix=/usr --sysconfdir=/etc --mandir=/usr/share/man

But I'm not totally sure since I've never built a package before so I don't have a ton of information on how that works, although I can do some more research if it's helpful.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Aug 19, 2016

The last time someone had this problem, they grabbed the source and built
it from scratch. The problem went away. Could you do the same? You can get
the latest source distribution here:

http://download.joedog.org/siege/siege-4.0.2.tar.gz

(The distribution already has a configure script so you don't need
autotools)

On Thu, Aug 18, 2016 at 5:00 PM, ebbletarts notifications@github.com
wrote:

I installed it from the ArchLinux Community repo via pacman:

https://www.archlinux.org/packages/community/x86_64/siege/

From looking at the pkgbuild, it may be getting compiled:

./configure --prefix=/usr --sysconfdir=/etc --mandir=/usr/share/man

But I'm not totally sure since I've never built a package before so I
don't have a ton of information on how that works, although I can do some
more research if it's helpful.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#66 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFMT5nUQLhJwvBDGEFJIzS9KQwMUHu4zks5qhMfcgaJpZM4IkvJu
.

@ebbletarts

This comment has been minimized.

Copy link

ebbletarts commented Aug 22, 2016

I compiled from source and have run 4 half-hour long tests and none of them have had the issue.

Weird, haha.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Aug 23, 2016

Well now you know why I can't reproduce it. The problem is with that
distribution.....

On Mon, Aug 22, 2016 at 3:31 PM, ebbletarts notifications@github.com
wrote:

I compiled from source and have run 4 half-hour long tests and none of
them have had the issue.

Weird, haha.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#66 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFMT5pgmUaTWAn4WWtTp0gU8TW0h7h6Uks5qifkagaJpZM4IkvJu
.

@RickMoynihan

This comment has been minimized.

Copy link

RickMoynihan commented Aug 24, 2016

I get the same behaviour too.

Running on

$ uname -a
Darwin mccarthy.lan 13.4.0 Darwin Kernel Version 13.4.0: Wed Mar 18 16:20:14 PDT 2015; root:xnu-2422.115.14~1/RELEASE_X86_64 x86_64

Tried installing from brew and also building from source.

It always hangs for me whenever I run

siege -b -t5s http://some.webserver/

@ebbletarts

This comment has been minimized.

Copy link

ebbletarts commented Aug 24, 2016

I'm running into this with the source compile as well now.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Aug 24, 2016

And this is on arch linux?

On Wed, Aug 24, 2016 at 11:16 AM, ebbletarts notifications@github.com
wrote:

I'm running into this with the source compile as well now.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#66 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFMT5tHatdD7d3Fs0TWltkgXwzva-0j5ks5qjGBSgaJpZM4IkvJu
.

@mrimann

This comment has been minimized.

Copy link

mrimann commented Jul 18, 2017

I also ran into this issue. At first, I thought it's some kind of garbage that stays after running siege against several targets as it got worse over time, but a reboot didn't help, so I guess it must be something different.

OS: Mac OS El Capitan, v10.11.6
Siege: v4.0.2 (installed via brew)

Commands I ran were:

siege -c 5 -t 30s URL
siege -c 15 -t 30s URL

(where URL is a full URI like https://www.domain.tld/foo/bar.html)

In my case, the problem occured more often, the higher the number of concurrent simulated users were. And, but that's just an impression so far, I tend to think the problem occurs more often, the more single items/requests are handled - e.g. an optimized site with just a HTML-page, 1-2 CSS and 1-2 JS files will run through - where a page with >20 assets (CSS/JS/Images) will more likely cause siege to run into that problem. But as said, I cannot prove this impression with data so far.

@pkallos

This comment has been minimized.

Copy link

pkallos commented Jul 18, 2017

e.g. an optimized site with just a HTML-page, 1-2 CSS and 1-2 JS files will run through - where a page with >20 assets (CSS/JS/Images) will more likely cause siege to run into that problem. But as said, I cannot prove this impression with data so far.

I don't think siege goes and fetches CSS, JS and images on a given page, just makes the initial HTTP request.

@mrimann

This comment has been minimized.

Copy link

mrimann commented Jul 18, 2017

@pkallos I just checked again, and at least the siege output says it does fetch the assets:

➜  ~ siege -c 1 -t 30s https://www.somedomain.tld/de/startseite.html
** SIEGE 4.0.2
** Preparing 1 concurrent users for battle.
The server is now under siege...
HTTP/1.1 200     0.20 secs:   15255 bytes ==> GET  /de/startseite.html
HTTP/1.1 200     0.06 secs:    4644 bytes ==> GET  /typo3conf/ext/nezzoworldmap/Resources/Public/JS/richMarker.js?1463484247
HTTP/1.1 200     0.06 secs:    7705 bytes ==> GET  /typo3conf/ext/nezzoworldmap/Resources/Public/JS/markerclustered.js?1463478857
HTTP/1.1 200     0.06 secs:    2698 bytes ==> GET  /typo3conf/ext/nezzoworldmap/Resources/Public/JS/jquery.waypoints.min.js?1439930412
HTTP/1.1 200     0.06 secs:    9220 bytes ==> GET  /typo3conf/ext/nezzoworldmap/Resources/Public/JS/nezzoworldmap.js?1464189710
(...)
@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Jul 18, 2017

@mrimann

This comment has been minimized.

Copy link

mrimann commented Jul 18, 2017

@JoeDog OK, got it - wasn't aware of the config file being present - it was either auto-created by brew upon installing siege, or upon the first usage of siege (don't know). And yes, it's using the default value therefore, which is "parser = true". (Which by the way is absolutely fitting for my use-case, so I don't consider that option to be wrong)

@pkallos

This comment has been minimized.

Copy link

pkallos commented Jul 18, 2017

oh wow, TIL!

@jeff1985

This comment has been minimized.

Copy link

jeff1985 commented Aug 4, 2017

had same problem here on macosx sierra and bower install, disabled parsing in siege conf => works now!

@javorszky

This comment has been minimized.

Copy link

javorszky commented Sep 14, 2017

I used nettop to figure out what's happening. Still hanging on mac. All threads are in CloseWait state.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Sep 14, 2017

@javorszky

This comment has been minimized.

Copy link

javorszky commented Sep 14, 2017

I was using 4.0.2 through homebrew. I downloaded the files, but building siege leaves out libraries, and I'm not experienced enough to figure out how to do that properly. It can't use ssl. INSTALL file refers to the directory where ssl is installed. which openssl gives me /usr/local/opt/openssl/bin/openssl

This is how I installed siege based on the above info:

./configure --with-ssl=/usr/local/opt/openssl/
make
make install

Still won't test ssl urls :/

@wjn740

This comment has been minimized.

Copy link

wjn740 commented Sep 15, 2017

SUSE Linux Enterprise also encounter same problem.
siege-4.0.2.tar.gz ae03c9072d4ada7309f31d14d0e24518

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Sep 18, 2017

wjn740,

Could you please try siege-4.0.4?

http://download.joedog.org/siege/

javorszky,

Could you look at the configure output to see if it's actually finding your ssl headers and libraries?

@javorszky

This comment has been minimized.

Copy link

javorszky commented Sep 18, 2017

@JoeDog got it working, thank you. Had to do make clean before rerunning make after having configured correctly above.

I'm going to test it in a minute with 404 (now with SSL!), bear with.

UPDATE: a siege that previously didn't close connections finished without a problem with 4.0.4. <3

@wjn740

This comment has been minimized.

Copy link

wjn740 commented Sep 18, 2017

@JoeDog I trying but I'm in SUSE Lab conf feedback later

@mralexho

This comment has been minimized.

Copy link

mralexho commented Sep 18, 2017

@JoeDog turning off parsing solves this issue for me. Thanks!

Mac OS X 10.12.6
Siege 4.0.2 (via homebrew)

@wjn740

This comment has been minimized.

Copy link

wjn740 commented Sep 19, 2017

@JoeDog ,Hello sir.
I use v4.0.4 command line:
/usr/share/qa/qa_test_siege/siege/bin/siege -c 8 -t1m http://127.0.0.1/qa_test_siege/index.html -i -b
It will be hang at 7/10 times,
In v4.0.2 it blocked at 3/10 times.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Sep 19, 2017

@wjn740

This comment has been minimized.

Copy link

wjn740 commented Sep 19, 2017

@JoeDog Hi sir, I use siege on x86_64 arch, kernel version = 4.12.11

@wjn740

This comment has been minimized.

Copy link

wjn740 commented Sep 19, 2017

@JoeDog ,If I don't use -t option, it will be fine.
Because I need change the benchmark so I found this problem. Originally, I use -r option it's OK in 2 years.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Sep 19, 2017

@wjn740

This comment has been minimized.

Copy link

wjn740 commented Sep 19, 2017

@JoeDog, did you try continue run several times ?

@wjn740

This comment has been minimized.

Copy link

wjn740 commented Sep 19, 2017

0x00007f6915acc8ec in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f6915acc8ec in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f6915ac5d65 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x000055d03f16743d in cookies_destroy ()
#3 0x000055d03f161519 in main ()
(gdb) info thread
Id Target Id Frame

  • 1 Thread 0x7f6915ee3700 (LWP 9279) "siege" 0x00007f6915acc8ec in __lll_lock_wait () from /lib64/libpthread.so.0
@wjn740

This comment has been minimized.

Copy link

wjn740 commented Sep 19, 2017

When It hang, the last thread wait mutex that hold by another guy, but the guy have been gone.

@MadeInChina

This comment has been minimized.

Copy link

MadeInChina commented Oct 1, 2017

Same issue with SIEGE 4.0.2 on mac.

@yn-academia

This comment has been minimized.

Copy link

yn-academia commented Nov 7, 2017

Seeing this w. 4.0.4 on an Ubuntu 14.04 machine on AWS

@jonross

This comment has been minimized.

Copy link

jonross commented Mar 21, 2018

Same issue, siege 4.0.4, CentOS 7.4.1708
This is built from source btw. (Yes I'll try -r, adding some info to this comment as I glean it.)

Have some sockets in CLOSE_WAIT but not that many... 20 sockets like that while running with -c100.

strace on process leader & threads:

% strace -p 1723
strace: Process 1723 attached
futex(0x7f79cfc219d0, FUTEX_WAIT, 1736, NULL^Cstrace: Process 1723 detached
 <detached ...>
% strace -p 1736
strace: Process 1736 attached
futex(0x84cef8, FUTEX_WAIT_PRIVATE, 2, NULL^Cstrace: Process 1736 detached
 <detached ...>
% strace -p 1822
strace: Process 1822 attached
futex(0x84cef8, FUTEX_WAIT_PRIVATE, 2, NULL^Cstrace: Process 1822 detached
 <detached ...>

Update: gdb info stack on process leader and threads. Maybe SSL-related?

#0  0x00007f79d7f07f57 in pthread_join () from /lib64/libpthread.so.0
#1  0x000000000040b76b in crew_join (crew=crew@entry=0x866ab0, finish=finish@entry=boolean_true, payload=payload@entry=0x7ffd8c6f8d70) at crew.c:280
#2  0x0000000000403d35 in main (argc=<optimized out>, argv=<optimized out>) at main.c:498

#0  0x00007f79d7f0d42d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f79d7f08dcb in _L_lock_812 () from /lib64/libpthread.so.0
#2  0x00007f79d7f08c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000417145 in SSL_pthreads_locking_callback (mode=<optimized out>, type=<optimized out>, file=<optimized out>, line=<optimized out>) at ssl.c:246
#4  0x00007f79d7695e37 in CRYPTO_add_lock () from /lib64/libcrypto.so.10
#5  0x00007f79d7729e19 in RSA_free () from /lib64/libcrypto.so.10
#6  0x00007f79d775a53b in EVP_PKEY_free_it () from /lib64/libcrypto.so.10
#7  0x00007f79d775acd8 in EVP_PKEY_free () from /lib64/libcrypto.so.10
#8  0x00007f79d776d1a0 in pubkey_cb () from /lib64/libcrypto.so.10
#9  0x00007f79d7772e8a in asn1_item_combine_free () from /lib64/libcrypto.so.10
#10 0x00007f79d777311f in ASN1_template_free () from /lib64/libcrypto.so.10
#11 0x00007f79d7772faa in asn1_item_combine_free () from /lib64/libcrypto.so.10
#12 0x00007f79d777311f in ASN1_template_free () from /lib64/libcrypto.so.10
#13 0x00007f79d7772faa in asn1_item_combine_free () from /lib64/libcrypto.so.10
#14 0x00007f79d7773065 in ASN1_item_free () from /lib64/libcrypto.so.10
#15 0x00007f79d7acfcbe in ssl_cert_clear_certs () from /lib64/libssl.so.10
#16 0x00007f79d7ad048a in ssl_cert_free () from /lib64/libssl.so.10
#17 0x00007f79d7acc8f7 in SSL_CTX_free () from /lib64/libssl.so.10
#18 0x0000000000416892 in socket_close (C=0x7f79640008c0) at sock.c:686
#19 0x0000000000407dc0 in __http (U=0x84d7c0, this=0x863c90) at browser.c:580
#20 __request (this=this@entry=0x863c90, U=U@entry=0x84d7c0) at browser.c:406
#21 0x0000000000408719 in start (this=0x863c90) at browser.c:295
#22 0x000000000040b1fe in crew_thread (crew=0x866ab0) at crew.c:141
#23 0x00007f79d7f06e25 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f79d714734d in clone () from /lib64/libc.so.6

#0  0x00007f79d7f0d42d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f79d7f08dcb in _L_lock_812 () from /lib64/libpthread.so.0
#2  0x00007f79d7f08c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000417145 in SSL_pthreads_locking_callback (mode=<optimized out>, type=<optimized out>, file=<optimized out>, line=<optimized out>) at ssl.c:246
#4  0x00007f79d7695e37 in CRYPTO_add_lock () from /lib64/libcrypto.so.10
#5  0x00007f79d7729e19 in RSA_free () from /lib64/libcrypto.so.10
#6  0x00007f79d775a53b in EVP_PKEY_free_it () from /lib64/libcrypto.so.10
#7  0x00007f79d775acd8 in EVP_PKEY_free () from /lib64/libcrypto.so.10
#8  0x00007f79d776d1a0 in pubkey_cb () from /lib64/libcrypto.so.10
#9  0x00007f79d7772e8a in asn1_item_combine_free () from /lib64/libcrypto.so.10
#10 0x00007f79d777311f in ASN1_template_free () from /lib64/libcrypto.so.10
#11 0x00007f79d7772faa in asn1_item_combine_free () from /lib64/libcrypto.so.10
#12 0x00007f79d777311f in ASN1_template_free () from /lib64/libcrypto.so.10
#13 0x00007f79d7772faa in asn1_item_combine_free () from /lib64/libcrypto.so.10
#14 0x00007f79d7773065 in ASN1_item_free () from /lib64/libcrypto.so.10
#15 0x00007f79d7acfcbe in ssl_cert_clear_certs () from /lib64/libssl.so.10
#16 0x00007f79d7ad048a in ssl_cert_free () from /lib64/libssl.so.10
#17 0x00007f79d7acc8f7 in SSL_CTX_free () from /lib64/libssl.so.10
#18 0x0000000000416892 in socket_close (C=0x8f4970) at sock.c:686
#19 0x0000000000407dc0 in __http (U=0x84d7c0, this=0x8720e0) at browser.c:580
#20 __request (this=this@entry=0x8720e0, U=U@entry=0x84d7c0) at browser.c:406
#21 0x0000000000408719 in start (this=0x8720e0) at browser.c:295
#22 0x000000000040b1fe in crew_thread (crew=0x866ab0) at crew.c:141
#23 0x00007f79d7f06e25 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f79d714734d in clone () from /lib64/libc.so.6

FWIW

% openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

Update: It certainly looks like SSL is the culprit. I've been trying to work through a 5x5 grid of delay (-d) and concurrency (-c) and never get past the first few cases. With SSL turned off, all 25 complete. (By SSL I mean I'm using a client certificate, not just hitting an HTTPS endpoint.)

Update: If I switch to connection = keep-alive it works with client certs.

@JoeDog

This comment has been minimized.

Copy link
Owner

JoeDog commented Mar 21, 2018

@jonross

This comment has been minimized.

Copy link

jonross commented Mar 21, 2018

@JoeDog -- some new info above, hope it helps.

@jonross

This comment has been minimized.

Copy link

jonross commented Mar 21, 2018

Unrelated to the SSL issue, I've also seen it hang after lifting the siege and printing statistics. In this case there is only one thread active:

#0  0x00007f3d061d0eec in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007f3d0617b7bc in _L_lock_2546 () from /lib64/libc.so.6
#2  0x00007f3d0617b5f7 in __tz_convert () from /lib64/libc.so.6
#3  0x0000000000411ed1 in write_to_log (count=18742, elapsed=9.97000027, elapsed@entry=0, bytes=bytes@entry=0, ttime=365.660828, ttime@entry=0, code=18742, failed=0) at log.c:66
#4  0x00000000004121b5 in log_transaction (D=D@entry=0x1bbe3c0) at log.c:40
#5  0x000000000040419d in main (argc=<optimized out>, argv=<optimized out>) at main.c:547
@benzvan

This comment has been minimized.

Copy link

benzvan commented Dec 24, 2018

In case it helps, I found the lifting siege issue in a centos container. Here's my dockerfile:

FROM centos

RUN yum install -y gcc openssl-devel make

RUN curl -O http://download.joedog.org/siege/siege-latest.tar.gz && \
    tar xvzf siege-latest.tar.gz && \
    rm -f siege-latest.tar.gz && \
    cd siege-* && \
    ./configure && \
    make && \
    make install

Here's the command I ran in the container

siege --log=./siegelog.txt -t 500S -r 1000 -c 100 "${URL}"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment