I am using CocoaHTTPServer in my thesis project https://github.com/ArloL/IntAirAct. During a performance evaluation I noticed a weird behavior after a certain number of requests. To test this I created the project https://github.com/ArloL/HelloCocoaHTTPServer. I am using jMeter to test. The configuration can be found here https://github.com/ArloL/IntAirAct-Performance.
I tested three times with 100.000 requests on my MacBook Pro, 10.7.4 with 8 GB of RAM.
Test #1 has 6 requests that took about 20s each.
Test #2 has 6 requests that took about 20s each.
Test #3 has 6 requests that took about 8s each.
The interesting part is, is that the sample numbers of the long requests are nearly the same.
Test #1: 16374, 32747, 49123, 65498, 81873, 98252
Test #2: 16376, 32750, 49122, 65494, 81866, 98240
Test #3: 16364, 32704, 48976, 65223, 81537, 97824
As a control I tested with Sinatra and there I don't have these issues. All the numbers are close to multiples of 16384 (2^14). I would guess at a memory problem or something similar.
I tried running the same test on my machine, but wasn't able to reproduce the problem. That is, I cloned the HelloCocoaHTTPServer, downloaded jMeter, and ran the configuration you provided.
I tested on a MacBook Air, 10.7.4 with 4 GB of RAM.
I have also recently installed the software update "Java for OS X 2012-004" (not sure if this would affect jMeter).
Is the test easily reproducible on your machine? Do you suspect any issues with jMeter?
Oh wait, I just realized that "100.000" meant "100,000". (Silly americans with our miles and pounds. Lol) Let me retest.
OK, so retested after changing test parameters from original "100" value to "100000" requests. But I'm getting similar results as before. Are you also testing with original setting of "users=1" (which I take to mean one http request at a time)?
Of note: during the test I opened up the activity monitor. I noticed that my CPU for HelloCocoaHTTPServer stayed around 9 - 12%. Meanwhile, jMeter was around 112%.
I've seen something similar in the past. On my previous machine running ApacheBench for 100,000 in memory responses with a concurrency value of 1 would sometimes hang between 10,000 and 20,000 responses. Monitoring the process it went to 0% CPU for several seconds before it started responding to requests again.
On my current machine I'm not able to reproduce the problem with HelloCocoaHTTPServer but I can reproduce it with SimpleHTTPServer fetching a file from disk:
ab -c 1 -n 100000 http://127.0.0.1:55481/item.html
This again hangs between 10k and 20k responses for me, again with the process going idle. Repeating the test with Nginx shows no delays. I don't know if it is the same issue since the disk is involved, but the behavior seems similar.
Sorry about that number mix-up :)
Yes, it is very easily reproducible. I just tested again and I got the same results.
No, I do not suspect jMeter. For one mattstevens had similar results using ab and I tested against Sinatra using this configuration https://github.com/ArloL/IntAirAct-Performance/blob/master/servers/Sinatra/server.rb without problems.
Before I tested I checked for updates and I have all updates installed. Additionally a paste of my running processes http://pastebin.com/XMgPi2jY via ps aux.
I also opened up Activity Monitor to see about the CPU usage and I notice approx. 70-80% of HelloCocoaHTTPServer, approx. 30% of jMeter and about 50% of kernel_task as the main CPU users. If we suspect this to be an issue I will look into using https://code.google.com/p/jmeter-plugins/ to record proper results. I see the exact same behavior during the hickups as mattstevens said (everything going to 0% until the request is over).
Are you running jMeter with the GUI interface? I use the non-GUI version.
I made sure that the run.sh in IntAirAct-Performance shows the exact configuration that I use. So yes, I use users=1 which means that there is one thread that sends requests, waits for a response and then sends another request.
I was testing with jMeter GUI. I'll test again with jMeter non-GUI and/or ab. This sounds fairly reproducible.