heavy performance degradation from 3.31 -> 3.52/3.53

[jdv145]

Hi all,

Recently i upgraded my netty library from 3.31 to 3.52. After this upgrade i noticed a heavy performance drop. I also tried 3.53 but it seemed to perform on par with 3.52.

I am running a dual flash socket server together with a http server. The http is only for controlling the socketserver so performance is not that important for me. It used to do only http-1.0 but i changed it so it will respond with http-1.1 if the client is using this.

With keep-alive both 3.31 and 3.52 perform on par.

When i don't use keep-alive the performance drop is 90%.

The way i tested it was using ab.exe (from apache) with and without the -k flag.

The output from ab is posted below. The only thing i changed was including either 3.31 or 3.52 (did it a couple of times).

I used the netbeans profiler (and made screenshots) to see if i could see something jumping, but that didn't help me very much.

specs:
win7 x64
jdk 1.6u32
i7-2600 12GB ram

slow

Concurrency Level: 1
Time taken for tests: 6.121 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 111000 bytes
HTML transferred: 13000 bytes
Requests per second: 163.36 #/sec
Time per request: 6.121 ms
Time per request: 6.121 [ms](mean, across all concurrent requests)
Transfer rate: 17.71 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.5 0 1
Processing: 4 6 0.6 6 10
Waiting: 3 5 0.6 5 10
Total: 5 6 0.4 6 11

Percentage of the requests served within a certain time (ms)
50% 6
66% 6
75% 6
80% 6
90% 6
95% 7
98% 7
99% 7
100% 11 (longest request)

faster

Concurrency Level: 1
Time taken for tests: 0.795 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 111000 bytes
HTML transferred: 13000 bytes
Requests per second: 1257.79 #/sec
Time per request: 0.795 ms
Time per request: 0.795 [ms](mean, across all concurrent requests)
Transfer rate: 136.34 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 1
Processing: 0 0 0.5 0 1
Waiting: 0 0 0.5 0 1
Total: 0 1 0.4 1 2

Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 1
95% 1
98% 1
99% 1
100% 2 (longest request)

edit
If i must supply more info, i'm more then willing to help. I think the info i supplied is a bit scarce, but im not familiar enough with netty to be able to proficiently search for the cause of this issue. I did however tried other versions of the library and after testing twice i noticed 3.3.1 is the last version which runs with normal performance. 3.4.0 beta and 3.4.0 final show the lower then normal performance.

for reference, with the -k option i get
Requests per second: 8886.01 #/sec
which seems normal

[normanmaurer]

@jdv145 could you post the exact ab command you used ? Also would it possible to reproduce it with a simple server app ? Or could you provide us your app to test against ?

[jdv145]

The server is pretty big and spread in different parts with a lot of dependencies. I wouldn't be surprised if i did something wrong in the handlers or bootstrapping or something. It was my assumption the error was on my side, but then i exchanged the libraries with the results i posted.

I am working like crazy to to finish a new release of a game server, but as soon as i can, i'll try to make a minimal server based on my code and look how it performs (which i can easily share if needed). Isn't there some bare-bones (but correctly configured) http server with netty. I assume if its something with the library a degradation of 80-90% will be easy to spot. I can then compare my bootstrapping and stuff with the bare-bones one.

3.31
ab -n 1000 localhost:844/
Requests per second: 1141.49 #/sec

ab -k -n 1000 localhost:844/
Requests per second:    4565.96 [#/sec] (mean)

3.53
ab -n 1000 localhost:844/
Requests per second: 167.31 #/sec

ab -k -n 1000 localhost:844/
Requests per second:    4405.03 [#/sec] (mean)

ps, i now get 4400 req/s (and not 8800) because the page i tested with is more complex then my previous hello world! response.

[jdv145]

Hello again.

Finally i got some time on my hands so i decided to quickly copy/paste some of my original code to get a working http server and use it to benchmark it like i did before.

The results are 300 req/sec vs 1300req/sec using ab.exe with the commandline and output included in the zip (netty_3.3.1.txt and netty_3.5.3.txt). The only difference is switching between netty 3.3.1 and netty 3.5.3.

I tried to keep it as simple as possible without any dependecies.

system specs are again the same as before.

Im pretty sure i should be able to upload the zip somewhere here but i was unable to find it. Instead i uploaded it to one of the first hits in google:
(you dont need to register, just press download)
http://www.filedropper.com/testnetty

regards,

[CruzBishop]

Hi @jdv145

I was planning on writing a few projects with Netty, and I happened to come across this bug. I'll have a look and see if I can discover anything :)

[CruzBishop]

Server siege using 3.5.4-SNAPSHOT started. Time to go make a coffee

[CruzBishop]

Hmm. Just pulled in the latest changes in netty/3, using HTTP 1.1

Going to test with HTTP 1.0 in a minute.

Transactions: 100000 hits
Availability: 100.00 %
Elapsed time: 30.76 secs
Data transferred: 1.91 MB
Response time: 0.00 secs
Transaction rate: 3250.98 trans/sec
Throughput: 0.06 MB/sec
Concurrency: 9.82
Successful transactions: 100000
Failed transactions: 0
Longest transaction: 0.05
Shortest transaction: 0.00

[CruzBishop]

HTTP 1.0, 10 concurrent requests

Transactions: 100000 hits
Availability: 100.00 %
Elapsed time: 36.97 secs
Data transferred: 1.91 MB
Response time: 0.00 secs
Transaction rate: 2704.90 trans/sec
Throughput: 0.05 MB/sec
Concurrency: 9.84
Successful transactions: 100000
Failed transactions: 0
Longest transaction: 0.25
Shortest transaction: 0.00

[CruzBishop]

Doesn't seem to be applying with Ubuntu 12.04 in Netty 3.5.4 so far, but I'll do some small changes anyway :)

[CruzBishop]

With HTTP 1.1, no profiling, and 10 concurrent requests...

6443.30 transactions per second.

Seems like this was fixed somewhere in 3.5.4, unless it's a Windows issue - perhaps @trustin or @normanmaurer would be able to test that? I don't have access to my Windows 7 VM at the moment

EDIT: And in comparison to Apache 2.2.22 and PHP 5.3 just doing a simple output of GET, POST, and SERVER vars, 5% failed requests with 843 requests per second. Wow.

[jdv145]

Thanks cruz for also testing.

The strange thing was that http 1.1 (with the keepalive aka -k option in ab) was performing equally in 3.3.1 and 3.5.3.

Seeing the results from cruz, i also tested with the snapshot from 2 hours ago with http1.0 and now the results are about the same between 3.3.1 and snapshot, so i think whatever caused the degradation is fixed in 3.5.4.

I think the issue can be closed?

I dont know much about the testing process with netty, but i was wondering if it would be an idea to do a small benchmark before any release. A performance drop of 50%+ is probably easy to spot this way, and you get to see if all the different parts of netty are still playing along nicely with each other.

[normanmaurer]

@jdv145 thanks for let us know..

@CruzBishop thanks again also :)

[CruzBishop]

Any time, @jdv145 and @normanmaurer :)