New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Best value for nginx's ssl_buffer_size option? #63

Closed
robn opened this Issue Feb 6, 2015 · 19 comments

Comments

Projects
None yet
5 participants
@robn

robn commented Feb 6, 2015

@igrigorik, in your Velocity EU 2014 talk you recommend 4k as a reasonable value for nginx's ssl_buffer_size option (in the absence of dynamic record sizing).

However in the nginx.conf in this repo, its set to 1400, to fit in one MTU.

Has your recommendation on this changed or is there some other reason for it to be 1400 here?

@igrigorik

This comment has been minimized.

Show comment
Hide comment
@igrigorik

igrigorik Feb 6, 2015

Owner

1400 bytes (actually, it should probably be even a bit lower) is the recommended setting for interactive traffic where you want to avoid any unnecessary delays due to packet loss/jitter of fragments of the TLS record. However, packing each TLS record into dedicated packet does add some framing overhead.. and you probably want larger record sizes if you're streaming larger (and less latency sensitive) data. 4K is an in between value that's "reasonable" but not great for either case.

For more, see: http://chimera.labs.oreilly.com/books/1230000000545/ch04.html#TLS_RECORD_SIZE

Owner

igrigorik commented Feb 6, 2015

1400 bytes (actually, it should probably be even a bit lower) is the recommended setting for interactive traffic where you want to avoid any unnecessary delays due to packet loss/jitter of fragments of the TLS record. However, packing each TLS record into dedicated packet does add some framing overhead.. and you probably want larger record sizes if you're streaming larger (and less latency sensitive) data. 4K is an in between value that's "reasonable" but not great for either case.

For more, see: http://chimera.labs.oreilly.com/books/1230000000545/ch04.html#TLS_RECORD_SIZE

@igrigorik igrigorik closed this Feb 6, 2015

@robn

This comment has been minimized.

Show comment
Hide comment
@robn

robn Feb 6, 2015

Thank you @igrigorik, good answer. We'll look at our traffic and make a choice from there.

robn commented Feb 6, 2015

Thank you @igrigorik, good answer. We'll look at our traffic and make a choice from there.

@skynet

This comment has been minimized.

Show comment
Hide comment
@skynet

skynet commented Oct 12, 2015

@mechalas

This comment has been minimized.

Show comment
Hide comment
@mechalas

mechalas Oct 22, 2015

Setting it to 64k or larger will let nginx take advantage of the multibuffer feature in OpenSSL when using AES in CBC mode which has a significant impact on performance.

https://software.intel.com/en-us/articles/improving-openssl-performance

mechalas commented Oct 22, 2015

Setting it to 64k or larger will let nginx take advantage of the multibuffer feature in OpenSSL when using AES in CBC mode which has a significant impact on performance.

https://software.intel.com/en-us/articles/improving-openssl-performance

@skynet

This comment has been minimized.

Show comment
Hide comment
@skynet

skynet Oct 22, 2015

But how about "To minimize Time To First Byte it may be beneficial to use smaller values" here:
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_buffer_size

skynet commented Oct 22, 2015

But how about "To minimize Time To First Byte it may be beneficial to use smaller values" here:
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_buffer_size

@mechalas

This comment has been minimized.

Show comment
Hide comment
@mechalas

mechalas Oct 22, 2015

It depends on your data. If you are serving larger files the gains from multibuffer on the block encryption are going to be significant and will dominate this latency. If you are serving mostly small files, then not so much. If you are using a different encryption mode like GCM, then buffer size will have less of an impact.

mechalas commented Oct 22, 2015

It depends on your data. If you are serving larger files the gains from multibuffer on the block encryption are going to be significant and will dominate this latency. If you are serving mostly small files, then not so much. If you are using a different encryption mode like GCM, then buffer size will have less of an impact.

@skynet

This comment has been minimized.

Show comment
Hide comment
@skynet

skynet Oct 22, 2015

@mechalas please define larger files. We have web sites behind @nginx and most files are somewhere in the 10-100k range. And in regard to what encryption mode is used by @nginx I guess that's not even under our control, if I am not mistaken. ( also see #83 )

skynet commented Oct 22, 2015

@mechalas please define larger files. We have web sites behind @nginx and most files are somewhere in the 10-100k range. And in regard to what encryption mode is used by @nginx I guess that's not even under our control, if I am not mistaken. ( also see #83 )

@igrigorik

This comment has been minimized.

Show comment
Hide comment
@igrigorik

igrigorik Oct 22, 2015

Owner

@skynet @mechalas there isn't one perfect value. This is why other servers implement dynamic record size - e.g. http://chimera.labs.oreilly.com/books/1230000000545/ch04.html#tls_optimizations_at_google.

Owner

igrigorik commented Oct 22, 2015

@skynet @mechalas there isn't one perfect value. This is why other servers implement dynamic record size - e.g. http://chimera.labs.oreilly.com/books/1230000000545/ch04.html#tls_optimizations_at_google.

@mechalas

This comment has been minimized.

Show comment
Hide comment
@mechalas

mechalas Oct 22, 2015

Well, yeah. But nginx doesn't, which is why this was a question. :)

mechalas commented Oct 22, 2015

Well, yeah. But nginx doesn't, which is why this was a question. :)

@skynet

This comment has been minimized.

Show comment
Hide comment
@skynet

skynet Oct 23, 2015

@igrigorik - have you seen this study: http://conferences.sigcomm.org/co-next/2013/program/p303.pdf
It was referenced to me in one of my comments ( https://www.nginx.com/blog/nginx-open-source-reflecting-back-and-looking-ahead/#comment-2279271562 ) answered by Valentin V. Bartenev, the author of SPDY and HTTP/2 modules @nginx
From that perspective and in this ssl_buffer_size context I have no clue whatsoever what's the best configuration for mobile devices. And maybe he's right with this statement: "There is a lot of noise around HTTP/2, but this noise is driven by marketing and works for some sort of PR."

skynet commented Oct 23, 2015

@igrigorik - have you seen this study: http://conferences.sigcomm.org/co-next/2013/program/p303.pdf
It was referenced to me in one of my comments ( https://www.nginx.com/blog/nginx-open-source-reflecting-back-and-looking-ahead/#comment-2279271562 ) answered by Valentin V. Bartenev, the author of SPDY and HTTP/2 modules @nginx
From that perspective and in this ssl_buffer_size context I have no clue whatsoever what's the best configuration for mobile devices. And maybe he's right with this statement: "There is a lot of noise around HTTP/2, but this noise is driven by marketing and works for some sort of PR."

@igrigorik

This comment has been minimized.

Show comment
Hide comment
@igrigorik

igrigorik Oct 23, 2015

Owner

@skynet not sure I follow, what does Android have to do with this? There are two things to consider here: latency and throughput. Larger records improve throughput, smaller records run much lower risk of imposing latency penalties. Depending on the type of traffic you're serving you'll want to adjust these..

Owner

igrigorik commented Oct 23, 2015

@skynet not sure I follow, what does Android have to do with this? There are two things to consider here: latency and throughput. Larger records improve throughput, smaller records run much lower risk of imposing latency penalties. Depending on the type of traffic you're serving you'll want to adjust these..

@skynet

This comment has been minimized.

Show comment
Hide comment
@skynet

skynet Oct 23, 2015

@igrigorik - I get the feeling that mobile devices running on cellular networks are not to be considered by any means identical in (their networking) nature to devices running on wired or WiFi networks, and therefore web servers should dynamically adjust for delivery to these devices.

While everyone's looking at HTTP/2 as an "out of the box - performance improvement" ... conclusion of the aforementioned study sounds bitter:

"In cellular networks, there are fundamental interactions across protocol layers that limit the performance of both SPDY as well as HTTP. As a result, there is no clear performance improvement with SPDY in cellular networks, in contrast to existing studies on wired and WiFi networks."

I can open another issue specifically on cellular networks vs wired and WiFi networks, hoping that you, as a top Web authority and Master of web performance can put some light on this topic.

I understand that engineers at Google have implement dynamic record size for ssl_buffer_size. From that perspective maybe we should encourage (@nginx) to do the same, or, otherwise, our hangup on things like ssl_buffer_size is absolutely irrelevant. I simply cannot adjust these dynamically with NGINX when serving web traffic, as you suggested.

"For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled." - Richard P. Feynman

skynet commented Oct 23, 2015

@igrigorik - I get the feeling that mobile devices running on cellular networks are not to be considered by any means identical in (their networking) nature to devices running on wired or WiFi networks, and therefore web servers should dynamically adjust for delivery to these devices.

While everyone's looking at HTTP/2 as an "out of the box - performance improvement" ... conclusion of the aforementioned study sounds bitter:

"In cellular networks, there are fundamental interactions across protocol layers that limit the performance of both SPDY as well as HTTP. As a result, there is no clear performance improvement with SPDY in cellular networks, in contrast to existing studies on wired and WiFi networks."

I can open another issue specifically on cellular networks vs wired and WiFi networks, hoping that you, as a top Web authority and Master of web performance can put some light on this topic.

I understand that engineers at Google have implement dynamic record size for ssl_buffer_size. From that perspective maybe we should encourage (@nginx) to do the same, or, otherwise, our hangup on things like ssl_buffer_size is absolutely irrelevant. I simply cannot adjust these dynamically with NGINX when serving web traffic, as you suggested.

"For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled." - Richard P. Feynman

@igrigorik

This comment has been minimized.

Show comment
Hide comment
@igrigorik

igrigorik Oct 23, 2015

Owner

@skynet I've suggested the dynamic approach to nginx on many occasions. ATS and Haproxy are using that approach, others are looking into it as well, I'm hoping nginx will (eventually) follow.

Re, mobile: as a broad statement, yes of course that's true, but at the end of the day you're back to optimizing same things: latency and bandwidth. In that regard nothing fundamentally changes, at least as far as record size is concerned.

Owner

igrigorik commented Oct 23, 2015

@skynet I've suggested the dynamic approach to nginx on many occasions. ATS and Haproxy are using that approach, others are looking into it as well, I'm hoping nginx will (eventually) follow.

Re, mobile: as a broad statement, yes of course that's true, but at the end of the day you're back to optimizing same things: latency and bandwidth. In that regard nothing fundamentally changes, at least as far as record size is concerned.

@skynet

This comment has been minimized.

Show comment
Hide comment
@skynet

skynet Oct 23, 2015

Thanks, appreciate the insights. We hope that @nginx will catch-up somehow and provide an even better HTTP2 implementation.

skynet commented Oct 23, 2015

Thanks, appreciate the insights. We hope that @nginx will catch-up somehow and provide an even better HTTP2 implementation.

@jult

This comment has been minimized.

Show comment
Hide comment
@jult

jult Apr 12, 2016

But for now, what did you guys end up with using for the ssl_buffer_size ?

jult commented Apr 12, 2016

But for now, what did you guys end up with using for the ssl_buffer_size ?

@igrigorik

This comment has been minimized.

Show comment
Hide comment
@igrigorik

igrigorik Apr 12, 2016

Owner

For interactive traffic (e.g. HTML, CSS, etc), 4KB or less.

Owner

igrigorik commented Apr 12, 2016

For interactive traffic (e.g. HTML, CSS, etc), 4KB or less.

@jult

This comment has been minimized.

Show comment
Hide comment
@jult

jult Apr 12, 2016

And for a seafile reverse proxy? Files transferred are generally pretty big there, average 2 MB per file.

jult commented Apr 12, 2016

And for a seafile reverse proxy? Files transferred are generally pretty big there, average 2 MB per file.

@igrigorik

This comment has been minimized.

Show comment
Hide comment
@jult

This comment has been minimized.

Show comment
Hide comment
@jult

jult Apr 12, 2016

Good article. I think I'll go with 8k then for this particular use-case scenario. We barely if at all suffer from congestion in the networks I use.

jult commented Apr 12, 2016

Good article. I think I'll go with 8k then for this particular use-case scenario. We barely if at all suffer from congestion in the networks I use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment