Enable upstream keepalives by default

[sjberman]

By default, NGINX opens a new connection to an upstream server for every new incoming request. This is safe but inefficient, because NGINX and the server must exchange three packets to establish a connection and three or four to terminate it.

At high traffic volumes, opening a new connection for every request can exhaust system resources and make it impossible to open connections at all.

Right now we allow users to enable upstream keepalives using an UpstreamSettingsPolicy. However, it would be nice if we could have NGINX be performant right out of the box.

Acceptance

• turn upstream keepalives on by default, set to 16
  • The reference guide linked below recommends settings this to 2x the number of upstream servers. However, with one server, I actually noticed that this degraded performance unless I also increased the keepalive_requests number. But with a value of 16, which seems to be commonly used elsewhere, performance improved
• if UpstreamSettingsPolicy defines a keepalive value, that value is used instead of the default
  • value of 0 should disable it (policy minimum needs to be updated to 0, currently 1)
• Connection header should be unset (this logic should already be in place when keepalives are defined, but ensure it still works when a default is implemented)
• Update UpstreamSettings guide to mention this default

Reference: https://www.f5.com/company/blog/nginx/avoiding-top-10-nginx-configuration-mistakes#no-keepalives