Introduce httpcommon package in libbeat (add support for Proxy)

[urso]

• Enhancement

What does this PR do?

We introduces a libbeat/common/transport/httpcommon package. The package provides structures that can be used to extract common http settings from a configuration object and construct a http.RoundTripper or a http.Client. Currently the types configure:

• Proxy settings
• TLS
• Timeout

The HTTPTransportSettings and HTTPClientProxySettings implement Unpack, which will be used automatically when a configuration is unpacked into a configuration structure. The Unpacker implemetnations directly initialize TLS and parse URLs and other settings that can fail. If some setting is missconfigured, Beats will fail early, reporting the full path of the setting (input, output, ...) that was missconfigured. Initializing the proxy URL and TLS, also reduces the responsiblities for the inputs/outputs to provide copy and paste code to do so.

Example usage:

type mySettings struct {
  ...

  Transport httpcommon.HTTPTransportSettings `config:",inline"`
}

...

settings := defaultSettings()
if err := config.Unpack(&settings); err != nil { return .... }

client := settings.Transport.Client(
  httpcommon.WithAPMHTTPInstrumentation(),
)

...

resp, err := client.Get(...)

Features already provided by httpcommon:

• common dialer configuration
• common TLS configuration (already pre initialized)
• out of the box proxy support for everything HTTP
• (optionally) instrument your HTTP client with APM
• Timeout settign
• Copy some Transport defaults from http.DefaultTransport. This initializes the idle connection limit and idle connection timeout (don't leak connections).

By default (can be disabled via functional options), proxy support defaults to the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables. The environment variables are automatically ignored if the URLs host is localhost or the loopback device (by IP).

Many features that use an http.Client have been update to use the httpcommon package in order to extract "common" setting (this list only documents changes on existing features):

• Heartbeat http monitor:
  • Added setting: proxy_headers
  • Added APM instrumentation
• Elasticsearch output:
  • Added settings: proxy_headers, proxy_disable
• Internal Elasticsearch monitoring reporter (legacy x-pack monitoring):
  • Added settings: proxy_disable, proxy_headers
• Kibana client (setup.kibana.* namespace):
  • Added proxy support
  • automatically close idle connection
• Filebeat httpjson input:
  • Add proxy settings
  • Automatically instrument the http client with APM HTTP support
• Filebeat/Metricbeat cloudfoundry:
  • Add proxy support
  • APM HTTP instrumentalization
  • automatically close idle connections
• Metricbeat http helper:
  • Added proxy support
  • automatically Close idle connections (default was no limits on number of idle connection and no timeout)
  • Added APM HTTP auto instrumentation
  • Affected modules (Note: I didn't check lightweight modules):
    • Consul
    • HTTP
    • Beat
    • Elasticsearch
    • Prometheus
    • NATS
    • Ceph
    • Traefik
    • Logstash
    • Rabbit MQ
    • Jolokia
    • Envoyproxy
    • PHP FPM
    • nginx
    • dropwizard
    • apache
    • couchbase
    • kubernetes
    • Kibana
    • etcd
    • couchdb
    • golang
    • haproxy
• Agent:
  • Updated: Downloader supports
    • Adds proxy support
  • Updated: remote.Config, which is also used fleet server config namespace
    • Adds proxy support

Missing (left over for follow up PRs):

• More settings:
  • make connection pool settings configurable?
  • Headers
  • Authentication:
    • Basic Auth
    • OAuth
    • Bearer token (must be configured or read from store... not from disk)
  • More fine grained timeout settings
  • read/write buffer sizes
• Testing: the package is more or less indirectly tested. E.g. proxy support via the Elasticsearch output. New tests for Unpack behavior and proxy tests from the output should be moved to the httpcommon package
• Add proxy settings to enroll command (enroll reads some settings from the config file and applies those, but CLI flags are missing)
• Documentation (many inputs/modules have been touched)

Why is it important?

The change improves consistency between different features that use HTTP. Plus the common copy and past boilerplate is removed and concentrated in a single place.

All Features that use HTTP will gain proxy support.

More APM HTTP instrumentation for HTTP based features.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
  - [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Setup an HTTP proxy and try to connect via said proxy:

• Elasticsearch output
• Kibana dashboard setup
• Elastic Agent artifact download
• Metricbeat modules (chose one... e.g. Elasticsearch module)

Related issues

• Relates Elastic Agent should have official support for HTTP_PROXY #25021 [Fleet] Add support for proxy in Elastic Agent #25319
• Closes Add support for proxy_url to the dashboard setup task #8276

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: urso commented: /test

• Start Time: 2021-06-29T18:48:22.377+0000

• Duration: 86 min 25 sec

• Commit: 7e7bb06

Test stats 🧪

Test	Results
Failed	0
Passed	29616
Skipped	3876
Total	33492

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	29616
Skipped	3876
Total	33492

[ph]

This is a super great start! cc @mostlyjason @alvarolobato

[urso]

Argh... I forgot to adapt legacy Central Managent support and broke the build :(

[michalpristas]

http2 change with fallback looks good

[urso]

/test

[blakerouse]

Does this need to be http2.Client? Or will the httpcommon.WithForceAttepmtHTTP2(true) switch it to http2 transparently?

[urso]

the net/http package can upgrade to http2. HTTP 1 or 2 is handled by the transport.

[blakerouse]

Changes look good.