Add basic auth to /metrics endpoint

[zwass]

Goal

Though the /metrics endpoint doesn't contain anything particularly sensitive, it is desirable to keep this behind authentication.

Prometheus supports scraping with basic auth (see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).

Tasks

• Add a prometheus_username and prometheus_password Server configuration option and use this to authenticate the /metrics endpoint with basic auth.
• If any of the two configuration options are not-defined/empty, then metrics collection and the /metrics endpoint will be disabled.

[noahtalerman]

@gavinelder I'm adding your comment from the separate "[FR] Separate internal / external endpoints" issue here:

The reasoning for /version /metrics was to limit unauthenticated which could expose information about an estate that could be used for reconnaissance purposes but there may be more.

Maybe the first step is to add basic auth for both /metrics and /version? This way, all Fleet endpoints are kept behind basic auth.

Later, Fleet might want to make it easier to allow more granular control over which endpoints are public v. internal.

[lucasmrod]

If the configuration option is not defined, then the user experiences the current unauthenticated behavior

What about the following three options:

1. auth'd /metrics by default: fleet errors out if FLEET_PROMETHEUS_METRICS_USERNAME, FLEET_PROMETHEUS_METRICS_PASSWORD credentials are not set.
2. allow unauth'd /metrics: via explicit setting of FLEET_PROMETHEUS_METRICS_DISABLE_AUTH.
3. no /metrics: via explicit setting of FLEET_PROMETHEUS_METRICS_ENABLE. (see Allow /metrics route to be enabled/disabled via config #4934)

The above is the secure by default approach but we should discuss because it's sort of a "breaking change".

[zwass]

What if we go with metrics are turned off by default and only enabled if the credentials are provided?

I suspect not many teams are actually using the metrics, so we'll break the fewest installations if we go that way.

[lucasmrod]

What if we go with metrics are turned off by default and only enabled if the credentials are provided?

Works too, it's a good trade off between security <-> deployment-ease.

[lucasmrod]

I've updated the issue description.

[gavinelder]

@gavinelder I'm adding your comment from the separate "[FR] Separate internal / external endpoints" issue here:

The reasoning for /version /metrics was to limit unauthenticated which could expose information about an estate that could be used for reconnaissance purposes but there may be more.

Maybe the first step is to add basic auth for both /metrics and /version? This way, all Fleet endpoints are kept behind basic auth.

Later, Fleet might want to make it easier to allow more granular control over which endpoints are public v. internal.

This would probably close down the linked issue in #3614