Multiple prometheus_scrape endpoints not scraped in parallel

[wjordan]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I am using Vector to scrape metrics from a large number (thousands) of instances using the prometheus_scrape source component.

I expected the URLs set in the endpoints config to be scraped in parallel, but it appears to be scraping sequentially. This is not very useful since a single slow or broken endpoint prevents all other endpoints from being scraped properly. Other Prometheus agents scrape the provided endpoints in parallel.

A workaround is to create a separate prometheus_scrape component for each endpoint, but creating a large number (thousands) of components creates heavy CPU load and seems to go against Vector's design.

Configuration

Here's a minimal configuration that reproduces the issue:

[sources.internal_src]
type = "internal_metrics"

[transforms.internal]
type = "filter"
inputs = ["internal_src"]
condition = '.name == "component_sent_bytes_total"'

[sinks.prom_export]
type = "prometheus_exporter"
inputs = ["internal"]
address = "0.0.0.0:9598"

[sources.prom_scrape]
type = "prometheus_scrape"
endpoints = [
    "http://172.0.0.0/metrics",
    "http://127.0.0.1:9598/metrics"
]
endpoint_tag = "endpoint"

[sinks.console]
type = "console"
inputs = ["prom_scrape"]
encoding.codec = "text"

Version

vector 0.30.0 (x86_64-unknown-linux-gnu 38c3f0b 2023-05-22 17:38:48.655488673)

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[nullren]

Just came across this issue trying to configure about 1500 nodes for prometheus_scrape—completely unusable in its current state. I'm happy to pick up the work started in #17660. Seems like there are just a couple of tasks to do?

This still needs some additional work:

1. (blocking issue) Pending requests can start to pile up on the heap causing unbounded memory growth. Scrape requests should implement a timeout less than the interval duration to prevent this.

   • Alternatively, skip future requests for the same endpoint if a previous scrape request still hasn't completed.

• scrape requests with a maximum timeout of the scrape interval value seems like a sane default, eg Request timeouts hyperium/hyper#1097 (comment)

noticed there are a couple more issues related to this:

• http scraping with an IntervalStream piles up requests if the server takes > interval_secs to respond #14087
• prometheus_scrape source hangs indefinitely if the endpoint never responds #14132

2. Each request should spawn in a separate short-lived task to spread out the request-processing load across many threads.

   • Alternatively, each endpoint could spawn and reuse a single long-lived task which could be more efficient.

long-lived tasks might be a larger refactor as you'd need to clean them up if the config changes and removes an endpoint.

3. The timing of endpoint requests should be distributed across the scrape interval instead of all executing at the same time, to spread out the scrape-request load more evenly.

"should" but letting tokio do its thing like @jszwedko mentioned seems more pragmatic.

I'm happy to submit a new PR.

[wjordan]

Thanks for picking this up!