Can relabelling be used to query multiple services on the same server?

[gouthamve]

Hi,

We have a unique use case, where we are running 10 services (All the same type) on one instance. Can I use EC2 SD relabelling to ask prometheus to scrape 10 ports for the same instance (instanceId)?

Thanks,
Goutham

[brian-brazil]

You'll need 10 scrape configs to do that.

[gouthamve]

Hmm,

Sounds okay as long as I can relabel the job tag to point to the same thing. Thats possible right? I am just about to start a POC on this.

Also, I am thinking instead of having 10 different time series for the same instance, can I relabel the address for all the 10 to be the same address? Or will it change how the metrics are handled?

Thanks,
Goutham.

[brian-brazil]

Yes, that's possible. Job names in scrape configs must be unique, so you'll need to use relabelling on 9 of them to get the same value.

The instance labels will be different for them, which will distinguish them.

[gouthamve]

What if I don't want to distinguish them? I will running a large cluster with 10 services on each machine.

Having the details on a service level will increase the number of series 10x and it doesn't make sense for me to monitor on a service level. Server level monitoring make sense. Also not having a metric level retention policy makes me keep all the series even after using a recording rule to aggregate.

That was what made me go for unifying the instance label for all services on that instance, but I am not sure how it impacts storage/metrics.

[gouthamve]

Hmm, for the 10 scrape-configs, in each scrape config, I am specifying the ec2_sd_config. Will that make 10 lookups?

The describe instances call is throttled easily if a lot of instances are returned. Can we have a global ec2 lookup as, if a lot of jobs are there, throttling will become an issue.

[brian-brazil]

For 10 servers I doubt you have enough time series to run into performance issues.

Yes, it'll do 10 ec2 lookups. If you wish to avoid that I'd suggest doing your own lookups and writing to file_sd.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.