search: trace and observe each zoekt host

[keegancsmith]

This commit moves tracing and observability from being around the
aggregated zoekt client to being both the aggregated client and a trace
per zoekt replica.

To do this we adjust the prometheus metric to have a hostname label. We
also change the category / family name to indicate if its a search
against a specific zoekt replica or its the aggregation.

[pecigonzalo]

In theory, each service should already have an identifier as prometheus adds it to it from discovery. Ill verify.

---

Yeah, they have instance already. I think Im getting this the other way around and this is the hostname of the target, is that the case?

[uwedeportivo]

@pecigonzalo yes, correct

[pecigonzalo]

I would be careful with sort of metrics as they can create a cardinality explosion for the metrics DB. In most cases, I believe the downstream service should actually provide the metrics if possible, and this service only provide his health (general latency to the upstream, etc)

[uwedeportivo]

@pecigonzalo yes, you are right. you would be surprised of our labelling in general :-). this one is "only" a multiplier of 15 at the bigdata cluster. we have far worse offenders in our code base. we need to clean and reduce sometime. this one is super useful for our current debug situation with our bigdata customer.