[e2ealerting] Support pass through labels & custom histogram buckets

[callumj]

We're using e2ealerting across our k8s fleet to monitor latency from ingestor to AM and want to be able to break that down by specific tags like cluster name or AZ. We could solve this by generating one rule for each but this becomes harder to manage as our cluster grows automatically. Instead this adds the ability to pass through custom labels from AM to the histogram and improves the timestamp seen logic to track this by the particular facets.

It also provides customization for the histogram buckets for those that want to get more granular on specific details.

[gotjosh]

Hi @callumj thank you very much for your contribution! This is gladly something we can bake into e2ealerting.

I started to review this but found the changes a bit hard to follow all at once, do you mind if we split this PR into smaller increments so that we can work together?

From what I can see we have three changes:

1. Configurable Histogram Buckets
2. Healthcheck endpoint
3. Support tracking of custom labels

It'd be my preference if we can start with 1/2 and then move into 3. This tool is quite sensitive to changes given it's nature and I want to make sure we get it right from the get go 🙏

[callumj]

Good call @gotjosh, will open some new PRs.