Adding the ability to set custom labels on metrics

[jnovick]

Is your feature request related to a problem? Please describe.
When creating batches of jobs, I want the batch number to be a label on the same metric not a part of the metric name. This will make it easier to report both individual batches and all batches over a given time.

Describe the solution you'd like to see
Something like this: #19 (comment)

If you look at the Prometheus best practices, specifically the documentation about the data model and the part about Metric and label naming, you'll see that Prometheus advocates using more generic metric names

I know I can set labels statically using *.sink.<your-sink>.labels=label1=value1,label2=value2 but I want to apply labels where the value is determined at runtime.

Describe alternatives you've considered
I can continue to make the metric name longer by adding to the accumulator, but it would be more in line with Prometheus best practices to use job labels.

Additional context
None that I can think of