Broken example for a custom Counter metrics

[feeeper]

📚 The doc issue

The example in the section Add Counter based metrics shows how to add custom Counter metric:

# Create a counter with name 'LoopCount' and dimensions, initial value
metrics.add_counter('LoopCount', 1, None, dimensions)

# Increment counter by 2 
metrics.add_counter('LoopCount', 2 , None, dimensions)

# Decrement counter by 1
metrics.add_counter('LoopCount', -1, None, dimensions)

I tried to copy this example to my custom handler:

dims = [Dimension('ModelName', 'doc_model')]
self.metrics.add_counter('LoopCount', 1, None, dimensions=dims)
# Increment counter by 2 
self.metrics.add_counter('LoopCount', 2 , None, dimensions=dims)
# Decrement counter by 1
self.metrics.add_counter('LoopCount', -1, None, dimensions=dims)

When I call API for inference I got an error in the terminal:

2024-02-28T15:23:57,011 [ERROR] W-9000-doc_model_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Failed to parse metrics line: "[METRICS]Failed to update metric with name:LoopCount and dimensions: ModelName:doc_model,Level:Model with value: -1: Counter metric update value cannot be negative".

Suggest a potential alternative/fix

No response

[namannandan]

@feeeper This error is expected since counter metrics cannot have negative values. You will need to use a gauge metric for this use case by setting metric_type = MetricTypes.GAUGE: https://github.com/pytorch/serve/blob/v0.9.0/docs/metrics.md#function-api-to-add-generic-metrics-with-default-dimensions.

Torchserve metrics are modeled based on prometheus metrics semantics: https://prometheus.io/docs/concepts/metric_types/

A more detailed examples to emit various types of custom metrics: https://github.com/pytorch/serve/tree/master/examples/custom_metrics

[feeeper]

@namannandan yeah, I got it. But what I want to say that I got this exact code snippet from your documentation. And this snippet doesn't work.

[namannandan]

Metrics documentation will be fixed as a part of this tracking issue: #3001