Add prepared statements performance optimisation to increase observation performance x2

CHANGELOG.md

@@ -8,10 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- New optional ``label_keys`` parameter for ``histogram()`` metrics
+- New optional ``label_keys`` parameter for ``summary()`` metrics
+- Prepared statements feature for performance optimization: ``:prepare()`` method on collectors
+  to cache ``label_pairs`` and reduce GC pressure from ``make_key()`` string operations
+
 ### Changed
 
 ### Fixed
 
+- Turnend `Shared:make_key` into a Lua table method
+- Make it impossible to override the `label_keys` when calling `Shared:make_key`
+
 ### Removed
 
 # [1.6.1] - 2025-10-20

doc/monitoring/api_reference.rst

@@ -244,7 +244,7 @@ The metric also displays the count of measurements and their sum:
 
 The design is based on the `Prometheus histogram <https://prometheus.io/docs/concepts/metric_types/#histogram>`__.
 
-..  function:: histogram(name [, help, buckets, metainfo])
+..  function:: histogram(name [, help, buckets, metainfo, label_keys])
 
     Register a new histogram.
 
@@ -254,6 +254,8 @@ The design is based on the `Prometheus histogram <https://prometheus.io/docs/con
                           The infinity bucket (``INF``) is appended automatically.
                           Default: ``{.005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0, 7.5, 10.0, INF}``.
     :param table metainfo: collector metainfo.
+    :param table label_keys: predefined label keys to optimize performance.
+        When specified, only these keys can be used in ``label_pairs``.
 
     :return: A histogram object.
 
@@ -415,7 +417,7 @@ Also, the metric exposes the count of measurements and the sum of observations:
 
 The design is based on the `Prometheus summary <https://prometheus.io/docs/concepts/metric_types/#summary>`__.
 
-..  function:: summary(name [, help, objectives, params, metainfo])
+..  function:: summary(name [, help, objectives, params, metainfo, label_keys])
 
     Register a new summary. Quantile computation is based on the
     `"Effective computation of biased quantiles over data streams" <https://ieeexplore.ieee.org/document/1410103>`_
@@ -449,6 +451,8 @@ The design is based on the `Prometheus summary <https://prometheus.io/docs/conce
         Default value: ``{max_age_time = math.huge, age_buckets_count = 1}``.
 
     :param table metainfo: collector metainfo.
+    :param table label_keys: predefined label keys to optimize performance.
+        When specified, only these keys can be used in ``label_pairs``.
 
     :return: A summary object.
 
@@ -528,6 +532,78 @@ The example above allows extracting the following time series:
 You can also set global labels by calling
 ``metrics.set_global_labels({ label = value, ...})``.
 
+..  _metrics-api_reference-prepared_statements:
+
+Prepared statements
+-------------------
+
+When working with metrics intensively, the ``make_key()`` function used internally
+to create observation keys can cause GC pressure due to string operations.
+To optimize performance, each collector provides a ``:prepare()`` method that
+creates a prepared statement object.
+
+A prepared statement caches the ``label_pairs`` and the internal key, allowing
+repeated operations with the same labels without the overhead of key generation.
+Prepared objects have the same methods as their parent collectors, but without
+the ``label_pairs`` parameter since the labels are already cached.
+
+..  method:: collector_obj:prepare(label_pairs)
+
+    Create a prepared statement for the given ``label_pairs``.
+
+    :param table label_pairs: table containing label names as keys,
+                              label values as values. Note that both
+                              label names and values in ``label_pairs``
+                              are treated as strings.
+
+    :return: A prepared object with methods specific to the collector type.
+
+    :rtype: prepared_obj
+
+..  class:: prepared_obj
+
+    Prepared objects have methods corresponding to their collector type:
+
+    * **Counter prepared object**: ``inc(num)``, ``reset()``, ``remove()``
+    * **Gauge prepared object**: ``inc(num)``, ``dec(num)``, ``set(num)``, ``reset()``, ``remove()``
+    * **Histogram prepared object**: ``observe(num)``, ``remove()``
+    * **Summary prepared object**: ``observe(num)``, ``remove()``
+
+    All methods work the same as their collector counterparts, but without
+    the ``label_pairs`` parameter since labels are already cached.
+
+    **Example usage:**
+
+    ..  code-block:: lua
+
+        local metrics = require('metrics')
+
+        -- Create a counter
+        local requests_counter = metrics.counter('http_requests_total')
+
+        -- Prepare a statement for specific labels
+        local post_requests = requests_counter:prepare({method = 'POST', status = '200'})
+
+        -- Use the prepared statement (no label_pairs needed)
+        post_requests:inc(1)
+        post_requests:inc(5)
+
+        -- Another prepared statement for different labels
+        local get_requests = requests_counter:prepare({method = 'GET', status = '200'})
+        get_requests:inc(1)
+
+    **Performance considerations:**
+
+    Prepared statements are most beneficial when:
+
+    * The same ``label_pairs`` are used repeatedly
+    * Metrics are updated in performance-critical code paths
+    * You want to reduce GC pressure from string operations in ``make_key()``
+
+    For one-off operations or infrequently used label combinations, using
+    the regular collector methods with ``label_pairs`` is simpler and
+    doesn't require managing prepared statement objects.
+
 ..  _metrics-api_reference-functions:
 
 Metrics functions

metrics/api.lua

@@ -77,20 +77,20 @@ local function gauge(name, help, metainfo, label_keys)
     return registry:find_or_create(Gauge, name, help, metainfo, label_keys)
 end
 
-local function histogram(name, help, buckets, metainfo)
-    checks('string', '?string', '?table', '?table')
+local function histogram(name, help, buckets, metainfo, label_keys)
+    checks('string', '?string', '?table', '?table', '?table')
     if buckets ~= nil and not Histogram.check_buckets(buckets) then
         error('Invalid value for buckets')
     end
 
-    return registry:find_or_create(Histogram, name, help, buckets, metainfo)
+    return registry:find_or_create(Histogram, name, help, buckets, metainfo, label_keys)
 end
 
-local function summary(name, help, objectives, params, metainfo)
+local function summary(name, help, objectives, params, metainfo, label_keys)
     checks('string', '?string', '?table', {
         age_buckets_count = '?number',
         max_age_time = '?number',
-    }, '?table')
+    }, '?table', '?table')
     if objectives ~= nil and not Summary.check_quantiles(objectives) then
         error('Invalid value for objectives')
     end
@@ -107,7 +107,7 @@ local function summary(name, help, objectives, params, metainfo)
         error('Age buckets count and max age must be present only together')
     end
 
-    return registry:find_or_create(Summary, name, help, objectives, params, metainfo)
+    return registry:find_or_create(Summary, name, help, objectives, params, metainfo, label_keys)
 end
 
 local function set_global_labels(label_pairs)

metrics/collectors/counter.lua

@@ -1,19 +1,19 @@
 local Shared = require('metrics.collectors.shared')
 
-local Counter = Shared:new_class('counter')
+local Counter = Shared:new_class('counter', {'inc', 'reset'})
 
-function Counter:inc(num, label_pairs)
+function Counter.Prepared:inc(num)
     if num ~= nil and type(tonumber(num)) ~= 'number' then
         error("Counter increment should be a number")
     end
     if num and num < 0 then
         error("Counter increment should not be negative")
     end
-    Shared.inc(self, num, label_pairs)
+    Shared.Prepared.inc(self, num)
 end
 
-function Counter:reset(label_pairs)
-    Shared.set(self, 0, label_pairs)
+function Counter.Prepared:reset()
+    Shared.Prepared.set(self, 0)
 end
 
 return Counter

metrics/collectors/histogram.lua

@@ -7,7 +7,7 @@ local INF = math.huge
 local DEFAULT_BUCKETS = {.005, .01, .025, .05, .075, .1, .25, .5,
                          .75, 1.0, 2.5, 5.0, 7.5, 10.0, INF}
 
-local Histogram = Shared:new_class('histogram', {'observe_latency'})
+local Histogram = Shared:new_class('histogram', {'observe', 'observe_latency'})
 
 function Histogram.check_buckets(buckets)
     local prev = -math.huge
@@ -20,19 +20,25 @@ function Histogram.check_buckets(buckets)
     return true
 end
 
-function Histogram:new(name, help, buckets, metainfo)
+function Histogram:new(name, help, buckets, metainfo, label_keys)
     metainfo = table.copy(metainfo) or {}
-    local obj = Shared.new(self, name, help, metainfo)
+    local obj = Shared.new(self, name, help, metainfo, label_keys)
 
     obj.buckets = buckets or DEFAULT_BUCKETS
     table.sort(obj.buckets)
     if obj.buckets[#obj.buckets] ~= INF then
         obj.buckets[#obj.buckets+1] = INF
     end
 
-    obj.count_collector = Counter:new(name .. '_count', help, metainfo)
-    obj.sum_collector = Counter:new(name .. '_sum', help, metainfo)
-    obj.bucket_collector = Counter:new(name .. '_bucket', help, metainfo)
+    obj.count_collector = Counter:new(name .. '_count', help, metainfo, label_keys)
+    obj.sum_collector = Counter:new(name .. '_sum', help, metainfo, label_keys)
+
+    local bkt_label_keys = table.copy(label_keys)
+    if bkt_label_keys ~= nil then
+        table.insert(bkt_label_keys, 'le')
+    end
+
+    obj.bucket_collector = Counter:new(name .. '_bucket', help, metainfo, bkt_label_keys)
 
     return obj
 end
@@ -44,10 +50,28 @@ function Histogram:set_registry(registry)
     self.bucket_collector:set_registry(registry)
 end
 
+function Histogram:prepare(label_pairs)
+    local buckets_prepared = table.new(0, #self.buckets)
+    for _, bucket in ipairs(self.buckets) do
+        local bkt_label_pairs = table.deepcopy(label_pairs) or {}
+        if type(bkt_label_pairs) == 'table' then
+            bkt_label_pairs.le = bucket
+        end
+
+        buckets_prepared[bucket] = Counter.Prepared:new(self.bucket_collector, bkt_label_pairs)
+    end
+
+    local prepared = Histogram.Prepared:new(self, label_pairs)
+    prepared.count_prepared = Counter.Prepared:new(self.count_collector, label_pairs)
+    prepared.sum_prepared = Counter.Prepared:new(self.sum_collector, label_pairs)
+    prepared.buckets_prepared = buckets_prepared
+
+    return prepared
+end
+
 local cdata_warning_logged = false
 
-function Histogram:observe(num, label_pairs)
-    label_pairs = label_pairs or {}
+function Histogram.Prepared:observe(num)
     if num ~= nil and type(tonumber(num)) ~= 'number' then
         error("Histogram observation should be a number")
     end
@@ -58,32 +82,26 @@ function Histogram:observe(num, label_pairs)
         cdata_warning_logged = true
     end
 
-    self.count_collector:inc(1, label_pairs)
-    self.sum_collector:inc(num, label_pairs)
-
-    for _, bucket in ipairs(self.buckets) do
-        local bkt_label_pairs = table.deepcopy(label_pairs)
-        bkt_label_pairs.le = bucket
+    self.count_prepared:inc(1)
+    self.sum_prepared:inc(num)
 
+    for bucket, bucket_prepared in pairs(self.buckets_prepared) do
         if num <= bucket then
-            self.bucket_collector:inc(1, bkt_label_pairs)
+            bucket_prepared:inc(1)
         else
             -- all buckets are needed for histogram quantile approximation
             -- this creates buckets if they were not created before
-            self.bucket_collector:inc(0, bkt_label_pairs)
+            bucket_prepared:inc(0)
         end
     end
 end
 
-function Histogram:remove(label_pairs)
-    assert(label_pairs, 'label pairs is a required parameter')
-    self.count_collector:remove(label_pairs)
-    self.sum_collector:remove(label_pairs)
+function Histogram.Prepared:remove()
+    self.count_prepared:remove()
+    self.sum_prepared:remove()
 
-    for _, bucket in ipairs(self.buckets) do
-        local bkt_label_pairs = table.deepcopy(label_pairs)
-        bkt_label_pairs.le = bucket
-        self.bucket_collector:remove(bkt_label_pairs)
+    for _, bucket_prepared in pairs(self.buckets_prepared) do
+        bucket_prepared:remove()
     end
 end