Adding Custom metrics

[esatterwhite]

I'm trying to add additional prometheus metrics from a plugin. I'm close, but its generally complaining about the metrics being uninitialized

local exporter = require('kong.plugins.prometheus.exporter')

local kong = kong
local table_new = kong.table.new
local get_prometheus = exporter.get_prometheus

local register = table_new(0, 3)

local function init()
  local prometheus = get_prometheus()
  register.kong_request_ratelimit_reached = prometheus:counter(
    'kong_request_rate_limit_reached'
  , 'total request that have reached a rate limit threshold'
  , {'service', 'route', 'type'}
  )

  return register
end

return {
  init = init
, register = register
}

and my plugin is doing this

  local Plugin = {Priority = 900}
  function Plugin:init_worker(conf)
    metrics.init()
  end

When I run a test on it (pongo)

I get this

2022/01/07 17:26:03 [error] 90#0: *2 [lua] prometheus.lua:839: log_error(): counter not initialized! Have you called Prometheus:init() from the init_worker_by_lua_block nginx phase?, context: ngx.timer

From what I can tell, Prometheus machinery is loaded. In the same test I hit the metrics endpoint and dump the out put. It is all there.

The test may be doing something wrong. but is this a thing that is possible?

[esatterwhite]

This is possible, generally doing what was described. I was trying to require the counter instance directly in the test and increment it, but it generally does not like that.

You have to load the plugin + start kong using the test helpers and send a request. That seems to work just fine

[sydpz]

do you solve this problem now?

[esatterwhite]

Yes, I was able to get this to work

[chenmaolin928]

Could you tell me how to solve it in detail? I have also met this problem with you at present, and I have been unable to define metrics, thank you very much

[esatterwhite]

Sure. The thing to keep in mind about these kong plugins, is that "its just lua", and as long as you play by the rules of the kong PDK - anything goes.

Basically what I did was separate each phase handler into separate lua modules

kong
└── plugins
    └── my-plugin
        ├── access.lua
        ├── handler.lua
        ├── metrics.lua
        └── schema.lua

in handler.lua where the plugin class lives, you initialize your metrics lib, and pass it to your phase functions in addition to the plugin config

-- handler.lua

local access = require('kong.plugins.my-plugin.access')
local metrics = require('kong.plugins.my-plugin.metrics')
local version = require('kong.plugins.my-plugin.version')

local register = metrics.register

local Plugin = {
  PRIORITY = 1001, -- determines plugin execution order
  VERSION = version,
  init_worker = nil,
  access = nil,
}

function Plugin:init_worker()
  register = metrics.init()
end

-- runs in the 'access_by_lua_block'
function Plugin:access(conf)
  access(conf, register)
end

-- return our plugin object
return Plugin

So in simple lua terms, I just pass the metrics register as a function parameter

[esatterwhite]

We enable the kong metrics plugin globally, so we don't have to worry about if all the metrics internals within kong are loaded or not, we just assume they are

[chenmaolin928]

Thank you for your answer. My current approach is to directly copy the prometheus source code of kong, and rename it as a custom plug-in. This is also registered successfully, but the built-in indicators of kong are overwritten and when I do not enable the built-in prometheus plugin, my custom plugin will not register metrics. Will you encounter such a problem, can you show me all the codes of other files, thank you

[esatterwhite]

I don't have that problem with my approach because it's using the built in Kong register. It's singleton.

You don't need to copy it, you can just require it like any other lua module.

[chenmaolin928]

I finally copied your code and now found that the indicator can be successfully registered, thank you very much indeed. This question has been bothering me for days.

[flowdopip]

Can someone share the official documentation to add custom metrics on custom plugins?

[esatterwhite]

There isn't any

[abhi-gaur]

Hie, I am using victoriametrics in the replacement of Prometheus, does kong support other tools than Prometheus, because I tried a lot on victoria metrics i am not able to list the job for kong ???

[Blaimi]

For everyone who came here from google, searching for a simple solution:

This exports a new counter-metric kong_my_requests_total. You can also add gauges and histograms.

Hint: To pass values between plugins, you can use kong.log.serialize() and kong.log.set_serialize_value(…)

schema.lua

return {
  name = "my-metrics",
  fields = { },
}

handler.lua

local exporter = require('kong.plugins.prometheus.exporter')
local metrics = {}
local prometheus

local my_labels = { 0, 0 } -- one for every label

local MyMetricsHandler = {
  NAME = 'my-metrics',
  VERSION = "1.0.0",
  PRIORITY = 15,
}

function MyMetricsHandler:init_worker()
  prometheus = exporter.get_prometheus()
  metrics.my = prometheus:counter("my_requests_total", -- kong_ prefix is already set global
    "my labeled request information",
    {"label1", "label2"})
end

function MyMetricsHandler:log(conf)
  my_labels[1] = "content of label1"
  my_labels[2] = "content of label2"
  metrics.my:inc(1, my_labels)
end

return MyMetricsHandler

[flowdopip]

Works as expected! Thanks.
Maybe a official documentation will help all, and now looking to the prometheus lib is the only way to understand what we can build

[vanigupta82]

@Blaimi , i am trying to create custom metrics in my custom rate limit plugin for monitoring rate limit. I tried to implement solution given by you but after implementing custom metrics in ratelimit plugin, its actually stopped working completely other prometheus plugin related metrics as well. below is the implementation i did in my handler file. please help with the solution how to fix prometheus plugin metrics along with custom metrics so that metrics scraping can work properly.

handler.lua

`

local timestamp = require "kong.tools.timestamp"
local exporter = require "kong.plugins.prometheus.exporter"
local prometheus

RateLimitingHandler.VERSION = kong_meta.version
RateLimitingHandler.PRIORITY = 910

function RateLimitingHandler:init_worker()
  -- Register your custom metrics with the Prometheus plugin
  prometheus = exporter.get_prometheus()
  if prometheus then
    prometheus:counter("custom_rate_limit_remaining_quota", "Remaining quota for rate limiting", 0)
  else
    kong.log.err("Prometheus library is not available")
  end
end

local function get_usage(conf, identifier, current_timestamp, limits)
  local usage = {}
  local stop
  local metric_name = "custom_rate_limit_remaining_quota_" .. current_timestamp .. "_" .. identifier
  for period, limit in pairs(limits) do
    local current_usage, err = policies[conf.policy].usage(conf, identifier, period, current_timestamp)
    if err then
      return nil, nil, err
    end

    -- What is the current usage for the configured limit name?
    local remaining = limit - current_usage
    prometheus:counter(metric_name, "Remaining quota for rate limiting", remaining)
    -- Recording usage
    usage[period] = {
      limit = limit,
      remaining = remaining,
    }
   
    if remaining <= 0 then
      stop = period
    end
  end

  return usage, stop
end


local function get_identifier(conf)
 --- some logic to find identifier
end

function RateLimitingHandler:access(conf)
  local current_timestamp = time() * 1000

  -- Consumer is identified by ip address or authenticated_credential id
  local identifier = get_identifier(conf)
  kong.log.info("identifier from the kong request :", identifier)

  local fault_tolerant = conf.fault_tolerant
  
  local limits = get_limits(conf, identifier)
  
  -- apply_rate_limits(conf, identifier, limits, current_timestamp)
  local dotCount = select(2, identifier:gsub("%.", ""))
  
  local usage, stop, err = get_usage(conf, identifier, current_timestamp, limits)

  
end


return RateLimitingHandler
`

[pavan-kumar-06]

i have enabled prometheus plugin as well as this plugin , but i am not able to see the custom metric in prometheus console did any one face the same issue ??

[flowdopip]

The metricname: local metric_name = "custom_rate_limit_remaining_quota_" .. current_timestamp .. "_" .. identifier is different from the defined on INIT prometheus:counter("custom_rate_limit_remaining_quota", "Remaining quota for rate limiting", 0)

Please remind that dynamic metrics name can cause lot of issues on prometheus

[SimonKpf]

Hi everyone,

I'm still struggling with my custom metrics plugin—despite numerous attempts and trying different approaches, my custom metrics still do not show up on the /metrics endpoint.

Here’s my current implementation:

handler.lua

local metrics = require("kong.plugins.custom-metrics.metrics")
local register = metrics.register

local plugin = {
  PRIORITY = 10,  -- tried out different numbers
  VERSION = "0.1.0",
}

function plugin:init_worker()
  register = metrics.init()
  kong.log.info("[custom-metrics] Metrics successfully initialized")
end

function plugin:access(conf)
  local consumer = "test_consumer"
  register.chat_usage_prompt_tokens:inc(27, { consumer = consumer })
  register.chat_usage_total_tokens:inc(74, { consumer = consumer })
  register.chat_usage_completion_tokens:inc(47, { consumer = consumer })

  kong.log.info("[custom-metrics] Updated metrics for consumer: ", consumer)
end

return plugin

metrics.lua

local exporter = require("kong.plugins.prometheus.exporter")
local kong = kong

local register = {}

local function init()
  kong.log.info("[custom-metrics] Initializing Prometheus metrics in metrics.lua")
  local prom = exporter.get_prometheus()
  
  if not prom then
    kong.log.err("[custom-metrics] Error: Prometheus object not obtained!")
    return register
  end

  register.chat_usage_prompt_tokens = prom:counter(
      "chat_usage_prompt_tokens",
      "Prompt tokens used by chat completions",
      {"consumer"}
  )
  register.chat_usage_total_tokens = prom:counter(
      "chat_usage_total_tokens",
      "Total tokens used by chat completions",
      {"consumer"}
  )
  register.chat_usage_completion_tokens = prom:counter(
      "chat_usage_completion_tokens",
      "Completion tokens used by chat completions",
      {"consumer"}
  )

  kong.log.info("[custom-metrics] Metrics initialized: chat_usage_prompt_tokens, chat_usage_total_tokens, chat_usage_completion_tokens")
  return register
end

return {
  init = init,
  register = register,
}

schema.lua

return {
  name = "custom-metrics",
  fields = {
    { config = {
        type = "record",
        fields = {}
      }
    },
  },
}

I'm running on Kong Gateway version 3.3.1.0

Although the log messages indicate that the metrics are initialized and updated, I don’t see any of my custom counters (chat_usage_prompt_tokens, chat_usage_total_tokens, chat_usage_completion_tokens) in the /metrics output.

Despite all this, none of my custom counters appear in the /metrics output. I have tried numerous different approaches and comments above to resolve the issue.

Has anyone encountered this behavior or have any suggestions on how to get my custom metrics to show up? Any help or debugging tips would be greatly appreciated!

Thanks in advance!

[Blaimi]

IIRC, you have to do it in the log phase instead of the access phase.

[SimonKpf]

I tried changing the access phase to the log phase in my Lua files, but it didn't work—I still don't see the custom metrics. I also attempted to use your solution from November 21, 2023, but that didn't work either, as I still can't see your my_requests_total metric
Thanks for taking the time to reply