Skip to content

Add prepared statements performance optimisation to increase observation performance x2 #534

@maksimuimin

Description

@maksimuimin

I've faced an issue with tarantool/metrics poor performance in a following scenario:

local function observability_middleware(next)
    local called_func = _G.onedb_state.initing_func
    local called_app = _G.onedb_state.loading_app

    return function (...)
        local timer = es_timer:new()
        timer:start()

        local result = { pcall(next, ...) }
        local ok = result[1]

        timer:stop()
        onedb_stat.api_requests_latency:observe(timer:latency(), { method = called_func, app = called_app })

        if not ok then
            onedb_stat.api_requests_counter:inc(1, { method = called_func, app = called_app, state = 'fail' })

            error(result[2])
        end

        onedb_stat.api_requests_counter:inc(1, { method = called_func, app = called_app, state = 'success' })

        return unpack(result, 2, table.maxn(result))
    end
end

^ The idea is to wrap each applications' API function with a given middleware to track latency/throughput/errors on a per-function basis

I've performed a synthetic performance test, on my setup I've had the following results:

  • tarantool with no middleware got 40k RPS, flamegraph;
  • tarantool with middleware got 15k RPS, flamegraph.

In the second falmegraph I see an issue with slow histogram observations. Three main reasons for that are:

  1. intensive label_pairs copying to fill in the le label;
  2. intensive work with strings in make_key for each observation;
  3. GC pressure caused by temporary strings/tables deallocation.

My idea is to introduce "prepared statements". My labels have low cardinality, which means I could make an object that have:

  1. a collector reference;
  2. immutable label_paris set;
  3. cached key, that is computed once and then could be used for infinite number of new observations.

Usage example:

local Histogram = require('metrics.collectors.histogram')
local h = Histogram:new('hist', 'some histogram', {2, 4})
local prepared= h:prepare({label = 1})
prepared:observe(3)
prepared:observe(5)
prepared:observe(7)

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions