macropro

A generic, language-agnostic Go library for parsing and expanding Grafana macros in query strings.

Grafana macros take the form $__name or $__name(arg1, arg2, …) and are expanded at query time to inject dynamic values — time ranges, grouping intervals, table names, and so on. macropro provides the parsing engine and a set of dialect-neutral defaults; datasource backends register their own handlers on top.

Installation

go get github.com/grafana/macropro

Requires Go 1.23 or later (uses generics and maps.Copy).

Quick start

package main

import (
    "fmt"
    "time"

    "github.com/grafana/macropro"
)

func main() {
    ctx := macropro.QueryContext[struct{}]{
        TimeRange: macropro.TimeRange{
            From: time.Now().Add(-time.Hour),
            To:   time.Now(),
        },
        Interval:   5 * time.Minute,
        IntervalMS: 300_000,
        Table:      "metrics",
    }

    result, err := macropro.Interpolate(
        "SELECT * FROM $__table WHERE $__timeFilter(created_at) GROUP BY $__timeGroup(created_at, $__interval)",
        macropro.DefaultMacros[struct{}](),
        ctx,
    )
    if err != nil {
        panic(err)
    }
    fmt.Println(result)
    // SELECT * FROM metrics WHERE created_at >= '2024-01-01T23:00:00Z' AND created_at <= '2024-01-02T00:00:00Z' GROUP BY FLOOR(UNIX_TIMESTAMP(created_at)/300)*300
}

Datasource-specific macros

Provide datasource-specific handlers by merging them on top of the defaults. The generic type parameter T carries any extra context fields your handlers need.

type ClickHouseCtx struct {
    Database string
}

overrides := macropro.MacroMap[ClickHouseCtx]{
    // Override $__timeFilter with ClickHouse-native syntax.
    "timeFilter": func(ctx macropro.QueryContext[ClickHouseCtx], args []string) (string, error) {
        if len(args) != 1 {
            return "", fmt.Errorf("timeFilter requires 1 argument")
        }
        col := args[0]
        from := ctx.TimeRange.From.Unix()
        to := ctx.TimeRange.To.Unix()
        return fmt.Sprintf("%s >= toDateTime(%d) AND %s <= toDateTime(%d)", col, from, col, to), nil
    },
    // Add a new datasource-specific macro.
    "database": func(ctx macropro.QueryContext[ClickHouseCtx], _ []string) (string, error) {
        return ctx.Extra.Database, nil
    },
}

macros := macropro.MergeMacros(macropro.DefaultMacros[ClickHouseCtx](), overrides)

ctx := macropro.QueryContext[ClickHouseCtx]{
    // ... populate common fields ...
    Extra: ClickHouseCtx{Database: "default"},
}

result, err := macropro.Interpolate(query, macros, ctx)

Migrating from sqlds

Many Grafana SQL datasources use grafana/sqlds for query execution and macro interpolation. sqlds.MacroFunc has a different signature to macropro.MacroFunc[T]:

// sqlds signature
type MacroFunc func(query *sqlutil.Query, args []string) (string, error)

// macropro signature
type MacroFunc[T any] func(ctx QueryContext[T], args []string) (string, error)

The recommended pattern keeps sqlds.Driver for query execution and framing but replaces its interpolation engine entirely: expand macros upstream in the driver's MutateQueryData hook, then return an empty sqlds.Macros{} from Driver.Macros() so sqlds's own scan is a no-op on already-expanded SQL.

Step 1 — Bridge *sqlutil.Query to QueryContext

Write a contextFrom function that maps the sqlds query struct to a macropro context:

func contextFrom(q *sqlutil.Query) macropro.QueryContext[struct{}] {
    return macropro.QueryContext[struct{}]{
        TimeRange: macropro.TimeRange{
            From: q.TimeRange.From,
            To:   q.TimeRange.To,
        },
        Interval:   q.Interval,
        IntervalMS: q.Interval.Milliseconds(),
        Table:      q.Table,
        Column:     q.Column,
    }
}

If your datasource needs extra context (e.g. a database name, cluster ID), define an Extra struct and populate it here.

Step 2 — Define handlers and expose an Interpolate function

Write your macro handlers as macropro.MacroFunc[T], then expose a standalone Interpolate helper so call sites don't need to know about sqlds:

var MyMacros = macropro.MergeMacros(
    macropro.DefaultMacros[struct{}](),
    macropro.MacroMap[struct{}]{
        "timeFilter": func(ctx macropro.QueryContext[struct{}], args []string) (string, error) {
            // ... datasource-specific SQL ...
        },
        // ...
    },
)

func Interpolate(rawSQL string, q *sqlutil.Query) (string, error) {
    return macropro.Interpolate(rawSQL, MyMacros, contextFrom(q))
}

Step 3 — Expand macros in MutateQueryData, return empty sqlds.Macros

In the sqlds.Driver, rewrite each query's rawSql before sqlds sees it, and report no macros so sqlds does not scan a second time:

// Macros satisfies sqlds.Driver. Returning an empty map disables sqlds's own
// macro scan; macros are fully expanded by the time sqlds receives the query.
func (d *Driver) Macros() sqlds.Macros { return sqlds.Macros{} }

func (d *Driver) MutateQueryData(ctx context.Context, req backend.QueryDataRequest) (context.Context, backend.QueryDataRequest) {
    for i, q := range req.Queries {
        req.Queries[i] = expandMacrosInQuery(q)
    }
    return ctx, req
}

func expandMacrosInQuery(q backend.DataQuery) backend.DataQuery {
    var sqq sqlutil.Query
    if err := json.Unmarshal(q.JSON, &sqq); err != nil {
        return q
    }
    if sqq.RawSQL == "" {
        return q
    }
    sqq.TimeRange = q.TimeRange
    sqq.Interval = q.Interval

    expanded, err := Interpolate(sqq.RawSQL, &sqq)
    if err != nil {
        backend.Logger.Error("macro expansion failed", "refId", q.RefID, "error", err)
        return q
    }
    if expanded == sqq.RawSQL {
        return q
    }

    // Round-trip through a raw map so plugin-specific JSON fields
    // (meta.timezone, format, etc.) are preserved verbatim.
    var raw map[string]json.RawMessage
    if err := json.Unmarshal(q.JSON, &raw); err != nil {
        return q
    }
    newRawSQL, err := json.Marshal(expanded)
    if err != nil {
        return q
    }
    raw["rawSql"] = newRawSQL
    newJSON, err := json.Marshal(raw)
    if err != nil {
        return q
    }
    q.JSON = newJSON
    return q
}

Why not keep sqlds's built-in macro handling? sqlds delegates to sqlutil's defaults, several of which emit SQL that is not valid in every dialect — $__timeGroup produces SQL-Server-style datepart() calls, $__timeFrom/$__timeTo emit RFC 3339 string literals that rely on implicit string→DateTime coercion. Expanding with your own MacroMap upstream lets you override those defaults with dialect-correct output and drop the sqlds-side macro wiring entirely.

Expansion errors cannot be returned from MutateQueryData, so the handler logs the error and passes the original query through unchanged; the database then rejects it with its normal syntax error. If you need first-class error reporting, expand in a CallResource or custom query path instead.

Default macros

These are provided by DefaultMacros[T](). The parsing engine is language-agnostic and works on any string (SQL, KQL, ES|QL, PromQL, etc.). DefaultMacros provides SQL-flavoured implementations as a starting point — non-SQL datasources should define their own MacroMap with only the macros they need, and SQL datasources should override any macro that requires dialect-specific syntax.

Macro	Arguments	Default output
$__interval	—	Interval as a duration string, e.g. 5m
$__interval_ms	—	Interval in milliseconds, e.g. 300000
$__timeFrom()	—	Start of time range as RFC 3339, e.g. 2024-01-01T00:00:00Z
$__timeTo()	—	End of time range as RFC 3339
$__timeFilter(col)	column name	col >= 'from' AND col <= 'to'
$__timeGroup(col, interval)	column name, duration	FLOOR(UNIX_TIMESTAMP(col)/N)*N
$__table	—	Table name from QueryContext.Table
$__column	—	Column name from QueryContext.Column

Comment stripping

Interpolate automatically strips standard SQL line comments (--) and block comments (/* */) before expanding macros, so macro tokens hidden inside comments are never evaluated. For finer control — disabling stripping, stripping MySQL-style hash comments, or Flux-style // — pass WithComments (see below).

You can also call StripComments directly. It is length-preserving: comment regions are replaced with spaces so line/column positions in error messages remain accurate.

clean := macropro.StripComments(query, macropro.HashComment)
result, err := macropro.Interpolate(clean, macros, ctx)

Available CommentStyle flags:

Flag	Strips / preserves
LineComment	Strips -- … until end of line
BlockComment	Strips /* … */
HashComment	Strips # … until end of line (MySQL-style)
SlashComment	Strips // … until end of line (Flux-style)
DollarQuote	Preserves PostgreSQL $tag$…$tag$ dollar-quoted strings
BacktickQuote	Preserves MySQL `col name` backtick identifiers (with `` escape)
BracketQuote	Preserves T-SQL [col name] bracket identifiers (with ]] escape)
BackslashEscape	Treats \x as a two-byte escape inside '…' and "…" — required for MySQL with NO_BACKSLASH_ESCAPES=OFF (the default)

Flags can be combined with \|. Pass 0 to strip nothing.

Dialect recipes:

Dialect	Style
Generic SQL (default)	LineComment | BlockComment
PostgreSQL	LineComment | BlockComment | DollarQuote
MySQL	LineComment | BlockComment | HashComment | BacktickQuote | BackslashEscape
MSSQL / T-SQL	LineComment | BlockComment | BracketQuote
InfluxDB Flux	SlashComment | BlockComment

Options

Both the default $__ prefix and the default comment-stripping set can be overridden per-call via functional options. This is essential for query languages that don't follow Grafana SQL conventions.

// Flux uses "v." as its variable prefix and "//" for line comments.
result, err := macropro.Interpolate(query, fluxMacros, ctx,
    macropro.WithPrefix("v."),
    macropro.WithComments(macropro.SlashComment|macropro.BlockComment),
)

// Legacy InfluxQL bare "$" macros (e.g. $timeFilter, $interval):
result, err := macropro.Interpolate(query, legacyMacros, ctx,
    macropro.WithPrefix("$"),
)

Available options:

Option	Purpose	Default
WithPrefix(string)	Macro prefix to scan for	"$__"
WithComments(CommentStyle)	Comment styles auto-stripped before expansion; 0 disables	LineComment|BlockComment

If your query language uses more than one prefix family in the same query (as InfluxDB Flux does, with both $__interval and v.timeRangeStart), call Interpolate multiple times with different prefixes. Each call operates on the output of the previous one.

API reference

// Interpolate replaces all recognised <prefix><name> macros in query.
// Unknown macros are left unchanged. By default, the prefix is "$__" and
// standard SQL comments are stripped. Use WithPrefix / WithComments to
// override. Returns the first handler error encountered, with the macro
// name included.
func Interpolate[T any](query string, macros MacroMap[T], ctx QueryContext[T], opts ...Option) (string, error)

// WithPrefix overrides the macro prefix used when scanning. Default "$__".
func WithPrefix(prefix string) Option

// WithComments overrides the set of comment styles auto-stripped before
// expansion. Default LineComment|BlockComment. Pass 0 to disable stripping.
func WithComments(style CommentStyle) Option

// MergeMacros returns a new MacroMap with overrides merged on top of base.
// For identical names, the override wins. The base map is not mutated.
func MergeMacros[T any](base, overrides MacroMap[T]) MacroMap[T]

// DefaultMacros returns the standard set of Grafana macros with RFC 3339 /
// generic SQL implementations.
func DefaultMacros[T any]() MacroMap[T]

// StripComments removes comment regions from query, replacing them with
// spaces to preserve byte positions.
func StripComments(query string, style CommentStyle) string

Types

type QueryContext[T any] struct {
    TimeRange  TimeRange
    Interval   time.Duration
    IntervalMS int64
    Table      string
    Column     string
    Extra      T // datasource-specific fields
}

type TimeRange struct {
    From time.Time
    To   time.Time
}

type MacroFunc[T any] func(ctx QueryContext[T], args []string) (string, error)
type MacroMap[T any] map[string]MacroFunc[T]

Implementation notes

• Single-pass greedy name reading: the parser makes one forward pass over the input. At each prefix occurrence it consumes the longest run of identifier characters and then consults the MacroMap, so $__interval_ms is always matched in full before $__interval is ever considered — no longest-first sort required.
• No-prefix fast paths: if the prefix does not occur in the input, Interpolate returns the original string without allocating or stripping comments. If the prefix only occurs inside a comment, the post-strip check short-circuits before allocating a strings.Builder.
• Bracket-depth argument parsing: arguments are split by , while tracking bracket depth and respecting single- and double-quoted strings, so $__wrap(COALESCE(a, b), c) correctly yields two arguments.
• Panic isolation: a panic inside a MacroFunc is caught and returned as an error. Handlers are a trust boundary; one buggy handler will not crash the caller.
• Error safety: if a handler returns an error, Interpolate returns the original unmodified query alongside the error.
• No SQL assumptions: the parser works on any string; only the default macro implementations produce SQL.

Security considerations

macropro is a string-templating engine. It has no notion of SQL syntax, no notion of identifiers vs. literals, and it does not escape anything. Treat it accordingly.

Macro arguments are spliced, not escaped

The built-in handlers interpolate arguments directly into their output:

// $__timeFilter(col) expands to:
//   col >= 'from' AND col <= 'to'
return fmt.Sprintf("%s >= '%s' AND %s <= '%s'", col, from, col, to), nil

There is no quoting, no identifier validation, and no type checking. If the string passed as col contains SQL syntax, that syntax ends up in the final query. The same applies to every custom handler you write — the output string is a raw fragment spliced into the query.

Trust model

Interpolate is safe only if the query template and macro arguments originate from a trusted author (typically the dashboard editor). A concrete way to think about it: the strings you pass to Interpolate are code, not data.

The library is not safe if you:

• Concatenate template-variable values, HTTP parameters, or any other untrusted input into the query string before calling Interpolate. The attacker can trivially close a macro argument and inject arbitrary SQL — this is plain SQL injection, not a macropro bug, but the macro layer does nothing to mitigate it.
• Populate QueryContext.Table, QueryContext.Column, or any Extra field used by a handler from unsanitised input. These values are spliced into output with no escaping.
• Accept arbitrary macro definitions from untrusted code. A malicious MacroFunc has full control over the rewritten query string.

Responsibilities by layer

Layer	Must do
Caller of Interpolate	Sanitise any untrusted input before it reaches the query string or QueryContext. Use parameterised queries at the driver level where possible.
Handler author	Quote, escape, or validate arguments as required by the target dialect. If an argument is expected to be an identifier, reject or quote non-identifier input.
Library	Parse $__name(args) tokens correctly, strip comments so hidden macros don't execute, consume the longest valid name at each prefix occurrence, contain handler panics, terminate. Nothing else.

Known parser limitations

These are not security boundaries, but are worth understanding when reasoning about what the parser treats as a macro vs. what it leaves alone:

• Quote tracking in StripComments recognises SQL-standard doubled-quote escapes ('', "") by default. Backslash escapes (\', \") are only honoured when BackslashEscape is set — callers targeting MySQL's default NO_BACKSLASH_ESCAPES=OFF mode must include the flag, or a macro token appearing after a backslash-escaped quote can be misparsed.
• Error messages returned by Interpolate may include raw argument text from the query. If those errors are logged, treat them as query fragments (potentially sensitive) and scrub accordingly.

License

Apache 2.0 — see LICENSE.