Add label manipulation functions

[brian-brazil]

We need functions in promql to allow playing around with labels, to allow for more advanced use cases such as dealing with mismatched label names.

I propose the following functions:

• label_set(vector, labelname, labelvalue)
• label_copy(vector, dest_labelname, src_labelname)
• label_join(vector, dest_labelname, separator, src_labelname...)
• label_replace(vector, dest_labelname, replacement, src_labelame, regex)

Combining label_join and label_replace you can do any merging and mixing you want of labels. The alternative to having two functions would be to offer a way to use go templates, which I don't think is wise for performance reasons.

[brian-brazil]

Set and copy can be done with replace, so we could have just join and replace.

[juliusv]

I'm going to implement label_replace now as a first thing. label_join having an arbitrary number of source labels is IIUC the only reason why we want the dest_..., src_... arguments in that order, right?

[fabxc]

dest first is also the order of Go's builtin copy(). I find it easier to read in this context, too.

[juliusv]

Yeah, that order is common in some contexts. Ok, will keep it like that then.

[brian-brazil]

label_join having an arbitrary number of source labels is IIUC the only reason why we want the dest_..., src_... arguments in that order, right?

Yes

[varunpatro]

Hi what's the status of this PR? Only label_replace has been completed am I right?
I am keen on working on this PR for Hacktoberfest.

[brian-brazil]

That's correct. label_join is what's needed to complete this PR, as it allows all possible functionality. At some vague future date we may add the other two as syntactic sugar around label_replace.

[varunpatro]

Alright thanks. I'll work on label_join then. Should I work on master or some other branch?

[fabxc]

Awesome, you can file that PR straight to master.

On Mon, Oct 5, 2015 at 6:54 PM Varun Patro notifications@github.com wrote:

Alright thanks. I'll work on label_join then. Should I work on master or
some other branch?

—
Reply to this email directly or view it on GitHub
#959 (comment)
.

[brian-brazil]

To answer your question on IRC: https://github.com/prometheus/prometheus/blob/master/promql/functions.go#L819

Allow for up to say 8 labels to be specified, that should be more than enough.

[hryamzik]

How about label_remove? I'm playing around labels and miss this function indeed.

P.S.: the scenario as a bit complicated.
I have two prometheus hosts for redundancy set up to two alert managers with nginx upstreams in backup mode. Both servers have some server-side checks implemented as shell scripts with output to prom files for node_exporter. So two servers export the same checks but the instance label is different. It results in duplicated alerts.

[brian-brazil]

That's not something that should be handled at the PromQL level. You should either exclude instance from your grouping labels in the alertmanager, or use aggregation to remove the label in the alert.

[hryamzik]

I can't do this as other metrics are instance-related. But I've found a way to do this:

label_replace(certificate_expiration_date,"instance", "", "instance", ".*")

[brian-brazil]

May I suggest using the blackbox exporter for this? It exports SSL cert expiry dates, and the labels should be nicer to work with.

[hryamzik]

@brian-brazil good point, I didn't mention this exporter, thank you!

But I can't see a way to check domain expiration and ssl certificates on multiple hosts.

[brian-brazil]

It does support multiple hosts, but doesn't do domain expiration currently. Would need to be careful on that one, don't want to overwhelm Whois.

[hryamzik]

I simply run cron scripts once an hour. Whois is a nightmare with it's format so I had to implement failing on parse error.

#!/bin/bash
# Scipt checks domain expiration dates.
# Checked with reg.ru and GoDaddy.com
# whois is required

scriptname=${0##*/}
metricname=${scriptname%.*}


function get_date_from_string {
    if exp_date=$(date -d $2 +%s)
    then
        printf 'domain_check_expiration_date{name="%s"} %s\ndomain_check_success{name="%s"} 1\n' $1 $exp_date $1
    else
        echo "invalid date '$2'" > /dev/stderr
        printf 'domain_check_expiration_date{name="%s"} 0\ndomain_check_success{name="%s"} 0\n' $1 $1
    fi
}

function check_expiration {
    whois=$(whois $1)
    if exp_date_raw=$(echo "$whois"|egrep -i 'Registry Expiry Date|Registrar Registration Expiration Date')
    then
        # generic & godaddy
        # echo "godaddy or generic" > /dev/stderr
        get_date_from_string $1 $(echo $exp_date_raw|awk -F ': ' {'print $2'})
    elif exp_date_raw=$(echo "$whois"|egrep -i 'free-date')
    then
        # reg.ru
        # echo "reg.ru" > /dev/stderr
        get_date_from_string $1 $(echo $exp_date_raw|awk -F ': ' {'print $2'}|tr '.' '-')
    else
        echo "Unsupported whois format for $1" > /dev/stderr
        printf 'domain_check_expiration_date{name="\s"} 0\ndomain_check_success{name="\s"} 0\n' $1 $1
    fi
}


echo -n '' > /tmp/${metricname}.prom.$$
{% for domain in check_domains %}
check_expiration {{ domain }} >> /tmp/${metricname}.prom.$$
{% endfor %}

cat <<EOF>> /tmp/${metricname}.prom.$$
${metricname}_completion_time $(date +%s)
EOF
mv -v /tmp/${metricname}.prom.$$ /var/lib/node_exporter/${metricname}.prom

[leonerd]

Would there be any mileage in allowing multiple destination labels somehow too? E.g. if you wanted to split an "address" label into separate host + port fields. Or to specifically remove a label by setting its value to the empty string.

Also another thing is I quite often I end up doing a simple match of (.*) and a replacement of $1, effectively just renaming a label. Maybe a convenient shortcut of

label_rename(vector, dest_labelname, src_labelname)

with the added effect of removing the old label name at the same time.

[brian-brazil]

We have join and replace now which are the minimal set of functions.

[bric3]

Adding a label don't work if there is no label. I'm trying to integrate a metric into a table of existing values. However the label_replace don't activate when there's no label.

label_replace(
  sum(http_requests_total{service="$service",handler=~".*"}),
  "handler", "all", "", ".+"
  )

---

Prom 1.8

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.