metric_relabel_configs does not drop __-prefixed labels as documented

[hoffie]

Bug Report

What did you do?
When using a two-step relabeling in metric_relabel_configs, it may be necessary to use temporary labels for storing state. The docs suggest using the __-prefix (or even __tmp prefix) for such temporary labels and states that such labels will be removed after completing relabeling:

Labels starting with __ will be removed from the label set after relabeling is completed.

This does not seem to be the case.

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
    metric_relabel_configs:
      - source_labels: [__name__] 
        regex: 'process_open_fds'
        action: replace
        target_label: __tmp_foo
        replacement: 1

What did you expect to see?
The relabeled metric should not contain a __tmp_foo label.

What did you see instead? Under which circumstances?
The relabeled metric still contains the __tmp_foo label:

$ curl -s 'http://localhost:9090/api/v1/query?query=process_open_fds' | python -m json.tool
{
    "status": "success",
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "__name__": "process_open_fds",
                    "__tmp_foo": "1",
                    "instance": "localhost:9090",
                    "job": "prometheus"
                },
                "value": [
                    1547384438.198,
                    "15"
                ]
            }
        ]
    }
}

Environment

• System information: Linux 4.19.8-arch1-1-ARCH x86_64
• Prometheus version:

  build user:       root@bf5760470f13
  build date:       20181217-15:14:46
  go version:       go1.11.3

This behavior can be worked around by another metric relabeling step, but I think this is an actual bug in Prometheus (or at least a documentation bug, if the cited sentence is supposed to be restricted to relabel_configs).

[brian-brazil]

That logic only applies to target relabelling, the latest docs clarify this.

[hoffie]

That logic only applies to target relabelling, the latest docs clarify this.

I'm afraid I don't fully understand -- I did not find a clarification in the live docs or the Git repo.
From @brian-brazil's comment I assumed that a fix is already in the pipeline somewhere. From @simonpasquier's label change I would assume that someone should send a PR...?

Can someone clarify? I'm happy to send a pull request.

[brian-brazil]

It's there: 2e725a1#diff-7878dab1819673d180662f2d2ff34a22

[simonpasquier]

The documentation website will get the update once prometheus/docs#1273 is merged. @hoffie hope you don't mind if I close this issue now.