promtail: replace stage source field not used, causes stage to be ignored

[dpkirchner]

Describe the bug
There does not appear to be a way to replace the value in a specific field using the replace stage, as suggested in the documentation.

This could very well be an issue with my config -- I wouldn't be surprised -- but I think I followed the gist of what is in the documentation at https://grafana.com/docs/loki/latest/clients/promtail/stages/replace/ .

To Reproduce
Steps to reproduce the behavior:

1. Start promtail: docker run --rm --entrypoint bash -it grafana/promtail:2.2.1
2. Create the following config file
3. Run echo -n "lvl=info" | promtail --stdin --dry-run --config.file /tmp/promtail.yaml --log.level debug 2>&1

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  pipeline_stages:
  - regex:
      expression: 'lvl=(?P<level>\w+)'
  - labels:
      level:
  - replace:
      source: level
      expression: "(.+)"
      replace: xxx
  static_configs:
  - labels:
      job: my-stdin-logs

Expected behavior
I expected the output to include level=xxx.

Environment:

• Infrastructure: Docker (but also GKE, not part of this repro)
• Deployment tool: Docker

Screenshots, Promtail config, or terminal output

Note the level="info" and lvl=info below:

root@75dc2b5e5c33:/# echo -n "lvl=info" | promtail --stdin --dry-run --config.file /tmp/promtail.yaml --log.level debug 2>&1
Clients configured:
----------------------
url: http://localhost:3100/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
backoff_config:
  min_period: 500ms
  max_period: 5m0s
  max_retries: 10
timeout: 10s
tenant_id: ""

level=debug ts=2021-05-07T20:12:52.4533081Z caller=manager.go:58 msg="configured to read from stdin"
2021-05-07T20:12:52	{job="my-stdin-logs", level="info"}	lvl=info
level=info ts=2021-05-07T20:12:52.4545358Z caller=server.go:229 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2021-05-07T20:12:52.4547656Z caller=main.go:112 msg="Starting Promtail" version="(version=2.2.1, branch=HEAD, revision=babea82ef)"

If you drop the source: level from the config file you'll see:

root@75dc2b5e5c33:/# echo -n "lvl=info" | promtail --stdin --dry-run --config.file /tmp/promtail.yaml --log.level debug 2>&1
Clients configured:
----------------------
url: http://localhost:3100/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
backoff_config:
  min_period: 500ms
  max_period: 5m0s
  max_retries: 10
timeout: 10s
tenant_id: ""

level=debug ts=2021-05-07T20:14:14.2116357Z caller=manager.go:58 msg="configured to read from stdin"
level=info ts=2021-05-07T20:14:14.2122519Z caller=server.go:229 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
2021-05-07T20:14:14	{job="my-stdin-logs", level="info"}	xxx
level=info ts=2021-05-07T20:14:14.213329Z caller=main.go:112 msg="Starting Promtail" version="(version=2.2.1, branch=HEAD, revision=babea82ef)"

where it replaces the entire output.

[3Xpl0it3r]

I look into this, I found is source filed provided, then replace stage will replace the extracted labels with value of replace filed, if source filed is empty, then stage will replace the log with the value of replace filed in config.
and extracted labels will not be passed to client(Loki/log client), that's the reason that you saw {job="my-stdin-logs", level="info"} rather than {job="my-stdin-logs", level="xxxxx"}
.
it not bug ,https://grafana.com/docs/loki/latest/clients/promtail/pipelines/#pipeline
Parsing stages parse the current log line and extract data out of it. The extracted data is then available for use by other stages
replace is one of parsing stages.