How to properly setup remoteWrite credentials in vmagent

[evertonspader-tomtom]

Is your question request related to a specific component?

vmagent

Describe the question in detail

What is the proper way to set up the remoteWrite credentials using environment variables? I can get a successful connection using

remoteWrite.url: https://<username>:<password>@<prometheus_endpoint>

Or by splitting the username and password in remoteWrite.username and remoteWrite.password.
But, of course, for security reasons, this is not ideal. So I am trying to use environment variables to pass extra flags.

So I added the following:

remoteWrite.url: <prometheus_endpoint>
envflag.enable: "true"
envflag.prefix: "VM_"

env:
  - name: VM_remoteWrite_basicAuth_username
    valueFrom:
      secretKeyRef:
        name: prometheus-secret
        key: username
  - name: VM_remoteWrite_basicAuth_password
    valueFrom:
      secretKeyRef:
        name: prometheus-secret
        key: password

But it doesn't seem to pick up the credentials as now I see a 401 in the export requests. What am I missing here? Needless to day that the secret is already created with the correct keys an values.

{"ts":"2024-05-14T09:21:16.539Z","level":"error","caller":"VictoriaMetrics/app/vmagent/remotewrite/client.go:461","msg":"unexpected status code received after sending a block with size 1398 bytes to \"1:secret-url\" during retry #4: 401; response body=\"{\\\"status\\\":\\\"error\\\",\\\"error\\\":\\\"authentication error: invalid authentication credentials\\\"}\"; re-sending the block in 16.059 seconds"}

Troubleshooting docs

• General - https://docs.victoriametrics.com/troubleshooting/
• vmagent - https://docs.victoriametrics.com/vmagent/#troubleshooting
• vmalert - https://docs.victoriametrics.com/vmalert/#troubleshooting

[Haleygo]

Hey @evertonspader-tomtom ,
The above deployment yaml looks right to me. Do you see those auth flags been printed in logs when vmagent start, should be like:

2024-05-16T06:52:15.552Z	info	lib/logger/flag.go:12	build version: 
2024-05-16T06:52:15.552Z	info	lib/logger/flag.go:13	command-line flags
2024-05-16T06:52:15.552Z	info	lib/logger/flag.go:20	  -envflag.enable="true"
2024-05-16T06:52:15.552Z	info	lib/logger/flag.go:20	  -envflag.prefix="VM_"
2024-05-16T06:52:15.552Z	info	lib/logger/flag.go:20	  -httpListenAddr=":8420"
2024-05-16T06:52:15.552Z	info	lib/logger/flag.go:20	  -promscrape.config="/Users/haley/go/src/github.com/VictoriaMetrics-1/VictoriaMetrics/app/vmagent/scrape.yml"
2024-05-16T06:52:15.552Z	info	lib/logger/flag.go:20	  -remoteWrite.basicAuth.password="secret"
2024-05-16T06:52:15.552Z	info	lib/logger/flag.go:20	  -remoteWrite.basicAuth.username="admin"
2024-05-16T06:52:15.552Z	info	lib/logger/flag.go:20	  -remoteWrite.url="secret"
2024-05-16T06:52:15.552Z	info	app/vmagent/main.go:134	starting vmagent at "[:8420]"...

If you can see them, it might be caused by some wrong escape characters in your username or password.

[evertonspader-tomtom]

@Haleygo yeah, that is the problem, I don't see the flags being added, I thought the flags from env variables could be inherited in a different way. See the full logs:

{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:13","msg":"command-line flags"}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":"  -envflag.enable=\"true\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":"  -envflag.prefix=\"VM_\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":"  -loggerFormat=\"json\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":"  -promscrape.config=\"/config/scrape.yml\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":"  -remoteWrite.label=\"owner=team-everton@groups.tomtom.com,release=metric-proxy-vm\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":"  -remoteWrite.tmpDataPath=\"/tmpData\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":"  -remoteWrite.url=\"secret\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/app/vmagent/main.go:127","msg":"starting vmagent at \"[:8429]\"..."}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/memory/memory.go:42","msg":"limiting caches to 2457600000 bytes, leaving 1638400000 bytes to the OS according to -memory.allowedPercent=60"}
{"ts":"2024-05-16T08:11:42.465Z","level":"info","caller":"VictoriaMetrics/lib/persistentqueue/fastqueue.go:66","msg":"opened fast persistent queue at \"/tmpData/persistent-queue/1_2F652A4689A1A365\" with maxInmemoryBlocks=400, it contains 0 pending bytes"}
{"ts":"2024-05-16T08:11:42.623Z","level":"info","caller":"VictoriaMetrics/app/vmagent/remotewrite/client.go:166","msg":"the remote storage at \"1:secret-url\" doesn't support VictoriaMetrics remote write protocol. Switching to Prometheus remote write protocol. See https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol"}
{"ts":"2024-05-16T08:11:42.623Z","level":"info","caller":"VictoriaMetrics/app/vmagent/remotewrite/client.go:202","msg":"initialized client for -remoteWrite.url=\"1:secret-url\""}
{"ts":"2024-05-16T08:11:42.623Z","level":"info","caller":"VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:276","msg":"removing dangling queue \"1_075B615FC191DB76\""}
{"ts":"2024-05-16T08:11:42.626Z","level":"info","caller":"VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:283","msg":"removed 1 dangling queues from \"/tmpData\", active queues: 1"}
{"ts":"2024-05-16T08:11:42.626Z","level":"info","caller":"VictoriaMetrics/app/vmagent/main.go:152","msg":"started vmagent in 0.179 seconds"}
{"ts":"2024-05-16T08:11:42.626Z","level":"info","caller":"VictoriaMetrics/lib/httpserver/httpserver.go:119","msg":"starting server at http://127.0.0.1:8429/"}
{"ts":"2024-05-16T08:11:42.626Z","level":"info","caller":"VictoriaMetrics/lib/promscrape/scraper.go:113","msg":"reading scrape configs from \"/config/scrape.yml\""}
{"ts":"2024-05-16T08:11:42.626Z","level":"info","caller":"VictoriaMetrics/lib/httpserver/httpserver.go:120","msg":"pprof handlers are exposed at http://127.0.0.1:8429/debug/pprof/"}
{"ts":"2024-05-16T08:11:42.627Z","level":"info","caller":"VictoriaMetrics/lib/promscrape/config.go:140","msg":"starting service discovery routines..."}
{"ts":"2024-05-16T08:11:42.627Z","level":"info","caller":"VictoriaMetrics/lib/promscrape/config.go:146","msg":"started 0 service discovery routines in 0.000 seconds"}
{"ts":"2024-05-16T08:12:21.148Z","level":"error","caller":"VictoriaMetrics/app/vmagent/remotewrite/client.go:461","msg":"unexpected status code received after sending a block with size 700 bytes to \"1:secret-url\" during retry #1: 401; response body=\"{\\\"status\\\":\\\"error\\\",\\\"error\\\":\\\"authentication error: no credentials provided\\\"}\"; re-sending the block in 2.016 seconds"}

[Haleygo]

@evertonspader-tomtom Okay, that's not expected.
Could you share the full vmagent deployment yaml for me to reproduce?

[evertonspader-tomtom]

@Haleygo I am using the Helm chart with the values below:

extraArgs:
  remoteWrite.url: <remote_write_endpoint>

  # This was another attempt of using env variables
  # remoteWrite.basicAuth.username: "%{BASIC_AUTH_USERNAME}"
  # remoteWrite.basicAuth.password: "%{BASIC_AUTH_PASSWORD}"

  envflag.enable: "true"
  envflag.prefix: "VM_"

env:
  - name: VM_remoteWrite_basicAuth_username
    valueFrom:
      secretKeyRef:
        name: prometheus-secret
        key: username

  - name: VM_remoteWrite_basicAuth_password
    valueFrom:
      secretKeyRef:
        name: prometheus-secret
        key: password

  # - name: BASIC_AUTH_USERNAME
  #   valueFrom:
  #     secretKeyRef:
  #       name: prometheus-secret
  #       key: username
  # - name: BASIC_AUTH_PASSWORD
  #   valueFrom:
  #     secretKeyRef:
  #       name: prometheus-secret
  #       key: password

service:
  enabled: true

resources:
  limits:
    memory: 4096M
    cpu: 2000m
  requests:
    memory: 2048M
    cpu: 1000m

persistence:
  enabled: true
  size: 18Gi

config:
  scrape_configs: []

I can see the variables created with the correct values if I exec into the container:

VM_remoteWrite_basicAuth_username=<username>
VM_remoteWrite_basicAuth_password=<password>

And I'm installing the chart like this:

helm upgrade --install \                        
            --version 0.10.7 \
            --namespace vmagent \
            --create-namespace \
            --values values.yaml \
            vmagent \
            vm/victoria-metrics-agent

[Haleygo]

@evertonspader-tomtom
Weird, I tested with your values and it still works

Do you see any difference from my command?

[evertonspader-tomtom]

I see it now using the value directly in the environment variable:

env:
  - name: VM_remoteWrite_basicAuth_username
    value: username
  - name: VM_remoteWrite_basicAuth_password
    value: password

{"ts":"2024-05-16T11:40:27.858Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":"  -remoteWrite.basicAuth.password=\"secret\""}
{"ts":"2024-05-16T11:40:27.858Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":"  -remoteWrite.basicAuth.username=\"\\\"username\\\\n\\\"\""}

but I still get a 401. I've tried to change the username and password to not have any special characters.

[evertonspader-tomtom]

I also see this in the logs when trying to apply changes to the chart but I don't know what it means or if it's related at all:

{"ts":"2024-05-16T12:04:55.734Z","level":"panic","caller":"VictoriaMetrics/lib/fs/fs.go:345","msg":"FATAL: cannot create lock file: cannot acquire lock on file \"/tmpData/persistent-queue/1_2F652A4689A1A365/flock.lock\": resource temporarily unavailable; make sure a single process has exclusive access to \"/tmpData/persistent-queue/1_2F652A4689A1A365\""}```

[Haleygo]

I see it now using the value directly in the environment variable:
but I still get a 401.

And you still can get it work by specifying cmd-flags -remoteWrite.basicAuth.username and -remoteWrite.basicAuth.password directly?
I tested the env with our docker-compose yaml, and everything works.

You can find the full yaml in https://github.com/VictoriaMetrics/VictoriaMetrics/blob/test-vmalert-remotewrite-auth/deployment/docker/docker-compose.yml.
Could you try using this docker-compose yaml to reproduce your problem? You can test it with your version of vmagent.

I also see this in the logs when trying to apply changes to the chart but I don't know what it means or if it's related at all:

It's a panic level log which will exit the process immediately, it means the vmagent process can't acquire exclusive lock for persistent-queue directory. It could happen if you mount one filesystem to vmagent, and new pod is running while old pod hasn't finished terminating.

[evertonspader-tomtom]

Ok, I've noticed that after making any changes to the values file and applying it with helm upgrade ... the new pod enters in a CrashLoopBackOff state because of the error and stays there forever. After manually removing both pods, authentication seems to work.