Closed
Description
Component(s)
No response
What happened?
Description
The Prometheus receiver fails to scrape and endpoint with basic authentication enabled.
Steps to Reproduce
- Set up basic authentication for a Prometheus endpoint, in this case 192.168.0.8:5572/metrics
- Configure basic_auth in the scrape config
- Start collector
Expected Result
Prometheus receiver is able to scrape the endpoint.
Actual Result
Scrape fails with 401 Unauthorized
Collector version
v0.127.0
Environment information
Environment
OS: Ubuntu 24.04.2 LTS
OpenTelemetry Collector configuration
receivers:
prometheus:
config:
scrape_configs:
- job_name: integrations/rclone
metrics_path: /metrics
scrape_interval: 30s
static_configs:
- targets: ['192.168.0.8:5572']
basic_auth:
username: "USR"
password: "PWD"
relabel_configs:
- source_labels: [__address__]
action: replace
target_label: instance
metric_relabel_configs:
- source_labels: [__name__]
action: keep
regex: "up|rclone_.*"
processors:
cumulativetodelta:
resourcedetection/dynatrace:
override: false
detectors: [dynatrace]
Log output
2025-06-06T09:48:43.196Z debug Scrape failed {"resource": {}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "scrape_pool": "integrations/rclone", "target": "http://192.168.0.8:5572/metrics", "err": "server returned HTTP status 401 Unauthorized"}
2025-06-06T09:48:43.196Z warn internal/transaction.go:150 Failed to scrape Prometheus endpoint {"resource": {}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "scrape_timestamp": 1749203323194, "target_labels": "{__name__=\"up\", instance=\"192.168.0.8:5572\", job=\"integrations/rclone\"}"}
Additional context
- Verified that the enpoint is reachable
- Verified that enpont returns 401 Unauthorized without providing basic authentication using curl 192.168.0.8:5572/metrics
- Verified that the endpoint returns the Prometheus metrics using curl -u USER:PWD 192.168.0.8:5572/metrics
Activity
github-actions commentedon Jun 6, 2025
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself.
VihasMakwana commentedon Jun 6, 2025
@Ltty Does your prometheus serve show any errors why this might happen?
You mentioned that basic authentication for urls stopped working. Did it work in older versions and not working now?
Ltty commentedon Jun 6, 2025
Unfortunately not. I just see that trying to scrape the endpoint returns a 401 Unauthorized with debug logs.
This has been working for almost a year since I initially set up the server. It stopped working with the latest distro update though.
And 401 is only returned if basic authentication is not working. Since I verified that that username and password is correct and that the endpoint is reachable, I concluded that the scrape is not authenticating correctly.
dashpole commentedon Jun 9, 2025
What versions did you upgrade to/from?
jage commentedon Jun 11, 2025
We're also experiencing this, broke when upgrading from 0.126.0 to 0.127.0. Still broken in 0.128.0.
When trying to understand what happened we captured this traffic:
Decoding Base64
YWRtaW46PHNlY3JldD4=
givesadmin:<secret>
which led me to https://github.com/prometheus/common/blob/95acce133ca2c07a966a71d475fb936fc282db18/config/config.go#L25 but haven't figured out if/how it's used in opentelemetry.We are using a YAML config, but seeing same issue with JSON config, and putting the secret in a file, same issue.
This is our own build of otel-collector that fails:
dashpole commentedon Jun 11, 2025
#40103 seems like the most likely culprit based on timing, and was the only relevant PR I can see.
cc @ArthurSens or @npordash who might have more context on why basic auth scrape config could have stopped working.
erikburt commentedon Jun 16, 2025
I was able to replicate locally by pointing a scrape config at a local http server that logged the basic auth. Saw the same thing @jage did. It seems to use the configured
username
but uses<secret>
for the password, instead of the configured one.Config
Simple go http server
erikburt commentedon Jun 17, 2025
I think I've root caused the issue. The problem lies with the new
reloadPromConfig
function.opentelemetry-collector-contrib/receiver/prometheusreceiver/config.go
Lines 147 to 158 in dfc894e
The
password
field in thebasic_auth
is of typeSecret
, and if you try and Marshal it, by default it will return the placeholder<secret>
.https://github.com/prometheus/common/blob/95acce133ca2c07a966a71d475fb936fc282db18/config/config.go#L36-L44
I believe the issue is because
reloadPromConfig
is now marshalling the config to then callpromconfig.Load
. Based on the PR comment this was needed to supporttarget_allocator
configs.Here's a patch for a minimal reproduction in
config_test
.Patch
6 remaining items