Skip to content

[receiver/prometheusreceiver] basic authentication for urls stopped working #40520

Closed
@Ltty

Description

@Ltty

Component(s)

No response

What happened?

Description

The Prometheus receiver fails to scrape and endpoint with basic authentication enabled.

Steps to Reproduce

  • Set up basic authentication for a Prometheus endpoint, in this case 192.168.0.8:5572/metrics
  • Configure basic_auth in the scrape config
  • Start collector

Expected Result

Prometheus receiver is able to scrape the endpoint.

Actual Result

Scrape fails with 401 Unauthorized

Collector version

v0.127.0

Environment information

Environment

OS: Ubuntu 24.04.2 LTS

OpenTelemetry Collector configuration

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: integrations/rclone
          metrics_path: /metrics
          scrape_interval: 30s
          static_configs:
            - targets: ['192.168.0.8:5572']
          basic_auth:
            username: "USR"
            password: "PWD"
          relabel_configs:
            - source_labels: [__address__]
              action: replace
              target_label: instance
          metric_relabel_configs:
            - source_labels: [__name__]
              action: keep
              regex: "up|rclone_.*"

processors:
  cumulativetodelta:
  resourcedetection/dynatrace:
   override: false
   detectors: [dynatrace]

Log output

2025-06-06T09:48:43.196Z        debug   Scrape failed   {"resource": {}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "scrape_pool": "integrations/rclone", "target": "http://192.168.0.8:5572/metrics", "err": "server returned HTTP status 401 Unauthorized"}

2025-06-06T09:48:43.196Z        warn    internal/transaction.go:150     Failed to scrape Prometheus endpoint    {"resource": {}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "scrape_timestamp": 1749203323194, "target_labels": "{__name__=\"up\", instance=\"192.168.0.8:5572\", job=\"integrations/rclone\"}"}

Additional context

  • Verified that the enpoint is reachable
  • Verified that enpont returns 401 Unauthorized without providing basic authentication using curl 192.168.0.8:5572/metrics
  • Verified that the endpoint returns the Prometheus metrics using curl -u USER:PWD 192.168.0.8:5572/metrics

Activity

added
bugSomething isn't working
needs triageNew item requiring triage
on Jun 6, 2025
github-actions

github-actions commented on Jun 6, 2025

@github-actions
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

VihasMakwana

VihasMakwana commented on Jun 6, 2025

@VihasMakwana
Contributor

@Ltty Does your prometheus serve show any errors why this might happen?

You mentioned that basic authentication for urls stopped working. Did it work in older versions and not working now?

Ltty

Ltty commented on Jun 6, 2025

@Ltty
Author

Unfortunately not. I just see that trying to scrape the endpoint returns a 401 Unauthorized with debug logs.

This has been working for almost a year since I initially set up the server. It stopped working with the latest distro update though.

And 401 is only returned if basic authentication is not working. Since I verified that that username and password is correct and that the endpoint is reachable, I concluded that the scrape is not authenticating correctly.

dashpole

dashpole commented on Jun 9, 2025

@dashpole
Contributor

What versions did you upgrade to/from?

jage

jage commented on Jun 11, 2025

@jage

We're also experiencing this, broke when upgrading from 0.126.0 to 0.127.0. Still broken in 0.128.0.

When trying to understand what happened we captured this traffic:

$ sudo tcpdump -i lo -A -s 10240 'tcp port 15672 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
...
Host: 127.0.0.1:15672
User-Agent: otelcol/0.0.42
Accept: application/openmetrics-text;version=1.0.0;escaping=allow-utf-8;q=0.6,application/openmetrics-text;version=0.0.1;escaping=allow-utf-8;q=0.5,text/plain;version=1.0.0;escaping=allow-utf-8;q=0.4,text/plain;version=0.0.4;escaping=allow-utf-8;q=0.3,*/*;q=0.2
Accept-Encoding: gzip
Authorization: Basic YWRtaW46PHNlY3JldD4=
X-Prometheus-Scrape-Timeout-Seconds: 5

Decoding Base64 YWRtaW46PHNlY3JldD4= gives admin:<secret> which led me to https://github.com/prometheus/common/blob/95acce133ca2c07a966a71d475fb936fc282db18/config/config.go#L25 but haven't figured out if/how it's used in opentelemetry.

We are using a YAML config, but seeing same issue with JSON config, and putting the secret in a file, same issue.

This is our own build of otel-collector that fails:

dist:
  name: otelcol
  description: CloudAMQP OTel Collector
  output_path: ./otelcol
  version: 0.0.42

exporters:
  - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.127.0
  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/rabbitmqexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/googlecloudexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/azuremonitorexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/coralogixexporter v0.127.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor v0.127.0
  - gomod: go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/cumulativetodeltaprocessor v0.127.0

receivers:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/journaldreceiver v0.127.0

extensions:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding/otlpencodingextension v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension v0.127.0

providers:
  - gomod: go.opentelemetry.io/collector/confmap/provider/envprovider v1.33.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/fileprovider v1.33.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/yamlprovider v1.33.0
dashpole

dashpole commented on Jun 11, 2025

@dashpole
Contributor

#40103 seems like the most likely culprit based on timing, and was the only relevant PR I can see.

cc @ArthurSens or @npordash who might have more context on why basic auth scrape config could have stopped working.

erikburt

erikburt commented on Jun 16, 2025

@erikburt
Contributor

I was able to replicate locally by pointing a scrape config at a local http server that logged the basic auth. Saw the same thing @jage did. It seems to use the configured username but uses <secret> for the password, instead of the configured one.

2025/06/16 15:29:11 Authentication attempt: Username='username', Password='<secret>'
Config
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: test
          basic_auth:
            username: username
            password: password
          metrics_path: /metrics
          scrape_interval: 1s
          static_configs:
            - targets:
              - localhost:8080

exporters:
  debug:

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [debug]
Simple go http server
package main

import (
    "flag"
    "fmt"
    "log"
    "net/http"
)

// basicAuth is a middleware function to handle Basic Authentication.
func basicAuth(handler http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        user, pass, ok := r.BasicAuth()
        if !ok {
            log.Println("Authentication failed: No authentication provided.")
            w.Header().Set("WWW-Authenticate", `Basic realm="Restricted"`)
            http.Error(w, "Unauthorized.", http.StatusUnauthorized)
            return
        }

        // Log the username and password regardless of what they are.
        log.Printf("Authentication attempt: Username='%s', Password='%s'\n", user, pass)

        // In a real application, you would validate user and pass here.
        // For this example, we accept any authentication.

        handler(w, r)
    }
}

// metricsHandler handles requests to the /metrics endpoint.
func metricsHandler(w http.ResponseWriter, r *http.Request) {
    log.Println("Accessed /metrics endpoint successfully.")
    fmt.Fprintf(w, "Metrics data would be here.\n")
}

func main() {
    // Define a command-line flag for the port.
    port := flag.String("port", "8080", "Port to listen on")
    flag.Parse()

    // Register the /metrics endpoint with Basic Authentication.
    http.HandleFunc("/metrics", basicAuth(metricsHandler))

    // Start the HTTP server.
    listenAddr := ":" + *port
    log.Printf("Server starting on port %s\n", *port)
    log.Printf("Access the metrics endpoint at http://localhost:%s/metrics\n", *port)
    if err := http.ListenAndServe(listenAddr, nil); err != nil {
        log.Fatalf("Could not start server: %s\n", err)
    }
}
   
erikburt

erikburt commented on Jun 17, 2025

@erikburt
Contributor

I think I've root caused the issue. The problem lies with the new reloadPromConfig function.

func reloadPromConfig(dst *PromConfig, src any) error {
yamlOut, err := yaml.Marshal(src)
if err != nil {
return fmt.Errorf("prometheus receiver: failed to marshal config to yaml: %w", err)
}
newCfg, err := promconfig.Load(string(yamlOut), slog.Default())
if err != nil {
return fmt.Errorf("prometheus receiver: failed to unmarshal yaml to prometheus config object: %w", err)
}
*dst = PromConfig(*newCfg)
return nil
}

The password field in the basic_auth is of type Secret, and if you try and Marshal it, by default it will return the placeholder <secret>.

https://github.com/prometheus/common/blob/95acce133ca2c07a966a71d475fb936fc282db18/config/config.go#L36-L44

I believe the issue is because reloadPromConfig is now marshalling the config to then call promconfig.Load. Based on the PR comment this was needed to support target_allocator configs.

Here's a patch for a minimal reproduction in config_test.

Patch
diff --git i/receiver/prometheusreceiver/config_test.go w/receiver/prometheusreceiver/config_test.go
index c020b53744..5a5e6ac862 100644
--- i/receiver/prometheusreceiver/config_test.go
+++ w/receiver/prometheusreceiver/config_test.go
@@ -5,6 +5,7 @@ package prometheusreceiver
 
 import (
        "context"
+       "log/slog"
        "path/filepath"
        "strings"
        "testing"
@@ -12,6 +13,7 @@ import (
 
        promConfig "github.com/prometheus/common/config"
        promModel "github.com/prometheus/common/model"
+       promconfig "github.com/prometheus/prometheus/config"
        "github.com/stretchr/testify/assert"
        "github.com/stretchr/testify/require"
        "go.opentelemetry.io/collector/component"
@@ -399,3 +401,79 @@ func TestLoadPrometheusAPIServerExtensionConfig(t *testing.T) {
        require.NoError(t, sub.Unmarshal(cfg))
        require.Error(t, xconfmap.Validate(cfg))
 }
+
+func TestReloadPromConfigSecretHandling(t *testing.T) {
+       // This test specifically focuses on the secret marshaling issue mentioned
+       // where config entries might get marshaled to "<secret>" instead of proper values
+
+       tests := []struct {
+               name       string
+               configYAML string
+               checkFn    func(t *testing.T, dst *PromConfig)
+       }{
+               {
+                       name: "basic auth password preservation",
+                       configYAML: `
+scrape_configs:
+  - job_name: "test-basic-auth"
+    basic_auth:
+      username: "testuser"
+      password: "mysecretpassword"
+    static_configs:
+      - targets: ["localhost:8080"]
+`,
+                       checkFn: func(t *testing.T, dst *PromConfig) {
+                               require.Len(t, dst.ScrapeConfigs, 1)
+                               scrapeConfig := dst.ScrapeConfigs[0]
+                               assert.Equal(t, "test-basic-auth", scrapeConfig.JobName)
+
+                               // The critical check: ensure the password is not "<secret>"
+                               require.NotNil(t, scrapeConfig.HTTPClientConfig.BasicAuth, "basic auth should be configured")
+                               password := string(scrapeConfig.HTTPClientConfig.BasicAuth.Password)
+                               assert.NotEqual(t, "<secret>", password, "password should not be marshaled as '<secret>'")
+                               assert.Equal(t, "mysecretpassword", password, "password should preserve original value")
+                               assert.Equal(t, "testuser", scrapeConfig.HTTPClientConfig.BasicAuth.Username)
+                       },
+               },
+               {
+                       name: "bearer token preservation",
+                       configYAML: `
+scrape_configs:
+  - job_name: "test-bearer-token"
+    authorization:
+      type: "Bearer"
+      credentials: "mySecretBearerToken123"
+    static_configs:
+      - targets: ["localhost:9090"]
+`,
+                       checkFn: func(t *testing.T, dst *PromConfig) {
+                               require.Len(t, dst.ScrapeConfigs, 1)
+                               scrapeConfig := dst.ScrapeConfigs[0]
+                               assert.Equal(t, "test-bearer-token", scrapeConfig.JobName)
+
+                               // Check that bearer token is preserved
+                               require.NotNil(t, scrapeConfig.HTTPClientConfig.Authorization, "authorization should be configured")
+                               credentials := string(scrapeConfig.HTTPClientConfig.Authorization.Credentials)
+                               assert.NotEqual(t, "<secret>", credentials, "credentials should not be marshaled as '<secret>'")
+                               assert.Equal(t, "mySecretBearerToken123", credentials, "credentials should preserve original value")
+                               assert.Equal(t, "Bearer", scrapeConfig.HTTPClientConfig.Authorization.Type)
+                       },
+               },
+       }
+
+       for _, tt := range tests {
+               t.Run(tt.name, func(t *testing.T) {
+                       // First, load the config using promconfig.Load to get a proper prometheus config
+                       initialCfg, err := promconfig.Load(tt.configYAML, slog.Default())
+                       require.NoError(t, err)
+
+                       // Now test reloadPromConfig by passing the loaded config through it
+                       // This tests the marshaling/unmarshaling round-trip that could corrupt secrets
+                       dst := &PromConfig{}
+                       err = reloadPromConfig(dst, initialCfg)
+                       require.NoError(t, err)
+
+                       tt.checkFn(t, dst)
+               })
+       }
+}

6 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Participants

      @jage@Ltty@dashpole@erikburt@VihasMakwana

      Issue actions

        [receiver/prometheusreceiver] basic authentication for urls stopped working · Issue #40520 · open-telemetry/opentelemetry-collector-contrib