Skip to content

security(observability): pino logger has no redact paths so octokit error stacks can leak App JWTs and installation tokens #52

Description

@chrisleekr

Finding

The root pino instance at src/logger.ts:10-13 is configured with only
two options -- level and (in development) transport -- and has NO
redact path list. Every logger.error(err, message) site therefore
serialises errors through pino's default err serializer, which walks
enumerable own properties of the Error object. That matters because
the hot error paths in this codebase catch errors that carry secrets
on custom properties:

  • src/app.ts:82 forwards raw errors from @octokit/webhooks via
    app.webhooks.onError. When signature verification fails or a
    downstream handler throws, the thrown error can retain the webhook
    HTTP request, including the X-Hub-Signature-256 header and the
    raw body used to build the prompt.
  • src/orchestrator/connection-handler.ts:537 logs the error returned
    by ctx.octokit.auth called with installation type. Octokit
    surfaces these as RequestError instances with a request
    property: err.request.headers.authorization holds the App JWT
    used to mint the installation token, and err.response.data can
    echo the JWT back in the GitHub 401 body. With no redaction, both
    end up in the structured log.
  • src/daemon/job-executor.ts:307 logs any throw from runPipeline.
    The pipeline at src/core/pipeline.ts:140-142 minted the
    installation token, and a second new Octokit is built with that
    installationToken at src/daemon/job-executor.ts:219. A
    RequestError from any call on that client carries
    err.request.headers.authorization with value token ghs_...
    verbatim.
  • src/orchestrator/valkey.ts:128 logs the error on Valkey connect
    failure. Bun's RedisClient surfaces the connection URL in
    err.message on ECONNREFUSED, and the existing redactValkeyUrl
    helper at src/orchestrator/valkey.ts:64 is only applied at one
    info log site (valkey.ts:33) -- not to the error path.

The codebase already demonstrates awareness of the problem with two
point-solution redactors: redactGitHubTokens at
src/utils/sanitize.ts:77-89 (applied only to prompt content, not
logs) and redactValkeyUrl at src/orchestrator/valkey.ts:62-73
(applied only at one call site). Neither runs on log output. The
improvement is to add a single pino redact paths list plus an err
serializer that feeds err.message, err.stack, and err.request
through the already-tested redactGitHubTokens regex -- a ~15-line
change to src/logger.ts with no new dependencies.

Diagram

flowchart LR
    A1["octokit webhook error<br/>app.ts:82"]:::src
    A2["orchestrator mint<br/>connection-handler.ts:537"]:::src
    A3["daemon pipeline<br/>job-executor.ts:307"]:::src
    A4["valkey connect<br/>valkey.ts:128"]:::src
    ERR["Error instance<br/>err.request.headers.authorization<br/>err.message with secrets"]:::leak
    SER["pino.stdSerializers.err<br/>walks enumerable own props"]:::risk
    CUR["Current logger<br/>logger.ts 10-13<br/>NO redact configured"]:::risk
    OUT["stdout JSON<br/>shipped to log aggregation<br/>retained for weeks"]:::risk
    PROP["Existing but unused<br/>redactGitHubTokens<br/>sanitize.ts 77-89"]:::ok
    FIX1["Add pino redact paths<br/>authorization, token,<br/>webhookSecret, privateKey"]:::fix
    FIX2["Compose err serializer<br/>scrub message and stack"]:::fix
    OUT2["stdout JSON with<br/>Redacted placeholders"]:::ok

    A1 --> ERR
    A2 --> ERR
    A3 --> ERR
    A4 --> ERR
    ERR --> SER --> CUR --> OUT
    PROP -. reuse .-> FIX2
    FIX1 --> OUT2
    FIX2 --> OUT2

    classDef src fill:#1f4e79;color:#ffffff;stroke:#0b2545;stroke-width:1px
    classDef leak fill:#7a0f12;color:#ffffff;stroke:#4a0000;stroke-width:1px
    classDef risk fill:#8a4b00;color:#ffffff;stroke:#5a2f00;stroke-width:1px
    classDef ok fill:#196f3d;color:#ffffff;stroke:#0b3a1e;stroke-width:1px
    classDef fix fill:#1a5490;color:#ffffff;stroke:#0b2545;stroke-width:1px
Loading

Rationale

The project is a multi-tenant webhook server that handles three
classes of long-lived secrets: GitHub App private keys, the daemon
shared secret (DAEMON_AUTH_TOKEN), and -- for single-tenant OAuth
deployments -- CLAUDE_CODE_OAUTH_TOKEN, a personal Max/Pro
subscription credential that does not auto-rotate. Installation
tokens (the ghs_ prefix) rotate every 60 minutes so their blast
radius is bounded, but the App JWT used to mint them and the OAuth
and daemon tokens are not. docs/OBSERVABILITY.md:3 confirms
structured JSON logs are the primary signal and that log fields are
preserved end-to-end; any long-lived secret that lands in stdout is
then shipped to centralised aggregation and persists for weeks.

Pino's own documentation states that path-based redaction without
wildcards adds roughly 2% overhead to JSON serialization -- effectively
free relative to the cost of a single secret rotation. Because
src/utils/sanitize.ts:77-89 already contains a tested
redactGitHubTokens regex covering ghp_, gho_, ghs_, ghr_
and fine-grained github_pat_ prefixes, the fix can reuse that
helper inside a composed err serializer rather than duplicating
logic.

The scope of change is small and bounded: a single edit to
src/logger.ts plus a matching note in docs/OBSERVABILITY.md. It
does not require a new dependency, a new env var, or any changes to
the ~27 files that call logger.* today.

References

Internal:

  • src/logger.ts:10-13 -- pino root logger with no redact option
  • src/app.ts:82 -- webhook error handler that logs the raw err
  • src/daemon/job-executor.ts:219,307 -- Octokit construction with
    installationToken and an unredacted error log on any pipeline throw
  • src/orchestrator/connection-handler.ts:537 -- logs the error for
    Failed to mint installation token for job with raw err
  • src/orchestrator/valkey.ts:62-73,128 -- existing redactValkeyUrl
    helper not applied on the error-logging path at line 128
  • src/utils/sanitize.ts:77-89 -- tested redactGitHubTokens regex
  • src/core/pipeline.ts:140-142 -- installation-token retrieval via
    ctx.octokit.auth
  • src/config.ts:13-15,37-38,55,152,500-501 -- secret-bearing config
    fields: privateKey, webhookSecret, anthropicApiKey,
    claudeCodeOauthToken, awsBearerTokenBedrock, daemonAuthToken
  • docs/OBSERVABILITY.md:3 -- no redaction guidance present today

External:

Suggested Next Steps

  1. Extend src/logger.ts with a redact.paths list covering
    authorization, *.authorization, headers.authorization,
    *.headers.authorization, req.headers.authorization,
    request.headers.authorization, token, installationToken,
    privateKey, webhookSecret, anthropicApiKey,
    claudeCodeOauthToken, daemonAuthToken, awsSecretAccessKey,
    awsSessionToken, awsBearerTokenBedrock, and *.password.
    The default censor value Redacted is acceptable.
  2. Compose a custom err serializer that calls
    pino.stdSerializers.err first, then runs redactGitHubTokens
    from src/utils/sanitize.ts over the resulting message and
    stack strings, catching secrets embedded in free-form text
    rather than in named fields.
  3. Add one unit test per serializer path: (a) an Error with
    request.headers.authorization set, (b) an Error whose message
    contains a literal ghs_ token, (c) a plain log call with a
    privateKey PEM block field. Assert the emitted JSON contains
    no secret material.
  4. Update docs/OBSERVABILITY.md with a new Redaction section
    listing the redacted paths so operators know what will NOT appear
    in logs, and drop a one-line reminder next to redactValkeyUrl
    and redactGitHubTokens that the logger is now the canonical
    chokepoint.
  5. Optionally, fold redactValkeyUrl into the logger configuration
    so the remaining ad-hoc call at src/orchestrator/valkey.ts:33
    stops diverging from the global policy.

Areas Evaluated

  • src/logger.ts -- root pino logger configuration (the focus)
  • All 27 files under src/ that import logger or call
    logger.* / ctx.log.*, sampled for error-logging patterns
  • src/utils/sanitize.ts -- existing redaction helpers for prompts
  • src/orchestrator/valkey.ts -- existing URL redaction helper
  • src/config.ts -- nine secret-bearing config fields
  • src/app.ts -- top-level webhook error handler
  • src/daemon/job-executor.ts + src/core/pipeline.ts -- the
    installation-token lifecycle and the errors that can surface it
  • src/orchestrator/connection-handler.ts -- token mint path
  • docs/OBSERVABILITY.md -- confirmed no redaction guidance today
  • Duplicate check: gh issue list --label research --state all
    returned no area: observability entries

Generated by scheduled research workflow run #24850942493 on 2026-04-23

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions