Scrubber Consumes 45+ MB Memory per Run with Large Credentials

[midigofrank]

Monitoring shows Lightning.Scrubber processes consuming 45-50 MB of memory per run when credentials contain many sensitive values (~500+).

Claude says:

Root Cause
File: lib/lightning/scrubber.ex:136-140

The cartesian_pairs/1 function creates N² combinations of all sensitive values:

elixir
defp cartesian_pairs(items) do
Enum.flat_map(items, fn item ->
items |> Enum.map(&[item, &1])
end)
end
Example:

Credential with 549 sensitive values
Creates 549 × 549 = 301,401 pairs
Each pair gets base64 encoded
Result: ~47 MB memory per Scrubber process

[midigofrank]

From claude:

The Performance Problem

The issue is in the Lightning.Scrubber module at lib/lightning/scrubber.ex:111-140. Here's what's happening:

Current Flow

1. When a worker requests a credential via fetch:credential (line 131 in run_channel.ex), the channel handler calls:

   • Credentials.sensitive_values_from_body(credential.body) - extracts all sensitive values
   • Scrubber.encode_samples(samples, basic_auth) - encodes them for scrubbing

2. The Problem: encode_samples/2 calls cartesian_pairs/1 which generates ALL possible pairs of sensitive values:

   defp cartesian_pairs(items) do
     Enum.flat_map(items, fn item ->
       items |> Enum.map(&[item, &1])
     end)
   end

3. The Math: With your credential example:

   • ~25 API keys × 4 fields each = 100 values
   • ~18 cloud credentials × 4 fields = 72 values
   • ~20 database connections × 4 fields = 80 values
   • Plus encryption keys, JWT secrets, OAuth tokens, service accounts, etc.
   • Total: ~500+ sensitive values

4. Cartesian Explosion:

   • 500 values → 500² = 250,000 pairs
   • Each pair is Base64 encoded as "value1:value2"
   • This takes forever and uses massive memory

Why Was It Designed This Way?

Looking at line 180-201 in the test file, the original intent was to catch secrets that might appear in various encoded forms, including:

• Base64(secret1:secret2) - for any combination of secrets

But this is over-engineering. The realistic scenarios are:

1. Basic Auth: Base64(username:password) - but these are KNOWN pairs from the credential structure (already handled by the basic_auth parameter)
2. Individual secrets: Just the plain values and their Base64 encodings

The Solution

Remove the cartesian product generation. We should only generate:

1. Plain sensitive values
2. Base64 of individual sensitive values
3. Base64 of actual username:password combinations (already provided via basic_auth parameter)

This changes complexity from O(N²) to O(N), so for 500 values:

• Current: 250,000 operations
• Proposed: 1,000 operations (500 original + 500 base64 encoded)

Trade-offs

What we lose: The ability to catch a credential where someone manually constructs Base64("api_key_1:secret_key_7") combining two unrelated secrets.

Why that's okay:

• This scenario is extremely unlikely in practice
• No legitimate system would encode two random secrets together like this
• The performance cost (minutes of processing) vastly outweighs this theoretical edge case
• We still catch all real-world cases: individual secrets, basic auth pairs, and their encodings

[midigofrank]

This issue should also cover limiting the number of keys in the credential body to avoid bloating the memory