Feature Request: Redact private data from application logs

[devkit415]

Description

Speakr writes user-derived content and personal data to its application log at INFO level. INFO is the default production level.

Operators, administrators, and anyone with access to container or host logs can read uploaded filenames, transcription titles and summaries, raw LLM responses, chat queries and chat content, hotwords, speaker names, tag and folder names, emails, usernames, and absolute filesystem paths of uploaded audio.

This is a direct privacy exposure under GDPR/CCPA. It also contradicts the OWASP Logging Cheat Sheet, which requires logs to record what happened (events, IDs, counts, durations) rather than the data itself.

The defect was corroborated on a live instance. Uploading a short audio file produced an INFO-level log of the full multi-line raw LLM response. It spanned roughly twelve physical lines.

The goal is to keep INFO and above limited to operational metadata. Full payloads move to DEBUG behind an explicit opt-in toggle. Phase 1 is the desired production fix; phases 2-4 are optional hardening.

Affected Version

v0.10.21-alpha

Requested Feature

Introduce a shared redaction helper. Phase 1 (CONTENT) is desired: remove user-derived content from INFO and above. Phases 2-4 are optional follow-ups for PII, paths, and connector details. Full payloads are available only in DEBUG behind an explicit opt-in toggle and a size cap. IP addresses stay in security events per OWASP as an explicit, documented exception.

The helper is a new module src/utils/log_redaction.py. It exposes pure functions: compact_json, redact_path, redact_user, sanitize_text, and truncate. It also exposes thin debug_payload and log_payload wrappers.

Two environment toggles are added. LOG_RAW_PAYLOADS defaults to false. LOG_RAW_MAX_CHARS defaults to 2000.

File references reduce to recording, session, and staging IDs at INFO. Basename and absolute path appear only in gated DEBUG.

User identifiers reduce to numeric user_id at INFO. Username and email appear only in gated DEBUG.

IP addresses stay in security events per OWASP. This is an explicit, documented exception.

Connector request URLs drop the query string at INFO.

Proposed Implementation

• Add src/utils/log_redaction.py as the single owner of the redaction policy. Path redaction reduces to internal IDs. User redaction reduces to numeric ID. Text sanitization collapses whitespace and newlines. Truncation follows LOG_RAW_MAX_CHARS. Payloads moved to DEBUG are serialized as compact single-line JSON. All remediated call sites use the helper instead of ad-hoc f-strings.
• Read LOG_RAW_PAYLOADS (default false) and LOG_RAW_MAX_CHARS (default 2000) from the environment in src/app.py:148-161, next to LOG_LEVEL. Document them in docs/getting-started/installation.md:273.
• Phase 1 (CONTENT, desired): remove transcriptions, summaries, titles, prompts, raw LLM responses, hotwords, snippets, queries, chat content, response bodies, and tag and folder names from INFO and above. Touch src/tasks/processing.py:856, src/api/inquire.py:195, src/api/recordings.py:3178, src/services/llm.py:453, and related modules. Move full payloads to DEBUG through debug_payload, which checks both the toggle and the effective level.
• Phase 2 (PII, optional): reduce emails and usernames to user_id in INFO. Touch src/api/inquire.py:195, src/api/auth.py:478, src/api/recordings.py:3178, and admin paths. Speaker names become IDs or counts.
• Phase 3 (PATH, optional): replace absolute paths and user-derived filenames with recording, session, or staging IDs. Touch src/services/recording_ingestion.py:1018, src/utils/audio_conversion.py:167, src/utils/ffprobe.py:141, and src/file_exporter.py:285.
• Phase 4 (CREDENTIAL/OTHER, optional): apply the existing src/utils/error_body.py:65 (redact_secrets) and src/utils/error_body.py:79 (summarize_error_body) redaction consistently to connector error bodies and miscellaneous logged bodies. Check connector logs for leaked tokens or secrets.
• Drop the query string from ASR and LLM connector request URLs logged in src/services/transcription/connectors/asr_endpoint.py:195 (optional, part of phase 4). The httpx-emitted request URL lines are third-party and are covered by the backstop filter. Keep host and path only.
• Keep the existing text formatter and level defaults unchanged for backward compatibility.
• Reuse prior art. src/utils/error_body.py:65 already redacts credentials and absolute directories. src/utils/error_formatting.py:349 provides truncate_middle.

Concrete pytest cases:

• tests/test_log_redaction.py::test_compact_json_single_line: input {"a": "x\ny"}; assert output contains no \n or \r and parses as JSON.
• tests/test_log_redaction.py::test_truncate_caps_length: input a 5000-character string with cap 2000; assert result length <= 2000 plus marker and contains [truncated].
• tests/test_log_redaction.py::test_redact_path_keeps_id: input /data/uploads/foo.mp3 with recording id 29; assert output is recording 29 and contains no slash or foo.
• tests/test_log_redaction.py::test_redact_user_keeps_id: input username alice and email alice@example.com with user id 7; assert output is 7 and contains neither alice nor example.com.
• tests/test_log_redaction.py::test_debug_payload_respects_toggle: with LOG_RAW_PAYLOADS=false, call debug_payload for a payload; assert no record emitted even at DEBUG.
• tests/test_cov_processing.py caplog case: run the summary path with caplog at INFO; assert "Raw LLM response" not in any record message and no record message contains a newline.
• tests/test_cov_inquire.py caplog case: run an inquire request with caplog at INFO; assert the user message text and username are absent from record messages.
• Static check: grep src/ for logger\.(info|warning|debug|error) calls containing \n or \r literals or f-string interpolation of filenames, titles, summaries, or emails; expect zero matches after remediation.

Expected Behavior

• Phase 1 (desired): INFO and above logs contain no transcription text, summaries, titles, prompts, raw LLM responses, hotwords, snippets, queries, chat content, or response bodies.
• Phases 2-4 (optional): reduce emails, usernames, speaker names, user-derived filenames, and absolute filesystem paths to IDs or counts.
• Full payloads appear in DEBUG only when both LOG_RAW_PAYLOADS=true and LOG_LEVEL=DEBUG are set. They are truncated to LOG_RAW_MAX_CHARS characters.
• IP addresses remain in security events (src/services/login_lock.py:96, authentication paths) as an explicit OWASP exception.
• Existing consumers, dashboards, and tests that depend on the FECHA - app - LEVEL - mensaje text format keep working unchanged.
• No new endpoints, database columns, migrations, or configuration beyond the two environment toggles are introduced.

Root Cause Analysis

Logging is configured in src/app.py:148-161 with a single logging.StreamHandler and a plain text formatter. There is no shared redaction policy.

Call sites build messages with f-strings that interpolate user-derived values.

Concrete leak sites confirmed in the code and in the live log include src/services/recording_ingestion.py:1018 (File saved to {filepath}), src/services/recording_ingestion.py:413 and src/utils/audio_conversion.py:167 (Detected codec for {original_filename}), src/services/recording_ingestion.py:1150 (Final MIME type: ... for file {filepath}), src/tasks/processing.py:633 (AI title generated for recording ...: {title}), src/tasks/processing.py:445 (Title set for recording ...: {final_title}), src/tasks/processing.py:856 (Raw LLM response for recording ...: '{raw_response}'), src/tasks/processing.py:859 (Processed summary length ...), src/api/inquire.py:195 (Inquire chat - User: {user.username}, Query: '{user_message}'), src/api/inquire.py:393 and src/api/inquire.py:404 (Search query ... returned ... chunks), src/api/recordings.py:3178 (Created ... auto-shares for recording ... via group tag '{tag.name}'), src/file_exporter.py:285 and src/file_exporter.py:726 (export paths), and many filename-interpolating warnings in src/utils/ffprobe.py:141 and src/utils/audio_conversion.py:167.

The raw LLM response log at src/tasks/processing.py:856 is the site that produced the roughly twelve-line INFO record observed in production. The LOG_LEVEL default is INFO (src/app.py:149), so all of these execute in a default deployment.

There is no existing redaction utility for log messages. src/utils/error_body.py:65 is the closest prior art but is scoped to error bodies returned by external services.

Additional Context

Affected modules and representative sensitive sites by phase, in citecheck form (path:line).

Phase 1 (CONTENT): src/tasks/processing.py:445, src/tasks/processing.py:633, src/tasks/processing.py:856, src/tasks/processing.py:859, src/tasks/processing.py:2989, src/api/inquire.py:195, src/api/inquire.py:393, src/api/inquire.py:404, src/api/inquire.py:563, src/services/llm.py:254, src/services/llm.py:278, src/services/llm.py:305, src/services/llm.py:453, src/services/llm.py:484, src/services/llm.py:487, src/services/llm.py:489, src/services/llm.py:498, src/services/llm.py:599, src/services/llm.py:630.

Phase 2 (PII): src/api/inquire.py:195, src/api/recordings.py:3178, src/api/recordings.py:3249, src/api/auth.py:478, src/api/auth.py:519, src/api/auth.py:773, src/api/auth.py:794, src/api/auth.py:803, src/services/inquire_agent.py:1573.

Phase 3 (PATH): src/services/recording_ingestion.py:1018, src/services/recording_ingestion.py:413, src/services/recording_ingestion.py:828, src/services/recording_ingestion.py:1150, src/utils/audio_conversion.py:167, src/utils/audio_conversion.py:206, src/utils/audio_conversion.py:215, src/utils/audio_conversion.py:266, src/utils/audio_conversion.py:271, src/utils/audio_conversion.py:277, src/utils/audio_conversion.py:285, src/utils/audio_conversion.py:297, src/utils/ffprobe.py:141, src/utils/ffprobe.py:271, src/utils/ffprobe.py:338, src/file_exporter.py:285, src/file_exporter.py:726.

Phase 4 (CREDENTIAL/OTHER): src/services/transcription/connectors/asr_endpoint.py:195 (request URL log), and any connector or service that logs raw error bodies. The httpx-emitted request URL lines are third-party and are covered by the backstop filter. Reuse src/utils/error_body.py:79 (summarize_error_body) and src/utils/error_body.py:65 (redact_secrets).

IP addresses are retained in security events per OWASP. Notably src/services/login_lock.py:96 (Password sign-in locked for user id ... from ...) and src/api/auth.py:342. This is an explicit, documented exception to PII redaction.

Tests should be added as tests/test_log_redaction.py. Cover the helper: no newlines in output, truncation, path and user redaction, JSON validity.

Add per-phase caplog assertions of absence of sensitive substrings at INFO. Add a static check that no logger.* call interpolates \n, \r, or sensitive values.

Manual smoke should run against the test instance on port 8899. Use LOG_LEVEL=INFO, then LOG_LEVEL=DEBUG with LOG_RAW_PAYLOADS=true.

Phase 1 is the desired, reviewable unit. Phases 2-4 are optional and can be delivered separately. Rollback is a revert of the relevant phase commits.

No schema, API, UI, or formatter change is required beyond the two environment toggles.

[murtaza-nasir]

Thank you. I agree on the first phase: transcript, summary, title, prompt and query text will be logged at DEBUG only. I do not plan the helper module, the two toggles or phases 2 to 4. Usernames, file names and paths in the server log are operational data for the person who runs the instance.

[devkit415]

Thank you. I agree that phase one is enough, the full plan is too ambitious.