Harden redaction and rewrite the test suite - #1
Merged
Conversation
- list/get/dry-run share a span-aware reader; PEM bodies no longer leak - dry-run prints a unified diff, not the whole file - new: envctl redact filters stdin; masks known values literally plus base64/URL/JSON-escaped variants; --no-env for heuristics only - detectors: fixed-point Shannon entropy, Authorization values, secret URL query params, private JWKs, connection strings, WEBHOOK_URL, PuTTY, A3T; values dequoted before matching - drop envctl.sh, the bash port of the tool - tests/run.sh: 226 named cases, per-failure output, shellcheck clean
- oversized PEM labels no longer bypass suppression; the key body was printing in the clear - a `-----BEGIN` boundary sharing a line keeps the surrounding text, so JSON objects survive redaction instead of being replaced wholesale - redacted `--dry-run` hunk counts match the emitted body; the diff was malformed whenever a multiline span collapsed - trailing `}`/`]` is trimmed only when unbalanced, so JSON values keep their own braces - literal mask floor 8 -> 6 behind a Shannon entropy gate, plus hex encoding variants - `list --values` prints the whole span instead of the first line - indented assignments are visible to `list` and `get` - `read_stream_line` dies on a read error instead of truncating silently - `scan_text_line` returns a status rather than a `-1` length sentinel - help synopsis splits the command and bare forms; `--raw` records that `redact` rejects it
- text around PEM boundaries goes through the normal scanners, so secrets beside a BEGIN or END marker mask instead of passing raw, and prose after a boundary is kept instead of dropped - PuTTY key files: private line blocks and the MAC mask; public lines and comments pass - private JWK objects mask anywhere in a line, not only as `KEY=` - a pending `KEY=` or `KEY:` consumes a quoted value whole, so `export DB_PASSWORD="correct horse"` masks all four words; an unterminated quote carries across lines until it closes - `KEY:` tokens keep their key context behind log prefixes - Authorization values behind prefixes mask from 8 chars when the Authorization context is present; bare Bearer/Basic prose keeps the 20-char bar - lowercase values under strong secret keys enter the literal mask set; `changeme` joins the placeholder list - a public-PEM substring no longer exempts a strong secret key; the exemption needs a complete standalone public block - the first line after an abandoned PEM carry is scanned, not eaten - write failures on stdout exit non-zero (`get`, `list`, dry-run, `redact`) - `list --values` is one record per key again, matching the README - suite runs from paths with spaces; `wc -c` compare survives BSD padding; `eq` requires exit 0; 265 cases
- install.sh detects OS and arch, downloads the right release asset, and verifies it against SHA256SUMS; ENVCTL_INSTALL_DIR and ENVCTL_VERSION override destination and tag, ENVCTL_ATTEST=1 also requires a passing gh attestation verify - releases attest build provenance via actions/attest@v4; the publish job gets id-token and attestations permissions, the workflow drops to contents: read elsewhere - release assets upload by glob under working_directory: dist - CONTRIBUTING.md: build, layout, test harness, lint, labels, releasing - SECURITY.md: threat model, private vulnerability reporting, supported versions, artifact verification - README keeps usage and points at both
- `tests/run.sh` is a harness holding no test data: it reads a case file, runs envctl, and compares byte-exact - every case is a file in `tests/cases/`, every input and expectation a file under `tests/fixtures/`; `%% args` takes one argv element per line so a value never meets shell word splitting - cases run under `env -i`, so agent detection starts from a clean environment and `%% setenv` opts back in - new coverage: all 16 agent-detection signals plus the empty-value rule, the agent+TTY automatic masking path through a real pty, the EPIPE write-failure path through a fifo, and locale stability - `%% mode pty` and `%% mode nosigpipe` report as skipped where `script` or GNU `env` is missing, rather than passing silently - failures print a labelled diff plus the first differing byte offset, which is the only signal when a trailing space or CR is the defect - `.gitguardian.yml` ignores `tests/**`; fixtures are deliberately full of credential-shaped strings Known defect, tracked but not fixed here: `crlf.txt` and `crlf-token.txt` are stored without their CRs, so `filter-crlf` and `filter-crlf-masking` fail on a fresh checkout.
- preserve CRLF and final-newline semantics across env and filter paths - mask literals spanning lines and parse multiline JSON, JWK, PEM, PuTTY, and quoted secret forms without leaking malformed input - honor `--`, aliases, implicit `.env`, commented keys, and strict argument errors while reporting stdout failures consistently - expand data-driven regression coverage and fix byte-exact fixtures
- buffered JSON at EOF rescans through the plain scanner instead of
printing raw; benign truncated JSON still passes through
- `Authorization=Basic ...` masks as an assignment; the Basic shortcut
applies only to `:` headers, and an explicit Authorization key accepts
4-byte Basic and short Bearer credentials
- JSON property names decode strict ASCII escapes (`api_key`)
before key matching, and pairs parse across CR/LF whitespace
- private JWKs nested below the top level mask again (`{"keys":[...]}`)
- PuTTY: a completed private block expects the MAC next; an unrelated
line resumes normal scanning while undeclared body-shaped lines stay
dropped, so undercounted `Private-Lines` cannot leak key material
- filter mode stops at the first failed write instead of draining stdin
- harness: stdout compares unconditionally against an empty default,
and the `epipe-open` mode proves the EPIPE exit with stdin held open
under a watchdog instead of relying on EOF
- `list`, `get`, and mutations free their line sets; ASan/UBSan suite
runs clean; CFLAGS gain `-Werror -Wpedantic -Wshadow`
- help and README describe the real PEM carry recovery; the long help
is a shared fixture asserted for `--help` and the no-args path
- binary mode is set once at the top of `main` for stdin, stdout, and stderr; the copy inside `act_redact` is gone - the mingw CRT rewrote every `\n` as `\r\n` on text-mode streams, so help, `list`, `get`, and dry-run output failed byte-exact comparison everywhere off the filter path, and stderr diagnostics would miss their LF fixtures the same way
- a native binary reading stdin from an MSYS fifo dies with a read error, so the handshake case can never observe the EPIPE exit there; the write-side coverage stays with `epipe-reports-write-failure`, whose stdin is a regular file - the skip reason says "no usable fifo" since `mkfifo` itself succeeds
This comment was marked as low quality.
This comment was marked as low quality.
This comment was marked as spam.
This comment was marked as spam.
- macos ships bash 3.2: no `declare -A`, no `mapfile`, and expanding an
empty array trips `set -u`
- `fail_case` dedupes by scanning the failures array, argv/setenv load
through `read` loops, and empty `envv`/`argv` expand through the
`${arr[@]+...}` guard
kjanat
added
bug
- BSD `script` has no `-E never`, so the EOF it reads from /dev/null echoes into the pty log as a leading `^D` and broke the four tty cases on macos - the pty postprocess drops it from the start of line one, in both the caret-visualization and raw-EOT encodings, alongside the existing CR strip
kjanat
force-pushed
the
redaction-hardening
branch
from
July 29, 2026 02:23
a659788 to
d792cae
Compare
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as low quality.
This comment was marked as low quality.
- a review claimed the line closing a streamed object never reaches the buffer, so a private member sharing it would escape; the case proves that shape masks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
envctl redactfilter mode: masks the env file's literal values plus their base64, URL-encoded, JSON-escaped, and hex forms, then runs value-shape heuristics over the rest of the stream; streaming parsers handle multiline JSON, private JWKs at any depth, PEM, PuTTY, and quoted secrets without leaking on malformed or truncated input--,ls/rmaliases, implicit./.env, bareKEY [VALUE]forms, strict argument errors, non-zero exit on stdout write failure, and filter mode stops at the first failed write instead of draining stdin%% sectioncase file per scenario, byte-exact compares), unconditional stdout/stderr comparison, pty and fifo modes that skip honestly where the platform cannot express themVerification
-Werror -Wpedantic -Wshadowclean