feat(remediation): outcome vocabulary so a changed host is never reported as unchanged - #761
Merged
Merged
Conversation
… feature
The Request Remediation modal told the operator "Applying the fix on the host
is an OpenWatch Enterprise feature: the free tier governs the request and
approval only." That is false, and has been since single-rule remediation
shipped free in v0.2.0-rc.11.
Verified against the registry rather than the copy: audit_export is the ONLY
license_gated permission in auth/permissions.yaml. remediation:execute and
remediation:rollback carry no license gate, and api-remediation C-06 states it
outright ("Execute/rollback are FREE core (Tier A), NOT license-gated ... No
remediation act endpoint returns 402"). frontend-remediation-tab AC-07 already
required this upsell copy to be gone; it was removed from the tab but not from
this modal.
The practical effect is a Community user being told to buy something they
already have, on the screen where they decide whether to act.
Also reformats the paragraph: PR #755's "OpenWatch+" -> "OpenWatch Enterprise"
rename pushed the line past the print width and merged unformatted, because CI
does not run prettier (only the local pre-commit hook does). Worth noting as a
gap rather than a one-off.
…rted as unchanged
Remediation reported two terminal outcomes, executed and failed, and "failed"
meant five different things carrying one message: "Remediation did not
complete. No host change was committed."
That message is false for Kensa v0.8.0's new `staged` status. On a host whose
audit config is immutable (auditctl -s reports `enabled 2`) an audit_rule_set
apply writes the reboot-deferred persist layer and terminates staged. The host
IS mutated. OpenWatch absorbed the unknown status through a default branch,
marked the request failed, told the operator nothing had changed, journalled it
as 'skipped', then refused rollback because rollback required 'executed'. The
change sat on the host, invisible, with no route back through the UI. Verified
reachable on 2 of 6 reachable dev-fleet hosts (owas-tst01 RHEL 8.10, owas-tst02
RHEL 9.6, both enabled 2), across 103 audit_rule_set rules.
Outcomes now map one-to-one onto operator actions:
executed runtime converged, host protected
staged persist written, NOT converged, host changed, needs reboot
reverted validation failed, Kensa restored pre-state, host clean
not_applied engine declined, host untouched
partially_applied stranded steps, host state unknown
failed errored or unreachable
Three behaviours follow, and they are the point of the change:
- staged does NOT flip host_rule_state to pass. The kernel has not loaded
the change, a re-scan correctly still fails the rule, and claiming a pass
would assert a protection the host does not have.
- staged IS rollback-eligible. Kensa captured pre-state and reverses a
staged drop-in byte-perfect; the precondition and the rollback-handle
lookup both widened, since either alone still stranded the change.
- reverted is not red and does not page. Validation failing and the host
being restored is the atomic model working. Alerting on it teaches
operators to ignore the alert.
remediation.OutcomeOf is now the single place a Kensa status is interpreted,
returning (Status, known bool). An unknown status logs at WARN naming the value
and fails closed to failed, never to a success-shaped outcome. That guard is
AC-11, and its absence is what caused this bug: a default branch silently
swallowed a new terminal state.
Deliberately NOT done: detecting Kensa's "engine refused without mutating"
case (the v0.8.0 duplicate-audit-action guard). Kensa surfaces it as
StepResult.Success=false plus human-readable Detail with no distinguishable
TransactionStatus, so telling it from a real failure would mean matching step
text on a path that runs as root. Refusals map by status to reverted, which is
truthful if less specific. StatusNotApplied and the DB enum already accept the
value, so wiring it is one line once Kensa exposes a signal. Same reasoning for
already-compliant runs, which Kensa reports as committed with a synthetic check
step. Both filed upstream.
Migration 0054 widens both CHECK constraints. The one-open-request partial
unique index is deliberately untouched: every new value is terminal, so a
staged request must not block a fresh request for the same host and rule.
.secrets.baseline is refreshed for two reasons, neither a new finding: line
drift in the generated server.gen.go from the enum addition, and one
pre-existing false positive at api/openapi.yaml:127 (the word "password" in
prose about which profile fields are editable). That finding fires on
unmodified main too; the baseline predates it.
Specs: api-remediation 1.1.0 -> 1.2.0 (C-09, AC-09/10/11),
frontend-remediation-tab 1.1.0 -> 1.2.0 (AC-08). 116 specs, 116 passing.
Verified: gofmt, go vet, go build, full go test ./internal/..., tsc, prettier,
vitest 361/361, OpenAPI drift gate green after regeneration.
remyluslosius
added a commit
that referenced
this pull request
Jul 28, 2026
Unblocked by the remediation outcome vocabulary (#761). Bumping before that landed would have written audit rules to hardened RHEL hosts while reporting "No host change was committed" and refusing rollback, on 2 of 6 reachable dev-fleet hosts. Payload, all corpus-side (the frozen api/ change is additive and already handled): - STIG Ubuntu coverage ~6% -> 84.0% (22.04) and 85.6% (24.04) - CIS Ubuntu 41.8% (22.04) and 38.9% (24.04), campaign in progress - corpus 748 -> 769 unique rule ids - config_value gains opt-in dropin_dir: the pwquality family read only the base file and false-FAILED a host hardened via pwquality.conf.d/, which is where CIS/STIG remediation writes - exact audit-rule matching: -F path=/usr/bin/su no longer substring-matches a loaded /usr/bin/sudo, which was a false-PASS on the su control - audit_rule_set stages reboot-deferred on immutable-audit hosts instead of failing, which is the change #761 exists to handle - golang.org/x/text v0.38.0 -> v0.39.0 (GO-2026-5970) The AC-11 guard was checked against the real module rather than assumed: every TransactionStatus v0.8.0 can produce is mapped by remediation.OutcomeOf, so nothing falls through to the fail-closed branch. EXPECT SCORE MOVEMENT ON FIRST SCAN. The verdict corrections are strictly more correct but they do change shipped RHEL cis/stig results: the drop-in reads, maxrepeat/maxsequence now failing an explicit =0 (previously a false-PASS), and framework-divergent thresholds defaulting to the stricter requirement (e.g. pwquality difok, CIS >= 2 vs STIG >= 8, ships 8). A CIS-only operator loosens via --var. Hosts will flip state and drift alerts will fire; that is the corpus becoming honest, not a regression. Doc counts follow the BACKLOG DOC-2 rule that a bare number is the bug: every count now names its scope and its Kensa version ("769 rules in the corpus, Kensa v0.8.0"), and the guides state explicitly that a single host is measured against the subset applying to its OS, not all 769. That ambiguity is what put three different rule counts into circulation (508, 539, 748). kensa-rules needs no manual bump: build-kensa-rules.sh derives its version from the module via stage-kensa-rules.sh, verified resolving to 0.8.0 / 769 rules. Spec: system-kensa-executor 2.7.0 -> 2.8.0 (AC-10 version pin). Verified: gofmt, go vet, go build, full go test ./internal/..., 116 specs 116 passing.
remyluslosius
added a commit
that referenced
this pull request
Jul 28, 2026
Unblocked by the remediation outcome vocabulary (#761). Bumping before that landed would have written audit rules to hardened RHEL hosts while reporting "No host change was committed" and refusing rollback, on 2 of 6 reachable dev-fleet hosts. Payload, all corpus-side (the frozen api/ change is additive and already handled): - STIG Ubuntu coverage ~6% -> 84.0% (22.04) and 85.6% (24.04) - CIS Ubuntu 41.8% (22.04) and 38.9% (24.04), campaign in progress - corpus 748 -> 769 unique rule ids - config_value gains opt-in dropin_dir: the pwquality family read only the base file and false-FAILED a host hardened via pwquality.conf.d/, which is where CIS/STIG remediation writes - exact audit-rule matching: -F path=/usr/bin/su no longer substring-matches a loaded /usr/bin/sudo, which was a false-PASS on the su control - audit_rule_set stages reboot-deferred on immutable-audit hosts instead of failing, which is the change #761 exists to handle - golang.org/x/text v0.38.0 -> v0.39.0 (GO-2026-5970) The AC-11 guard was checked against the real module rather than assumed: every TransactionStatus v0.8.0 can produce is mapped by remediation.OutcomeOf, so nothing falls through to the fail-closed branch. EXPECT SCORE MOVEMENT ON FIRST SCAN. The verdict corrections are strictly more correct but they do change shipped RHEL cis/stig results: the drop-in reads, maxrepeat/maxsequence now failing an explicit =0 (previously a false-PASS), and framework-divergent thresholds defaulting to the stricter requirement (e.g. pwquality difok, CIS >= 2 vs STIG >= 8, ships 8). A CIS-only operator loosens via --var. Hosts will flip state and drift alerts will fire; that is the corpus becoming honest, not a regression. Doc counts follow the BACKLOG DOC-2 rule that a bare number is the bug: every count now names its scope and its Kensa version ("769 rules in the corpus, Kensa v0.8.0"), and the guides state explicitly that a single host is measured against the subset applying to its OS, not all 769. That ambiguity is what put three different rule counts into circulation (508, 539, 748). kensa-rules needs no manual bump: build-kensa-rules.sh derives its version from the module via stage-kensa-rules.sh, verified resolving to 0.8.0 / 769 rules. Spec: system-kensa-executor 2.7.0 -> 2.8.0 (AC-10 version pin). Verified: gofmt, go vet, go build, full go test ./internal/..., 116 specs 116 passing.
remyluslosius
added a commit
that referenced
this pull request
Jul 28, 2026
Unblocked by the remediation outcome vocabulary (#761). Bumping before that landed would have written audit rules to hardened RHEL hosts while reporting "No host change was committed" and refusing rollback, on 2 of 6 reachable dev-fleet hosts. Payload, all corpus-side (the frozen api/ change is additive and already handled): - STIG Ubuntu coverage ~6% -> 84.0% (22.04) and 85.6% (24.04) - CIS Ubuntu 41.8% (22.04) and 38.9% (24.04), campaign in progress - corpus 748 -> 769 unique rule ids - config_value gains opt-in dropin_dir: the pwquality family read only the base file and false-FAILED a host hardened via pwquality.conf.d/, which is where CIS/STIG remediation writes - exact audit-rule matching: -F path=/usr/bin/su no longer substring-matches a loaded /usr/bin/sudo, which was a false-PASS on the su control - audit_rule_set stages reboot-deferred on immutable-audit hosts instead of failing, which is the change #761 exists to handle - golang.org/x/text v0.38.0 -> v0.39.0 (GO-2026-5970) The AC-11 guard was checked against the real module rather than assumed: every TransactionStatus v0.8.0 can produce is mapped by remediation.OutcomeOf, so nothing falls through to the fail-closed branch. EXPECT SCORE MOVEMENT ON FIRST SCAN. The verdict corrections are strictly more correct but they do change shipped RHEL cis/stig results: the drop-in reads, maxrepeat/maxsequence now failing an explicit =0 (previously a false-PASS), and framework-divergent thresholds defaulting to the stricter requirement (e.g. pwquality difok, CIS >= 2 vs STIG >= 8, ships 8). A CIS-only operator loosens via --var. Hosts will flip state and drift alerts will fire; that is the corpus becoming honest, not a regression. Doc counts follow the BACKLOG DOC-2 rule that a bare number is the bug: every count now names its scope and its Kensa version ("769 rules in the corpus, Kensa v0.8.0"), and the guides state explicitly that a single host is measured against the subset applying to its OS, not all 769. That ambiguity is what put three different rule counts into circulation (508, 539, 748). kensa-rules needs no manual bump: build-kensa-rules.sh derives its version from the module via stage-kensa-rules.sh, verified resolving to 0.8.0 / 769 rules. Spec: system-kensa-executor 2.7.0 -> 2.8.0 (AC-10 version pin). Verified: gofmt, go vet, go build, full go test ./internal/..., 116 specs 116 passing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Remediation reported two terminal outcomes,
executedandfailed, and "failed" meant five different things carrying one message:That message is false for Kensa v0.8.0's new
stagedstatus, and this PR is the gate on the v0.8.0 bump (v0.7 exit criteria 4 and 5).The defect
On a host whose audit config is immutable (
auditctl -sreportsenabled 2) anaudit_rule_setapply writes the reboot-deferred persist layer and terminatesstaged. The host is mutated. OpenWatch absorbed the unknown status through adefault:branch and:failed,skipped, andexecuted.The change sat on the host, invisible, with no route back through the UI. Recovery meant SSH plus
kensa rollback, by someone who had been told there was nothing to reverse.Not theoretical. Probed the dev fleet with a read-only
auditctl -s: 2 of 6 reachable hosts are immutable (owas-tst01RHEL 8.10,owas-tst02RHEL 9.6), both hardened RHEL withimmutable.rules. Kensa ships 103audit_rule_setrule files.owas-tst02's Remediation tab already showsaudit-cmd-chshandaudit-sudoersas Failed for this reason; under v0.7.6 that verdict is honest (the engine rolls back to nothing), under v0.8.0 those exact rows become silent host mutations.The vocabulary
executedstagedrevertednot_appliedpartially_appliedfailedThree behaviours follow, and they are the point:
stageddoes NOT fliphost_rule_stateto pass. The kernel has not loaded the change and a re-scan correctly still fails the rule. Flipping it would claim a protection the host does not have.stagedIS rollback-eligible. Both the precondition and the rollback-handle lookup widened; either one alone still stranded the change.revertedis not red and does not page. Validation failing and the host being restored is the atomic model working. Paging on it teaches operators to ignore the alert.The guard that was missing
remediation.OutcomeOfis now the single place a Kensa status is interpreted, and it returns(Status, known bool). An unknown status logs at WARN naming the value and fails closed, never to a success-shaped outcome. That is AC-11, and its absence is precisely what caused this bug.Deliberately not done
Detecting Kensa's "engine refused without mutating" case (the v0.8.0 duplicate-audit-action guard). Kensa surfaces it as
StepResult.Success=falseplus human-readableDetail, with no distinguishableTransactionStatus. Telling it from a real failure would mean pattern-matching step text on a path that runs as root, where an upstream wording change would silently reclassify real failures. Refusals map by status toreverted(host untouched), which is truthful if less specific.StatusNotAppliedand the DB enum already accept the value, so wiring it later is one line.Same reasoning for already-compliant runs, which Kensa reports as
committedwith a synthetic check step (its own docs flag the overload as a known follow-up). Today that shows a green "Fixed" chip and a Roll back button for a change that never happened. Both are filed upstream rather than worked around here.Also fixed
RequestRemediationModaltold operators "Applying the fix on the host is an OpenWatch Enterprise feature." False since single-rule remediation shipped free in v0.2.0-rc.11. Verified against the registry:audit_exportis the onlylicense_gatedpermission. Separate commit.SDD
api-remediation1.1.0 -> 1.2.0 (C-09, AC-09/10/11, AC-11 is the negative path)frontend-remediation-tab1.1.0 -> 1.2.0 (AC-08)Migration 0054
Widens both CHECK constraints (
remediation_requests.statusandremediation_transactions.phase_result). The one-open-request partial unique index is deliberately untouched: every new value is terminal, so a staged request must not block a fresh request for the same host and rule. Down-migration folds the new values back before restoring the narrower constraint.Verification
gofmt,go vet,go build ./..., fullgo test ./internal/...,tsc,prettier,vitest 361/361, OpenAPI drift gate green after regeneration.Note on
.secrets.baselineRefreshed for two reasons, neither a new finding: line drift in the generated
server.gen.gofrom the enum addition, and one pre-existing false positive atapi/openapi.yaml:127(the word "password" in prose about editable profile fields). That finding fires on unmodifiedmaintoo; the baseline predates it.Follow-on for v0.7
This unblocks the Kensa v0.8.0 bump (exit criterion 6), which is a separate PR:
KensaModuleVersion, two spec refs, and the 748 -> 769 corpus counts in the tracked guides.