Skip to content

v3.27.0: every CI gate can fail the job, baseline staleness in CI, gate results in MCP

Choose a tag to compare

@BartWaardenburg BartWaardenburg released this 17 Sep 21:32
· 646 commits to main since this release
Immutable release. Only release title and notes can be modified.
v3.27.0
adff2c9

Features

  • An armed gate reports its result in the JSON output. A run that arms a gate gets gate_outcomes, an object keyed by gate name. Each entry has status (pass, warn, fail or skipped) and enforced. Where a comparison happened, you also get observed, threshold and threshold_label. Only the gates the run armed appear in the object. A dead-code run can exit 1 on the default severity rules with no object at all. The key set is open, and a gate name you do not recognise means "some gate". (#2680, #2681, #2683, #2685)
  • The JSON output reports a stale baseline, and CI can fail on it. 3.26.0 printed the stale-baseline warning and --fail-on-stale-baseline on stderr only, and --quiet removes that output. The GitHub Action and the GitLab template both run with --quiet. dead-code / check, the bare run, dupes and health report one baseline_staleness object, in grouped output too. It holds the entry counts, change_scoped, the warning result and gate_trips. Both integrations show a stale baseline as a warning and in the job summary. On a pull request the main run covers changed files only. The integrations therefore re-read the baseline once over the whole project. That re-read took 0.11s with a cache and 0.24s with type-aware analysis on an 870-file TypeScript project. Whether a stale baseline fails the job depends on the new fail-on-stale-baseline input and the FALLOW_FAIL_ON_STALE_BASELINE variable. The pull-request comment and the merge-request note do not include the warning yet. Thanks @cloud-walker for the report, which traced the gap through the action scripts. (#2673)
  • min-score is an Action input and a GitLab variable. Before this release you could set it only through args: / FALLOW_ARGS. --min-score implies --score, so the integrations add --complexity when no health section input is set. That keeps the annotations, the SARIF upload and the pull-request comment populated. The CLI turns its own findings rule off for such a run, and the integrations do the same, which leaves the decision to the score. min-score and min-severity apply to command: health and exit 2 elsewhere. (#2682)
  • A run that analyzed no source file reports it. The run prints a warning and passes by default. fail-on-empty-analysis: true (FALLOW_FAIL_ON_EMPTY_ANALYSIS on GitLab) makes it fail. The JSON output has a no-source-files-analyzed diagnostic and the new workspace_diagnostics[].degrades_analysis field. When findings cover less than the whole project, the integrations print one combined warning with the diagnostic kinds and their counts. (#2686)
  • An MCP tool result reports what each gate concluded. The tools run the CLI with --quiet and turn exit 1 into a successful result, which hid every gate result from the agent. The root warnings array reports a stale baseline together with the re-save remedy. It also lists every gate that concluded fail or warn, with its numbers. A run that covered less than the whole project gets one entry too. The subprocess, Code Mode and typed routes all do this. find_dupes with a threshold takes the route that can evaluate it. A response with nothing to report is unchanged. (#2676)
  • fallow report --from shows the gate results in CI. It prints them as a notice annotation and as a line in the job summary, the pull-request comment and the merge-request note. The line is informational and never fails a step. (#2684)

Changed

  • security-gate fails the job independently of fail-on-issues. In both integrations the security branch sat inside the fail-on-issues conditional, and fail-on-issues: false meant that branch never ran.
  • The Action's inline Check threshold step is gone. Its logic moved into the analyze step, and the gates-failed output names the gates that decided the result.
  • In combined mode the duplication threshold does not fail the run, and the JSON output has enforced: false for it. Standalone dupes exits 1 as before. The default GitLab job forwards the threshold in combined mode, and that pipeline prints a warning with the reason.
  • --fail-on-stale-baseline changes one field in the output. That field is gate_outcomes["stale-baseline"].enforced. Nothing in baseline_staleness depends on the flag, not even gate_trips.
  • health --report-only sets enforced: false on every gate it evaluated.
  • One stale baseline can produce two lines for a pull request. The Action's warning comes from its whole-project re-read. The gate line from fallow report describes the scoped run, where the gate fails nothing.
  • No schema_version in the JSON output changed. gate_outcomes and workspace_diagnostics[].degrades_analysis are additive and optional.

Bug fixes

  • fail-on-regression, threshold, min-severity and the security gate fail the job. All four were documented as gates. Each one reported its result on stderr, which --quiet removes. Both integrations also drop the exit code when stdout parses as JSON. The integrations read gate_outcomes instead. A gate fails the build when its status is fail and enforced is true, and only when the input that owns it was set. A flag passed through args: prints a warning and cannot override fail-on-issues: false. (#2680, #2681, #2683, #2685)
  • Every failing gate is reported before the step exits. Both integrations stopped at the first failure, and a run with a tripped gate and findings reported one of the two. They print every reason, write the outputs and artifacts, and exit once. The security gate keeps exit 8, which outranks the generic 1.
  • The duplication threshold applies to the bare command on GitHub. The Action forwarded it on command: dupes only.

Upgrade notes

  • If you set fail-on-regression, threshold, min-severity or a security gate and relied on the job passing, expect failures. With security-gate and fail-on-issues: false, unset security-gate to keep the old behaviour.
  • Remove --fail-on-stale-baseline from args / FALLOW_ARGS, delete any separate unscoped gate step added as a workaround, and set the fail-on-stale-baseline input.
  • fail-on-stale-baseline with no baseline set, or on fix or security, exits 2.
  • Do not point baseline and save-baseline at the same file. The run saves before it compares, and such a baseline can never have a stale entry. The integrations warn about it.
  • If a workflow references the Check threshold step by name (continue-on-error, steps.*.outcome), point it at the analyze step and the gates-failed output.
  • With a pinned fallow older than 3.27.0, the integrations use the fields that version writes. A gate with no field in that version passes, and the integrations print one warning.
  • If you import npm/fallow/types in TypeScript, HealthBaselineStaleness is now BaselineStaleness. The old name remains as a deprecated alias.

Full Changelog: v3.26.0...v3.27.0