v0.25.0

lapilli serve run by the people who use it
(docs/design/serveops.md): quiet windows every
month or by cron, and a team's own; the weekly digest set in the UI; roles
given in the UI, of every team or of one; the range check by each job's own
scrape interval; and Grafana's notification templates checked beside each
rule. Ledger schema 014: grants, and a quiet window's team and schedule;
nothing changes on upgrade until someone uses them.

• Quiet windows every month or by a cron expression, beside every day
  and every week: every: month with on a day (15), last, or a
  weekday of the month (first sat, last fri), at the start's time of
  day; every: cron with on five cron fields, hourly at most. The first
  repeat is the window's start and end, and must be one of the schedule's
  times (a refusal names the next); each repeat is shorter than the
  shortest time between two over the four years from the start, a leap
  year's cycle. A firing is looked up once: in the last repeat to start by
  it, or by the end of Until's day, whatever the gap. Daylight saving is
  the reports' (internal/schedule, which also gains the weekday of a
  month for reports: monthly first sat@09:00).
• A team's quiet window (team_id) excuses only the firings of rules
  the team owns, worked out in the ledger as the pages work it out; a
  change to who owns what judges every team's window again, and a team
  removed stops its windows, as it stops its reports. A window's page
  names the team of each firing it excused.
• Who owns a rule is decided by its name first: a note's team, or else
  the first team, by name, whose pattern matches it, and only then the
  first whose labels it has. Before, the first team by name whose pattern
  or labels matched did, so a team owning by labels could take a rule
  another team's pattern named. Where teams overlap so, upgrading moves
  such a rule to the team its pattern names, and with it everything done
  by team: the rule's page and lists filtered by team, the teams' tallies
  and overview, its team's digest and reports, whose hours count its
  firings asleep or off hours, the incidents its catches give a team, and
  which team's editors may change its notes, tasks and quiet windows.
• Roles given in the UI (/roles, from the Teams page, and
  /api/v1/roles), with --admins: an admin gives an editor's role, of
  every team or of one, to user:NAME or group:NAME. Never admin or
  approver (the table refuses any other role), never everyone, and a role
  the flags give can't be taken away. A team's editors change their team's
  tasks, quiet windows and reports (to the team's own destination), notes
  on rules it owns (keeping their team) and comments on what is its; the
  ledger checks each inside the write, before and after, so a row can't be
  moved between teams. Only admins see who holds a role; a role taken away
  stops at the next request; a team removed takes its editors with it;
  the runs page shows roles given and taken to admins alone. A team's
  editors make ten of its reports at most, and the last ten of the
  ledger's fifty are an admin's.
  GET /api/v1/me gives the teams the reader edits, and the pages offer
  each reader what they may.
• The weekly digest set in the UI: on the Reports page and with GET,
  PUT and DELETE /api/v1/reports/digest. The flags define it until an
  admin sets it; from then on the ledger does, --digest isn't used (the
  log says so at start), and it can go to any named destination, a
  --report-dest too. It stays weekly, and its suggestions are worked out
  at its time whether it is sent or not. Upgrading changes nothing: the
  weeks sent are kept as before, so no week is sent twice and none is lost;
  a week due while it is off is passed over; moving its day never sends an
  older week. "Back to the flags" gives it back.
• Reports of a team can be made by its editors and by editors, sent to
  the team's own destination (team:NAME); reports for everyone, or sent
  elsewhere, stay an admin's. A week of the digest sent isn't sent again
  to a destination added after it went; the digest is on the Reports page
  to set whenever a report can go anywhere.
• The range check by each job's own scrape interval: serve and
  lapilli verdict --check-content read each job's scrape_interval from
  /api/v1/status/config and the job label its targets end up with from
  /api/v1/targets (a ServiceMonitor's job is its service's name). A range
  over a selector naming its job counts with that job's interval (the
  longest a job=~ can be); any other with the global, or the one every
  job has. The finding says whose: rate[3m] < 2 × 2m (job "batch").
  Checked against a real Prometheus 3.5.0.
• Grafana's notification templates, with --grafana: its templates,
  contact points and notification policies, read once an hour through the
  provisioning API with the rules' token (a Viewer's reads them), 8 MB an
  answer, no redirects. A rule's notifications are written with the
  templates of its own contact point (notification_settings) or of those
  the policies route it to by its labels, alertname and
  grafana_folder; a matcher on a label the rule doesn't set is followed
  both ways, and a template counts only when every way uses it. Three new
  checks: notify_no_text (an annotation a template shows the rule
  hasn't; fact), notify_label_never_seen (measured) and
  notify_label_not_given (heuristic, by a rule's one query). Policies'
  match and match_re are read; a rule's label written as a template,
  and policies past the 2,000 or 32 levels read, route both ways or not at
  all; {{ or .x "fallback" }} is guarded. Contact points are numbered
  and a way's are a bitset; routing is charged by the work done, a
  million a rule and twenty million a check, rules alike in the labels the
  policies match on are routed once, and the check routes for five
  seconds at most: the 1,697-policy tree a real Grafana accepted gives
  each rule up in about 20 ms, and 500 rules take 0.3 s. A rule given up
  on says so on its page, and why, and the log says how many, once a
  check. Templates are parsed, never run. Checked against a real Grafana
  13.3.1.
• Korean: quiet windows are 메인터넌스 윈도우, as DevOps teams say it.
• Security (from before this batch): a name the sign-in proxy gave
  with a character that doesn't show (a zero-width space, a soft hyphen, a
  word joiner) had it taken out, so it could become another person's, an
  admin's. Such a name, or one over 200 bytes, is no one's now: it may
  only look. Refused are control characters; format characters, but the
  zero-width joiner and non-joiner between two letters of one script or
  two emoji, as Persian names and emoji use them; characters that show
  nothing on their own (the combining grapheme joiner, the Hangul
  fillers); and a variation selector but after a symbol. A grant's name
  is held to the same, 200 bytes.
• The weekly digest's outbox guard read a key TellOnce didn't write,
  so from the second week every round worked the whole digest out again
  before finding it told (also in 0.23 and 0.24). It reads what TellOnce
  keeps, or the week 0.24 marked as passed over, when later.
• Reviewed three times, once as an attacker would and twice for
  correctness and cost, then the fixes verified and what that found fixed
  too: no way to an admin's or an approver's role was found; what was found
  is in docs/design/serveops.md, "What the reviews found". Every refusal
  has a test, and each fix and key check, taken out, fails one. A ledger
  from 0.24 migrates to schema 014 with the same stones.

v0.24.0

Changing a cloud's own alarms, step 3 of changing rules with a person's
approval (docs/design/alarms.md): Amazon CloudWatch's alarms, Azure
Monitor's alert rules, Google Cloud Monitoring's alerting policies and OCI
Monitoring's alarms, read through each cloud's API with --alarms and
changed through it, once a person approves, only those --changes-in
names. The four clouds gain the same; none of the four paths has run
against an account, only against stand-ins answering as each cloud's
documents show. No ledger schema change; but a CloudWatch alarm whose
actions are off now reads as off, so a ledger reading such an alarm from a
file gains a version on upgrade.

• --alarms SOURCE=URL, for lapilli verdict and lapilli serve,
  repeatable, beside --rules, --loaded-rules, --grafana or --ruler,
  or alone: a cloud's alarms through its API, the URL's host saying which
  (https://monitoring.REGION.amazonaws.com, Azure Resource Manager's
  subscription or resource group, a Google Cloud project's
  alertPolicies, OCI's telemetry.REGION with a compartmentId), read
  with that cloud's credentials whatever --auth says. One document an
  alarm, its state left out, so a version is a change to an alarm, not to
  its state; a read that fails anywhere keeps the version as it was. An
  alarm and a rule of one name are told apart by the source
  (HighCPU [cloudwatch-prod]). lapilli serve reads them every
  --alarms-every (10m), so the clouds' request quotas hold.
• Read too: an Azure metric alert in PromQL (one static criterion) is
  replayed, from files too; OCI's alarms say what they say (GetAlarm has
  the body the list hasn't); a CloudWatch alarm with ActionsEnabled: false
  reads as off.
• An alarm and its history, one rule. An alarm Lapilli doesn't replay,
  read through --alarms, and its history copied in under its source are
  one rule, judged under the alarm's name: tied by the alarm's own ID,
  which each firings recipe now labels its alerts with (alarm_id: the
  ARN, the resource id, the policy's name, the OCID), or by name where none
  is given, and at the severity the alarm has now where it changed (one
  that fired at the other within twelve weeks has no record). Google's
  policies are compared by the policy, their project named by ID or by
  number. Its record is twelve weeks its history was copied in for
  without a gap, up to now: each write's period is kept (in the ledger's
  meta table; no schema change), so copies that stopped for a while, or
  stopped longer ago than --source-quiet, are no record until the
  weeks missed are copied in. A ledger from before keeps its source's
  last period alone; a job copying weekly wants --source-quiet 8d.
  Without one, it has no record: it isn't among the rules that never
  fired, and retiring it isn't proposed.
• The alarm path. A proposal for such an alarm is applied through its
  cloud's API when --changes-in names it as SOURCE:NAME (CloudWatch's
  alarm name, Azure's RG/NAME, Google's display name, OCI's display name;
  a prefix ends in *) and no tag says another tool keeps it
  (CloudFormation's own, Config Connector's and Terraform's labels, or one
  --changes-managed names); otherwise it is for a person, and its page
  says why. A for and a threshold where the alarm is PromQL, backtested;
  what it says on all four, a runbook its own form; turning it off. What
  is written is the least each API takes: CloudWatch's whole alarm as read
  (PutMetricAlarm), its actions alone off and on, Azure's properties
  changed (PATCH), Google's fields in updateMask, OCI's fields with
  if-match. The rest of the alarm changed since, the change is made
  afresh; the rule itself changed, it is refused as stale. Read back, once
  more a round later where the cloud may be a moment behind.
• Not every alarm: CloudWatch's whose actions scale or act on an
  instance aren't changed; one that warms up isn't rewritten; a composite
  alarm is turned off and on alone; one with a field this Lapilli's model
  doesn't have isn't rewritten, nor deleted from here (its undo couldn't
  make it again whole). A Google policy of several conditions takes a
  condition's for and threshold alone.
• Retiring: an alarm Lapilli can't replay is turned off and no
  further, never deleted from here: the page says to delete it in the
  console once it has been off a while. A PromQL alarm is turned off, then
  deleted four weeks later, its replay checked.
• Turning off and deleting wait while the rule fires, on every path:
  the git path, Grafana, the rulers and the alarms alike, while the ledger
  has an alert of it still on (an alarm's by its ID too), or the cloud
  says so, read just before the write: CloudWatch's ALARM; an Azure
  alert of the rule whose monitor condition is Fired and that no one
  closed (one of a stateless rule, an activity log alert's or one that
  doesn't resolve itself, only within its evaluation window); an open
  Google alert of the policy; OCI's FIRING. A cloud that throttles or
  fails a while holds the change a round, uncounted. It says it stopped
  once it is carried out.
• One API path with the rulers': an alarm is changed by the engine
  v0.23's rulers are, with each of their guarantees: one change to an
  alarm at a time, read back only where it was sent and again for two
  minutes, an undo put back only over the change as it left it and
  checked as any write. A CloudWatch alarm is written only in its own
  account (the role's, or the pod's own as STS says), and a Google policy
  deleted and made again by a person isn't made a second time by an undo;
  made again twice alike after a lost answer, it is closed as settled in
  the console. One rule whose API won't answer leaves the week's other
  proposals made.
• Write identities: --changes-aws-role and --changes-azure-client-id
  for alarms too; --changes-google-service-account, a service account
  impersonated through IAM Credentials; --changes-oci-config, an OCI
  config file and its API key. Each needs an --alarms of its cloud.
• The chart: rules.alarms, rules.alarmsEvery, and
  serve.changes.googleServiceAccount and ociConfigSecret, the key
  mounted only with changes on. Its README gives each cloud's least
  privilege.
• The weekly fuzzing reads the four clouds' alarm answers too.
• Reviewed as an attacker would and for correctness; what the reviews
  found is fixed: a notification ties to its alarm's firing by alarm_id;
  an undo is refused where the change it undoes would be; a composite
  alarm's parents are asked as AWS takes it; the waits above; ties by
  severity and across a policy's conditions; a record up to now; an Azure
  description added and undone; the chart's OCI URL with compartmentId
  anywhere in its query. A verification of those fixes found nine more,
  fixed too: Azure's stateless alerts, a record across two severities and
  across a gap in the copies, Google's two forms of a policy's name, an
  alarm with a field this Lapilli doesn't know deleted, a cloud that
  throttles, a failing lookup stopping the week's proposals, and the two
  policies alike. Each is held by a test that fails without it.

v0.23.0

Changing rules through a managed Prometheus ruler, step 2 of changing rules
with a person's approval (docs/design/rulers.md): Mimir and Cortex, Amazon
Managed Service for Prometheus and Azure's Prometheus rule groups, read
with --ruler and changed through their own APIs, once a person approves,
only in the namespaces --changes-in names. Mimir's path runs against a
real Mimir 3.2.4 in CI; Cortex's, Amazon's and Azure's against stand-ins
answering as their documents show, not an account. No ledger schema
change: a rule read otherwise keeps its key, so no ledger sees a new
version of its rules on upgrade.

• --ruler URL, for lapilli verdict and lapilli serve: the alert
  rules a ruler holds, as written, from the API that writes them. Mimir's
  and Cortex's configuration API (--ruler-token-file, and
  --ruler-tenant for one with tenants, sent as X-Scope-OrgID), Amazon's
  control plane (https://aps.REGION.amazonaws.com/workspaces/ID, with
  --auth aws), or Azure Resource Manager's rule groups in a subscription
  or a resource group (with --auth azure). A rule knows its namespace;
  rules of one name in two namespaces are told apart by both
  (ScrapeSlow [payments/shop]).
• The ruler path. A proposal for a rule read with --ruler is applied
  through the ruler's API when --changes-in names its namespace (Azure:
  RG/NAME; a prefix ends in *) and no tag says another tool keeps it
  (CloudFormation's own, or one --changes-managed KEY=VALUE names);
  otherwise it is for a person, and its page says why. What one write
  replaces (a group, an Amazon namespace, an Azure rule group) is read
  again just before: the rule itself changed since is refused as stale;
  other rules in it changed, the change is made afresh. It is kept for
  undo, written with its kind's write credential
  (--changes-ruler-token-file, --changes-aws-role,
  --changes-azure-client-id), and read back as rules: an Amazon namespace
  stays applying while it is UPDATING, and a unit that reads as neither
  it was nor as proposed fails. A group goes with its last rule; an Amazon
  namespace's or an Azure rule group's last rule is for a person.
• Undo puts back the rule as it was kept in the group as it is now,
  others' changes since left as they are.
• Retiring on Azure turns the rule off (its own enabled: false) and
  deletes it four weeks later, approved again, as Grafana's.
• Every request goes to --ruler's host only, follows no redirect
  (nor an Azure nextLink elsewhere), and reads at most 64 MB for a list
  and 4 MB for one unit, within 30 seconds; names read from a ruler go into
  a path as one escaped segment.
• The chart: rules.ruler (url, tenant, tokenSecret), and
  serve.changes.in, managed, rulerTokenSecret, awsRole and
  azureClientId, the write credential mounted only with changes on. Its
  README gives the IAM policy and the Azure role.
• Google Cloud's Rules, ClusterRules and GlobalRules stay on the git
  path: a test now holds that a new for and a new threshold are written
  into one, the rest of the file as it was.
• test/e2e/changes.sh starts Mimir 3.2.4, which its Prometheus writes to,
  and runs the ruler path against it. The weekly fuzzing reads the three
  rulers' answers too.
• Reviewed before release as an attacker would and for correctness; what
  the reviews found is fixed: undo puts back one rule, only over the rule
  as the change left it, every key of it, never a group's other rules
  with it; a unit is read back by its rules' whole entries; one change to
  a unit at a time, read back only where it was sent; names counted by
  namespace and group together and escaped, so two can't read alike, and
  none with a .. segment written to, nor a rule's own name taken for one
  Lapilli makes; one group Mimir writes in a form YAML can't read back
  (|4), or one Amazon namespace whose file doesn't read, is passed over
  and logged, not the whole read, and a Mimir namespace of more than 128
  bytes is read; an Azure rule group's id is kept in one letter case; a
  Mimir read back as it was is read again for two minutes.

v0.22.2

Fixed

• A rule Lapilli doesn't replay was proposed for retirement. A cloud
  alarm that isn't PromQL, a Grafana rule on Loki: its firings, copied in,
  are kept under a name of their own, so after twelve weeks the rule read
  as never fired. With --changes, Lapilli proposed deleting it, its
  record empty, and the rules page listed it as never fired. Such a rule
  is neither now; one turned off still is.
• One rule with a proposal waiting stopped the week's proposals.
  Lapilli's own proposals stopped at the first rule a person had a
  proposal open for, and the rest weren't made that week. That rule is
  passed over now.

v0.22.1

Changes to rules carried out against real servers, not only stand-ins:
Prometheus 3.5, Gitea 28.3, Grafana 13.3 and GitLab CE 19.5. Two of the
paths v0.22.0 shipped didn't work against them; both are fixed, and
test/e2e/changes.sh now runs the git and Grafana paths against real
servers in CI.

Fixed

• Grafana: every change was refused as stale. A rule was read again
  before it was changed from its own export, which Grafana 13 gives without
  its folder, so its group read differently from the export of every rule
  it was proposed against. It is read from that export now.
• GitLab: recipes/trackers/gitlab.sh stopped at the first task.
  GitLab 18 and later link an issue as a work item (-/work_items/N), which
  the recipe didn't take as its own, after making the issue. Both links are
  taken. Quick actions in a task's text (/assign, /confidential), which
  GitLab 19.5 was seen to carry out on an issue made through its API, stay
  escaped, as v0.22.0 made them.
• Grafana: turning a rule off takes an Admin service account. Grafana
  13 answers a pause from an Editor as taken and leaves the rule running.
  Lapilli found it when it read the rule back, and the change failed; now
  it also says why. The help, the design and the chart say which role is
  needed: an Editor's token changes a for or what a rule says.
• A Grafana rule's deletion proposed before the rule has been off four weeks
  is refused then, not once approved.

v0.22.0

Lapilli reads by default, and now changes only what a person approves: the
first principle in GOVERNANCE.md, rewritten with its design (#40). Changes
are off until turned on. With them: reports people schedule themselves, what
an alert says checked, and tasks kept in step with a tracker. The ledger
gains schemas 012 (reports) and 013 (proposals and approvals). Reviewed
before release as an attacker would and for correctness, twice, then
checked again once merged; what the reviews found is fixed in what is
described here.

• Changing rules, with a person's approval (docs/design/changes.md,
  step 1). With --changes, Lapilli proposes changes to rules from what it
  finds: a longer for for a noisy rule, a threshold or for for a near
  miss, and retiring a spare rule or one that never fires (off first, then
  deleted no sooner than four weeks later, approved again). People propose
  too, and agents over MCP behind --mcp-propose. Each proposal comes with
  its backtest over twelve weeks, on /proposals and the rule's page, and
  can be made into a task. A person approves it in the UI only, never
  through the API, MCP or a token, and never their own (--approvers;
  --approvals 2 for two). An approved change is read and replayed again,
  then opened as a pull request by recipes/changes/ (GitHub, GitLab,
  Bitbucket, Gitea or Forgejo, Azure DevOps), which hold the repository's
  credentials and take changes with --changes-token-file, or applied
  through Grafana's provisioning API (--changes-grafana-token-file), with
  the rule as it was kept for undo. Limits (--changes-limits, 5 deletions
  a day and 20 changes an hour by default) and a stop (--changes-paused, a
  file, or an admin's switch, looked at again just before each write) apply.
  One proposal waits for a rule at a time, and a person, with their agents,
  makes twenty an hour at most. A change sent to Grafana is kept as
  applying, with the rule as it was and counted against the limits, until
  it is read back. A Grafana rule is deleted only after its disable, and
  only while it is still off. The change recipes take only their own pull
  requests (same repository, into BASE, by the token's own account; Gitea
  tokens need read:user, and GITHUB_LOGIN names a GitHub App), and
  write only files under RULES_PATHS, never one a CI system runs, so a
  Lapilli someone took over can't push a workflow that would run with the
  repository's secrets.
  A pull request changed in review is amended (told as change.amended)
  rather than holding up its rule, and an admin can close one stuck opened,
  merged or applying, with a reason. Raising --approvals holds proposals
  approved by fewer people. --changes needs roles (--editors or
  --admins) and someone to approve (--approvers or --admins). The
  recipes take the pull request already open from their branch when the host
  refuses another; a proposal's task shows its pull request but keeps its
  own link. The chart adds
  serve.changes; its pause is a ConfigMap named FULLNAME-changes-paused,
  and helm install prints the command that makes it.
  A fix to what a rule says is proposed the same way: one annotation
  (summary, description, runbook_url or runbook) set to text a
  person or an agent wrote, or taken out, offered on the rule's page beside
  each content-check finding text can fix and carrying that finding as its
  evidence; it is written as one quoted line, the rest of the file byte for
  byte, and must parse as a template in the rule's language, call only that
  language's own functions (for Prometheus, those every version since 2.45
  has) and no query, and can't grow without bound when the alert fires
  (no range, no templates, no variables set; an escape or a replacement
  only on a value as it is). Lapilli never writes the text itself.
• Reports people schedule themselves (docs/design/reports.md). A Reports
  page and /api/v1/reports take reports on a preset (daily@09:00,
  weekly mon@09:00, monthly 1@09:00 or last) or a cron expression,
  hourly at most, in the ledger's zone or the team's, over the calendar span
  before it or a span given (24h, 7d). A report is for everyone or one
  team and holds the sections picked: summary, noise, missed, woken,
  notifications, replay, sources, tasks, content and, for a week or longer,
  suggestions. It goes by name, never a URL, to --report-dest NAME=FILE
  (the chart's serve.reports.destinations), slack, team:NAME, or
  webhook as report.ready. It is sent once for each span through the
  outbox, when the ledger has the span, or after it fell due by six hours or
  half the time to the one before, whichever is less; after a restart, only
  the last one due, if it isn't later than that time again. A time daylight
  saving skips falls due when the clocks jump, and a repeated time the first
  time. A report whose team is removed stays stopped. Admins write reports;
  everyone sees them and a preview. The weekly digest is unchanged and shown
  with them.
• What an alert says, checked (docs/design/content.md). Lapilli now
  reads what each rule says when it fires: summaries, descriptions, runbook
  links, templates and severities, from every source's own fields. Azure
  Monitor's alert rules and OCI's alarms are now read as rules too, listed
  but not replayed. It checks them. Facts: no summary, no runbook, a
  template that doesn't parse, no severity or one not in --severities, the
  same condition as another rule. Measured: a template label no firing had,
  a noisy rule with no for. Guesses, kept apart, each with why: a label an
  aggregation can't give, a rare severity, a range shorter than two scrapes,
  absent() with no for. Templates are parsed, never run; links are
  checked, never visited. Each rule's page has "What it says", the rules
  list marks problems of fact, /metrics has
  lapilli_rule_content_problems{rule, check}, and MCP carries it all as
  data; a report's content section gives the problems now, by check. The
  check takes time in proportion to an expression's length, is worked out
  again only when the rules or firings change, and leaves out a rule it
  can't read rather than failing. lapilli verdict --check-content does the
  same without a ledger. --check-content=false turns it off.
• Tasks kept in step with a tracker (docs/design/trackers.md).
  --tasks-token-file lets a job, on the --incidents-listen port, read
  tasks (with who set each one's status, status_by and status_at, and
  updated_after=) and set their link and status, and nothing else. The
  link must be https, at most 2,048 characters, and is set once. The token
  has its own rate and turn and is refused if another token's file holds the
  same value.
  Its writes are recorded as tasks-token, don't count as anyone's use, and
  aren't sent to the notice webhook. recipes/trackers has scripts for
  GitHub, GitLab, Jira and Linear. Each run makes an issue for every open
  task without one and links it. A task follows its issue when the issue is
  closed, and reopens if the job had closed it. With CLOSE_ISSUES=true, an
  issue is closed when a person closes its task. A recipe keeps in step only
  the issues its own account made, labelled lapilli where labels are by
  name: an issue filed ahead with a task's marker, or one linked by hand, is
  neither followed nor closed. An issue is a task's only by the recipe's own
  last marker line, which the task's text can't forge; Jira team labels are
  lapilli-team-NAME. A GitHub App or Actions token names its account with
  GITHUB_LOGIN. A task the tracker refuses is skipped and the run fails at
  the end; a deleted issue is reported and its task left as it is. The
  chart's serve.trackers runs the script as a CronJob. Task links typed in
  the UI can now be 2,048 characters long.

Security

• go 1.26.11. Go's net/http could crash an HTTP/1 server whose
  handler reads a request's body while writing its headers, past a
  MaxBytesReader (GO-2026-6859, CVE-2026-97054, published on the day
  v0.21.0 was). Lapilli's handlers read a body before they answer, so no
  way to reach it was found, but v0.21.0's archives and image were built
  with Go 1.27.3 and carry the flaw. This release's are built with Go
  1.27.4, and go install now needs 1.26.11 at least.

Fixed

• lapilli serve --read-only=false and --notify-slack=false were taken as
  --read-only and --notify-slack: the value after = was ignored. They
  now read it, as --check-content=false does.

v0.21.0

The replay set beside what really fired, as it goes; rule versions from
before the ledger began; a cloud's own alarms' notifications, every cloud
alike; and four more incident recipes. The first release reviewed as an
attacker would before it shipped, as CONTRIBUTING.md now asks.

• The replay beside what really fired, kept (docs/design/fidelity.md).
  lapilli serve sets each day the runs won't replay again beside
  ALERTS and GRAFANA_ALERTS (or --check-series), each firing counted
  on the day it started, and keeps it (schema 010). A rule's page says, over
  twelve weeks, how many were alike, how many only in the replay and how
  many only in what really fired, with the last of those; the rules list
  and the dashboard mark a rule whose replay differed over the last week
  (--check-differs N/RATIO, 5/0.9 by default), and replay.differs tells
  the webhook and Slack, once a week while it does. The runs page says how
  far it has got, which series held nothing, and what really fired with no
  rule by that name. lapilli_rule_replay_check{rule, way} in /metrics,
  in MCP's rules and rule, and a line in the weekly digest. No stone
  changes. On with rules to replay; --check-replay=false, or the chart's
  serve.checkReplay.enabled: false, turns it off.
• Rule versions from before the ledger began (docs/design/history.md).
  --rules-history DIR, for serve and verdict, reads earlier versions
  of the rules, one directory each, named by when it took effect
  (2026-10-14T16:40:31Z or 20261014T164031Z). verdict reads it as each
  version given as --rules DIR@TIME. serve adds them to a ledger that
  has no version yet, so its first backfill replays each day with that
  day's rules rather than all of it with today's; a ledger with versions of
  its own keeps them, and a version that can't be read stops the start with
  the ledger as it was. recipes/versions/git.sh writes the directory from
  a clone: the rules as they stood at SINCE, then each commit on the
  branch that changed them, a merge at the merge's time, with RENDER for
  rules rendered from a chart. The chart's serve.rulesHistory runs it in
  an init container on each start.
• A cloud's own alarms' notifications, every cloud alike
  (docs/design/notifications.md). CloudWatch alarms' messages to SNS,
  directly or through EventBridge, are taken on /v1/sns, told from Amazon
  Managed Service for Prometheus's by their shape, a contributor's
  attributes among an alarm's labels. Azure Monitor's other alerts are
  taken on /v1/azure, their resource now the alert's target, lower-cased
  here and in recipes/firings/azure-monitor.jq. Google Cloud's other
  policies are taken on /v1/google. OCI Notifications is taken on a new
  /v1/oci/SOURCE, with the token as the password, checked before the body
  is read; a subscription is confirmed only from such a request, only over
  https to a Notifications API host, each URL once; the path waits two
  seconds for its turn, then answers 503. ?source=NAME (the path for OCI)
  ties an alert to the rule copied in from that source by the name the
  alarm sent, though the ledger names the rule apart; an OCI alert, by its
  alarm's OCID, which recipes/firings/oci.jq now gives its firings as
  alarm_id. A message of an alarm resolving or repeating takes its start
  from the last notification of its group. Schema 011.
• What takes input from outside is reviewed as an attacker would
  before a release, as CONTRIBUTING.md now says. This release's reviews
  found, and it fixes before shipping: a notification whose alerts don't
  say their start is read with one bounded query (it held the ledger's
  writer for up to a minute with 1,000 alerts against a busy rule), from
  notifications of the same kind and labels in the last month. An OCI
  alert's alarm id names its rule when read, so a message that comes before
  its firing is copied in still ties. EventBridge's other CloudWatch events
  are kept as nothing. Azure resources are compared without case, so
  v0.20's rows still tie. The SNS certificate and OCI confirmation fetches
  reach only internet addresses, and only Oracle's own realm domains. A
  history version that links outside itself is refused, one dated in the
  future is skipped, and an empty directory, or a restart that can't reach
  git, starts with today's rules; git.sh keeps versions in the branch's
  order, refuses links, and runs RENDER without the token. verdict's
  --rules beside --rules-history is the newest version. Rootly's recipe
  waits out a 429, Rootly's and FireHydrant's de-duplicate across pages,
  and Compass's stops before 20,000. The replay check counts rules that
  share an alert name together, starts from the first whole day the runs
  replayed, pairs firings across midnight alike from either day, and keeps
  a day a day where clocks skip midnight (the ledger's midnight is now a
  day's first instant there). A review of those fixes found more, fixed
  too: a history version that is itself a link is checked where it leads;
  the chart's RENDER runs in an init container without the token
  mounted; a commit dated after now stops git.sh rather than hide every
  later version; a link elsewhere in the repository no longer stops
  RENDER; rules that share an alert name only some days are summed day
  by day; a start is found however long ago its alarm went off; the fetches
  refuse a few more ranges that aren't the internet (NAT64, 6to4,
  benchmarking, reserved, Azure's host address) and keep no connection
  open.
• Four more incident recipes: Grafana IRM, Rootly, FireHydrant, and
  Compass's alerts, for Opsgenie's users as its support ends on 5 April 2027
  (its incidents move to Jira Service Management, which jira.sh reads).
  Each checks SINCE before asking, bounds its paging, and writes nothing
  when the tool fails; tested against each API's published schema with a
  server standing in, not against real accounts.

v0.20.1

A security release for the two ports 0.20.0 opened, after an audit from an
attacker's point of view: the notifications port, which may face the
internet, and the firings a token copies in. Nothing let anyone past a
token or a signature; what was found let them take the port down, drop
SNS's messages, or make a write cost far more than it should.

• The notifications port read a body before its proof (high). Anyone
  who could reach it could send many bodies of 1 MB at once, unproven, and
  run lapilli out of memory. A path a token proves is now refused before
  its body is read, and only 64 bodies signed by SNS or Grafana are read
  at once, each within ten seconds, and only when its headers say it could
  be: SNS's message type and a topic taken, or Grafana's signature. Past
  that the port answers 503, which SNS sends again.
• SNS's messages could be made to fail (medium). A certificate URL
  spelled many ways filled the certificate cache, and fetching spent a
  budget all shared, so a genuine message got 403, which SNS drops for
  good. A certificate is now taken only from the SNS host of its topic's
  region, spelled as SNS spells it; one that has verified a message stays
  kept; one that can't be fetched now gets 503; a URL with no certificate
  behind it doesn't use up the fetches. A message signed more than
  an hour ago isn't taken, and the same message, or the same Grafana
  signature, sent again is answered as kept without using a turn. Each
  source has its own rate, and a topic not taken and a bad signature get
  the same answer.
• A firing with a far end judged the whole ledger again (medium). One
  short write could hold the ledger's writer for seconds, each time. A
  write now judges again no further than a period past its period, nothing
  in it may start or end more than an hour ahead of lapilli's clock, and
  the firings token writes twenty periods at once, then one each five
  seconds.
• A write's cost grew with the square of the rules copied in (medium).
  Finding a source's rule didn't use its index, and each new name read
  every version of the rules. Sent again unchanged with 52,000 rules, 2,000
  rules took 8 s; now the index is used and the names are read once a
  write.
• What a token could add had no bound (medium). The ledger keeps 5,000
  rules from each source and 20,000 from all, 500,000 alerts from each, and
  200,000 alerts a day from each source of notifications. A rule's name is
  512 characters at most (CloudWatch's are 255, Google Cloud's 512), a
  receiver's 200, an alert's labels 64 and 4 KB; anything else is refused
  with 400. A group key past 1,000 characters is kept as its hash. /metrics counts
  notifications only for the ledger's rules, and each rule's ten busiest
  receivers, so a flood of names can't flood the Prometheus that scrapes
  it.
• go 1.26.10 (medium). go install with Go 1.26.0 to 1.26.9 built
  lapilli with 37 known flaws in the standard library, among them
  net/http's and html/template's. The release archives and the image
  were built with Go 1.27.3 and weren't affected.
• Low. Names with a character that doesn't show (a line break, a
  zero-width space, a mark that turns text right to left) are refused, so
  a rule copied in can't look like another or forge a line of the digest.
  A rule copied in no longer takes a name a note, task or comment waits
  under, which a replayed rule coming later would have lost to it. A
  source is as far as the furthest period it has written, so writing 1970
  each hour can't hold the digest back. The notifications token may not
  be the incidents or firings token, at start and on each request. Slack's
  one-line notices are posted as written, with no formatting or link
  previews. MCP tells agents that receivers, sources and the labels of
  firings copied in and of notifications are data too. put.sh prints
  what the server says as JSON. A period sent past now counts as written
  up to now. The firings token's 429 says to try again in five seconds. A
  body cut off is answered 408, not 413.
• The AWS SDK's updates (#34). SQLite stays at 3.53.4 for now: 3.54.1
  wrote a week of 100,000 firings a fifth slower, past the budget on
  macOS.

v0.20.0

Firings recorded elsewhere, kept in the ledger and judged with the rest, past
what the cloud keeps; notifications as they were sent, tied to the firings
they told of; and both told to Slack and the webhook.

• Firings recorded elsewhere, kept in the ledger (docs/design/recorded.md).
  With --firings-token-file, the port --incidents-listen opens also takes
  PUT /api/v1/firings/SOURCE?from=&to=: a source's firings that start in
  the period, one a line as --firings reads them, in place of those the
  ledger had from it then. Sent again as they are, nothing changes or is
  judged again. The token is its own: one that copies incidents can't write
  firings, and the reverse. They are judged as replayed firings are, and
  their rules are named as their sources name them, told apart from another
  rule with the name (checkout-5xx [cloudwatch-prod]); a replayed rule
  coming later takes the name. A recorded rule's page says where it was
  recorded, and that what needs its condition isn't there. Runs leave
  recorded firings alone, --keep drops them and forgets a source not
  written since, and a write is 31 days in whole seconds, 16 MB and 100,000
  firings at most, with 50 sources. Schema 008: rules.source and rules.sent, and
  sources.
• lapilli serve on firings copied in alone. With the firings token,
  --rules and --prometheus are optional: nothing is replayed, and what
  comes in is judged as it comes. --impact still needs --prometheus.
• Told to Slack and the webhook. A source nothing has been copied in
  from for --source-quiet (default 26h; the chart's
  serve.firings.quietAfter) is told of once, as source.quiet, to the
  webhook and, with --notify-slack, to Slack. The weekly digest says which
  sources firings were copied in from, how many each brought and which has
  gone quiet, and what the notifications sent came to (how many, repeats,
  while people slept, for nothing, the rules that sent the most for
  nothing); a team's digest, for its rules. The webhook's digest.ready
  carries both as sources and notifications.
• recipes/firings/put.sh sends a recipe's output as a source's firings for
  SINCE to UNTIL. The runs page lists the sources, /api/v1/sources
  has them, and /metrics has lapilli_serve_source_last_timestamp_seconds
  and lapilli_serve_source_firings by source. The chart takes
  serve.firings.tokenSecret, with rules and prometheus optional then.
• Notifications as they were sent. lapilli serve --notify-listen ADDR
  opens a port that takes notifications, and nothing else, a path for each
  source: /v1/alertmanager (a webhook, with the token in
  --notify-token-file as its bearer token), /v1/grafana (a webhook
  contact point, with the token or its HMAC signature with the secret in
  --notify-grafana-secret-file), /v1/sns (Amazon SNS's signature, from a
  topic --notify-sns names, for Amazon Managed Service for Prometheus; the
  subscription is confirmed at an SNS host only), /v1/azure (an action
  group's webhook, common alert schema, the token as the URL's password) and
  /v1/google (a Cloud Monitoring webhook channel, basic auth after a 401).
  Each alert is tied to its rule's firing whose labels are all among its own,
  started within two evaluations; a notification the same as one from
  another replica within a minute is kept once; repeats are as the source
  says, or the same alerts again with nothing new and nothing resolved. Rule
  pages and the rules list show notifications, repeats, those while people
  slept and those for nothing, by receiver; firings, how many told of each,
  or that one paged no one; the dashboard's "Woken for nothing" counts
  notifications once there are any. lapilli_rule_notifications{rule, receiver, reason} in /metrics, and in MCP's rules and rule. Schema
  009. The chart's serve.notificationsAPI.

v0.19.0

A cloud's own alarms: replayed where they are written in PromQL, on every
cloud that has such alarms, and judged from their history where they aren't.

• CloudWatch PromQL alarms. --rules reads the AWS CLI's
  describe-alarms answer: a PromQL alarm is replayed with its pending
  period as for, its recovery period as keep_firing_for and its
  evaluation interval; metric and composite alarms are listed as not
  replayed. --prometheus https://monitoring.REGION.amazonaws.com --auth aws
  queries CloudWatch's PromQL API, signed for the service monitoring, as
  is its dual-stack (REGION.api.aws) or VPC endpoint name; aps stays for
  Amazon Managed Service for Prometheus. A period given as NaN is refused.
• Google Cloud alerting policies with a PromQL condition. --rules
  reads them as gcloud monitoring policies list writes them, in YAML or
  JSON, or as the API's list answers: the condition's duration is for,
  its evaluation interval (30 s when it doesn't say) the rule's, its
  alertRule and ruleGroup the name and group (the policy's display name
  otherwise), its labels the rule's, and the policy's severity becomes
  severity. An alert closes at the first evaluation at least the larger of
  270 s and two evaluations after its condition last held, as Google Cloud
  closes them. A disabled policy, and one with another kind of condition, is
  listed as not replayed. Replay with --gcp-project, whose PromQL API
  answers for the same metrics, and --step 30s, of which every policy's
  interval is a multiple.
• An answer cut short is an error. An API that caps the series in an
  answer and says so only in a warning (CloudWatch: 500) no longer gives a
  replay of what was left: the rule's query fails, saying why.
• Firings from a cloud's alarm history. recipes/firings has a script
  for each cloud that writes its alarms' history as --firings takes it:
  cloudwatch.sh (describe-alarm-history), azure-monitor.sh (Azure
  Resource Graph), google-cloud.sh (projects.alerts.list) and oci.sh
  (get-alarm-history), each with the cloud's own CLI, tested against the
  clouds' documented answers with stand-ins for the CLIs. Times are checked
  to be times before they reach a query or a filter (UTC for Azure), the
  token goes only to https, a CloudWatch move that says neither its old nor
  its new state stops the script, and a stateless Azure alert (activity log,
  service and resource health) ends as it fires.