sd-mqtt 0.8.0

Changed

• The licence is now the Apache License 2.0. It was MIT. Apache 2.0
  is as permissive and adds an explicit patent grant from contributors.
  Releases before this one remain available under MIT for anyone who
  has them; this release and everything after it are Apache-2.0. Every
  crate, the web package, the image label and the notices headers say
  so.

• The documentation is split into features and a setup guide. The
  README is the licence, a table of features, a quick start and the
  configuration reference. Each feature has a page under
  docs/features/ saying what it does, how to configure it and what it
  reports; docs/setup.md goes from nothing to a three-node cluster;
  docs/README.md indexes it all. scripts/test_docs_links.py follows
  every relative link.

• The working documents are gone from the repository. The milestone
  plans, the reference-broker analyses and the milestone 3 demonstration
  were how it was built, not what ships, and the decision records'
  conclusions were already in the design's sections 15 and 17. The design
  is now docs/design.md and the component selection docs/components.md.

Before you upgrade a cluster

• Two replication frames are new, Snapshot and SnapshotFrom.
  They are sent only to a node that is behind its peers' log
  truncation, which is a new node joining or one rebuilt from an empty
  store. A node from an earlier release cannot read them and closes the
  link, so upgrade every existing node before adding or rebuilding one.
  No other frame changed and the store format did not change.
• A peer whose clock is more than sixty seconds ahead is refused.
  Its replication frames are logged and dropped rather than letting it
  win every conflict. Run NTP on every node.

Fixed

• At the node a session is moving to, a forward that arrives right
  behind the drain, before the session is installed, is held for up to two
  seconds and tried again, rather than dropped: sdmqtt_forwards_held_total.
  Only after a drain, so a new subscriber is never given a message published
  before it subscribed.

Found by a review of every crate, each fixed with a test that failed
first.

• Security.
  • A will is authorised as a publication at CONNECT. A client refused
    a topic could publish to it by setting it as its will and dropping
    the connection.
  • An access rule for fleet/+ no longer admits a subscription to
    fleet/#. A subscribe is covered only by a rule whose filter
    covers every topic the requested filter does.
  • A password login for an unknown user costs what a wrong password
    costs, so timing no longer tells which user names exist. The decoy
    it is checked against is made on the first such login, not at
    startup, so an idle broker does not hold argon2's working memory.
  • SCRAM data with a multi-byte character where an attribute name
    belongs is refused as malformed instead of panicking the
    connection.
  • A WebSocket message or frame is bounded by the packet limit. It was
    bounded by the library's 64 MiB default.
  • The admin Authorization scheme is read without regard to case. On
    a loopback listener with no token, bearer <key> was treated as no
    key and admitted with every right.
• Connections.
  • A connection displaced by a takeover stops acting on the session.
    One whose writer was blocked could subscribe, and receive
    acknowledgements, on behalf of the client that replaced it.
  • A write that makes no progress ends the connection, so a client
    that stops reading cannot hold its task, descriptor and connection
    slot for ever. See the listeners page.
  • A session's outbox exists from the moment the session does, so a
    publication arriving in between is no longer dropped.
  • Packets the shell answers itself, a refused publication during a
    partition or an AUTH step, count as keep-alive activity.
  • A takeover of a connection whose session expiry is zero starts a
    new session, as the specification says, instead of resuming one
    that ended.
  • A PUBCOMP or PUBACK for an identifier not outstanding is ignored
    instead of closing the connection without a reason it could encode.
  • A connection the broker closed is counted closed, not clean.
• Clustering.
  • Peers' stamps advance the local clock, so a node with a fast clock
    no longer wins every conflict, session ownership included.
  • A node joining after its peers truncated their logs is sent the
    registers it missed, and the exchange can no longer loop at link
    speed.
  • After a session moves, followers of the old owner start over under
    the new one instead of discarding its stream while acknowledging
    synchronous publications.
  • A drain is complete only when every message it announced arrived.
    One whose first chunk was lost was counted complete, with its queue
    and subscriptions gone.
  • An old owner that no longer holds a session says so, and the
    claimer pulls from a copy, instead of being told the drain was
    complete and empty.
  • The pull after a short drain waits under its own key, and a drain
    it makes whole keeps the drain's subscriptions, inflight window and
    inbound identifiers.
  • A follower stalled behind a missed change is repaired with a
    snapshot, and a stopped session's synchronous waiters are answered.
  • A late forward is redirected at most once and never back to the
    node that sent it.
  • Dialling and accepting a cluster link are bounded by the dead
    window.
  • A batch that removes a session carries no stream position for it,
    which the store would refuse along with the batch.
  • A repair snapshot is stamped with the position its queue was read
    at, from the same read, and a follower ignores one from its owner
    that is no newer than its copy. A snapshot cut a batch behind could
    replace a copy holding the newest message with one that lacked it,
    and a synchronous publish acknowledged against that copy was lost
    when the owner died. Found by CI.
  • A repair snapshot that reaches a follower ahead of a stream frame it
    already holds no longer makes the follower drop the copy. Frames go
    out once the store has taken their batch and repairs read the store,
    so the snapshot can overtake; the follower took the frame for a
    stream starting over and waited thirty seconds for another repair.
    A follower that has answered and fallen behind is repaired again
    after five seconds rather than thirty.
• Rules and the admin API.
  • Replacing or re-enabling a running rule leaves no window in which
    matching publications miss it.
  • Changing a rule's action kind is refused with 409 while its outbox
    holds envelopes, which would otherwise be stranded and make the
    rule undeletable.
  • A tap on a broker that refuses anonymous clients is answered 403
    instead of panicking the request.
  • A replicated rule this node refuses still stores its register.
  • The events stream tags each event with the run it came from, so a
    browser reconnecting after a restart is told to resynchronise
    instead of being replayed another run's events. A bare number is
    still accepted.
• Protocol.
  • A client PUBLISH carrying a Subscription Identifier is a protocol
    error.
  • A property not permitted on its packet type is a Malformed Packet,
    0x81, as the specification says. It was 0x82.
  • A Response Topic with a wildcard is refused.
  • An invalid filter in a SUBSCRIBE is refused on its own, 0x8F,
    and the other filters in the packet are granted. It closed the
    connection.
  • A DISCONNECT followed by the socket closing no longer stamps the
    session's expiry and will delay twice.
• Storage and configuration.
  • A stream position or replica mark for a session the store does not
    hold is refused, as every other per-session record is.
  • Finishing an outbox envelope after its rule was removed is a no-op
    instead of refusing the batch and leaking the payload.
  • A store from a later format is refused with the version message
    before any table is opened.
  • max_connections above what the runtime can count, and cluster
    waits of zero milliseconds, are refused by name at startup instead
    of panicking or looping.
  • A second SIGINT or SIGTERM during shutdown ends the process at
    once.
• The interface. Editing one rule and then another edits the
  second. It kept the first rule's values and saved them over it.

sd-mqtt 0.7.1

Fixed

• A publication forwarded to a session's node while the session was
  moving could be lost. Section 8.3 forwards a publication to the node
  whose register holds the subscriber; section 8.4 moves the session when
  its client reconnects elsewhere; the two arrive over different links.
  A forward that landed after the hand-over found no subscriber and was
  dropped with a debug line. The node now remembers, for thirty seconds,
  which filters each session it gave up held and where it went, and sends
  such a forward on to that node. sdmqtt_forwards_redirected_total
  counts those; sdmqtt_forwards_unroutable_total counts a forward
  nothing wanted and nothing could be done with, which was uncounted
  before. Found by a cluster test failing once on GitHub's runners; a new
  test makes the race happen on purpose by stalling the link.

• CI: the nightly fuzz job installed no fuzzer and used the wrong
  toolchain; the cluster tests' repair counts are measured from the kill;
  the cluster tests' ports are below Linux's ephemeral range and start
  apart under nextest; the development image has procps.

sd-mqtt 0.7.0

Before you upgrade: a store written before this milestone is refused

A broker built from this milestone will not open a store written by
milestone 6. The format moved from 10 to 11 for the user records, and
the refusal says so:

store format version 10 is not readable by this build, which reads version 11.

What to do is what every version move asks: stop the broker, move or
delete the store directory, start the new build. What that throws away
is the milestone 5 list plus, from milestone 6, the replica copies and
stream positions. On a cluster, one store at a time, letting the node
rejoin and be repaired before the next.

Added

• A directory of users, section 12.1, in a crate of its own. A user
  is durable metadata like a rule: written through POST /api/v1/users
  on any node, replicated as a register to every other, removed by a
  tombstone. The password is hashed with argon2id (PHC string) and,
  when the request enrols the user with "scram": true, derived into a
  SCRAM-SHA-256 verifier at the one moment the password is in hand,
  because a verifier cannot be made from an argon2 hash. Enrolment is
  opt-in because section 12.2 treats a verifier as a private key's
  equal. Neither form is ever returned by the API, and the record's Debug says
  <redacted>. The directory is a section 9.3 hook — authenticate —
  and the broker's connect path does not know users exist. A wrong
  password and an unknown name are one answer. auth.anonymous is an
  explicit switch, on by default, that creating a user does not flip;
  auth.scram_iterations sets the verifier's rounds and refuses fewer
  than a thousand. ClientInfo carries the authenticated username, for
  the access control rules that follow. argon2 is the one new direct
  dependency.

• SCRAM-SHA-256 over the AUTH exchange, section 12.2, with
  re-authentication. The arithmetic is pure functions in the server
  crate, pinned by RFC 7677's one published exchange byte for byte;
  the connection shell runs it before the broker is asked to connect
  anybody, and the CONNACK carries the server's final message so the
  client can verify the server knew its key too. An unknown name gets a
  challenge with a salt of its own and fails at the proof, the same
  answer at the same step as a wrong password. =2C and =3D in a
  name are unescaped per RFC 5802. Channel binding is not offered; a
  client asking for it, or for any other method, is told 0x8C, and so
  is one on a node with no directory. Re-authentication is the same
  exchange in AUTH packets, answered with Success and the server's
  final; a wrong proof, or a proof as another user, disconnects with
  0x87. The user name a SCRAM client proves is what the session
  remembers, whatever the CONNECT's username field said.

• Access control rules, section 12.1, deny by default once one
  exists. [[acl]] tables in the file and POST /api/v1/acl through
  the API take the same shape — identifier, order, subject (anyone,
  user:<name>, client:<id>, where a name may end in * to cover a
  prefix and user:* is every authenticated connection), filter, and an opinion on publishing
  and on subscribing — and are consulted together by order, first
  match deciding; the API's replicate as registers, the file's are the
  node's own. Filters match the way subscriptions do, through the topic
  crate, and a shared subscription is judged by the filter it shares. A
  refused subscribe is 0x87 (0x80 on 3.1.1), a refused publish 0x87 on
  its acknowledgement, and an at-most-once publish that was refused —
  by a rule or by the reserved namespace — is now dropped and counted
  as refused, where before nothing counted it. Paho's
  test_subscribe_failure passes on 3.1.1 now that the runner
  configures the rule it needs; on version 5 it stays in the baseline
  as an assumption, because the suite asserts 0x80 where the protocol
  lists 0x87.

• A client cannot publish into the dollar namespace, section
  12.3's other half, deferred since milestone 3. Judged in the broker
  before the authorizing chain and told to the hooks as a reserved
  refusal; 0x87 on an acknowledged publish, dropped and counted
  otherwise. Only the first level is reserved.

• A landing page, site/index.html, published with GitHub Pages by
  .github/workflows/pages.yml. scripts/test_site.py, in CI and before
  every deployment, holds it to its sources: every version it names is
  the workspace's, every figure it quotes is one the spec, a decision or
  the README records, and every link names this repository.

• Install documentation, checked. The README opens with installing
  from a release archive, the image, and Compose, and carries a table of
  every configuration section and every SDMQTT_* variable;
  tests/documented.rs fails naming any variable the configuration reads
  or any section it accepts that the README does not mention, which found
  ten variables and two sections missing. The store-format refusal no
  longer says nothing is released: it says releases before 1.0 do not
  migrate a store.

• Release archives for x86_64 and aarch64 Linux (static musl,
  cut from the image's own builder) and Intel and Apple silicon macOS:
  scripts/package.sh <target> writes dist/sdmqtt-<version>-<target>.tar.gz
  with the binary, LICENSE, README and both notices, and its .sha256.
  scripts/test_package.py unpacks one and runs it — natively, under
  Rosetta, or in a container of its architecture — and requires the
  version, a ready broker and the embedded interface. It caught macOS
  tar shipping AppleDouble ._ files in the first archive.

• A container image. services/broker/Dockerfile cross-compiles the
  broker as a static musl binary with cargo-zigbuild and ships it on
  scratch with its licence and notices: about 10 MB, uid 65532, store
  in /data, read-only root works. The admin listener binds every
  interface in the image, so it refuses to start without
  SDMQTT_ADMIN_TOKEN. scripts/test_image.py, in CI, checks a running
  container, including that a durable session and a message queued for
  it survive docker stop and docker start. The notices generator now
  covers every release target and audits a given --binary.

• Four decisions recorded in docs/decisions/, which section 17
  always said they would be: the SCRAM iteration count stays 4,096 by
  measurement (0002); SCRAM-SHA-256-PLUS is not offered (0003); the
  admin listener stays HTTP/1.1 (0004); release artifacts are Linux musl
  and macOS, not Windows (0005).

• The connection target is met. 100,000 MQTT connections opened in
  2.5 seconds and held for a minute on one node, at about 32 KB each;
  scripts/connection-target.sh measures it in the development image
  with sdmqtt-load, a generator that is a workspace member and never
  ships. See docs/decisions/0001-connection-target.md. Running out of
  file descriptors is now counted: sdmqtt_accept_pauses_total.

• mimalloc is the global allocator, section 10.4, on every target
  but Windows, on by default and removable with
  --no-default-features; static musl builds use its local dynamic TLS
  mode. sdmqtt_build_info{version, allocator} names the build a node
  runs.

• Two rate limiters, section 10.3, both the generic cell rate
  algorithm on the broker's coarse clock. Repeated authentication
  failures from one address are slowed: after
  auth.max_failures_per_minute (10 by default) a connect from it is
  refused with 0x9F before its credentials are read, until the minute
  drains; IPv6 is counted by /64, and the tick sweeps the map. A
  listener's max_publish_rate and publish_burst cap one session's
  publishes, off by default; over it, version 5 is disconnected with
  0x96 and 3.1.1 closed. sdmqtt_authentications_total{outcome}, a
  series registered at zero since milestone 3, now moves;
  sdmqtt_rate_limited_total{limiter} and
  sdmqtt_auth_failure_addresses are new.

• The Settings page, section 11.2's last: paste a key, then manage
  users, access control rules and API keys, and read this node's
  listeners, publish cap included. The key is kept in session storage
  for the tab's life and sent with every request the interface makes.
  GET /api/v1/listeners gains max_publish_rate and publish_burst.

• API keys, section 11.1, with read, write and admin scopes.
  POST /api/v1/keys mints one and shows it once; the broker stores a
  SHA-256 digest; DELETE /api/v1/keys/{id} revokes, and keys and
  revocations replicate as registers. Every mutating endpoint names the
  scope it needs. The configured admin.token is now the bootstrap key
  with admin scope. A write key cannot manage users, access
  control rules, keys or nodes, and is answered with the new forbidden
  error code and 403.

• A listener's maximum QoS and retain availability, section 5.4's
  two deferred properties. listener.max_qos (0, 1 or 2) is advertised
  to version 5 clients and enforced on every publish: above the cap is
  a protocol error, 0x9B on version 5 and a closed socket on 3.1.1,
  which has no code for it. A subscription above the cap is granted at
  the cap, as 3.8.4 allows. listener.retain_available = false refuses
  a retained publish with 0x9A and refuses the connect of a client whose
  will is retained, rather than accepting it and dropping the flag the
  client asked for. The defaults advertise nothing and refuse nothing,
  so the conformance baseline is unchanged.

• A durability mode per listener, section 8.6's override, deferred
  from milestone 6. listener.durability = "sync" makes publishes on
  that listener wait for a follower while the node's mode stays
  async, or the reverse. The timeout is the node's in both cases: a
  second timeout would be a second answer to "how long is too long" on
  one node. Refused at startup on a no...