Skip to content

tellus-v0.6.0

Choose a tag to compare

@tellus-rs-release-plz tellus-rs-release-plz released this 20 Nov 10:22
· 9 commits to main since this release
91dd01d

Release notes

Every actor declares its stash mode

Actor and EventSourced have a new associated type, Stash, next to Timers. It decides whether
an actor can stash: Stashed gives its context stash and unstash_all, Unstashed gives it
neither, and its run loop holds no stash code. An actor which does not stash declares Unstashed:

use tellus::{Actor, Unstashed, Untimed};

impl Actor for Greeter {
    type Message = Greet;
    type Timers = Untimed;
    type Stash = Unstashed;
    // ...
}

Nothing else changes for such an actor. ActorContext takes the stash mode as a third type
parameter, which defaults to Unstashed, so a signature naming ActorContext<Self::Message, Self::Timers> still compiles. The stash itself is described in
Stashing.

A cluster of 0.5 nodes cannot be upgraded node by node

The cluster protocol is now version 2, and a 0.6 node speaks nothing else. A 0.5 node and a 0.6
node refuse each other at the handshake, so they never form one cluster. To upgrade, stop every
node of the 0.5 cluster and start the 0.6 nodes as a new cluster. Rolling upgrades work from 0.6
on, which is what version 2 is for: every payload now carries the schema version of its type.

Every type which crosses nodes, as a message, a reply or a published item, implements
cluster::Schema, which Manifest now requires. For a type which keeps its shape, the defaults are
enough:

use tellus::cluster::Schema;

impl Schema for Greet {}

tellus implements Schema for the primitives, String, (), Option and Vec of those, and for
ActorRef and ReplyTo, but not for a container of an application type such as
Vec<Greet> or Result<Greet, Error>. Such a message crosses nodes in a type of the
application. How to change a type while old and new nodes run side by side is described in
Rolling upgrades.

The Json codec refuses NaN and infinite floats

Json used to write a NaN or an infinite float as null, and read Some(f64::NAN) back as None.
It now refuses to encode such a value. A command whose event holds one fails the actor under its
supervision, and nothing is appended. A snapshot of a state which holds one is skipped with a
warning.

A null which an earlier release stored for a float stays in the store, and the type which wrote it
cannot read it back. To recover such a stream, override Versioned::decode and decode a shape with
an Option<f64> in that place. Which non-finite value it was is lost.

Cluster durations which switch off a safeguard at zero are refused

A zero down_retention, leave_timeout or ambiguous_join_bound in EndpointConfig is now
refused, by start_endpoint and by deserialization, with InvalidEndpointConfig::ZeroDownRetention,
ZeroLeaveTimeout or ZeroAmbiguousJoinBound. Each of them used to switch off what the setting is
for: a zero leave_timeout, for example, severed a departing node's connections without waiting for
their drain.

PhiAccrual::with_warmup_deadline now returns a Result and refuses zero with
InvalidPhiAccrual::WarmupDeadline. A zero warmup deadline declared a peer unavailable one tick
after its first heartbeat. A config file with warmup_deadline: 0s fails to deserialize.

Where one of these four is zero, set a positive duration, or leave the key out for its default.

Zero stays valid for keep_majority.after, down_after_deadline.after, settle and
majority_with_fallback.after, and for acceptable_pause in PhiAccrual. An endpoint refuses a
zero acceptable_pause with the default intervals, though, unless min_std_deviation is raised.
docs/cluster.md says what zero means for each.

MemberState serializes in snake_case

cluster::MemberState used to serialize as "Up", "Left" and "Down". It now serializes as
"up", "left" and "down", like Lifecycle, like the tags of Transition and Change and like
its own Display. The state of a serialized Member changes with it, and with that the
serialized form of every type which holds a Member: ClusterState, BrokenLink, Transition and
Change.

Code which reads the JSON of these types without them has to match the new strings. Code which
deserializes into them reads only what 0.6 or later has serialized, and code built against an
earlier release does not read what 0.6 serializes. The same holds for every other format which
writes the name of a variant, such as CBOR. Gossip is not affected: a frame carries the state as the
index of its variant.

The payload codec is an enum, and a custom codec is no longer possible

cluster::codec::Codec is now an enum with one variant, Codec::Postcard, instead of a trait
implemented by Postcard. The trait had to be object safe, so every payload was serialized
through erased_serde, one dynamic call per field and per element. A message carrying a
Vec<u8> of 64 KiB decoded at 21 MiB/s on Linux. Without the trait, encoding and decoding are
static calls into postcard. On a 10-core arm64 machine, bulk transfer went from 97 to 713 MiB/s
and pipelined round trips from 534,000 to 1,462,000 per second. The bytes on the wire are
unchanged.

EndpointConfig::codec is a Codec instead of an Arc<dyn Codec>. Code assigning
Arc::new(Postcard) assigns Codec::Postcard, which is also the default. In a config file,
codec: postcard stays as it is. Postcard, DecodeVisitor and
InvalidEndpointConfig::CodecNameTooLong are gone, and an application's own impl Codec has no
replacement. Another format is a new variant of Codec; open an issue for the one you need.

A Schema override receives the codec by value, as codec: Codec, and decodes an older shape
with codec.decode::<OldShape>(payload).

Directory reads fail once this node has left or is down

receptionist::lookup, receptionist::lookup_at, Subscription::current and pubsub::publish
now fail with the new ReceptionistError::NoLongerMember once this node's own member entry is
Left or Down, because it left the cluster, was downed or downed itself. Before, they answered an
empty list or Ok(0), as if nobody were registered, since the directory lists the registrations
of Up members only and so left out this node's own. A node which is leaving answers as usual until
its entry is Left, and a node which never formed or joined a cluster is unaffected.

ReceptionistError has a new variant, so an exhaustive match on it needs an arm for
NoLongerMember. A caller which treated an empty answer after cluster::leave or a downing as
"nobody registered" handles the error instead, usually by stopping, since only a restarted process
can rejoin.

Oversize cluster config values are capped or refused

A duration in EndpointConfig or BootstrapConfig which the clock cannot represent, e.g.
Duration::MAX, is now capped at about thirty years, as a timer's delay is. It used to panic: in
start_endpoint and in deserialization for heartbeat_interval and watch_refresh_interval, in
leave and on every member learning of a departure for leave_timeout, on an ambiguous join for
ambiguous_join_bound, and in the probe task for probe_interval. With the default failure
detector, start_endpoint now refuses such a heartbeat_interval or watch_refresh_interval with
InvalidEndpointConfig::FailureDetectorTooEager.

change_capacity is now refused above 65536 with InvalidEndpointConfig::ChangeCapacity. The
channel allocates every entry when the endpoint starts, and the old bound, half the address space,
let start_endpoint panic allocating it, for example at 2^62.

PhiAccrual::with_max_sample_size now refuses a window above PhiAccrual::MAX_MAX_SAMPLE_SIZE,
10000, with the new variant InvalidPhiAccrual::MaxSampleSizeTooLarge, and so does
deserialization. The window grows by one interval of eight bytes per inbound frame and peer up to
that size, so a very large value let it grow for as long as the endpoint ran. A match on
InvalidPhiAccrual has to handle the new variant.

Where change_capacity or max_sample_size is above its new bound, lower it, or leave the key out
for its default.

QuicTransport::new takes the idle timeout as a new last parameter and sets it on the given
TransportConfig, replacing whatever that held. The idle timeout is what bounds a handshake toward
a silent address, and a config without one left such a handshake unbounded. new refuses an idle
timeout which is not at least one whole millisecond above the 10 s keep-alive it sets, since an idle
connection would then close between two keep-alives, with the new variant
QuicTransportError::IdleTimeoutTooShort. It refuses one QUIC cannot encode, above 2^62
milliseconds, with QuicTransportError::IdleTimeoutTooLong. mutual_tls, from_config and dev
use QuicTransport::DEFAULT_IDLE_TIMEOUT, 30 s, which is quinn's default and hence what they used
before. A caller of new passes QuicTransport::DEFAULT_IDLE_TIMEOUT to keep the previous
behavior, and a match on QuicTransportError has to handle the two new variants.

Leaving an endpoint which never formed or joined a cluster fails

cluster::leave and cluster::leave_on_terminated on an endpoint which has neither formed nor
joined a cluster now fail with the new LeaveError::NotAMember and leave the endpoint as it was, so
it can still form or join a cluster. They used to succeed with an empty Departure and latch the
endpoint downed for good, so a later form or join failed. A join in progress does not count as a
membership.

A match on LeaveError handles the new variant. Code which leaves only once it has formed or joined
needs no other change.

Sequence numbers refuse to overflow instead of saturating

SeqNo::succ and SeqNo::advanced_by now return an Option<SeqNo>, which is None beyond
u64::MAX. They used to saturate there, so two events could end up at the same position.

A store which calls either function handles None by failing the append, for example with
AppendError::Store and an error of its own, and never stores a position twice. The new
SeqNoOverflow error describes such an append. InMemoryStore returns it, so its EventStore::Error
is now SeqNoOverflow instead of Infallible. Code which matched that error as uninhabited, with
match error {}, now has to handle it.

The event-sourced run loop checks the position before it appends. A command whose events would
advance the stream beyond u64::MAX fails the actor under its supervision, and nothing is appended.
Recovery fails at a stored event at u64::MAX.

Bootstrap config fields, the formation view and the codec error constructors are renamed

These names in the cluster API now say what they are:

Old New
BootstrapConfig::min_peers BootstrapConfig::min_nodes
BootstrapConfig::DEFAULT_MIN_PEERS BootstrapConfig::DEFAULT_MIN_NODES
BootstrapConfig::formation BootstrapConfig::formation_provider
cluster::formation::Formation cluster::formation::FormationView
CodecError::encoding CodecError::make_encoding
CodecError::decoding CodecError::make_decoding

The YAML keys of BootstrapConfig change with its fields, and since unknown keys are refused, a
configuration file with the old keys no longer loads. Rename them:

bootstrap:
  min_peers: 5
  formation: majority

becomes

bootstrap:
  min_nodes: 5
  formation_provider: majority

A custom FormationProvider takes &FormationView in observe and form, and a custom Schema
implementation calls CodecError::make_encoding and CodecError::make_decoding.

Every store call of an event-sourced actor is bounded by a timeout

Persistence now bounds every store call by a timeout, Persistence::DEFAULT_TIMEOUT, 10 s, unless
Persistence::with_timeout sets another. with_timeout refuses zero with the new error
InvalidPersistence::ZeroTimeout. The timeout bounds the snapshot load and every page read of a
recovery, and the append and the snapshot save of a settlement, each on its own. A load, read or
append which exceeds it fails the actor into its supervision strategy, and a snapshot save which
exceeds it is logged like a failed save. A store which never answered used to keep the actor, and
every ancestor waiting for it, from terminating.

The stop guarantee changes. A shutdown used to wait for an append under way until it settled, so it
never left an append unsettled. It now waits at most the timeout for the append, and at most the
timeout again for a due snapshot save. An append which exceeds the timeout is ambiguous, and the
logged error says so: its events may have been stored. The replay of the next incarnation picks them
up if they have landed by then. An append which lands only after that replay makes the next append
conflict, and the command of that next append fails into supervision.

The timeout gives up on a call by dropping it, which does not end what the call started in a server.
A store hence bounds its own calls below the timeout. For PostgresStore, the README of
tellus-persistence-postgres says how. A store whose calls can take longer than 10 s, for example a
replay page of very large events over a slow link, raises the timeout with with_timeout.

Bootstrap bounds every join attempt and decides formation on a fresh view

BootstrapConfig has a new field, join_attempt_timeout, 5 s by default
(BootstrapConfig::DEFAULT_JOIN_ATTEMPT_TIMEOUT), with the YAML key of the same name. A join round
used to wait for each address until the transport gave up, which takes 30 s for QUIC towards a
silent address. Two silent addresses below a live member hence used up the 60 s timeout the
bootstrap READMEs recommend, so a restarted node could keep timing out without ever asking a
healthy member. Now a round goes on to the next address once an attempt has run for
join_attempt_timeout, and counts that address as unavailable. An attempt which has sent its Join
handshake still runs to its conclusion, so an admission or an ambiguous join is recorded as before.
A zero timeout is refused with the new variant InvalidBootstrapConfig::ZeroJoinAttemptTimeout.

Bootstrap also resolves the discovery again after a round it would decide formation on, and decides
only if that resolution shows the same settled universe. Before, it decided on the universe resolved
before the round. This node could hence form a cluster while a lower address which registered during
a long round formed a second one.

A BootstrapConfig built with a struct literal needs the new field, e.g. through
..BootstrapConfig::new(), and a match on InvalidBootstrapConfig has to handle the new variant.

The store contract suite checks persistence IDs and read limits at their extremes

persistence_tests has five new checks, and event_store_contract and snapshot_store_contract
run them. A store which passed the suite before can fail it now.

events_round_trip_under_persistence_ids_at_their_extremes and
snapshots_round_trip_under_persistence_ids_at_their_extremes use persistence IDs whose entity
type or entity ID is exactly 255 bytes long, in ASCII and in multi-byte characters. They also use
segments holding non-ASCII characters, %, _, single and double quotes, a backslash and spaces.
A store with keys shorter than 255 bytes, or with a character set short of four-byte characters,
fails them.

streams_are_isolated_between_lookalike_ids and snapshots_are_isolated_between_lookalike_ids
require two streams and two snapshots for IDs which differ only in case, in Unicode normalization,
in a trailing space or in the last byte of a 255-byte segment. The same holds for an ID holding %,
_ or a backslash and an ID it matches as a LIKE pattern, and for an ID and its prefix. A store
with a case-insensitive or padding collation, a normalizing or truncating key, a LIKE match or a
prefix match fails them.

reads_honor_limits_at_their_extremes reads with limit 1 and with limit usize::MAX. The latter
must return the whole rest of the stream. A store which casts the limit to a narrower integer
without clamping it fails this check.

What implementations of the store, connection and discovery traits must guarantee

The documentation of EventStore, SnapshotStore, Connection and SeedDiscovery now states
guarantees tellus relies on, and a third-party implementation has to meet them.

A store call may be dropped before it completes: at the timeout of its Persistence, and for read
and load also when the actor is stopped during recovery. A dropped call must not panic or corrupt
the store, and the store must stay usable for later calls. Dropping a call does not end what it
started in a server, so a store bounds its own calls below the Persistence timeout.
EventStore::append no longer promises that a failed append leaves the stream untouched; only a
conflict does. Any other failure may be ambiguous, like a dropped append, and replay reconciles it
as docs/persistence.md describes under "Store timeout".

The stream halves of a Connection must keep the connection open and stay usable after the
Connection value is dropped. Dropping the last half must close the connection.
Connection::accept_data must be cancel safe, so a dropped future must not lose a stream.
FrameReceiver::recv returns None once the peer has closed the stream, or once the connection
was closed between two frames and what arrived before has been received. A connection which fails,
or closes in the middle of a frame, makes recv fail.

SeedDiscovery::resolve must resolve one address per node and must bound itself. A source yielding
no address at all must fail rather than return an empty list. FixedSeeds::new hence refuses an
empty list with the new error NoSeeds, and so does deserializing FixedSeeds. An empty list used
to let a node with min_nodes of one form a cluster alone. Code calling FixedSeeds::new has to
handle the Result.

A child stopped before it ran no longer runs init

A child whose parent stops it before the child's task has run now terminates without running
init. It used to run init first, with every send, spawn and connection made there, and only then
stopped. A child spawned in the same receive which returns Control::Stop is the common case. The
same holds for a root whose ActorSystem::stop comes before the root ran. An event-sourced actor
stopped before it ran already skipped recovery, which is now documented: the store sees no load and
no read, and neither init, init_from_snapshot nor recovered runs.

Such an actor still runs the termination sequence. Its watchers get an ordinary terminated signal,
supervision is not consulted, and its parent terminates only behind it. Code which relied on init
running in this case, for example to hand out a reference or to acknowledge a start, must not count
on it: wait for the actor to be up before stopping its parent.

A departure lists the peers which left themselves

Departure has a fourth list, departed: the peers which left the cluster themselves before they
acknowledged this node's drain. Two nodes leaving at the same time used to report each other as
unacknowledged, or as timed out after leave_timeout. Now each one's DrainAck still counts after
the other's departure has been seen, and a drain towards a peer which has left is reported as
departed, since whether it got what was queued for it is unknown.

Departure::is_acknowledged stays strict and is false while departed is not empty, although
unacknowledged and timed_out may both be empty then. Code which checks only those two lists to
decide whether a departure failed should use the new Departure::is_complete instead, which accepts
acknowledged and departed peers alike.

A departure no longer waits for a node which is Down when it begins, nor lists it. Its death
already turned what was queued for it into dead letters or failed asks, so a leave right after a
node died used to wait out leave_timeout for nothing.

cluster-dev adds only the dev transport

The cluster-dev feature now adds QuicTransport::dev and nothing else.
cluster::sever_connections, cluster::gossip_bytes and cluster::drop_terminated_frames are no
longer part of it. They moved to __internal, tellus's own test and bench support, which is not
public API.

Changelog

Features

  • (persistence) [breaking] Bound every store call by a timeout (#409)
  • (host) Make HostConfig deserializable (#392)
  • Check more fencing, isolation and extremes in the store contract (#288)
  • [breaking] Replace the passivate function of Host with Hosted (#265)
  • (cluster) [breaking] Version message payloads for rolling upgrades (#262)
  • Add stashing, opt-in per actor via Actor::Stash (#201)

Bug Fixes

  • (cluster) Drain lanes toward unknown and restarted nodes on leave (#506)
  • (cluster) Resolve concurrent departures by incarnation (#453)
  • (examples) Wait for every view before leaving, sort seed addresses (#454)
  • (cluster) End a QUIC stream at a graceful close between frames (#451)
  • (cluster) Refuse joins as joining while pinned to a cluster (#448)
  • (cluster) Await abandoned join attempts before deciding formation (#450)
  • (cluster) Judge receptionist settled by the current cluster state (#447)
  • (persistence) Settle stash requests once the events are durable (#446)
  • (persistence) Stop before recovered when stopped during recovery (#444)
  • (examples) Wait for the passivation instead of sleeping (#435)
  • (examples) Bound every wait and check every exit status (#424)
  • [breaking] Never run init for a child its parent stopped before it ran (#414)
  • (cluster) [breaking] Refuse empty FixedSeeds and state the trait contracts (#415)
  • (cluster) [breaking] Bound bootstrap join attempts, decide on a fresh view (#410)
  • (persistence) [breaking] Refuse a sequence number overflow instead of saturating (#395)
  • (cluster) [breaking] Refuse to leave an endpoint which never formed or joined (#400)
  • (cluster) [breaking] Cap derived deadlines and bound oversize config values (#397)
  • (cluster) Keep a self-down from settling watchers and asks (#399)
  • (cluster) [breaking] Fail directory reads once this node has left or is down (#394)
  • (cluster) Send toward a peer at the smaller frame limit of both (#389)
  • (cluster) Release the reply destinations of a dead-lettered reply (#388)
  • (cluster) Name the asking incarnation in reply frames (#383)
  • (cluster) Refuse sends toward a dead incarnation through any lane (#382)
  • (cluster) [breaking] Serialize MemberState in snake_case (#339)
  • (cluster) [breaking] Refuse zero durations which switch off a safeguard (#332)
  • (cluster) Refuse a send toward another incarnation of this node (#331)
  • (cluster) Keep a panicking decode from closing the connection (#327)
  • (cluster) Keep a panicking reply encode from failing the responder (#316)
  • (inspect) [breaking] End event streams and the recorder on a shutdown future (#307)
  • [breaking] Refuse a NUL in a persistence ID, map positions beyond BIGINT (#308)
  • Keep a panicking reply_to conversion from failing the responder (#306)
  • (cluster) [breaking] Return a ClusterStateSubscription from cluster_state (#304)
  • Decode long CBOR text and exact JSON floats, refuse non-finite ones (#303)
  • (cluster) Warn of an overdue departure only if it was awaited (#287)
  • (cluster) Ask every seed before forming once a pin is released (#286)
  • (cluster) Hold back joins while this node is leaving (#285)
  • (cluster) Drop removed reply entries outside the pending lock (#274)
  • (cluster) Stop a lane from dialing once its endpoint is downed (#273)

Refactor

  • Replace Infallible with ! and require Rust 1.100 (#516)
  • (cluster) [breaking] Move test support from cluster-dev to __internal (#457)
  • (cluster) [breaking] Rename bootstrap fields, Formation and codec makers (#396)

Documentation

  • Regenerate changelogs with the unified config (#505)
  • (examples) Call the device manager's message DeviceManagerMessage (#502)
  • Correct statements found in review of the 0.6 docs (#483)
  • Clarify cluster guarantees and complete the READMEs (#475)
  • Fix review findings in READMEs, roadmap, demo and examples (#477)
  • Fix the review's core and persistence doc findings (#476)
  • (cluster) Name every node in FixedSeeds to keep a minority out (#460)
  • (cluster) State the order of gate and check in read_frames (#455)
  • (examples) Show pubsub, downing, rolling upgrades and formation (#431)
  • (examples) Show schema upcasting and event-sourced children (#429)
  • (examples) Show Host and passivation in a how-to example (#428)
  • (cluster) Limit the release of a lost frame to one not yet sent (#406)
  • (cluster) Say what releases a dead-lettered reply's destinations (#403)
  • Add release notes for the breaking changes since 0.5.3 (#321)
  • State the codecs' limits on nesting and on Some(None) (#313)
  • Add a how-to example for retrying failed writes (#270)
  • Retry the fourth lookup of the stash how-to instead of sleeping (#269)
  • Add a how-to example for stashing (#255)
  • State the stash exception to the unwatch contract (#257)
  • Mention clustering in the crate description (#245)
  • Show the build badge for main only (#244)

Performance

  • (cluster) [breaking] Make the payload codec a closed enum (#340)
  • Encode events in a plain loop (#236)

Styling

  • Trim example comments, rename the manager's message, per review (#474)
  • (cluster) Shorten paths and place docs in the remoting test (#473)
  • (cluster) Order and shorten failure, reachability, bootstrap, QUIC (#472)
  • (cluster) Order, scope and name the directory and reply code (#471)
  • (cluster) Order, scope and name the endpoint and membership code (#468)
  • (tests) Unify paths and helpers' names, regroup the budget test (#470)
  • (persistence) Order items, narrow fields, shorten paths per review (#469)
  • (examples) Paths, item order, uniform constants and contexts (#467)
  • (core) Narrow visibility, order items, trim docs per review (#464)
  • (cluster) Order, name and shorten the peer and frame code (#463)

Miscellaneous Tasks

  • Add the stash bench as a binary of its own (#252)
  • Rename docker-compose.yaml to compose.yaml (#241)

Other

  • (persistence) [breaking] Check IDs and read limits at their extremes (#412)