Skip to content

v3.1.1

Latest

Choose a tag to compare

@kiwidream kiwidream released this 29 Oct 18:47
v3.1.1
6046726

Release date: 2026-10-29. Prepared on 3.x.x (#810, then #818 to #821)
and tagged v3.1.1 on its promotion into main.

3.1.1 is a patch release of 3.1.0: fixes only, with
no new feature. Its one new setting is for a self-check run, not a frontend
(#808). It contains every change merged into 3.x.x after the v3.1.0 tag.
The dual writer stays off by default: with PRISM_DUAL_WRITER unset, a
frontend runs as one writer, as in 3.1.0.

Unlike the upgrade from 3.1.0-rc.1 to 3.1.0, this one is more than a binary
swap:

  • Every start requires migration 032. It adds two functions and changes
    no table. 3.1.1's migrate applies it while the 3.1.0 frontends serve, and
    they keep serving and restarting beside it (Upgrading).
  • Each frontend holds one more database connection, for its
    payout-revision listener (#795). Budget it before the upgrade
    (Database connections).
  • Five published alert rule UIDs changed (#793), and two warnings are
    gated differently (#800). Re-provision the rules as
    Alert rules says.

Everything in the 3.1.0 notes applies, with the
changes below.

Changes since 3.1.0

Share acceptance and the order lock

  • #788 (#716): a share admitted under a tip-change lease keeps its credit
    when its commit gate meets a writer on one of the chain-authority locks.
    The append rolls back, waits for the authority outside ORDER_LOCK, and
    runs again, logging share commit gate waited out contention for its authority; appending again at INFO. Before, the share was refused and
    never credited. Contention that outlasts the acknowledgement deadline is
    still answered current chain state is unavailable, now logged
    closure="authority-contended"; closure="authority-unavailable" now means
    a readiness or tip-observation loss only
    (commit-gate refusals).
    This is the refusal behind #779's L3 failures.
  • #801 (#738): a share append no longer writes the qbit_prism_cluster
    row. Each share is stamped with the greatest of the stored ledger clock,
    the database's clock and the stamp of the node's newest own share
    (qbit_prism_ledger_clock, migration 032). A snapshot held on
    the primary no longer leaves a dead row version per share, which
    saturated ORDER_LOCK in about 20 s at 400 shares/s. The snapshot anchor
    and the peer sync's own-log floor are the only writers of the stored clock.
  • #802 (#711): a share append without a block does its in-lock work in
    one server-side call, qbit_prism_append_share (migration 032). ORDER_LOCK
    is held for that call and the COMMIT, where it was held for the lock
    statement, three statements and the COMMIT. A refusal is raised before
    any write and mapped back to the miner answer the statements gave. The
    append's ORDER_LOCK wait and hold metrics now come from the server's own
    measures. An append abandoned while queued on ORDER_LOCK can now leave a
    gap in share_seq, as a rollback always could; window proofs tolerate
    gaps.

Blocks, offers and settlement

  • #797 (#762): a submitblock whose new connection isn't ready within
    half its deadline (PRISM_BLOCK_SUBMIT_RPC_TIMEOUT_SECONDS, 1 s by default,
    and never more than 4.9 s) fails as not sent: its block returns to pending
    and is offered again. Before, it was recorded as an unknown offer and never
    offered again, although it never left the frontend. last_error names the
    cut, as in no connection within 500ms, its share of the call's 1s deadline. The cut connect runs on and serves the next call, on Linux only
    and only while the kernel reports it open and unread. A call that took a
    pooled connection meanwhile keeps that request's outcome.
  • #798 (#630): a post-offer write (the landing, the finish, or an orphan
    settlement) that loses a payout-revision race is observed again and
    retried in place, at most 3 times, none starting 60 s or more after the
    first lost race. Before, every lost race logged ALERT: offered candidate could not be processed and went to reconciliation, about once every 3
    landings on the 400k two-frontend soak. A spent budget, and every other
    failure, keep the ALERT and reconciliation. candidates recover retries
    its finish too. The new counter
    qbit_prism_post_offer_revision_races_total counts both outcomes, and
    ledger operations
    has the rewritten runbook.
  • #786 (main's #781): each disposition poll under the block-proof
    acknowledgement bound runs in its own task, so the bound no longer drops a
    pooled connection partway through a checkout (3.1.0's known issue). #786
    merged the released main (#783) into 3.x.x, which also brought
    docs/releasing.md and the released 3.0.0 notes.

Work refresh

  • #795 (#752): when one frontend moves the payout revision without
    moving the tip, as a pool block's confirmation does, every other frontend's
    refresh wakes through PostgreSQL LISTEN/NOTIFY and republishes at once.
    Before, they kept the superseded work, and refused shares on it as
    stale-job, until their next PRISM_BLOCKPOLL_SECONDS tick, up to about
    2.6 s after each pool block. The revision fence is unchanged, and the tick
    stays as the fallback. Each frontend's listener holds its own database
    connection (Database connections).

Migration runner

  • #791 and #821 (#789): 3.1's migrate can no longer deadlock with frontends that
    serve the database while 027 and 028 run, as rc.6 frontends do during the
    pre-migrate from 3.0. It locks every table 027 and 028 alter in one
    statement that waits at most 100 ms, and on a timeout or deadlock rolls the
    whole migration transaction back and tries again, backing off from 100 ms
    to 2 s, for up to ten minutes. 027's and 028's SQL is unchanged. Past ten
    minutes it stops, records nothing, and names up to five holding
    transactions. PostgreSQL cancels an autovacuum pass that blocks a lock
    request only from the waiter's deadlock check after deadlock_timeout
    (1 s), which no 100 ms attempt reaches, so from its second attempt on
    migrate lowers its own deadlock_timeout to 50 ms for its lock
    requests only: PostgreSQL then cancels the worker blocking them, unless
    the pass prevents wraparound, as it did for 3.1.0's 027. The rest of the
    transaction, and every frontend, keeps its own deadlock_timeout.
    Lowering it takes a superuser, or from PostgreSQL 15 SET on the
    parameter, which migrate asks only after contention; a role without
    either waits a blocking pass out, warns once it sees an autovacuum
    worker, and its refusal names the worker and the remedies. The 100 ms
    bound holds for frontends on the primary only: a hot standby replays the
    locks each attempt took. Every connect that migrates takes the migration
    history's lock the same way, so a start with
    PRISM_POSTGRES_INIT_SCHEMA=1 that can't take it now retries for up to
    ten minutes, warning on the first attempt and every tenth, where it failed
    after PRISM_DATABASE_LOCK_TIMEOUT_MS (5 s)
    (the migration guide).

Database TLS: the vendored sqlx-core

  • #794 (#771): sqlx-core 0.8.6 is vendored in vendor/sqlx-core with
    transact-rs/sqlx#4251, so a large statement over TLS can no longer lose its
    wakeup and stall the ledger connection, and any advisory lock it holds,
    until a client deadline. Both workspaces patch sqlx-core to that
    directory, and lab/prism/Dockerfile copies it, so a build from the tag
    needs vendor/. CI checks that both lockfiles build the vendored crate
    and that every sqlx-core RustSec advisory is reviewed.
  • #807 and #818 (#713): sslmode=verify-ca behaves as libpq's verify-ca does.
    It checks the server's certificate chain against the root certificates in
    sslrootcert (or PGSSLROOTCERT) only, never against the public roots,
    and it does not check the host name. So it accepts a certificate your CA
    issued for another host name; before, it refused one, as verify-full
    does. verify-ca without sslrootcert, or with an sslrootcert that
    holds no certificate, is refused before the TLS handshake, with
    sslmode=verify-ca requires sslrootcert or holds no certificate
    (Upgrading). Use verify-full wherever the certificate's
    names match the host you connect to: it checks the name as well.
    verify-full is unchanged, so it still trusts the public roots beside
    sslrootcert, unlike libpq's verify-full; only verify-ca now matches
    libpq. This backports transact-rs/sqlx#3861 into the same vendored
    crate, with a local change that is not upstream's: sqlx-core trusts the
    public roots beside sslrootcert, so #3861 alone would have let
    verify-ca accept any publicly issued certificate, for any name.

Dual writer

  • #799 (#785): the peer sync's apply_prepared takes the blob-GC fence,
    in chunks of at most 1,000 jobs and 2 MiB each, so the blob collector can't
    delete a template or balance blob a pulled job reuses. Before, the job
    could commit without it, and a block the peer found on that job couldn't
    be adopted on this node.

Audits, self-check and the public API

  • #808 (#734): production self-check runs and reports every local check
    before its audit-completeness verdict, so it can gate a deployment while
    import-audits runs. Before, it stopped at audit completeness and skipped
    them. Its audit_completeness now reports a status (complete,
    legacy_import_pending or incomplete), and pending_legacy_rows and
    incomplete_native_rows beside the existing counts. With
    PRISM_SELF_CHECK_LEGACY_IMPORT_PENDING=1, set for that run only, legacy
    rows awaiting import are reported as pending instead of failing; a native
    row missing its bytes, or an unreadable count, still fails.
    make prism-self-check passes the setting into the container only when it
    is set. scripts/prism_legacy_import_admission_gate.sh stays, for earlier
    builds
    (the migration guide).
  • #796 (#735): /public/v1/artifacts/<sha> answers a no-store 404,
    not 500, when an audit's legacy body, or a share segment it names, isn't on
    the host yet, as before import-audits has stored it. A body that is there
    but can't be read or verified still answers 500, logged at warn. A row
    re-imported under another address between the lookup and the read answers
    404 instead of serving the other audit's body. file:// share segment URIs
    now resolve in the API, import-audits and the audit CLIs.
  • #806 (#733): a legacy segment_range or segment_prefix part verifies
    when its slot has a share_seq gap outside the range the part references.
  • #809 (#709, #731): legacy (1.x and 2.x) audit bodies verify without
    sidecars when their digests were computed as the Python writers computed
    them: range and prefix digests over Python's ASCII-escaped JSON (#709), and
    2.x audit-bundle.v2 parts digests in 2.x's key order (#731). A parts digest
    in sorted-key order, which no writer produced, is now refused.
    import-audits, the canonicalizer and the public API all verify through
    the fixed parser. The parts-digest checker keeps its statuses, so with this
    release its 2x-order-only rows are ones import-audits accepts, and the
    sidecar tool reports them ok-in-3x without writing a sidecar
    (#731 in the migration guide).

Alerting

  • #793 (#757): five native alert rule UIDs are shortened to Grafana's
    40-character limit (Alert rules). CI rejects any UID that
    is too long, has other characters or is duplicated.
  • #800 and #820 (#758): PrismAcceptedRevisionWorkPending and
    PrismAcceptedRevisionWorkTrackingUnknown count only on a frontend that
    had an authorized miner in the last 15 minutes, so an idle backup frontend
    no longer warns after every pool block. On the server, a frontend no
    longer keeps a block's wait unknown until restart when another settlement
    confirmed the block first (a peer's, or its own attempt whose reply was
    lost): its own later settlement proves a covering revision, and its next
    write at or above it ends the wait without a sample. Nor can an idle
    frontend's waits exhaust its 4,096 tracked block identities any longer:
    once a block matures, a wait that no authorized miner was connected to
    receive since its acceptance ends without a sample. Before, an idle
    backup latched tracking unknown after about 4,096 pool blocks, and after
    a failover PrismAcceptedRevisionWorkPendingCritical could not page on
    it until it restarted.

Tests and CI only

  • #775 (R1's pre-migrate checks), #792 (#729, the restart test's statement
    count), #804 (the block-only poll acquire tests) and #805 (#790, the
    real-node smoke's tip-to-notify gate). None changes the server.

No dependency version changes, and no package is added to or removed from
either lockfile: sqlx-core 0.8.6 now builds from vendor/sqlx-core. Two
manifest changes touch only crates already in the graph:

  • the server's tests name rustls and sqlx-core as dev-dependencies, both
    already in the graph through sqlx, with sqlx-core held at =0.8.6, the
    vendored copy (#794 added them; #807's verify-ca test uses them too);
  • the server's hyper-util dependency enables its client-legacy feature
    (#797).

Operator-facing changes

Migrations

  • New: 032 (crates/qbit-prism-server/migrations/032_share_append_call.sql,
    #711, #738). It creates two functions, qbit_prism_ledger_clock and
    qbit_prism_append_share, and alters no table. It declares no capability
    and needs no shutdown proof. It is applied in the migration transaction,
    with no up-front table lock: CREATE FUNCTION takes no lock a serving
    frontend waits for
    (migration 032).
    Every 3.1.1 frontend, tool and operator command but migrate refuses a
    database without it.
  • No released migration changed. git diff --stat v3.1.0 over
    crates/qbit-prism-server/migrations shows only the new 032, so the SQL of
    001 to 031 is unchanged. #791 changes the runner around 027 and 028, not
    their SQL.

Settings

  • Added, for self-check only: PRISM_SELF_CHECK_LEGACY_IMPORT_PENDING
    (#808). Set it for a self-check run while import-audits is storing
    legacy audits, and never in a frontend's environment: there it would hide
    an import that stopped, and a 3.1.0 run or check-config refuses a
    PRISM_* name it doesn't know in production mode.
  • Removed or renamed: none, and no default changes.
  • Bounding something more:
    • PRISM_BLOCK_SUBMIT_RPC_TIMEOUT_SECONDS also bounds submitblock's
      connect, at half the deadline and never more than 4.9 s (#797).
    • PRISM_DATABASE_LOCK_TIMEOUT_MS no longer bounds a migrating start's
      wait for the migration history's lock, which retries for up to ten
      minutes (#791).

Database connections

Each running frontend opens one LISTEN connection for its payout-revision
listener (#795), on its own one-connection pool outside the run pool and its
PRISM_DATABASE_MAX_CONNECTIONS. Its session is named
qbit-prism-revision-wake in pg_stat_activity. So each frontend uses up to
PRISM_DATABASE_MAX_CONNECTIONS + 1 connections, + 2 more in dual-writer
mode for its health pool. Raise the database's max_connections, or lower
PRISM_DATABASE_MAX_CONNECTIONS, by one per frontend before the upgrade, and
keep the reserve for migrations, monitoring, backup and failover
administration
(storage sizing).
LISTEN needs a session: if PRISM_DATABASE_URL goes through a
transaction-pooling proxy, the listener doesn't work and the frontend falls
back to its tick
(pool acquisition).

Metrics

  • Added: qbit_prism_post_offer_revision_races_total
    (phase=landing,settlement, outcome=retried,spent, #798). No rule fires
    on it.
  • Changed meaning: for a share append without a block,
    qbit_prism_database_advisory_lock_wait_seconds{lock="order"} and
    qbit_prism_database_order_lock_hold_seconds{holder="append"} record the
    server's own measures from qbit_prism_append_share, without the lock
    statement's round trip (#802). qbit_prism_rejections_total counts
    commit-gate contention only when it outlasts the acknowledgement deadline
    (#788).
  • Renamed or removed: none
    (metrics).

Alert rules

  • #793 (#757) shortens five UIDs in docs/prism-native-alert-rules.json.
    Titles, labels and expressions are unchanged:

    Rule Former UID UID
    PrismPoolResubmissionRejectRatioHigh qbit-prism-pool-resubmission-reject-ratio qbit-prism-pool-resubmit-reject-ratio
    PrismBlockCandidateLandingFailedCritical qbit-prism-candidate-landing-failed-critical qbit-prism-candidate-landing-failed-crit
    PrismBlockOfferStandbyUnconfirmed qbit-prism-block-offer-standby-unconfirmed qbit-prism-offer-standby-unconfirmed
    PrismDatabaseAdvisoryLockFailuresCritical qbit-prism-database-advisory-lock-failures-critical qbit-prism-advisory-lock-failures-crit
    PrismAcceptedRevisionWorkPendingCritical qbit-prism-revision-work-pending-critical qbit-prism-revision-work-pending-crit

    Grafana refused the former UIDs, so it never stored a rule under one, and
    there is nothing to delete for them. A deployment that renamed them at
    render time to exactly these UIDs, as qbit-tools' five aliases do, keeps
    its rules and alert state and can drop the renames. One that used other
    names must delete those UIDs when it adopts these: Grafana treats a new
    UID as a new rule, so state history and any silence matching
    __alert_rule_uid__ start over. Routing and label-based silences are
    unaffected.

  • #800 (#758) gates PrismAcceptedRevisionWorkPending and
    PrismAcceptedRevisionWorkTrackingUnknown on
    max_over_time(qbit_prism_authorized_clients[15m]) > 0. A deployment that
    gates PrismAcceptedRevisionWorkPending itself must turn its own gate off
    when it adopts these rules. For qbit-tools, that is the idle-backup gate
    switch and its render guard, which stops the next re-vendor until they are
    removed. This drops its carry-owner backstop, and an absent miner count
    goes quiet instead of warning.

Both are in
the alert migration.
The rules read only metrics 3.1.0 exports too, so they need no change if the
binaries roll back.

API behaviour

/public/v1/artifacts/<sha> answers a no-store 404 for an audit whose
legacy body or share segment isn't on the host yet, where it answered 500
(#796, #735). The OpenAPI document declares the 404's Cache-Control: no-store and its messages. The API verifies legacy bodies through the
parser #809 fixed, so on a host that holds the 2.x bodies it also stops
answering 500 for the 2.x audit-bundle.v2 bodies of #731.

Upgrading

  • From 3.1.0: migrate first, then replace the binaries.

    1. Migrate while 3.1.0 serves. Run 3.1.1's qbit-prism-server migrate
      once, as a one-off command with a frontend's database settings. It
      applies 032 alone, in one short transaction, and alters no table. The
      3.1.0 frontends keep serving, and restart, throughout: 032 declares no
      capability, so a 3.1.0 binary accepts the database and logs database has schema migrations this server does not know; a later release applied them, naming 32. CI's required prism-dual-writer lane runs
      the same check against 3.0.0-rc.6: its R1 scenario runs this release's
      migrate beside serving rc.6 frontends and restarts rc.6 at 32.
    2. Budget the listener's connection before the first 3.1.1 frontend
      starts (Database connections).
    3. Replace the binary on every frontend and tool, and on the
      public-api service. A 3.1.1 binary refuses the database until 32 is
      recorded. 3.1.1 adds no frontend setting, so the environments stay as
      they are.
    4. Re-provision the alert rules (Alert rules).

    A rolling upgrade is supported as a single writer. On one database,
    3.1.0 and 3.1.1 frontends can serve together, so they can be replaced one
    at a time. A 3.1.0 frontend never calls 032's functions: it appends with
    its own statements under the same ORDER_LOCK and stores each stamp in
    the cluster row, which 3.1.1 reads as its clock's floor, so both keep
    share_seq order with rising stamps (#801, #802). The one exception is a
    backward step of the database host's clock while both releases append,
    such as an NTP step or a promotion to a host whose clock lags: a 3.1.0
    frontend can then stamp a share below one 3.1.1 stamped just before. So
    replace every frontend of a database before a promotion to a host whose
    clock may lag
    (migration 032).
    Until the last 3.1.0 frontend is gone:

    • the 3.1.0 frontends still write the cluster row on every append (#738);
    • a revision bump by a 3.1.0 frontend sends no notification, and a 3.1.0
      frontend doesn't listen, so either waits for its tick as before (#795);
    • a 3.1.0 frontend's lost post-offer revision race still raises the ALERT
      and goes to reconciliation (#798);
    • only 3.1.1's public-api answers the 404s above, only 3.1.1's
      import-audits verifies the legacy bodies #806 and #809 fixed without
      sidecars, and only 3.1.1's self-check runs its local checks during an
      import (#808).

    In dual-writer mode, migrate and replace each node's database and
    frontends as above. 032 adds no table or column, so it changes nothing the
    peer sync copies or compares, and each node's listener hears only its own
    database's bumps. Nothing refuses a 3.1.0 node beside a 3.1.1 peer, but
    no test runs that pair. Each frontend runs its own peer sync, and a 3.1.0
    frontend's still applies a pulled prepared job to its own node's database
    without the blob-GC fence, so #785 can still leave that job without its
    blobs there until the node's last 3.1.0 frontend is gone (#799). Replace a
    dual-writer node's frontends together. 3.1.1 doesn't repair a job that
    lost its blobs earlier; it ages out 24 hours after it expires.

    Don't overlap the migrate with heavy disk jobs on the database host, as
    for 3.1.0.

  • From 3.0.0 or 3.0.0-rc.6: as in the
    3.1.0 notes, with 3.1.1 in 3.1.0's
    place. Its pre-migrate applies 027, 028 and 032 in one short transaction,
    then builds 031's index online, while 3.0 serves. #791 doesn't change the
    steps: it makes the pre-migrate deadlock-free. 3.1.0's pre-migrate could
    deadlock with a serving rc.6 frontend, and PostgreSQL then aborted one
    side after its 1 s deadlock_timeout (#789). 3.1.1's waits at most
    100 ms for each attempt's table locks and tries again. What you may see:

    • the share ledger and qbit_prism_share_hashes are held from the start
      of 027 to the commit, not only from their own ALTER; in two R1 runs
      027 and 028 committed 372 and 558 ms after migrate started, and no
      share waited over 39 ms for its answer;
    • while another transaction holds one of the tables for long, each retry
      queues the frontends on the primary behind its request for up to
      100 ms, so they see repeated short queues rather than one long one or a
      deadlock;
    • while an autovacuum pass holds one of the tables or a share ledger
      partition, PostgreSQL cancels it from the next attempt it blocks
      (canceling autovacuum task in the server log), because from the
      second attempt on the migrate's lock requests lower its own
      deadlock_timeout to 50 ms; it never cancels an anti-wraparound
      vacuum. With log_lock_waits on, each request that waits past 50 ms
      logs the wait. Lowering it takes a superuser, or from PostgreSQL 15
      GRANT SET ON PARAMETER deadlock_timeout TO <role>; check the role
      beforehand with SELECT has_parameter_privilege('deadlock_timeout', 'SET') or rolsuper. A migrate role without either waits the pass out
      attempt by attempt, warns an autovacuum worker holds a table the migration must lock. This role may not lower its deadlock_timeout once
      it sees one (Whether this role may lower its deadlock_timeout could not be checked where migrate could not ask, which it then asks again at
      the next contention), and if it is refused, the refusal names the
      worker: grant the parameter, or an administrator runs SELECT pg_cancel_backend(<pid>) (a superuser since
      CVE-2023-5870's fix in 16.1, 15.5 and 14.10, and from PostgreSQL 18 a
      member of pg_signal_autovacuum_worker), or keeps autovacuum off its
      table (the partition, for the share ledger) with ALTER TABLE <table> SET (autovacuum_enabled = off) until the migrate is done, then ALTER TABLE <table> RESET (autovacuum_enabled); let an anti-wraparound
      vacuum finish;
    • the 100 ms bound is for the primary. Each attempt that took some of its
      locks before giving up is replayed on every hot standby, so a reader
      there, the public-read replica prism-public-api reads included, can
      stall for up to that standby's max_standby_streaming_delay at each
      retry, or be cancelled with canceling statement due to conflict with recovery, and a primary that waits for a standby's apply
      (synchronous_commit = remote_apply) waits for that replay too. Run
      the pre-migrate while public traffic is low, or point the public API at
      the primary for the window;
    • if the locks can't be taken for ten minutes, migrate stops with
      refusing to wait any longer for the table locks of migration 27, 28,
      changes nothing, and names the holding transactions and autovacuum
      workers: end the transactions, or stop the frontends, deal with the
      workers as above, and migrate again.

    3.0.0 and rc.6 append with the same statement as 3.1.0, so while 3.0 and
    3.1.1 frontends serve one database, the clock caveat above applies to them
    too. Budget the listener's connection and re-provision the alert rules, as
    from 3.1.0.

  • From 2.0.x, or from a 3.0.0 candidate before rc.6: as the 3.1.0 notes
    say, with 3.1.1 in 3.1.0's place. Its migrate applies 032 with the rest.

If a database URL uses sslmode=verify-ca, check it before you
replace the binaries. 3.1.1's verify-ca trusts only the certificates in
sslrootcert (or PGSSLROOTCERT). 3.1.0's also trusted the public roots,
and checked the host name (#713). This covers any URL a 3.1.1 binary or
tool connects with, such as PRISM_DATABASE_URL,
PRISM_PUBLIC_DATABASE_URL or PRISM_PEER_DATABASE_URL. No configuration
in this repository uses verify-ca. A 3.1.1 binary refuses such a URL:

  • without sslrootcert, before the TLS handshake, with
    sslmode=verify-ca requires sslrootcert;
  • when sslrootcert does not anchor the server's chain, with
    invalid peer certificate: UnknownIssuer. For example, the database's
    certificate is publicly issued and 3.1.0 accepted it through the public
    roots, while sslrootcert names another CA.

Change each such URL in one of two ways:

  • To keep 3.1.0's check, switch the URL to sslmode=verify-full.
    3.1.0 checked a verify-ca connection as verify-full does, against the
    public roots and sslrootcert, and against the host name. verify-full
    behaves the same in 3.1.0, so the change survives a rollback.
  • To keep verify-ca, set sslrootcert to the CA that actually signs
    the server's certificate, at a path the binary can read. 3.1.0's
    verify-ca also checks the name, so a URL whose certificate does not
    name the host works only on 3.1.1.

Rolling back

  • To 3.1.0: replace the binaries with 3.1.0's, one at a time if need be.
    Nothing needs undoing: 032 stays applied, its functions unused, and 3.1.0
    starts on the database with the warning above. 3.1.0 refuses a newer
    database only through a capability it doesn't know, and 032 declares
    none. 3.1.1 adds no frontend setting, so no environment changes; a 3.1.0
    self-check doesn't read PRISM_SELF_CHECK_LEGACY_IMPORT_PENDING, and
    stops at audit completeness again while an import runs. A 3.1.0 frontend
    reads only the stored clock and the database's clock, so the rolling
    upgrade's clock caveat applies to its first appends after the rollback
    too. The server fixes above go with the binaries, and the issues they
    fixed return.
  • To 3.0.0 or 3.0.0-rc.6: as in the 3.1.0 notes'
    rollback. 3.0 accepts the database at
    32, as at 31, with the same warning. Never start a 3.0.0 or 3.0.0-rc.6
    binary on a personalised database.

Verification

Each fix's tests are in its pull request. For this release:

  • CI runs on the version-bump and promotion pull requests, including the
    required prism-dual-writer lane and its R1 pre-migrate scenario, and on
    the push of the promoted main commit the tag points at;
  • the L3 production-window matrix runs l3-full on each pull request that
    changes VERSION, the version bump and the promotion, and the tag
    promotes the run whose tree it matches (#608). The first l3-full runs on
    a tree with #788, on #810 and on #811's first head, passed all 33 planned
    runs, including the three repeats of
    throughput-400k-window-2fe-async-3-blocks that failed on 3.0.0's tree
    (#779);
  • cargo deny --locked check passes, and CI also checks that every
    sqlx-core RustSec advisory is reviewed (#794).

Known issues

As in the 3.1.0 notes, with these
changes.

Fixed in 3.1.1:

3.1.1 also fixes #733 (#806), #735 (#796), #785 (#799) and #789 (#791,
#821).

Still open:

  • #779, the L3 matrix on 3.0.0's tree: the
    throughput-400k-window-2fe-async-3-blocks preset failed its gate in four
    of six repeats, each with one valid share refused, #716's commit-gate
    class. #788 removes that refusal: the first l3-full runs on a tree with
    it, on #810 and on #811's first head, passed all three repeats. #779 stays
    open for 3.0.x, which doesn't have #788, and for a steady-state
    shortfall it records in one repeat of throughput-400k-window-2fe-sync,
    not yet traced to the code or the runner; that preset passed in both runs.
  • #739: backfill-ctv refuses every legacy CTV set. The cutover skips
    it.
  • #737's open part: fatal-state clear asks the node one call at a time
    (#690), and the /audit/carry-forward-integrity and
    /audit/ledger-integrity routes still run under
    PRISM_DATABASE_STATEMENT_TIMEOUT_MS, which the report can outlast at
    production size: don't poll them.
  • #712: the source export is still on the cutover's critical path.
  • #741: CI runs the admission gate's real-schema tests as a superuser.
    Tooling only.
  • The dual writer's known limits, as in the
    3.1.0-rc.1 notes.
  • The issues still open from the 3.0.0 candidates, as their notes describe
    them: #695, #602, #622, #670, #688 to #690 and #704.