Skip to content

Releases: sudolulo/truenas-truecloud-patch

v0.8.0-rc3

v0.8.0-rc3 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 21 Sep 17:11

Changed

  • CI's python matrix is green on the self-hosted Gitea runner again. The real
    failure was that the Gitea runner image executes jobs as root, and the two
    unreadable-sidecar tests build their scenario with chmod(0) — which cannot make
    a file unreadable for root (CAP_DAC_OVERRIDE). Those two tests now skip as root
    with that reason; GitHub's non-root runner still exercises them. The matrix also
    moved to uv-managed interpreters (one toolchain across both runners) and ruff is
    pinned to 0.16.1 so an upstream ruff release can't turn main red without a code
    change.

  • README badges point at the public GitHub mirror (workflow status and
    releases) instead of the private forge. The release badge had also been reading
    the stale Gitea v0.6.1 release instead of the current v0.7.0 on GitHub.

  • master is now labelled 27-dev, because it is not the next release. iX
    branches each major onto its own release/ line and master rolls straight on to the
    one after — on 2026-07-14 every recent commit on master targeted 27.0.0-BETA.1
    while 26 was still in beta. So a BROKEN master row, rendered as
    "master (unreleased)", read as "the version you are about to install is broken"
    when the breakage was a major release away on a line nobody can download. In a table
    whose entire job is helping somebody decide whether to trust this with their backups,
    that is a false alarm in the worst possible place. The label is derived from the
    newest major in the matrix plus one, so it rolls over to 28-dev by itself once 27
    branches.

    For the record, the breakage is NAS-141498 (2026-06-24), "Convert cloud_backup
    plugin to the typesafe pattern": it re-signatures restic_backup and
    get_restic_config, splitting entry/credentials out of the cloud_backup dict.
    It is deliberately not being chased while the 27 line is still churning.

Fixed

  • The patch survived being applied and then silently stopped existing, because
    something else remounted /usr four seconds later.
    On a box running
    TrueNAS 25.10.6 the boot of 2026-08-19 went: 16:41:56 apply.sh mounts its
    overlay on /usr/lib/python3/dist-packages, patches b2.py/restic.py, logs
    every step OK; 16:42:00 a second PREINIT hook runs systemd-sysext refresh over /usr — Unmerged '/usr'. / Merged extensions into '/usr'. —
    and our overlay, which lives inside that hierarchy, is torn off with it;
    16:47:24 our own deferred restart fires exactly as designed and middlewared
    imports the stock modules. Every B2 TrueCloud Backup task then failed with
    NotImplementedError from stock rclone/base.py for nineteen hours, across
    four scheduled runs, while apply.log and hook_status.json both said the
    patch was active.

    Nothing in the patch was wrong, which is the point: applying at PREINIT and
    restarting later is only sound if the patched files are still on the live path
    when middlewared re-imports them, and that is not something PREINIT can
    guarantee
    . Init scripts run sequentially in id order, so any hook registered
    after ours always wins. Worse, hook ordering cannot fix it either —
    middlewared's own docker.configure_nvidia merges a sysext over /usr at
    runtime, long after every PREINIT hook is finished.

    So the deferred restart no longer trusts the PREINIT pass. wait_restart.sh
    now re-applies immediately before it restarts middlewared — after boot has
    settled, which is also after every sysext merge and docker nvidia
    configuration — and verifies the marker is genuinely on the live path before
    restarting. It is no longer exec systemctl try-restart middlewared, because
    something has to run afterwards.

    What runs afterwards deliberately does not restart again. try-restart
    returns as soon as middlewared is READY, and middlewared then brings docker up
    — docker.configure_nvidia merges the stock nvidia sysext over /usr at that
    point, detaching the overlay after the patched modules have already been
    imported. A disk check there reports "missing" on a perfectly healthy system,
    and restarting on that signal would restart a correctly-patched middlewared
    straight back into the same race. So the overlay is re-mounted for the benefit
    of the next restart, and the question of whether this middlewared actually
    holds the patch is left to the one thing that can answer it exactly — the
    in-process alert below. That re-mount preserves hook_status.json's
    patched_at: create_task.py verify decides "loaded" by comparing
    middlewared's start time against that stamp, so a re-apply running after the
    restart would have made the stamp newer than the process which correctly
    imported the patch, and verify would have reported FAIL forever on every
    boot where the sysext merge detaches the overlay. Caught on hardware while
    validating the candidate — a new lying status introduced by the fix for a
    lying status.

    Two supporting fixes fell out of the same failure. _ensure_writable treated
    "one of our overlays is listed on this directory" as "already done" — but it
    only ever reaches that check when the directory is not writable, and a live
    overlay of ours always is. A shadowed overlay was therefore indistinguishable
    from a healthy one; it is now detached and re-mounted, reusing the same
    upperdir so everything patched earlier in the boot reappears intact, with a
    fresh workdir because overlayfs refuses one left behind by a detached mount.

  • middlewared now says so when it is running stock. The gap that let this
    cost nineteen hours was not the remount, it was that nothing could tell the
    difference between "patched on disk" and "patched in the running process".
    apply.log can only ever report the first. A new CRITICAL alert asks the
    second question from inside middlewared, hourly, where it is exact: the patch
    stamps the objects it replaces, so a missing stamp means this interpreter
    imported stock code. It checks both halves — restic.py's _truecloud_patched
    marker and whether B2RcloneRemote.get_restic_config is still the base class's
    — since either can go missing alone. It stays quiet when the kill switch is
    set or the providers module has been retired as native, and it is deliberately
    not silenced by update_alerts_disabled: that mutes release notifications,
    not a broken backup path.

    Boot-time diagnosis also no longer depends on the journal. wait_restart.sh
    logged only to the journal, and journald retention on a busy box is easily
    shorter than the interval between reboots — the 2026-08-19 boot had already
    rotated away by the time it was investigated. It now writes to apply.log
    alongside everything else.

  • The next maintenance release was never checked, and it is the one that reaches
    users.
    Shipped versions were discovered from TS-* tags and unreleased ones from
    release/* branches carrying -BETA/-RC. A branched-but-untagged maintenance
    release is neither: release/25.10.5 has no tag, and its line has already shipped,
    so the "a prerelease of a shipped line is history" filter discarded it. It was
    invisible — and it is precisely what a 25.10.4 box gets on its next update. A break
    there would have reached real users before the daily check ever looked at it, on the
    only line anybody is actually running.

    A plain release/X.Y.Z branch is now checked when its line has shipped and it
    sorts newer than that line's newest tag. Both things that must stay out fall out
    of the same rule: release/24.10-RC.2 sorts older than TS-24.10.2.4 (history, not
    a warning), and iX's typo branch release/25.20.2.2 is on a line that has no tag at
    all, so it is not a release line. This immediately surfaced two refs that had never
    been checked — release/25.10.5 and release/24.10.2.5 — both of which pass.

    is_unreleased() now keys off where a ref came from (branch = not yet shipped)
    rather than looking for -BETA/-RC in its name. Otherwise release/25.10.5 would
    count as shipped and a break in it would fail the build as a live outage — on a
    version nobody is running yet.

  • An unchanged fingerprint froze the bug report's body, not just its comments. Two
    questions were sharing one answer. Have the findings changed? gates comments —
    they notify, and a daily "still broken, same as yesterday" is what teaches everyone
    to ignore the one that finally matters. Is the body still true? gates the body —
    and editing an issue body notifies nobody on either forge, so keeping it honest is
    free. Conflated, the report could never be corrected while the findings held steady,
    and the fingerprint deliberately ignores everything that moves on its own — healthy
    rows, the hardware-verified column, point releases, and how a row is labelled. The
    master → 27-dev relabel above would have reached the README and never the issue
    anybody actually opens. The body is now rewritten whenever it is out of date (after
    normalising line endings, so a forge round-tripping \r\n does not cause a rewrite
    every run) and comments remain strictly a changelog of real changes.

  • A change to the publisher did not re-run the check. compat.yml's push: paths
    listed tools/compat.py but not tools/compat_publish.py — so the very commit that
    taught the bot to refresh a stale report body triggered no run, and the report stayed
    stale until the next scheduled one. A fix nobody runs is a fix nobody has.

  • The compatibility bot filed a new duplicate bug report on every Gitea run.
    find_issue() skipped pull requests by testing for the presence of the
    pull_request key. GitHub omits that key on a plain issue; Gitea sends it as
    null. So on Gitea every issue was discarded as a PR, the lookup always came back
    empty, and the bot took the "nothing filed yet" branch and opened a fresh report
    ...

Read more

v0.8.0-rc2

v0.8.0-rc2 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 21 Sep 17:11

Changed

  • CI's python matrix is green on the self-hosted Gitea runner again. The real
    failure was that the Gitea runner image executes jobs as root, and the two
    unreadable-sidecar tests build their scenario with chmod(0) — which cannot make
    a file unreadable for root (CAP_DAC_OVERRIDE). Those two tests now skip as root
    with that reason; GitHub's non-root runner still exercises them. The matrix also
    moved to uv-managed interpreters (one toolchain across both runners) and ruff is
    pinned to 0.16.1 so an upstream ruff release can't turn main red without a code
    change.

  • README badges point at the public GitHub mirror (workflow status and
    releases) instead of the private forge. The release badge had also been reading
    the stale Gitea v0.6.1 release instead of the current v0.7.0 on GitHub.

  • master is now labelled 27-dev, because it is not the next release. iX
    branches each major onto its own release/ line and master rolls straight on to the
    one after — on 2026-07-14 every recent commit on master targeted 27.0.0-BETA.1
    while 26 was still in beta. So a BROKEN master row, rendered as
    "master (unreleased)", read as "the version you are about to install is broken"
    when the breakage was a major release away on a line nobody can download. In a table
    whose entire job is helping somebody decide whether to trust this with their backups,
    that is a false alarm in the worst possible place. The label is derived from the
    newest major in the matrix plus one, so it rolls over to 28-dev by itself once 27
    branches.

    For the record, the breakage is NAS-141498 (2026-06-24), "Convert cloud_backup
    plugin to the typesafe pattern": it re-signatures restic_backup and
    get_restic_config, splitting entry/credentials out of the cloud_backup dict.
    It is deliberately not being chased while the 27 line is still churning.

Fixed

  • The patch survived being applied and then silently stopped existing, because
    something else remounted /usr four seconds later.
    On a box running
    TrueNAS 25.10.6 the boot of 2026-08-19 went: 16:41:56 apply.sh mounts its
    overlay on /usr/lib/python3/dist-packages, patches b2.py/restic.py, logs
    every step OK; 16:42:00 a second PREINIT hook runs systemd-sysext refresh over /usr — Unmerged '/usr'. / Merged extensions into '/usr'. —
    and our overlay, which lives inside that hierarchy, is torn off with it;
    16:47:24 our own deferred restart fires exactly as designed and middlewared
    imports the stock modules. Every B2 TrueCloud Backup task then failed with
    NotImplementedError from stock rclone/base.py for nineteen hours, across
    four scheduled runs, while apply.log and hook_status.json both said the
    patch was active.

    Nothing in the patch was wrong, which is the point: applying at PREINIT and
    restarting later is only sound if the patched files are still on the live path
    when middlewared re-imports them, and that is not something PREINIT can
    guarantee
    . Init scripts run sequentially in id order, so any hook registered
    after ours always wins. Worse, hook ordering cannot fix it either —
    middlewared's own docker.configure_nvidia merges a sysext over /usr at
    runtime, long after every PREINIT hook is finished.

    So the deferred restart no longer trusts the PREINIT pass. wait_restart.sh
    now re-applies immediately before it restarts middlewared — after boot has
    settled, which is also after every sysext merge and docker nvidia
    configuration — and verifies the marker is genuinely on the live path before
    restarting. It is no longer exec systemctl try-restart middlewared, because
    something has to run afterwards.

    What runs afterwards deliberately does not restart again. try-restart
    returns as soon as middlewared is READY, and middlewared then brings docker up
    — docker.configure_nvidia merges the stock nvidia sysext over /usr at that
    point, detaching the overlay after the patched modules have already been
    imported. A disk check there reports "missing" on a perfectly healthy system,
    and restarting on that signal would restart a correctly-patched middlewared
    straight back into the same race. So the overlay is re-mounted for the benefit
    of the next restart, and the question of whether this middlewared actually
    holds the patch is left to the one thing that can answer it exactly — the
    in-process alert below.

    Two supporting fixes fell out of the same failure. _ensure_writable treated
    "one of our overlays is listed on this directory" as "already done" — but it
    only ever reaches that check when the directory is not writable, and a live
    overlay of ours always is. A shadowed overlay was therefore indistinguishable
    from a healthy one; it is now detached and re-mounted, reusing the same
    upperdir so everything patched earlier in the boot reappears intact, with a
    fresh workdir because overlayfs refuses one left behind by a detached mount.

  • middlewared now says so when it is running stock. The gap that let this
    cost nineteen hours was not the remount, it was that nothing could tell the
    difference between "patched on disk" and "patched in the running process".
    apply.log can only ever report the first. A new CRITICAL alert asks the
    second question from inside middlewared, hourly, where it is exact: the patch
    stamps the objects it replaces, so a missing stamp means this interpreter
    imported stock code. It checks both halves — restic.py's _truecloud_patched
    marker and whether B2RcloneRemote.get_restic_config is still the base class's
    — since either can go missing alone. It stays quiet when the kill switch is
    set or the providers module has been retired as native, and it is deliberately
    not silenced by update_alerts_disabled: that mutes release notifications,
    not a broken backup path.

    Boot-time diagnosis also no longer depends on the journal. wait_restart.sh
    logged only to the journal, and journald retention on a busy box is easily
    shorter than the interval between reboots — the 2026-08-19 boot had already
    rotated away by the time it was investigated. It now writes to apply.log
    alongside everything else.

  • The next maintenance release was never checked, and it is the one that reaches
    users.
    Shipped versions were discovered from TS-* tags and unreleased ones from
    release/* branches carrying -BETA/-RC. A branched-but-untagged maintenance
    release is neither: release/25.10.5 has no tag, and its line has already shipped,
    so the "a prerelease of a shipped line is history" filter discarded it. It was
    invisible — and it is precisely what a 25.10.4 box gets on its next update. A break
    there would have reached real users before the daily check ever looked at it, on the
    only line anybody is actually running.

    A plain release/X.Y.Z branch is now checked when its line has shipped and it
    sorts newer than that line's newest tag. Both things that must stay out fall out
    of the same rule: release/24.10-RC.2 sorts older than TS-24.10.2.4 (history, not
    a warning), and iX's typo branch release/25.20.2.2 is on a line that has no tag at
    all, so it is not a release line. This immediately surfaced two refs that had never
    been checked — release/25.10.5 and release/24.10.2.5 — both of which pass.

    is_unreleased() now keys off where a ref came from (branch = not yet shipped)
    rather than looking for -BETA/-RC in its name. Otherwise release/25.10.5 would
    count as shipped and a break in it would fail the build as a live outage — on a
    version nobody is running yet.

  • An unchanged fingerprint froze the bug report's body, not just its comments. Two
    questions were sharing one answer. Have the findings changed? gates comments —
    they notify, and a daily "still broken, same as yesterday" is what teaches everyone
    to ignore the one that finally matters. Is the body still true? gates the body —
    and editing an issue body notifies nobody on either forge, so keeping it honest is
    free. Conflated, the report could never be corrected while the findings held steady,
    and the fingerprint deliberately ignores everything that moves on its own — healthy
    rows, the hardware-verified column, point releases, and how a row is labelled. The
    master → 27-dev relabel above would have reached the README and never the issue
    anybody actually opens. The body is now rewritten whenever it is out of date (after
    normalising line endings, so a forge round-tripping \r\n does not cause a rewrite
    every run) and comments remain strictly a changelog of real changes.

  • A change to the publisher did not re-run the check. compat.yml's push: paths
    listed tools/compat.py but not tools/compat_publish.py — so the very commit that
    taught the bot to refresh a stale report body triggered no run, and the report stayed
    stale until the next scheduled one. A fix nobody runs is a fix nobody has.

  • The compatibility bot filed a new duplicate bug report on every Gitea run.
    find_issue() skipped pull requests by testing for the presence of the
    pull_request key. GitHub omits that key on a plain issue; Gitea sends it as
    null. So on Gitea every issue was discarded as a PR, the lookup always came back
    empty, and the bot took the "nothing filed yet" branch and opened a fresh report
    each run — nine copies on the canonical forge, four of them filed after the
    commit that was meant to stop precisely this. The mirror was fine, which is why it
    went unnoticed: GitHub's payload shape is the one the filter was written against.

    It is the same failure the anti-spam fix was written to prevent, moved from
    comments to issues, and it survived because find_issue was the only function in
    compat_publish.py with no test. It now has one, per forge, and the daily cron —
    which had not ye...

Read more

v0.8.0-rc1

v0.8.0-rc1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 21 Sep 17:11

Changed

  • CI's python matrix is green on the self-hosted Gitea runner again. The real
    failure was that the Gitea runner image executes jobs as root, and the two
    unreadable-sidecar tests build their scenario with chmod(0) — which cannot make
    a file unreadable for root (CAP_DAC_OVERRIDE). Those two tests now skip as root
    with that reason; GitHub's non-root runner still exercises them. The matrix also
    moved to uv-managed interpreters (one toolchain across both runners) and ruff is
    pinned to 0.16.1 so an upstream ruff release can't turn main red without a code
    change.

  • README badges point at the public GitHub mirror (workflow status and
    releases) instead of the private forge. The release badge had also been reading
    the stale Gitea v0.6.1 release instead of the current v0.7.0 on GitHub.

  • master is now labelled 27-dev, because it is not the next release. iX
    branches each major onto its own release/ line and master rolls straight on to the
    one after — on 2026-07-14 every recent commit on master targeted 27.0.0-BETA.1
    while 26 was still in beta. So a BROKEN master row, rendered as
    "master (unreleased)", read as "the version you are about to install is broken"
    when the breakage was a major release away on a line nobody can download. In a table
    whose entire job is helping somebody decide whether to trust this with their backups,
    that is a false alarm in the worst possible place. The label is derived from the
    newest major in the matrix plus one, so it rolls over to 28-dev by itself once 27
    branches.

    For the record, the breakage is NAS-141498 (2026-06-24), "Convert cloud_backup
    plugin to the typesafe pattern": it re-signatures restic_backup and
    get_restic_config, splitting entry/credentials out of the cloud_backup dict.
    It is deliberately not being chased while the 27 line is still churning.

Fixed

  • The patch survived being applied and then silently stopped existing, because
    something else remounted /usr four seconds later.
    On a box running
    TrueNAS 25.10.6 the boot of 2026-08-19 went: 16:41:56 apply.sh mounts its
    overlay on /usr/lib/python3/dist-packages, patches b2.py/restic.py, logs
    every step OK; 16:42:00 a second PREINIT hook runs systemd-sysext refresh over /usr — Unmerged '/usr'. / Merged extensions into '/usr'. —
    and our overlay, which lives inside that hierarchy, is torn off with it;
    16:47:24 our own deferred restart fires exactly as designed and middlewared
    imports the stock modules. Every B2 TrueCloud Backup task then failed with
    NotImplementedError from stock rclone/base.py for nineteen hours, across
    four scheduled runs, while apply.log and hook_status.json both said the
    patch was active.

    Nothing in the patch was wrong, which is the point: applying at PREINIT and
    restarting later is only sound if the patched files are still on the live path
    when middlewared re-imports them, and that is not something PREINIT can
    guarantee
    . Init scripts run sequentially in id order, so any hook registered
    after ours always wins. Worse, hook ordering cannot fix it either —
    middlewared's own docker.configure_nvidia merges a sysext over /usr at
    runtime, long after every PREINIT hook is finished.

    So the deferred restart no longer trusts the PREINIT pass. wait_restart.sh
    now re-applies immediately before it restarts middlewared — after boot has
    settled, which is also after every sysext merge and docker nvidia
    configuration — verifies the marker is genuinely on the live path, restarts,
    and verifies again, retrying once if the patch was torn off in between. It is
    no longer exec systemctl try-restart middlewared, because something has to
    run afterwards to find out what that restart actually loaded.

    Two supporting fixes fell out of the same failure. _ensure_writable treated
    "one of our overlays is listed on this directory" as "already done" — but it
    only ever reaches that check when the directory is not writable, and a live
    overlay of ours always is. A shadowed overlay was therefore indistinguishable
    from a healthy one; it is now detached and re-mounted, reusing the same
    upperdir so everything patched earlier in the boot reappears intact, with a
    fresh workdir because overlayfs refuses one left behind by a detached mount.

  • middlewared now says so when it is running stock. The gap that let this
    cost nineteen hours was not the remount, it was that nothing could tell the
    difference between "patched on disk" and "patched in the running process".
    apply.log can only ever report the first. A new CRITICAL alert asks the
    second question from inside middlewared, hourly, where it is exact: the patch
    stamps the objects it replaces, so a missing stamp means this interpreter
    imported stock code. It checks both halves — restic.py's _truecloud_patched
    marker and whether B2RcloneRemote.get_restic_config is still the base class's
    — since either can go missing alone. It stays quiet when the kill switch is
    set or the providers module has been retired as native, and it is deliberately
    not silenced by update_alerts_disabled: that mutes release notifications,
    not a broken backup path.

    Boot-time diagnosis also no longer depends on the journal. wait_restart.sh
    logged only to the journal, and journald retention on a busy box is easily
    shorter than the interval between reboots — the 2026-08-19 boot had already
    rotated away by the time it was investigated. It now writes to apply.log
    alongside everything else.

  • The next maintenance release was never checked, and it is the one that reaches
    users.
    Shipped versions were discovered from TS-* tags and unreleased ones from
    release/* branches carrying -BETA/-RC. A branched-but-untagged maintenance
    release is neither: release/25.10.5 has no tag, and its line has already shipped,
    so the "a prerelease of a shipped line is history" filter discarded it. It was
    invisible — and it is precisely what a 25.10.4 box gets on its next update. A break
    there would have reached real users before the daily check ever looked at it, on the
    only line anybody is actually running.

    A plain release/X.Y.Z branch is now checked when its line has shipped and it
    sorts newer than that line's newest tag. Both things that must stay out fall out
    of the same rule: release/24.10-RC.2 sorts older than TS-24.10.2.4 (history, not
    a warning), and iX's typo branch release/25.20.2.2 is on a line that has no tag at
    all, so it is not a release line. This immediately surfaced two refs that had never
    been checked — release/25.10.5 and release/24.10.2.5 — both of which pass.

    is_unreleased() now keys off where a ref came from (branch = not yet shipped)
    rather than looking for -BETA/-RC in its name. Otherwise release/25.10.5 would
    count as shipped and a break in it would fail the build as a live outage — on a
    version nobody is running yet.

  • An unchanged fingerprint froze the bug report's body, not just its comments. Two
    questions were sharing one answer. Have the findings changed? gates comments —
    they notify, and a daily "still broken, same as yesterday" is what teaches everyone
    to ignore the one that finally matters. Is the body still true? gates the body —
    and editing an issue body notifies nobody on either forge, so keeping it honest is
    free. Conflated, the report could never be corrected while the findings held steady,
    and the fingerprint deliberately ignores everything that moves on its own — healthy
    rows, the hardware-verified column, point releases, and how a row is labelled. The
    master → 27-dev relabel above would have reached the README and never the issue
    anybody actually opens. The body is now rewritten whenever it is out of date (after
    normalising line endings, so a forge round-tripping \r\n does not cause a rewrite
    every run) and comments remain strictly a changelog of real changes.

  • A change to the publisher did not re-run the check. compat.yml's push: paths
    listed tools/compat.py but not tools/compat_publish.py — so the very commit that
    taught the bot to refresh a stale report body triggered no run, and the report stayed
    stale until the next scheduled one. A fix nobody runs is a fix nobody has.

  • The compatibility bot filed a new duplicate bug report on every Gitea run.
    find_issue() skipped pull requests by testing for the presence of the
    pull_request key. GitHub omits that key on a plain issue; Gitea sends it as
    null. So on Gitea every issue was discarded as a PR, the lookup always came back
    empty, and the bot took the "nothing filed yet" branch and opened a fresh report
    each run — nine copies on the canonical forge, four of them filed after the
    commit that was meant to stop precisely this. The mirror was fine, which is why it
    went unnoticed: GitHub's payload shape is the one the filter was written against.

    It is the same failure the anti-spam fix was written to prevent, moved from
    comments to issues, and it survived because find_issue was the only function in
    compat_publish.py with no test. It now has one, per forge, and the daily cron —
    which had not yet run once — no longer accumulates a report a day.

    The issue list is also requested with both paging parameters (per_page for
    GitHub, limit for Gitea). Each forge ignores the other's, and Gitea's default page
    is 30, so the lookup would have started missing the report again once the pile it
    was creating grew past one page.

v0.7.0

Choose a tag to compare

@github-actions github-actions released this 14 Jul 02:04

Added

  • TrueNAS 26 support, verified on a real TrueNAS 26 install. 26 deletes
    plugins/zfs_/ outright, taking the private zfs.dataset.query,
    zfs.snapshot.query and zfs.snapshot.delete with it. Every one of those was on
    the nested module's critical path, so nested snapshots were BROKEN on 26 and
    apply.sh correctly refused to apply the module there.

    Snapshot deletion now resolves its namespace at runtime — pool.snapshot on
    25.10 and 26, zfs.snapshot on 24.10 and 25.04, because no single namespace spans
    every supported release. tools/compat.py checks the same list the runtime uses,
    so what CI verifies and what runs cannot drift apart.

    Hardware-verified on TrueNAS 26.0.0-BETA.1: a 274-snapshot recursive backup of a
    292-dataset pool, then a byte-identical restore of a four-level-deep child
    dataset
    .

Fixed

  • Enumeration no longer trusts middleware's dataset and snapshot queries — they
    are filtered.
    This is the important one, and it is the bug that a test VM caught
    and no amount of source analysis ever could have.

    The obvious port of the deleted private zfs.dataset.query was the public
    pool.dataset.query. It exists, it is documented, it is covered by iX's
    deprecation policy — and it is not a like-for-like replacement. It applies a
    visibility policy: it hides the datasets TrueNAS considers its own — ix-apps/*,
    .system/*, .ix-virt/*. On a real pool that is 84 of 270 datasets, and
    ix-apps holds live application data.

    Staging from that view would have silently omitted every one of them. Worse,
    plan_staging() would never have seen them, so they would not have appeared in its
    skipped list either — no warning, no failure, just a green backup quietly missing
    data. That is precisely the failure this module exists to prevent. The snapshot
    query lies the same way (205 of 274), so the sweep would have orphaned one snapshot
    per hidden dataset, on every run, forever.

    The module now reads the truth from ZFS and makes changes through middleware:
    enumeration is zfs list, which no policy can filter and which behaves identically
    on every release; mutation stays a middleware call, so TrueNAS's own bookkeeping
    stays consistent. A failing zfs list raises rather than returning an empty list —
    "no datasets" and "the command broke" must never look the same.

    No shipped release is affected. v0.6.1 and earlier call the private
    zfs.dataset.query, which returns all 270 datasets. The bug existed only in the
    unreleased TrueNAS 26 port.

  • The patch now owns the snapshot sweep even when it does not stage anything.
    Stock decides whether to take a recursive snapshot by its own rule, and on
    TrueNAS 26 that rule stopped being ours.

    Up to 25.10, stock's create_snapshot called get_dataset_recursive() — the same
    function this module vendors — so "stock went recursive" and "we have something to
    stage" were the same question, and stock's non-recursive delete was correct for
    everything the patch declined to stage. TrueNAS 26 uses filesystem.statfs:
    recursive = (path == the dataset's mountpoint). The two rules now disagree for a
    dataset whose only descendants are ZVOLs or legacy/none-mountpoint datasets
    — stock snapshots it recursively, while the patch sees nothing to stage.

    The patch then handed the snapshot back to stock, which destroys the parent only.
    With no staging tree there was no sidecar, and the garbage collector only ever ran
    from the staging path — so nothing on the box would ever have found the children.
    Reproduced on the test VM: one orphaned snapshot per zvol, on every run, forever,
    with the backup reporting success. Ownership of the sweep is no longer conditional
    on staging.

  • The runtime resolved a namespace; the checker verified a method. Those are
    different questions, and the gap is a false "ok". get_service() only proves a
    namespace is registered — it says nothing about whether delete still exists on it.
    So if iX guts the method while keeping the service (they have already done exactly
    that to pool.snapshot.do_update on master), tools/compat.py would fall through
    to zfs.snapshot, report the box healthy, and let the patch apply — while the
    runtime picked pool.snapshot and failed every delete, orphaning the whole tree.
    Both sides now ask the same question, and a test binds the two lists together.

  • query_filesystems() dropped malformed zfs list rows silently — the last
    remaining silent-omission path, and a direct contradiction of this module's cardinal
    rule. It raises now. A missing zfs binary raised FileNotFoundError rather than
    ZfsError; also fixed.

  • The snapshot retry loop discarded the delete error and reported every survivor
    as "(still busy?)" — naming the one cause that is benign and self-healing, and
    hiding the ones that are permanent. It keeps and reports the real error.

  • The staging-failure handler could lose the original exception if its own cleanup
    sweep raised. An error handler must not be able to lose the error.

  • A snapshot delete that returns cleanly is not proof that anything was deleted.
    The recursive sweep's fast path took the call's word for it and returned "no
    survivors" — so cleanup_task read that as a clean sweep and removed the sidecar,
    the only record the tree ever existed. Roughly 250 snapshots would have been orphaned
    on every run, with nothing left able to find them, and the backup reporting success.

    This is not a hypothetical about a well-behaved API: iX has already gutted
    pool.snapshot.do_update on master into a no-op whose body is commented out and
    which returns None. A source check still sees the def; a runtime check still sees
    a callable method. Only asking ZFS can tell. The sweep now confirms against ZFS, and
    where it cannot confirm it keeps owning the tree rather than claiming success — a
    false survivor self-heals on the next run, a lost record never does.

  • _write_sidecar swallowed OSError. The sidecar is the only thing that survives
    a middlewared restart; failing to write it is not fatal, but it must never be
    invisible. _read_sidecar had the mirror bug — it conflated "there is no sidecar"
    with "I could not read the sidecar", and cleanup_task then took the empty branch
    and unlinked the only record of a tree it had failed to read.

  • A dataset from another tree, mounted inside the backup path, was omitted
    silently.
    The staging plan scopes by dataset name, which is correct — a dataset
    with no mountpoint cannot be scoped by path at all. But ZFS lets any dataset mount
    anywhere, so one from an unrelated tree can sit inside the path:

    Tank/photos   mountpoint=/mnt/Tap/apps/photos
    

    It holds data inside the backed-up path, and zfs snapshot -r Tap@… does not
    cover it: recursion follows the dataset tree, not the directory tree. So there is no
    snapshot of it to stage, and no way to capture it consistently with the rest. It fell
    out of the name filter and vanished — not staged, not in skipped, no error, backup
    green. Stock has the same blind spot, but stock also refuses the nested config
    outright; this patch is what relaxes that guard, so the hole is this patch's to close.
    It now refuses, and names the offending datasets.

v0.7.0-rc1

v0.7.0-rc1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 14 Jul 01:46

Added

  • TrueNAS 26 support, verified on a real TrueNAS 26 install. 26 deletes
    plugins/zfs_/ outright, taking the private zfs.dataset.query,
    zfs.snapshot.query and zfs.snapshot.delete with it. Every one of those was on
    the nested module's critical path, so nested snapshots were BROKEN on 26 and
    apply.sh correctly refused to apply the module there.

    Snapshot deletion now resolves its namespace at runtime — pool.snapshot on
    25.10 and 26, zfs.snapshot on 24.10 and 25.04, because no single namespace spans
    every supported release. tools/compat.py checks the same list the runtime uses,
    so what CI verifies and what runs cannot drift apart.

    Hardware-verified on TrueNAS 26.0.0-BETA.1: a 274-snapshot recursive backup of a
    292-dataset pool, then a byte-identical restore of a four-level-deep child
    dataset
    .

Fixed

  • Enumeration no longer trusts middleware's dataset and snapshot queries — they
    are filtered.
    This is the important one, and it is the bug that a test VM caught
    and no amount of source analysis ever could have.

    The obvious port of the deleted private zfs.dataset.query was the public
    pool.dataset.query. It exists, it is documented, it is covered by iX's
    deprecation policy — and it is not a like-for-like replacement. It applies a
    visibility policy: it hides the datasets TrueNAS considers its own — ix-apps/*,
    .system/*, .ix-virt/*. On a real pool that is 84 of 270 datasets, and
    ix-apps holds live application data.

    Staging from that view would have silently omitted every one of them. Worse,
    plan_staging() would never have seen them, so they would not have appeared in its
    skipped list either — no warning, no failure, just a green backup quietly missing
    data. That is precisely the failure this module exists to prevent. The snapshot
    query lies the same way (205 of 274), so the sweep would have orphaned one snapshot
    per hidden dataset, on every run, forever.

    The module now reads the truth from ZFS and makes changes through middleware:
    enumeration is zfs list, which no policy can filter and which behaves identically
    on every release; mutation stays a middleware call, so TrueNAS's own bookkeeping
    stays consistent. A failing zfs list raises rather than returning an empty list —
    "no datasets" and "the command broke" must never look the same.

    No shipped release is affected. v0.6.1 and earlier call the private
    zfs.dataset.query, which returns all 270 datasets. The bug existed only in the
    unreleased TrueNAS 26 port.

  • The patch now owns the snapshot sweep even when it does not stage anything.
    Stock decides whether to take a recursive snapshot by its own rule, and on
    TrueNAS 26 that rule stopped being ours.

    Up to 25.10, stock's create_snapshot called get_dataset_recursive() — the same
    function this module vendors — so "stock went recursive" and "we have something to
    stage" were the same question, and stock's non-recursive delete was correct for
    everything the patch declined to stage. TrueNAS 26 uses filesystem.statfs:
    recursive = (path == the dataset's mountpoint). The two rules now disagree for a
    dataset whose only descendants are ZVOLs or legacy/none-mountpoint datasets
    — stock snapshots it recursively, while the patch sees nothing to stage.

    The patch then handed the snapshot back to stock, which destroys the parent only.
    With no staging tree there was no sidecar, and the garbage collector only ever ran
    from the staging path — so nothing on the box would ever have found the children.
    Reproduced on the test VM: one orphaned snapshot per zvol, on every run, forever,
    with the backup reporting success. Ownership of the sweep is no longer conditional
    on staging.

  • The runtime resolved a namespace; the checker verified a method. Those are
    different questions, and the gap is a false "ok". get_service() only proves a
    namespace is registered — it says nothing about whether delete still exists on it.
    So if iX guts the method while keeping the service (they have already done exactly
    that to pool.snapshot.do_update on master), tools/compat.py would fall through
    to zfs.snapshot, report the box healthy, and let the patch apply — while the
    runtime picked pool.snapshot and failed every delete, orphaning the whole tree.
    Both sides now ask the same question, and a test binds the two lists together.

  • query_filesystems() dropped malformed zfs list rows silently — the last
    remaining silent-omission path, and a direct contradiction of this module's cardinal
    rule. It raises now. A missing zfs binary raised FileNotFoundError rather than
    ZfsError; also fixed.

  • The snapshot retry loop discarded the delete error and reported every survivor
    as "(still busy?)" — naming the one cause that is benign and self-healing, and
    hiding the ones that are permanent. It keeps and reports the real error.

  • The staging-failure handler could lose the original exception if its own cleanup
    sweep raised. An error handler must not be able to lose the error.

  • A snapshot delete that returns cleanly is not proof that anything was deleted.
    The recursive sweep's fast path took the call's word for it and returned "no
    survivors" — so cleanup_task read that as a clean sweep and removed the sidecar,
    the only record the tree ever existed. Roughly 250 snapshots would have been orphaned
    on every run, with nothing left able to find them, and the backup reporting success.

    This is not a hypothetical about a well-behaved API: iX has already gutted
    pool.snapshot.do_update on master into a no-op whose body is commented out and
    which returns None. A source check still sees the def; a runtime check still sees
    a callable method. Only asking ZFS can tell. The sweep now confirms against ZFS, and
    where it cannot confirm it keeps owning the tree rather than claiming success — a
    false survivor self-heals on the next run, a lost record never does.

  • _write_sidecar swallowed OSError. The sidecar is the only thing that survives
    a middlewared restart; failing to write it is not fatal, but it must never be
    invisible. _read_sidecar had the mirror bug — it conflated "there is no sidecar"
    with "I could not read the sidecar", and cleanup_task then took the empty branch
    and unlinked the only record of a tree it had failed to read.

  • A dataset from another tree, mounted inside the backup path, was omitted
    silently.
    The staging plan scopes by dataset name, which is correct — a dataset
    with no mountpoint cannot be scoped by path at all. But ZFS lets any dataset mount
    anywhere, so one from an unrelated tree can sit inside the path:

    Tank/photos   mountpoint=/mnt/Tap/apps/photos
    

    It holds data inside the backed-up path, and zfs snapshot -r Tap@… does not
    cover it: recursion follows the dataset tree, not the directory tree. So there is no
    snapshot of it to stage, and no way to capture it consistently with the rest. It fell
    out of the name filter and vanished — not staged, not in skipped, no error, backup
    green. Stock has the same blind spot, but stock also refuses the nested config
    outright; this patch is what relaxes that guard, so the hole is this patch's to close.
    It now refuses, and names the offending datasets.

v0.6.1

Choose a tag to compare

@github-actions github-actions released this 13 Jul 20:42

Fixed

  • A reboot mid-backup orphaned the entire snapshot tree, permanently. The sidecar
    is the record of which snapshots a run pinned — and it lives in /run, which is
    tmpfs. A reboot (or a crash) between taking the recursive snapshot and cleaning
    it up destroyed that record, leaving one snapshot per descendant dataset — 250+ on
    a real pool
    — with nothing left pointing at them. Nothing would ever have found
    them again.

    gc_stale_snapshots() is the backstop: it identifies leftovers by name, so it
    works when the record is gone. It runs at the start of every backup, after the
    sidecar reclaim — the recorded path stays authoritative, and the collector only ever
    mops up what the record lost.

    Because it deletes data on a name match — a weaker claim than a recorded fact — the
    selection is a pure function with the harshest tests in the suite. A snapshot is
    collected only if all of these hold:

    name is exactly <dataset>@<task>-<YYYYMMDDHHMMSS> so cloud_backup-5 never matches cloud_backup-50, an auto-* periodic snapshot, or anything a human made
    it is not the current run's parent and children are excluded
    nothing is mounted from it an in-flight run pins its own snapshots — this, not the age guard, is what protects a concurrent backup
    it is over an hour old covers the seconds-long window where a live run has snapshotted but not yet mounted

    Verified against the real pool: of 4,728 snapshots — including 2,341 periodic
    ones — it selects exactly the orphans of the task being run, and nothing else.

v0.6.1-rc2

v0.6.1-rc2 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 13 Jul 20:41

Fixed

  • A reboot mid-backup orphaned the entire snapshot tree, permanently. The sidecar
    is the record of which snapshots a run pinned — and it lives in /run, which is
    tmpfs. A reboot (or a crash) between taking the recursive snapshot and cleaning
    it up destroyed that record, leaving one snapshot per descendant dataset — 250+ on
    a real pool
    — with nothing left pointing at them. Nothing would ever have found
    them again.

    gc_stale_snapshots() is the backstop: it identifies leftovers by name, so it
    works when the record is gone. It runs at the start of every backup, after the
    sidecar reclaim — the recorded path stays authoritative, and the collector only ever
    mops up what the record lost.

    Because it deletes data on a name match — a weaker claim than a recorded fact — the
    selection is a pure function with the harshest tests in the suite. A snapshot is
    collected only if all of these hold:

    name is exactly <dataset>@<task>-<YYYYMMDDHHMMSS> so cloud_backup-5 never matches cloud_backup-50, an auto-* periodic snapshot, or anything a human made
    it is not the current run's parent and children are excluded
    nothing is mounted from it an in-flight run pins its own snapshots — this, not the age guard, is what protects a concurrent backup
    it is over an hour old covers the seconds-long window where a live run has snapshotted but not yet mounted

    Verified against the real pool: of 4,728 snapshots — including 2,341 periodic
    ones — it selects exactly the orphans of the task being run, and nothing else.

v0.6.1-rc1

v0.6.1-rc1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 13 Jul 20:00

Fixed

  • A reboot mid-backup orphaned the entire snapshot tree, permanently. The sidecar
    is the record of which snapshots a run pinned — and it lives in /run, which is
    tmpfs. A reboot (or a crash) between taking the recursive snapshot and cleaning
    it up destroyed that record, leaving one snapshot per descendant dataset — 250+ on
    a real pool
    — with nothing left pointing at them. Nothing would ever have found
    them again.

    gc_stale_snapshots() is the backstop: it identifies leftovers by name, so it
    works when the record is gone. It runs at the start of every backup, after the
    sidecar reclaim — the recorded path stays authoritative, and the collector only ever
    mops up what the record lost.

    Because it deletes data on a name match — a weaker claim than a recorded fact — the
    selection is a pure function with the harshest tests in the suite. A snapshot is
    collected only if all of these hold:

    name is exactly <dataset>@<task>-<YYYYMMDDHHMMSS> so cloud_backup-5 never matches cloud_backup-50, an auto-* periodic snapshot, or anything a human made
    it is not the current run's parent and children are excluded
    nothing is mounted from it an in-flight run pins its own snapshots — this, not the age guard, is what protects a concurrent backup
    it is over an hour old covers the seconds-long window where a live run has snapshotted but not yet mounted

    Verified against the real pool: of 4,728 snapshots — including 2,341 periodic
    ones — it selects exactly the orphans of the task being run, and nothing else.

v0.6.0

Choose a tag to compare

@github-actions github-actions released this 13 Jul 19:46

Added

  • release.sh — a two-stage release process, and a barrier that enforces it.
    A stable vX.Y.Z tag is now only publishable if a vX.Y.Z-rcN tag points at the
    same commit, and the release job re-runs the entire suite against that tagged
    commit before publishing. Candidates are invisible to users — update.sh and the
    update alert both take the newest plain vX.Y.Z tag — so debugging happens across
    rc1, rc2, rc3 at nobody's expense, instead of across v0.5.0, v0.5.1, v0.5.2 at
    everybody's.

    bash release.sh 0.6.0 --rc        # candidate. Invisible to users.
    bash release.sh 0.6.0 --promote   # stable. Refused unless an rc passed HERE.
    

    The rule is enforced in tools/release_gate.py, which release.sh runs locally
    (so you fail in 200 ms) and .github/workflows/release.yml runs again where it
    cannot be bypassed (so failing locally is not optional). "The candidate passed,
    then I pushed one more little fix" is refused by name — that is precisely how
    v0.5.1 happened.

  • TrueNAS compatibility is now checked, not hoped for.
    tools/compat.py is a written-down record of everything each
    module assumes about middlewared, checked in two places:

    • CI, daily — against iXsystems' source at every release line including
      master and the current BETA/RC
      . When an unreleased TrueNAS breaks the patch
      it files a bug report automatically, so there is time to fix it before that
      version reaches anyone. It also refreshes the README's support matrix, so the
      table cannot quietly become a false promise.
    • apply.sh, at every boot — against the middleware actually installed on the
      box. A module whose assumptions no longer hold is not applied. Stock TrueNAS
      without a feature beats TrueNAS with a broken backup.

    It immediately found two real breaks: TrueNAS 26 (below), and a nested-snapshot bug
    that had been shipping for two releases (below).

  • The compatibility check now covers the middlewared methods the patch calls,
    not only the symbols it wraps — and that gap was hiding a catastrophe.

    TrueNAS 26 deletes plugins/zfs_/dataset.py and plugins/zfs_/snapshot.py
    outright
    , taking zfs.dataset.query, zfs.snapshot.query and
    zfs.snapshot.delete with them (26 uses filesystem.statfs and zfs.resource.*).
    Nothing about the five cloud_backup files reveals that, so every other check went
    green. The patch would have applied perfectly and then failed on the first
    backup
    — or, far worse, snapshotted successfully and failed to delete,
    orphaning one snapshot per descendant dataset (250 on a real pool) on every
    single run, forever.

    This is now an assumption class of its own, so a method disappearing is a BROKEN
    verdict rather than a silent time bomb.

  • Groundwork for TrueNAS 26 (async→sync and the deleted helper — see below).
    26 is still reported BROKEN and the nested module will not apply there, because
    the ZFS API rewrite above is not yet ported. Porting it needs a real 26 box to
    verify against, and shipping a port nobody has run is exactly the failure this
    project exists to avoid. On 26, TrueNAS is left stock: B2/S3 keeps working, nested
    datasets are simply not covered.

Fixed

  • A few snapshots leaked on every nested run, forever. Found on real hardware, in
    the one place it could be: a 256-snapshot backup of /mnt/Tap swept 253 cleanly and
    left 3 behind with dataset is busy.

    The cause is ZFS's own automount. Reading anything under
    <dataset>/.zfs/snapshot/<snap>/ makes ZFS automount that snapshot, and it stays
    mounted for zfs_expire_snapshot seconds (300 by default) after the last access.
    teardown() unmounts our bind mounts — but not the automount underneath — so
    zfs destroy refuses for exactly the datasets restic read most recently. Then
    cleanup_task() removed the sidecar anyway, destroying the only record that those
    snapshots existed. Nothing would ever have reclaimed them.

    Three changes, and the third is the one that makes it safe rather than merely
    unlikely:

    • release_snapdirs() unmounts ZFS's own .zfs/snapshot automounts (deepest first)
      before deleting, so the snapshots are not busy in the first place.
    • delete_snapshot_tree() retries the transient busy, and returns the
      snapshots it could not delete
      instead of swallowing them.
    • The sidecar is now removed only on a confirmed-clean sweep — including on the
      staging-failure path, which used to remove it before the caller swept. The
      asymmetry is deliberate: a sidecar left behind when the tree is already gone costs
      one no-op delete on the next run, while a sidecar removed while the tree still
      exists is unrecoverable. Survivors are reclaimed by the next run.

    Expect the occasional straggler, and expect it to clean itself up. On a
    256-snapshot tree this reliably sweeps ~255 immediately and may leave one: it is
    whatever restic read last, so its 300-second window has barely opened. That one is
    logged, its sidecar is kept, and the next run reclaims it before doing anything else.
    The leak is bounded at a single cycle rather than growing without limit — which is
    the property that actually matters. Blocking a backup job for five minutes to chase
    the last snapshot would be a worse trade, so it is not made.

  • Installing the patch permanently blocked updating it. install.sh does
    chmod +x update.sh, and git recorded update.sh as 100644 — so the chmod was a
    tracked modification, and update.sh refuses to run over a dirty tree. Install
    once and you could never update again; the error even told you to run
    git checkout -- ., which just undoes the exec bit so the next install can re-dirty
    it. A real box sat on an old version for exactly this reason.

    Fixed on both sides: the scripts install.sh chmods are now executable in git (so
    the chmod is a no-op), and update.sh's dirty check now looks at content, not
    file mode — git diff --numstat reports 0 0 for a mode-only change. A test
    asserts every script in install.sh's chmod loop is already 100755 in git.

  • Nested snapshots were broken on TrueNAS 24.10 and 25.04, and had been all
    along.
    SYNC_BLOCK's wrapper spelled out the stock signature and forwarded five
    arguments — but those releases declare restic_backup(middleware, job, cloud_backup, dry_run); rate_limit only arrived in 25.10. Every nested backup on
    24.10/25.04 raised TypeError: restic_backup() takes 4 positional arguments but 5 were given. The wrapper now takes *args, **kwargs and forwards whatever it is
    handed, so a trailing parameter appearing or disappearing is a non-event.

    Found by the new compatibility check, not by a user — which is the whole argument
    for having it. The check it replaced only asked whether the parameter names still
    appeared somewhere in the signature, so it happily passed a call that could never
    work.

  • The nested module is now one synchronous implementation behind two thin
    wrappers.
    TrueNAS 26 rewrites cloud_backup from async to synchronous and
    separately deletes get_dataset_recursive(), which an injected block called out
    of the host module's namespace. Either alone is a broken backup found at restore
    time: an async def wrapper hands sync.py a coroutine where it unpacks a tuple,
    and the vanished helper is a straight NameError.

    The module now talks to middlewared through call_sync, and apply.sh reads which
    flavour the installed middleware declares and injects the matching wrapper —
    TrueNAS ≤ 25.10 reaches it via await middleware.run_in_thread(...); a synchronous
    TrueNAS, already in a worker thread, calls it directly. The logic that owns the
    snapshots, the bind mounts and the failure modes exists once; an async twin
    would mean every future fix had to land twice, and the one that got missed would be
    the one that eats a backup. A middleware whose three wrapped functions disagree
    about async-ness is refused outright rather than guessed at, and
    get_dataset_recursive is carried as our own copy — removing the dependency on both
    versions instead of asserting it.

  • The patch no longer reaches into CloudSync tasks it has no business touching.
    create_snapshot is module-global in plugins/cloud/snapshot.py and is imported by
    cloud_sync.py as well as cloud_backup/sync.py — so the wrapper sat in the
    path of every rclone/Storj CloudSync task with snapshot=true, and issued a
    zfs.dataset.query before deciding it had nothing to do. That added a brand-new
    failure mode to jobs that worked fine before this patch was installed, and worse: a
    CloudSync task that ever did get staged would never be torn down, because the
    teardown is wired into cloud_backup's restic_backup and CRUD_BLOCK
    deliberately leaves CloudSync's nesting guard intact — the bind mounts would pin the
    ZFS snapshot forever. The staging path now bails out immediately unless the snapshot
    is named cloud_backup-*, before any middleware call.

  • Teardown warnings are no longer silently swallowed on TrueNAS ≤ 25.10. The async
    wrapper's finally dropped the logger= kwarg that the sync one passes, so a
    cleanup that failed to unmount a bind mount or to delete a snapshot tree logged
    nothing at all — on the only platform anyone actually runs. run_in_thread
    forwards **kwargs via functools.partial; it was a regression, not a limitation.

  • do_delete is recognised as delete. TrueNAS 24.10 and 25.04 declare
    do_delete (the CRUDService convention); 25.10 renamed it to delete. Both
    answer to zfs.snapshot.delete. Accepting only the literal name reported both older
    releases as BROKEN — a false verdict that would have switched nested snapshots off
    on boxes where they work perf...

Read more

v0.6.0-rc6

v0.6.0-rc6 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 13 Jul 19:36

Added

  • release.sh — a two-stage release process, and a barrier that enforces it.
    A stable vX.Y.Z tag is now only publishable if a vX.Y.Z-rcN tag points at the
    same commit, and the release job re-runs the entire suite against that tagged
    commit before publishing. Candidates are invisible to users — update.sh and the
    update alert both take the newest plain vX.Y.Z tag — so debugging happens across
    rc1, rc2, rc3 at nobody's expense, instead of across v0.5.0, v0.5.1, v0.5.2 at
    everybody's.

    bash release.sh 0.6.0 --rc        # candidate. Invisible to users.
    bash release.sh 0.6.0 --promote   # stable. Refused unless an rc passed HERE.
    

    The rule is enforced in tools/release_gate.py, which release.sh runs locally
    (so you fail in 200 ms) and .github/workflows/release.yml runs again where it
    cannot be bypassed (so failing locally is not optional). "The candidate passed,
    then I pushed one more little fix" is refused by name — that is precisely how
    v0.5.1 happened.

  • TrueNAS compatibility is now checked, not hoped for.
    tools/compat.py is a written-down record of everything each
    module assumes about middlewared, checked in two places:

    • CI, daily — against iXsystems' source at every release line including
      master and the current BETA/RC
      . When an unreleased TrueNAS breaks the patch
      it files a bug report automatically, so there is time to fix it before that
      version reaches anyone. It also refreshes the README's support matrix, so the
      table cannot quietly become a false promise.
    • apply.sh, at every boot — against the middleware actually installed on the
      box. A module whose assumptions no longer hold is not applied. Stock TrueNAS
      without a feature beats TrueNAS with a broken backup.

    It immediately found two real breaks: TrueNAS 26 (below), and a nested-snapshot bug
    that had been shipping for two releases (below).

  • The compatibility check now covers the middlewared methods the patch calls,
    not only the symbols it wraps — and that gap was hiding a catastrophe.

    TrueNAS 26 deletes plugins/zfs_/dataset.py and plugins/zfs_/snapshot.py
    outright
    , taking zfs.dataset.query, zfs.snapshot.query and
    zfs.snapshot.delete with them (26 uses filesystem.statfs and zfs.resource.*).
    Nothing about the five cloud_backup files reveals that, so every other check went
    green. The patch would have applied perfectly and then failed on the first
    backup
    — or, far worse, snapshotted successfully and failed to delete,
    orphaning one snapshot per descendant dataset (250 on a real pool) on every
    single run, forever.

    This is now an assumption class of its own, so a method disappearing is a BROKEN
    verdict rather than a silent time bomb.

  • Groundwork for TrueNAS 26 (async→sync and the deleted helper — see below).
    26 is still reported BROKEN and the nested module will not apply there, because
    the ZFS API rewrite above is not yet ported. Porting it needs a real 26 box to
    verify against, and shipping a port nobody has run is exactly the failure this
    project exists to avoid. On 26, TrueNAS is left stock: B2/S3 keeps working, nested
    datasets are simply not covered.

Fixed

  • A few snapshots leaked on every nested run, forever. Found on real hardware, in
    the one place it could be: a 256-snapshot backup of /mnt/Tap swept 253 cleanly and
    left 3 behind with dataset is busy.

    The cause is ZFS's own automount. Reading anything under
    <dataset>/.zfs/snapshot/<snap>/ makes ZFS automount that snapshot, and it stays
    mounted for zfs_expire_snapshot seconds (300 by default) after the last access.
    teardown() unmounts our bind mounts — but not the automount underneath — so
    zfs destroy refuses for exactly the datasets restic read most recently. Then
    cleanup_task() removed the sidecar anyway, destroying the only record that those
    snapshots existed. Nothing would ever have reclaimed them.

    Three changes, and the third is the one that makes it safe rather than merely
    unlikely:

    • release_snapdirs() unmounts ZFS's own .zfs/snapshot automounts (deepest first)
      before deleting, so the snapshots are not busy in the first place.
    • delete_snapshot_tree() retries the transient busy, and returns the
      snapshots it could not delete
      instead of swallowing them.
    • The sidecar is now removed only on a confirmed-clean sweep — including on the
      staging-failure path, which used to remove it before the caller swept. The
      asymmetry is deliberate: a sidecar left behind when the tree is already gone costs
      one no-op delete on the next run, while a sidecar removed while the tree still
      exists is unrecoverable. Survivors are reclaimed by the next run.

    Expect the occasional straggler, and expect it to clean itself up. On a
    256-snapshot tree this reliably sweeps ~255 immediately and may leave one: it is
    whatever restic read last, so its 300-second window has barely opened. That one is
    logged, its sidecar is kept, and the next run reclaims it before doing anything else.
    The leak is bounded at a single cycle rather than growing without limit — which is
    the property that actually matters. Blocking a backup job for five minutes to chase
    the last snapshot would be a worse trade, so it is not made.

  • Installing the patch permanently blocked updating it. install.sh does
    chmod +x update.sh, and git recorded update.sh as 100644 — so the chmod was a
    tracked modification, and update.sh refuses to run over a dirty tree. Install
    once and you could never update again; the error even told you to run
    git checkout -- ., which just undoes the exec bit so the next install can re-dirty
    it. A real box sat on an old version for exactly this reason.

    Fixed on both sides: the scripts install.sh chmods are now executable in git (so
    the chmod is a no-op), and update.sh's dirty check now looks at content, not
    file mode — git diff --numstat reports 0 0 for a mode-only change. A test
    asserts every script in install.sh's chmod loop is already 100755 in git.

  • Nested snapshots were broken on TrueNAS 24.10 and 25.04, and had been all
    along.
    SYNC_BLOCK's wrapper spelled out the stock signature and forwarded five
    arguments — but those releases declare restic_backup(middleware, job, cloud_backup, dry_run); rate_limit only arrived in 25.10. Every nested backup on
    24.10/25.04 raised TypeError: restic_backup() takes 4 positional arguments but 5 were given. The wrapper now takes *args, **kwargs and forwards whatever it is
    handed, so a trailing parameter appearing or disappearing is a non-event.

    Found by the new compatibility check, not by a user — which is the whole argument
    for having it. The check it replaced only asked whether the parameter names still
    appeared somewhere in the signature, so it happily passed a call that could never
    work.

  • The nested module is now one synchronous implementation behind two thin
    wrappers.
    TrueNAS 26 rewrites cloud_backup from async to synchronous and
    separately deletes get_dataset_recursive(), which an injected block called out
    of the host module's namespace. Either alone is a broken backup found at restore
    time: an async def wrapper hands sync.py a coroutine where it unpacks a tuple,
    and the vanished helper is a straight NameError.

    The module now talks to middlewared through call_sync, and apply.sh reads which
    flavour the installed middleware declares and injects the matching wrapper —
    TrueNAS ≤ 25.10 reaches it via await middleware.run_in_thread(...); a synchronous
    TrueNAS, already in a worker thread, calls it directly. The logic that owns the
    snapshots, the bind mounts and the failure modes exists once; an async twin
    would mean every future fix had to land twice, and the one that got missed would be
    the one that eats a backup. A middleware whose three wrapped functions disagree
    about async-ness is refused outright rather than guessed at, and
    get_dataset_recursive is carried as our own copy — removing the dependency on both
    versions instead of asserting it.

  • The patch no longer reaches into CloudSync tasks it has no business touching.
    create_snapshot is module-global in plugins/cloud/snapshot.py and is imported by
    cloud_sync.py as well as cloud_backup/sync.py — so the wrapper sat in the
    path of every rclone/Storj CloudSync task with snapshot=true, and issued a
    zfs.dataset.query before deciding it had nothing to do. That added a brand-new
    failure mode to jobs that worked fine before this patch was installed, and worse: a
    CloudSync task that ever did get staged would never be torn down, because the
    teardown is wired into cloud_backup's restic_backup and CRUD_BLOCK
    deliberately leaves CloudSync's nesting guard intact — the bind mounts would pin the
    ZFS snapshot forever. The staging path now bails out immediately unless the snapshot
    is named cloud_backup-*, before any middleware call.

  • Teardown warnings are no longer silently swallowed on TrueNAS ≤ 25.10. The async
    wrapper's finally dropped the logger= kwarg that the sync one passes, so a
    cleanup that failed to unmount a bind mount or to delete a snapshot tree logged
    nothing at all — on the only platform anyone actually runs. run_in_thread
    forwards **kwargs via functools.partial; it was a regression, not a limitation.

  • do_delete is recognised as delete. TrueNAS 24.10 and 25.04 declare
    do_delete (the CRUDService convention); 25.10 renamed it to delete. Both
    answer to zfs.snapshot.delete. Accepting only the literal name reported both older
    releases as BROKEN — a false verdict that would have switched nested snapshots off
    on boxes where they work perf...

Read more