Skip to content

v0.18.0

Choose a tag to compare

View all tags
@soulteary soulteary released this 13 Oct 02:51
v0.18.0
1cd3f76
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v0.18.0

APT Proxy v0.18.0 adds complete offline repository snapshots, verified runtime mirror failover, persistent cache integrity, resource admission, explicit shared Unix disk coordination and a real APT client matrix. Review the upgrade notes below before deploying.

New capabilities

  • Complete repository snapshots and offline maintenance.
    apt-proxy repository prefetch collects the selected suite, components,
    binary architectures and optional Sources indexes, plus every referenced
    object. Sizes and SHA256 digests are verified before atomic publication.
    verify, repair, export, import and serve operate on that complete
    selected set. Offline serving never falls back to the network. APT continues
    to verify the preserved repository signatures and expiry.
    See snapshot commands and limits.
  • Snapshot-checked runtime mirror failover. Optional mirror_health
    mappings allow bounded switching among operator-approved equivalent public
    Debian-family archives. Replacement metadata must match every current,
    fully observed Release/InRelease fingerprint. Unknown, expired, divergent or
    private snapshots cannot switch. Probes have bounded time and size; status is
    available through GET /api/mirrors/status under the existing management API
    authentication policy. This is
    metadata equality checking, not a PGP verifier or an atomic repository snapshot.
    See runtime mirror health.
  • Persistent cache integrity. cache.integrity: sha256 is now the daemon
    default. Private records bind body digest/length, request key, status, headers
    and generation. Damaged records become misses without removing a concurrently
    published replacement. Verification also applies to S3 and survives restart.
    See integrity and storage I/O costs.
  • Real APT client compatibility tests. Debian Bookworm/Trixie and Ubuntu
    22.04/24.04 run signed update, installation, source downloads, SHA256 by-hash,
    asserted gzip fallback, ranges, retries, interrupted transfers, authenticated
    HTTPS and imported offline snapshots. The runner supports digest-pinned and
    long image references, rejects colliding evidence identifiers before cleanup
    and uploads failure evidence in CI. These fixtures do not establish universal
    public-mirror compatibility. See client matrix.
  • Shared temporary-storage and download admission. One per-process budget
    covers unknown-length spools, HTTP cache response copies, native publication
    stages and S3 inline/spilled payloads. Known-length downloads also obey object
    and active-download limits. Disk admission preserves free-space headroom.
    See resource limits.
  • Explicit repository identity and shared Unix disk coordination. Public
    SHA256 by-hash indexes can share identity across explicit repository roots
    only after the body matches the requested digest. Kernel file locks coordinate
    cache publication and miss followers across processes; durable invalidation
    barriers preserve newer generations. Packages and mutable metadata retain
    per-process identities when coordination is enabled. Private requests never
    join the shared cache. See configuration and maintenance.

Correctness and dependency updates

  • Own detached request metadata before Fiber/fasthttp reuses its request buffers,
    preserving URL fields, headers, Vary selectors and caller values/deadlines
    throughout asynchronous publication.
  • Cancel in-flight upstream work at application pre-shutdown, before connection
    draining and cache teardown. Reuse cancellation domains for requests with the
    same caller lifetime without accumulating one callback per completed request.
  • Recheck coordinated cache fills without counting internal retries as extra
    native hits/misses. Keep integrity and invalidation checks on those rechecks.
  • Preserve replacement ordering, cross-instance header generations, full Vary
    selection and client conditional semantics through the cache wrappers.
  • Keep GET length verification active for the released library's versioned Vary
    keys, including ordinary retrieval and coordinated rechecks when integrity
    digests are disabled; HEAD metadata behavior remains valid. A narrow bridge
    retains recognized apt-cacher-ng HTTPS/// rewrite markers with Fiber v3.6.0,
    preserving ordinary routed-target slash/escape checks, download access
    controls, stable cancellation contexts and streaming trailers.
  • Honor explicit 0 cache size, TTL and cleanup controls supplied by CLI or
    environment variables. Omitted controls keep the ordinary defaults of 10 GiB,
    seven days and one hour; YAML nonpositive cache fields retain those defaults.
    Disabling backend TTL does not override HTTP freshness or origin directives.
  • Replace interim cache/VFS commit pins with released
    httpcache-kit v4.1.0
    and vfs-kit v1.4.5.
  • Upgrade Fiber v3.5.0 to v3.6.0, MinIO's Go client v7.3.0 to v7.4.1,
    and the indirect klauspost/compress dependency v1.20.1 to v1.20.3.
    The Go toolchain remains v1.27.2. The release workflow pins GoReleaser v2.18.3
    instead of following an unversioned tool installation. Other resolved
    dependency versions are recorded in go.mod and go.sum.

Upgrade from v0.17.2

  1. Save the running binary/image reference and configuration. Stop the process
    before making a consistent disk-cache backup. Keep a dedicated snapshot
    directory separate from the daemon's disposable cache directory.
  2. Plan for a temporary increase in origin traffic. Legacy entries without
    private SHA256 metadata become misses and are lazily refetched. The private
    isolation namespace excludes older potentially authenticated entries without
    deleting their files. httpcache-kit v4.1.0 additionally versions its Vary
    namespace: persisted legacy variants safely miss and rewarm; ordinary
    non-Vary base keys are unchanged by that library migration. No manual purge
    is required. Normal single-instance cleanup can retire unreachable objects.
  3. Review the default verification cost. A hit reads the complete body before
    response delivery, including HEAD; on S3 this can require a full object
    transfer before the first byte is returned. cache.integrity: none is an
    explicit performance opt-out for ordinary mode. Enabling shared_cache
    forces SHA256 integrity even with that opt-out. Digests detect accidental
    corruption; an attacker able to rewrite records can recompute them, so APT
    signature and package verification remain essential.
  4. Review the new bounded defaults below. A large CUDA/kernel/toolchain artifact
    exceeding the default object limit requires an explicit larger disk limit.
    S3 is restricted to at most 5120 MiB per object for single PUT. Zero selects
    defaults; it does not disable limits. The legacy download spool limits
    remain additional bounds. If resources.total_temp_mb is omitted, an explicit
    download.total_spool_mb supplies the shared total.
  5. Provision writable temporary storage even with S3. Inline payloads and their
    overlapping spill copies consume the shared budget. This is a payload-copy
    budget, not a total RSS or filesystem quota. HTTP buffers, bookkeeping,
    persistent entries and other processes need additional headroom. Resource
    counters are per process, even on a shared volume.
  6. Enable optional features deliberately. Omitted mirror_health and
    shared_cache leave runtime failover and shared identities disabled.
    repository is an explicit command with independent limits and a separate
    store. Main YAML feature changes require restart; SIGHUP/API refresh reloads
    distribution routing and resets mirror health after a successful election.
  7. For shared Unix disk mode, stop and upgrade every sharing instance together
    with identical repository policy. Keep cache/lock directories private on a
    filesystem supporting process-shared flock. Do not remove lock files while
    an instance is running. TTL/MaxSize eviction, background cleanup and startup
    stage sweeping are disabled; online global purge is rejected. Stop all
    instances for cache/orphan maintenance. Shared S3/Windows coordination is
    unavailable; isolate S3 instances with separate prefixes.
cache:
  integrity: sha256
resources:
  max_object_mb: 1024       # MiB; 1 GiB
  total_temp_mb: 4096       # MiB; 4 GiB per process
  max_active_downloads: 32
  min_free_mb: 128
download:
  max_spool_mb: 1024        # additional unknown-length spool bound
  total_spool_mb: 4096
  timeout_seconds: 1200    # 20 minutes

An active-download or shared temporary-budget refusal returns 503 with
Retry-After: 1; the legacy unknown-length spool sub-limit returns 502.
A cache-copy refusal after a complete upstream response skips publication
without discarding that response. Cache hits need no upstream slot. Cacheable
budget-controlled GET responses may report X-Cache-Staging as an HTTP trailer;
ADMITTED indicates staging admission, not completion of asynchronous commit.

Snapshot and private-repository boundaries

A snapshot includes every package/source object from the selected indexes, not
the solved dependencies of one package. Other suites/components/architectures
require their own selection. Optional translations, Contents, AppStream and
installer trees are outside this snapshot subset. Historical generations may
accumulate beyond one operation's total-byte bound.

Selected xz indexes require xz in PATH. The ordinary distroless image does
not contain it; build docker/Dockerfile.offline for the documented offline
image with xz and CA certificates. Prefetch/import/repair need a writable store
owned by the service account; serving an existing verified store supports a
read-only mount. Authenticated private origins are outside the snapshot command.

Authenticated live requests still forward credentials only to their configured
upstream and use no-store cache bypass. Optional mirror failover excludes them;
shared-cache participation also excludes unrecognized custom headers and bodies.
Configure trusted origins and any private-network access exceptions explicitly.
See private repository isolation.

Validation

PR #157 was validated at
ebca8a078756:

  • The complete race-enabled Go suite passed with 82.9% statement coverage.
  • All 12 CI checks succeeded, and the matching Codex review reported no
    major issues with no unresolved review threads.
  • Six signed APT scenarios passed with one clean Go 1.27.2 Linux arm64
    binary: Debian Bookworm/Trixie and Ubuntu 22.04/24.04 with public optional
    policies enabled, plus Bookworm and Ubuntu 24.04 private-repository samples
    with those policies active. Signatures and TLS verification remained enabled.
    Repeated authenticated responses matched fixture bytes, carried no-store
    and never returned a cache HIT; proxy logs contained no fixture credentials.
  • The scenarios covered signed updates, tampered-signature rejection,
    SHA256 by-hash, asserted gzip binary/source indexes, installation, source
    retrieval, byte ranges, retries, interrupted transfers and imported offline
    snapshots after the origin and normal proxy were stopped.

The release merge
1cd3f766f3fd
has the same source tree, 30f8797ec172b55b9a0cb0b914cde285775687a0, as the
validated PR head. The local scenarios use controlled Linux arm64 repositories;
cross-instance coalescing and runtime failover decisions are covered by separate
Go regressions. Other public mirrors and architectures require their own
validation.

Rollback

Keep the prior executable and configuration. Stop every sharing instance before
rollback, restore its compatible configuration, and use a separate old cache
directory/prefix or a quiesced pre-upgrade backup. Do not let old and new
instances concurrently write a shared store. Old versions do not understand
new integrity/coordination policies; rollback must not assume new private
records or Vary keys remain usable. Keep offline snapshot stores separate from
daemon cache maintenance.

Changelog

Features

Bug fixes

  • ed5c483: fix(ci): share non-default OtterIO integration credentials (@soulteary)
  • fd668b6: fix(distro): match Contents pdiff indexes and patches (@soulteary)
  • 27e0529: fix: acknowledge cleanup errors in failed download paths (@soulteary)
  • db08e64: fix: apply repository cache defaults before cache storage (@soulteary)
  • f32d17b: fix: apply repository cache defaults before cache storage (@soulteary)
  • e79b319: fix: apply repository cache defaults before cache storage (@soulteary)
  • 5e25989: fix: apply repository cache defaults before cache storage (@soulteary)
  • 830d9c9: fix: apply upstream access policy to mirror probes and geo discovery (@soulteary)
  • ef9ece6: fix: cancel detached repository work before server drain (@soulteary)
  • 266450e: fix: clear APT evidence before compilation and container builds (@soulteary)
  • aea960e: fix: clear completed empty replays when selecting a recovered mirror (@soulteary)
  • 250ba46: fix: contain aborted downloads and verify unknown-length bodies (@soulteary)
  • 9916953: fix: contain aborted downloads and verify unknown-length bodies (@soulteary)
  • 4e8e4f8: fix: contain aborted downloads and verify unknown-length bodies (@soulteary)
  • 51f9010: fix: contain aborted downloads and verify unknown-length bodies (@soulteary)
  • 183d6b6: fix: contain aborted downloads and verify unknown-length bodies (@soulteary)
  • 8e183f4: fix: count incomplete downloads and bound mirror replay headers (@soulteary)
  • ef30926: fix: honor configured storage directories for download spools (@soulteary)
  • 9ef27d0: fix: integrate reviewed staging dependency and safe disk admission (@soulteary)
  • 1bca465: fix: isolate credential-bearing requests from shared repository caches (@soulteary)
  • 5caf289: fix: isolate credential-bearing requests from shared repository caches (@soulteary)
  • 011f75e: fix: isolate credential-bearing requests from shared repository caches (@soulteary)
  • 675bb65: fix: isolate credential-bearing requests from shared repository caches (@soulteary)
  • 9b1dcd7: fix: merge download limits through legacy configuration API (@soulteary)
  • 69fe6a5: fix: own asynchronous cache requests and preserve retry accounting (@soulteary)
  • 225d2e0: fix: pin reviewed conditional and Vary generation corrections (@soulteary)
  • bd7745a: fix: pin reviewed integrity library and explain operator-owned imports (@soulteary)
  • 55e2f21: fix: preserve client cache directives during repository routing (@soulteary)
  • be5ab0b: fix: preserve client cache directives during repository routing (@soulteary)
  • b790b42: fix: preserve explicit empty download ACL lists across YAML merges (@soulteary)
  • 8c91cb1: fix: preserve healthy routes and bound repository method and spill lifecycles (@soulteary)
  • 15d1e4e: fix: preserve upgrade deadline and half-close behavior (@soulteary)
  • a1d0844: fix: preserve upstream trailers when spooling fixed-length responses (@soulteary)
  • 6209990: fix: prevent unverified mirror errors from poisoning shared by-hash cache (@soulteary)
  • e3d589c: fix: propagate configured mirror credentials before cache lookup (@soulteary)
  • e8bc7f7: fix: race approved address families within a shared dial deadline (@soulteary)
  • a8aee21: fix: recognize Ubuntu legacy installer image directories (@soulteary)
  • 3754255: fix: reject cached body length mismatches and refetch complete objects (@soulteary)
  • 5feffcd: fix: reject cached body length mismatches and refetch complete objects (@soulteary)
  • fc91ddb: fix: reject cached body length mismatches and refetch complete objects (@soulteary)
  • e68b8fc: fix: reject cached body length mismatches and refetch complete objects (@soulteary)
  • 2f63071: fix: reject cached body length mismatches and refetch complete objects (@soulteary)
  • 9784b8b: fix: reject invalid filesystem block sizes before resource admission (@soulteary)
  • 5cb8b8c: fix: retain negative caching without repository freshness defaults (@soulteary)
  • 578adaf: fix: retain spool reservations until cleanup succeeds (@soulteary)

Dependencies

Others

  • e24be5e: Add bounded repository-verified runtime mirror failover (@soulteary)
  • 5041c0a: Add complete repository snapshots and offline maintenance commands (@soulteary)
  • d643e56: Bound all temporary payload copies and upstream download admission (@soulteary)
  • 3cbb547: Fix late connection cleanup test timing and update cache validation dependency (@soulteary)
  • 33e12c6: Fix private cache safety, diagnostics, and download spool cleanup (@soulteary)
  • 9de4a7c: Merge main and reconcile OtterIO integration setup (@soulteary)
  • bc4bf17: Merge the offline repository baseline and preserve integrity documentation (@soulteary)
  • 6e8269b: Pin reviewed Vary semantics and test dependent PR branches (@soulteary)
  • 68124e5: Redact complete repository credentials and address cache review feedback (@soulteary)
  • 790efe7: Require private SHA256 verification for cache hits (@soulteary)
  • 46ee432: Test actual signed APT clients and offline snapshots across Debian and Ubuntu (@soulteary)
  • ebca8a0: Update stable dependencies and prepare v0.18.0 documentation (@soulteary)

Contributors

soulteary and dependabot
Assets 32
Loading