perfcached 0.5.6.6

A multithreaded, self-clustering cache for Linux, with its own encrypted protocols and a Redis-compatible door. New here? Start with the README.

This page covers everything since 0.5.5: 0.5.6 and its point releases up to 0.5.6.6. The ChangeLog has each one.

A rolling restart. 0.5.6.6 joins a 0.4.5 or later fleet one node at a time: restart the members first and the master last, and read the fleet between nodes. The storage formats and the wire epoch are unchanged, and clients built on libperfd 0.4.0 keep working. Two things to know before you start:

• Check the memory guard on huge-page nodes first - see Known below. A node whose arena lives in a reserved hugetlb pool may need memory_floor = off.
• A config that uses a new key is refused by an older release (memory_floor, stats_reset, upstream_writes). To roll back, put the old config back with the old binary.

The daemon protects itself from the out-of-memory killer

On a 1 GB node, 50,000 idle RESP connections used to get the daemon killed. Now:

• A memory guard reads, once a second, the memory the node may still use: MemAvailable, or the headroom of a memory-limited cgroup (v2 or v1), whichever is tighter, less the arena's uncommitted part. Below the floor, new clients are refused exactly as at max_clients. Open connections and /stats are untouched, and refusing stops once a further 5% of the limit is free.

  [daemon]
  memory_floor = 15%      # the default; kept within 128 MB .. 4 GB.  Or: 512 MB | 2 GB | off
  

  A 300 MB cgroup now holds 24,457 of 25,000 connections and refuses the rest.
• max_clients, when unset, counts memory as well as descriptors: the smaller of the two wins. The boot line and /stats clients.max_basis say which. A configured max_clients above what memory holds stands, with a warning; the guard protects.
• An idle connection holds no buffers. After two quiet seconds its input and output buffers are released (14 KB down to 2.2 KB per idle client), and they grow back on its next request. Once a burst of closed or idle connections settles, the freed heap is handed back to the system, so the daemon's memory falls back after a mass disconnect.
• A busy client is no longer slowed by idle ones. Every request used to pay a walk over all of its worker's connections (with 20,000 idle: 60 -> 750 us a PING). Now it does not.
• Encrypted native clients past the limit are refused before the Noise handshake, with the RESP door's -ERR max number of clients reached. A storm of refused clients no longer costs a handshake each: 12,000 clients against a limit of 2,000 take a third less daemon CPU. A library that does not read the line sees its handshake fail, as on any close. A plaintext native door (loopback, unix socket) still answers an error frame to the first request.
• New counters: /stats clients.memory and clients.buffers. New /metrics: perfcached_memory_limit_bytes, _available_bytes, _floor_bytes, perfcached_memory_refusing, perfcached_clients_refused_memory_total, perfcached_conn_buffers_released_total, perfcached_heap_trims_total. The status page shows a red "memory floor" chip while the node is refusing.

libperfd 0.4.2

• 0.4.1: a blocking connect takes the key from the process-wide cache. It used to derive it on every connect: 150 ms and 64 MB each. The derive is single-flight across threads, and perfd_psk_prepare() derives before fork() so a forking server's children inherit it. Rebuild clients that connect often.
• 0.4.2: fits a musl thread's 128 KB default stack. A connect from a thread on Alpine overflowed it.

Fixes

• A crash on the metrics door (0.5.6 - 0.5.6.2): a connection that reached http_timeout in the very moment its request or close arrived was freed and then handed its event, a use-after-free ending in SIGSEGV. Seen under a monitor plus a Prometheus scraper.
• WAL, standalone edition (every standalone release since 0.4.0): a key that expired, was reaped and was written again on another thread could come back after a crash with its expiry instead of the new value. The clustered edition is not affected.
• WAL: an empty node restarted more times than its WAL has segments came back FAILED ("wal: FULL at startup").
• "perfcached ready" is logged once every worker's listeners are open; a client that connected the moment it appeared could be refused.
• A record moved back from the overflow chain expires on time.
• A native client with no secret is told so at once, instead of being taken for a long handshake.
• The per-door connection and request totals are counted to 64 bits.

Read-through

• Tracking stays on one server. Behind a round-robin proxy over several replicas, a fetching connection could land on another server than the tracking one, and its copies would never see a change. Each connection now reads its server's run_id. One that reached another server is not tracked; its copies live at most upstream_negative_ms, and stats counts them (upstream.tracking_other_server). The upstream user needs +info; the GUIDE's ACL line has it.
• upstream_writes = refuse: a read-through collection clients can only read. Every write to it, on every door, is refused with one message, and so are a drain sweep and a restore into it.
• Read-through works on the RESP door under a mapped db index (resp_collections mapping SELECT n onto a named collection).
• It counts what clients got: a read answered with a value is a hit, an absent answer a miss. The status page adds a line per read-through collection: fetched from the upstream (found / absent, average fetch time), answered from fetched copies, and answered absent from the negative cache.
• The drain sweep reads untracked, so the upstream no longer remembers every swept key.
• The <name>.nofall companion follows its parent's autoscale.
• tools/redis-replica-interop.sh runs read-through against a real Redis replica in throwaway containers.

Stats and the status page

• Every count on the status page is the last five minutes, kept by the daemon (win_<field> in /stats), so the window is true from the moment the page opens. Gauges stay current; maxima are the last five minutes'.
• /stats resets on a schedule: daily at local midnight by default (stats_reset = off | 1d .. 7d, stats_reset_at = HH:MM). The page's "reset stats" resets the whole fleet at one moment. /metrics is never reset, so Prometheus sees true counters across resets and collection resizes.
• perfcached_command_latency_seconds has one bucket per octave (3 us .. 262 ms): 18 rows per command instead of 129. Every bound is an edge of the daemon's own histogram, so the counts are exact.
• The header names hosts in trouble: it blinks red (failed) or amber (any other state but ready) and counts the fleet ("fleet 3/3 ready").
• One rule for collection names wherever one is made: 1-32 characters of A-Z a-z 0-9 _ -, not starting with -. A name from before the rule is kept and named in the log.

Known

• The memory guard counts a hugetlb arena twice. With the arena on MAP_HUGETLB huge pages, the pool is already outside MemAvailable, but the guard also subtracts the arena's uncommitted part. A node whose arena is large against its RAM can refuse every new client once the arena has handed memory back. For example: 2 GB of RAM, arena_mb = 1024 on a 512-page hugetlb pool. At a fresh start the arena commits the whole pool, so it shows up later, not at boot.
  • Check the boot line arena: ... reserved on MAP_HUGETLB huge pages and /stats clients.memory.budget.
  • On such nodes set memory_floor = off (the 0.5.6.4 behaviour) until the fix.
  • THP and 4K arenas are counted correctly.
• Read-through: KEYS and SCAN see only what the collection holds. proxy, shard and spread collections refuse an upstream. In a cluster, a read on another node in the instant before a change reaches it may answer the upstream's older value.
• No sorted sets, lists or streams yet. Keys hold strings, counters, JSON documents or hashes (a hash is one value, under 58,000 bytes).
• discovery = unicast (clouds, Kubernetes) is tested in lab network namespaces that drop IP fragments and block UDP, not yet in a real cloud deployment. spread mode has not yet carried production traffic.
• In eager mode a delete made on one side of a network partition can be undone when the partition heals.

Packages

Debian 12 and 13, Ubuntu 20.04 and 24.04, amd64, in both editions:

• perfcached: the clustered daemon, every cluster mode.
• perfcached-standalone: the single-node edition, built from the same tree without the cluster code. A [cluster] section in its config is refused rather than ignored.

The two install the same files under the same names and replace each other, so moving between editions is one apt install. Each package was built in a clean container and install-checked in a second one: the version and revision, the edition perfcached -V reports, libperfd 0.4.2, the shipped config passing perfcached -C (also with memory_floor set), and the systemd unit. For the standalone package it also checked that a [cluster] section is refused and that installing perfcached over it swaps the edition.

Alpine 3.24 (musl), x86_64, both editions, as tarballs of the tree's make install layout: binaries in usr/bin, libperfd and perfd.h, the example config, and the docs. No systemd unit or OpenRC service is shipped; run the daemon under your own supervisor.

apk add libsodium
tar -xzf perfcached-0.5.6.6-alpine3.24-x86_64.tar.gz
cp -a perfcached-0.5.6.6-alpine3.24-x86_64/usr perfcached-0.5.6.6-alpine3.24-x86_64/etc /

Built with gcc 15 in a clean Alpine container: noisetest, the selftest, and a deliberately broken-lock build that must fail. Each tarball was then checked in a second clean container:

• -V and the musl interpreter;
• the shipped config passing -C, and a [cluster] section refused by the standalone build;
• the daemon starting and answering ping, set and get over its encrypted native door from perfcli, which is the musl-stack path libperfd 0.4.2 fixes.

SHA256SUMS covers every file on this page.