Repository navigation
Manage Many Instances
sipnab fleet inventory asks every sipnab instance you list for its
inventory, and prints one table of the fleet. Use it to answer "which build
and which configuration does each instance run, and which ones are down?"
before and after an upgrade or a configuration change.
The table shows, for each instance:
- the build it runs, and the SHA-256 of its binary,
- the digest of its effective configuration: the file it loaded and its command line,
- the user it runs as, and how long it has been up.
Below the table it names the instances that did not answer, the instances on a different build from the rest, the HEP senders a collector receives from that are not in your list, and the listed instances whose HEP a listed collector refuses for authentication. Each instance also reports when each of its key, token and password files was last replaced, so you can check that a rotation took effect. It reports. It changes nothing on any instance.
sipnab fleet serve is the other direction: a fleet center that members
report to. Each member pushes a heartbeat, the center keeps the newest one per
member, and it says which members went silent, including members the
controller cannot reach. See Run a fleet center.
Some upgrades need every sender and its collector on the same build, such as the HEP HMAC version 2 change in Upgrade a fleet that uses HMAC authentication. The report shows which instances are still on the old build.
The commands and output on this page come from a run on 2026-11-03: four
instances in containers on the network 192.0.2.0/24, and the controller on
the host. The instances ran a development build of 0.5.232 with this feature,
and one ran release 0.5.130. The output of
A listed sender the collector refuses
and Check that a key rotation took effect
comes from a run on 2026-11-07 on one host, with a development build of
0.5.236: the collector hub on 127.0.0.2 and the sender edge-a on
127.0.0.1.
- Each instance serves
GET /v1/inventoryon its REST API (--api). See GET /v1/inventory for every field. -
sipnab fleet inventoryruns on one machine, the controller. It reads a hosts file and asks every instance in it at once. The instances never connect to the controller. - Each request carries a bearer token with scope
inventory. That token reads the inventory and nothing else: no dialogs, no messages, no audio. - The answer holds no configuration value and no command-line argument, only their SHA-256 digests.
- The command reads at most 16 KiB of each answer's status line and headers
(
MAX_HEAD_BYTES) and at most 4 MiB of its body (MAX_ANSWER_BYTES). The inventories in this page's run were under 2 KB each.
-
sipnab on each instance. A build whose
sipnab --versionlists theapifeature, and that servesGET /v1/inventory. An older build answers404there, and the report marks itrefused. -
sipnab on the controller. Any build with the
apifeature. It needs no capture privilege: it opens no capture. -
TLS for each instance's API. A certificate whose subject alternative
name is the address or name in the instance's URL, its key readable by the
sipnab user only, and the certificate of the issuing CA on the controller.
Plain
http://works too, and sends the token in clear text. - A signing key on each instance, in a file only the sipnab user can read. Set up authentication shows how to create one.
- Network. The controller reaches each instance's API port over TCP, 8443 in this page's examples. Open that port to the controller's address only. The instances need no route to the controller.
- Other systems. No change. OpenSIPS, Kamailio, Homer and a HEP collector keep their configuration. A collector that runs sipnab joins the list like any other instance.
-
[each instance] Add
--api, the TLS certificate and key, the signing key and a node name to the instance's command line. This is the instanceedge-afrom the run, which reads a capture file and sends HEP to the collectorhub:sipnab -N -I /var/lib/sipnab/sample.pcap --config /etc/sipnab/edge-a.toml \ --hep-send 192.0.2.10:9060 --hep-auth-mode hmac --hep-auth-file /etc/sipnab/hep.key \ --api 192.0.2.11:8443 --api-tls-cert /etc/sipnab/edge-a.crt --api-tls-key /etc/sipnab/edge-a.key \ --api-signing-key-file /etc/sipnab/edge-a.signing --node-name edge-a
The collector ran with the same API flags and its HEP listener:
sipnab -N --config /etc/sipnab/hub.toml \ --hep-listen 192.0.2.10:9060 --hep-auth-mode hmac --hep-auth-file /etc/sipnab/hep.key \ --api 192.0.2.10:8443 --api-tls-cert /etc/sipnab/hub.crt --api-tls-key /etc/sipnab/hub.key \ --api-signing-key-file /etc/sipnab/hub.signing --node-name hub
Each instance logs
REST API listening on <address>when its API is up.
-
[each instance] Mint a token with that instance's signing key:
sipnab --mint-token --token-scope inventory --token-id fleet-controller \ --api-signing-key-file /etc/sipnab/edge-a.signing --api-token-ttl 2592000
It prints one token, starting
s2., and exits.--api-token-ttl 2592000makes it valid for 30 days. -
[controller] Save each token in its own file, readable by the controller's user only:
install -m 600 /dev/null /etc/sipnab/fleet/edge-a.token
Then paste the token into it. sipnab refuses a token file that other users can read.
-
[controller] Write
/etc/sipnab/fleet/fleet.toml, one[[host]]table per instance. This is the file from the run:[[host]] name = "hub" url = "https://192.0.2.10:8443" token_file = "/etc/sipnab/fleet/hub.token" ca_file = "/etc/sipnab/fleet/internal-ca.pem" [[host]] name = "edge-a" url = "https://192.0.2.11:8443" token_file = "/etc/sipnab/fleet/edge-a.token" ca_file = "/etc/sipnab/fleet/internal-ca.pem" [[host]] name = "edge-b" url = "https://192.0.2.12:8443" token_file = "/etc/sipnab/fleet/edge-b.token" ca_file = "/etc/sipnab/fleet/internal-ca.pem"
-
nameis the name the report shows. Each name appears once. -
urlis the instance's REST API,https://orhttp://. -
token_fileholds the instance'sinventorytoken on one line. -
ca_fileis optional: a PEM CA file to trust instead of the host's CA bundle.
sipnab refuses an unknown key, an empty name, an empty URL and a name used twice, with exit
2. -
-
[controller] Run:
sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml
Output from the run:
NAME STATE VERSION COMMIT BINARY CONFIG UID UPTIME_S hub ok 0.5.232 29204db7 6bba53867ecd ec09522b49b4 1000 54 edge-a ok 0.5.232 29204db7 6bba53867ecd 6a434df0c5e2 1000 40 edge-b ok 0.5.232 29204db7 6bba53867ecd f63dfdce2c9d 1000 39It exited
0.-
BINARYandCONFIGare the first 12 hex digits ofbinary.sha256andconfig.effective_sha256. The three instances run one binary and three configurations: each has its own config file and node name. -
--jsonprints the whole report, with every instance's full inventory. -
--timeout <SECS>sets how long each instance has to answer. Default10.
-
-
Every instance is
ok, and there is nofindingline. That is exit0. -
Two instances you expect to match show one
CONFIG. Two instances with the sameCONFIGloaded the same configuration values and run the same command line. A different--node-namealone gives a differentCONFIG. -
The exit status says the same for a script:
Code Meaning 0Every instance answered and there is no finding. 1An instance did not answer, refused the token, or sent an inventory this build cannot read; or there is a finding. 2sipnab cannot read or use the hosts file.
Each output below comes from the same run, with one thing changed.
-
unreachableandconnection timed out. Nothing answered at that address within--timeout: the instance is down, the URL names the wrong address or port, or a firewall drops the controller's packets. The other instances are still listed:edge-c unreachable 192.0.2.14:8443: connection timed outConnection refusedinstead means the host answered and nothing listens on that port: check--apion the instance. -
unreachableandinvalid peer certificate: UnknownIssuer. The controller does not trust the CA that issued the instance's certificate. Setca_filefor that host:hub unreachable sending: invalid peer certificate: UnknownIssuer -
refusedwith HTTP401. The instance did not accept the token: it was minted with another instance's signing key, it expired, someone revoked it, or its scope is notinventory,full,actionsoradmin. Mint a new one on that instance (step 2). This run gaveedge-bthe hub's token:edge-b refused the instance answered HTTP 401 -
refusedwith HTTP404. The instance runs a build withoutGET /v1/inventory. Upgrade it. Release 0.5.130 answered:edge-old refused the instance answered HTTP 404Release 0.5.130 also refuses
--token-scope inventoryat mint time, withinvalid value 'inventory' for '--token-scope <SCOPE>'. -
unreachableandcan be read or written by other users. The token file's mode lets other users read it. Runchmod 600on it:edge-a unreachable token_file '/etc/sipnab/fleet/open.token' can be read or written by other users (mode 0644); chmod 600 it -
finding hep_sender_refused. A collector in the list refuses the HEP of an instance in the list for authentication. See A listed sender the collector refuses. -
finding unlisted_hep_sender. A collector in the list receives HEP from an address that no listed instance's URL resolves to. Add that sender to the hosts file, or find out why it sends. Withedge-bleft out of the file:finding unlisted_hep_sender: hub receives HEP from 192.0.2.12, which is no instance in the hosts fileThe check compares addresses only. A sender whose packets leave from a different address than its API URL names shows up as unlisted.
-
incompatibleand aschema_version. The instance answered an inventory format newer than the controller reads. Upgrade the controller's sipnab. No release answers a newer format yet, so this output comes from a test server that answeredschema_version2:edge-next incompatible the instance answered inventory schema_version 2; this sipnab reads up to 1
A collector that refuses a sender's HEP drops every packet it refuses. The
sender counts them as sent, because UDP reports no delivery, so neither end
fails. The report names that sender, and exits 1, while the collector
refused the latest packet it received from it.
-
[controller] Run the report. In this run
edge-asent with a key the collectorhubdoes not hold:sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml
NAME STATE VERSION COMMIT BINARY CONFIG UID UPTIME_S hub ok 0.5.236 fd7d466e d7cd2b7e5769 a5dc546e4913 1000 5 edge-a ok 0.5.236 fd7d466e d7cd2b7e5769 79480fa13223 1000 3 finding hep_sender_refused: hub refused 7 HEP packets from edge-a (127.0.0.1): refused: authentication (wrong key or HMAC version); the collector's reasons: hmac_bad_mac 7It exited
1.- The finding names the collector, every listed instance whose URL resolves to the refused address, the address, and the authentication refusals only.
-
refused: authentication (wrong key or HMAC version)names both causes and claims neither. A sender with the wrong key, and a sender on a different HEP HMAC version, can both arrive as a MAC that does not verify, depending on which build each end runs. -
the collector's reasonsare the collector's own refusal reasons, with the namesGET /v1/hep/sendersuses:hmac_bad_macfor a MAC that does not verify,hmac_unsupported_versionfor a version-1 token on a collector on 0.5.131 or later,auth_mismatchfor a wrongplainkey, andauth_missing,hmac_bad_format,hmac_timestamp_out_of_windowandhmac_replay. Refusals for other reasons, such as the rate limit or the allowlist, are not this finding.
-
[controller] See the collector's refusals, from the same report:
sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml --json \ | jq -c '.hosts[] | {name, refused: .inventory.hep_listener.refused_sources}'
{"name":"hub","refused":[{"by_reason":{"hmac_bad_mac":7},"last_seen":"2026-11-07T12:52:30.675Z","packets":7,"peer":"127.0.0.1"}]} {"name":"edge-a","refused":[]} -
[sender] Fix the sender. Give it the collector's key, or upgrade it to the collector's build when the reason is
hmac_unsupported_version, as Upgrade a fleet that uses HMAC authentication describes. Restart it. -
[controller] Verify. The collector keeps counting the old refusals for as long as it runs, but it has admitted the sender since its last refusal, so the report has no finding and exits
0:sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml --json \ | jq -c '.hosts[0].inventory.hep_listener | {senders: [.senders[] | {peer, last_seen}], refused: [.refused_sources[] | {peer, packets, last_seen}]}'
{"senders":[{"peer":"127.0.0.1","last_seen":"2026-11-07T12:52:43.016Z"}],"refused":[{"peer":"127.0.0.1","packets":7,"last_seen":"2026-11-07T12:52:30.675Z"}]}
Troubleshoot:
-
The collector refuses the sender, and there is no finding. The refused address is no
listed instance's: the check compares addresses, as
unlisted_hep_senderdoes. A sender whose packets leave from another address than its API URL names is not matched. Read the collector'srefused_sourcesas above. -
No
refused_sourcesat all. A collector on a build before this check does not report them. Itspackets_refusedstill counts every refusal. -
The finding stays after the fix. The collector has not admitted a packet
from that address since its last refusal: the sender is not sending yet, or
is still refused. Its
last_seeninsendersmust be later than inrefused_sources.
The configuration digests cover a key file's path only, so replacing the key in it changes no digest. Each instance therefore reports, for every secret or credential file it named, the file's inode and modification time when it started, under the setting that named it. The answer holds nothing derived from the key. See GET /v1/inventory for the fields.
-
[each instance] Rotate the key the usual way: write the new key to a new file, rename it over the old one, and restart sipnab. This run fixed
edge-a's HEP key:install -m 600 hep.key edge-hep.key.new && mv edge-hep.key.new edge-hep.key -
[controller] Before and after the restart, read each instance's stamps:
sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml --json \ | jq -c '.hosts[] | {name, secrets: [.inventory.config.secret_files[] | {setting, inode, mtime}]}'
Before:
{"name":"hub","secrets":[{"setting":"--api-signing-key-file","inode":6849703,"mtime":"2026-11-07T12:41:19.472148552Z"},{"setting":"--hep-auth-file","inode":6849697,"mtime":"2026-11-07T12:41:19.472148552Z"}]} {"name":"edge-a","secrets":[{"setting":"--api-signing-key-file","inode":6849705,"mtime":"2026-11-07T12:41:19.476148542Z"},{"setting":"--hep-auth-file","inode":6852341,"mtime":"2026-11-07T12:52:28.642390890Z"}]}After:
{"name":"hub","secrets":[{"setting":"--api-signing-key-file","inode":6849703,"mtime":"2026-11-07T12:41:19.472148552Z"},{"setting":"--hep-auth-file","inode":6849697,"mtime":"2026-11-07T12:41:19.472148552Z"}]} {"name":"edge-a","secrets":[{"setting":"--api-signing-key-file","inode":6849705,"mtime":"2026-11-07T12:41:19.476148542Z"},{"setting":"--hep-auth-file","inode":6817841,"mtime":"2026-11-07T12:52:41.958355975Z"}]}
Verify: the rotated file's inode or mtime changed on the instance you
rotated, and on no other. CONFIG did not change: the path is the same.
Troubleshoot:
- Nothing changed. The instance has not restarted: sipnab takes the stamp at start, and it describes the file the run read. Restart it.
-
Only
mtimechanged. The key was rewritten in place rather than replaced. That is a rotation too. -
errorholds a reason. The instance could not stat the file at start, for exampleNo such file or directory (os error 2). The reason never names the path.
A fleet center is a sipnab, sipnab fleet serve, that fleet members push
heartbeats to. Each heartbeat carries the member's inventory, the record
GET /v1/inventory answers. The center keeps the newest heartbeat of each
member, and answers "which members are there, what do they run, and which went
silent" on GET /v1/fleet/members, without asking any member.
Use it when the controller cannot reach every member, such as members behind
NAT, or to learn that a member stopped without polling it. A member can also
push a record of each call that ended badly, as
Push call records to the center shows. A
capture run becomes a member
with --fleet-report-url (or [fleet] report_url), as
Make an instance a fleet member shows.
Step 3 sends one heartbeat by hand, in the format a member sends.
The output in this section comes from a run on 2026-11-07 on one host, with a
development build of 0.5.236: the center on 127.0.0.1:8444 over HTTPS, and
one instance, edge-a, on 127.0.0.1:18080. The run kept its files in one
working directory. The commands name them as a deployment would, under
/etc/sipnab and /var/lib/sipnab, and the log lines show the names the run
used.
-
sipnab on the center. A build with the
apifeature. It opens no capture and needs no capture privilege. - A TLS certificate for the center, whose subject alternative name is the address or name members use, its key readable by the center's user only, and the certificate of the issuing CA on each member. The center refuses plain HTTP on any address but loopback, because members send their token and their inventory.
- A signing key for the center, in a file only the center's user can read. The center mints and verifies member tokens with it. It is not any member's REST API key.
- A store key for the center, in a file only the center's user can read. The center encrypts its store under it, and does not start without it. Step 1 creates it. Encrypt the center's store covers backup and rotation.
- Network. Each member reaches the center's port over TCP, 8444 in this section. The center needs no route to the members.
- Disk. The store file holds the newest heartbeat of each member: 1,884 and 1,883 bytes of JSON for the two members of this run. Encryption adds 21 bytes to each record and 120 bytes to the file.
-
Bounds. The center reads at most 4 MiB of one request
(
MAX_INGEST_BYTES) and answers413past it. Its rate limit counts at most 4096 addresses in one second (MAX_TRACKED_PEERS); a further address in that second gets503, with--rate-limit-per-peerabove0. - Other systems. No change. OpenSIPS, Kamailio, Homer and any HEP collector keep their configuration.
-
[center] Create the signing key, readable by the center's user only:
# Run all of these, in order. install -m 600 /dev/null /etc/sipnab/center.signing head -c 32 /dev/urandom | base64 > /etc/sipnab/center.signing
-
[center] Create the store key, and copy it to your backup. The command writes the file with mode
0600and prints the key's id:sipnab fleet keygen --out /etc/sipnab/center.storekey
store.key: key id ac7b02adecf75d31 -
[center] Start it:
sipnab fleet serve --listen 127.0.0.1:8444 --store /var/lib/sipnab/center.store \ --store-key-file /etc/sipnab/center.storekey \ --signing-key-file /etc/sipnab/center.signing \ --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.key
INFO sipnab::app::fleet_center: fleet center store 'center.store': 0 member(s), 0 past --retention-hours 72 removed INFO sipnab::app::fleet_center: fleet center serving HTTPS on 127.0.0.1:8444 INFO sipnab::output::api: REST API serves HTTPS only (TLS 1.2/1.3, ALPN http/1.1) INFO sipnab::output::api: REST API listening on 127.0.0.1:8444A center after 0.5.243 also logs the key it encrypts the store under, before the store line. This line is from a run on 2026-11-10:
INFO sipnab::app::fleet_center: fleet center store encryption sealed, key id ac7b02adecf75d31, 0 old key(s)It creates the store with mode
0600, encrypted. It removes a member it has not heard from for--retention-hours(default 72), at start and every minute.--silent-after-missed(default 3) sets how many of a member's heartbeat intervals may pass before it is silent. Members reaching the center by a name need that name in--allowed-host. The CLI reference lists every flag. -
[center] Start it as the user that owns
/var/lib/sipnab, not as root. Since 0.5.256 a center that root starts reads its key files and binds, then serves as--user, elsenobody. Run the center as a service has the unit and what a root start needs. -
[center] To refuse any member without a certificate from your own CA, as well as its token, add
--tls-client-ca /etc/sipnab/members-ca.pem. Require member certificates on a fleet center makes the CA and the member certificates, and says how a member presents one.
-
[center] A member's token has scope
fleet-report, and its id is the member's name. The center vouches for that name, and for nothing else the member says:sipnab --mint-token --token-scope fleet-report --token-id edge-a \ --api-signing-key-file /etc/sipnab/center.signing --api-token-ttl 2592000
It prints one token, starting
s2.. Afleet-reporttoken reaches the center's ingest and nothing else, on the center and on every other sipnab. Give each member its own token, so revoking one leaves the others working. -
[center] A token that reads the members list has scope
inventory, as forsipnab fleet inventory:sipnab --mint-token --token-scope inventory --token-id fleet-reader \ --api-signing-key-file /etc/sipnab/center.signing --api-token-ttl 2592000
A member writes its inventory as a heartbeat every [fleet] heartbeat_secs
(default 30) into its spool directory, and sends the spool to the center,
oldest first. While the center is away the heartbeats wait, and the capture
goes on: the member runs on a thread of its own and the capture never waits
on it. The spool keeps at most [fleet] spool_max_files (default 2880, a day
at 30 s) and drops the oldest past that. A heartbeat the center refuses for
good (400, 413) moves to failed/ in the spool.
The output below comes from a second run on 2026-11-07 on one host: the
center on 127.0.0.1:8444 over HTTPS, and edge-a reading a capture file
with its REST API on 127.0.0.1:18080, reporting every 5 seconds.
-
Other systems. The member opens TCP connections to the center's port: open it outbound from each member. Nothing else changes.
-
[member] Put the token minted in step 2 in a file only the sipnab user can read, then add a
[fleet]section to the member's configuration:[fleet] report_url = "https://127.0.0.1:8444" report_token_file = "/etc/sipnab/fleet/edge-a.report.token" report_ca_file = "/etc/sipnab/fleet/internal-ca.pem" spool_dir = "/var/lib/sipnab/fleet-spool" heartbeat_secs = 5
The flags
--fleet-report-url,--fleet-report-token-file,--fleet-report-ca-fileand--fleet-spool-dirset the same and override the keys.report_urltakeshttp://only to loopback: sipnab refuses plain HTTP to another machine and does not start (exit2for a flag,1for a key). -
[member] Start the run as usual. It logs where it reports:
INFO sipnab::app::fleet_member: fleet member reporting to https://127.0.0.1:8444/ every 5 s, spool 'fleet-spool' -
[controller] The center lists the member, with the member's own inventory under
self_reported:{"member":"edge-a","silent":false,"seconds_since_heartbeat":3,"verified":{"heartbeats":2,"peer":"127.0.0.1","received_at":"2026-11-07T18:31:28Z"},"node":"edge-a","interval":5,"seq":2}That line comes from the members list through
jq -c '.members[] | {member, silent, seconds_since_heartbeat, verified, node: .self_reported.inventory.node_name, interval: .self_reported.heartbeat_secs, seq: .self_reported.seq}'.
Verify the member:
-
The center goes away. This run stopped the center for 12 seconds. The member logged it once, its REST API still answered
200, and two heartbeats waited in the spool (00000000000000000005.jsonand00000000000000000006.json):WARN sipnab::app::fleet_member: fleet member: heartbeats wait in the spool: the center did not answer: 127.0.0.1:8444: Connection refused (os error 111) -
The center returns. The spool drained, and the center counted every heartbeat once:
heartbeats8 forseq8.INFO sipnab::app::fleet_member: fleet member: the center takes heartbeats again (was: the center did not answer: 127.0.0.1:8444: Connection refused (os error 111))The returned center first reported
edge-asilent, from the time of its last heartbeat before the outage, thenheard againonce the spool drained. -
The member stops. This run killed
edge-a. The center reported it silent three 5 s intervals later:WARN sipnab::app::fleet_center: fleet member 'edge-a' is silent: no heartbeat for 15 s, more than 3 of its 5 s intervals
Troubleshoot the member:
-
the center refused this member's token (HTTP 401). The token was not minted with this center's key, it expired, or the center revoked its id. The heartbeats wait in the spool. Mint a new token on the center (step 2), put it in the token file, and restart the member: the member reads the token once, at start. -
the center did not answer. The center is down, or a firewall drops the member's connections. The heartbeats wait; nothing needs doing on the member once the center is back. -
the center answered HTTP 503. The center is starting and rebuilds its member list, or its rate limit refused the member. The heartbeats wait, and the member tries again after 1 s, 2 s, 4 s and so on, at most oneheartbeat_secsapart (What the center answers while it starts). -
the center did not answer: sending: invalid peer certificate: UnknownIssuer. The member does not trust the center's certificate. Setreport_ca_fileto the certificate of the issuing CA. -
the spool holds [fleet] spool_max_files. The center was away long enough to fill the spool, and the member dropped the oldest heartbeats. A newer heartbeat says everything an older one did.
Purpose. A member can send the center one record for each call that
ended badly, so the center holds the calls an operator asks about without
anyone opening a session on the member. The record's call is the member's
own answer to GET /v1/dialogs/{call_id}, built by the same code, so the
center and the member cannot disagree about a call. The record leaves out
only the capture's identity, which the REST API stamps on each answer.
The outputs below come from a run on 2026-11-10 on one host, with a
development build after 0.5.243: a center on 127.0.0.1:18444 and edge-a
reading a capture file that holds one call that got a 486 and one call
that the callee answered and that then ended.
Before you start.
-
A center that takes call records. Upgrade the center to this release
first. An older center refuses a call record with
400, and the member moves that delivery tofailed/in its spool. - A fleet member that already reports heartbeats, as Make an instance a fleet member sets up.
- A key file on the member, apart from every other key. Call records hold numbers and URIs. While they wait for the center they are on the member's disk only in sealed files, under this key.
- Other systems. Nothing changes on OpenSIPS, Kamailio, Homer or any HEP collector. The member uses the connection to the center that its heartbeats already use.
-
Room at the center. The center keeps the records in one sealed file
beside its store,
<store>.intake, for--retention-hours, and holds at most 64 MiB there (MAX_INTAKE_BYTES); past that the oldest records go first. A center with the history store moves them from there into files it keeps for days, as Keep call history at the center shows.
Steps.
-
[member] Create the spool key. The file gets mode
0600:sipnab fleet keygen --out /etc/sipnab/fleet/spool.key
spool.key: key id 0728832701dd1d86 -
[member] Name the key in the member's configuration and restart the member. With a spool key and no
call_push, the member pushesexceptions:[fleet] spool_key_file = "/etc/sipnab/fleet/spool.key" call_push = "exceptions" exception_max_per_sec = 10
-
call_push = "exceptions"sends a call that ended on a final error response, or whose media diagnosis found one-way audio, no media, a NAT mismatch, an asymmetry between the two directions, late media or any other finding. The record says which inselected_by. -
call_push = "all"sends every ended call.call_push = "none"sends none. -
exception_max_per_sec(default 10,DEFAULT_EXCEPTION_MAX_PER_SEC; at most 10000,MAX_EXCEPTION_MAX_PER_SEC) caps the exceptions. The member keeps the unused allowance of at most 10 seconds (CAP_BURST_SECS), counts every call the cap refused, and does not send that call later.allis not capped.
The member's start line names the rule:
INFO sipnab::app::fleet_member: fleet member reporting to http://127.0.0.1:18444/ every 5 s, spool '/var/lib/sipnab/fleet-spool', call records: exceptions -
The member looks at its dialogs every 5 seconds. It sends a call once the
call has ended and has stood unchanged for 5 seconds, so the record is the
answer the call settled on. When a call changes after the member sent it,
such as one whose BYE arrives late, the member sends it again as it then
stands. One delivery holds at most 500 records (MAX_DELIVERY_RECORDS) and
2 MiB (MAX_DELIVERY_BYTES). The member deletes a delivery when the center
acknowledges it.
Verify.
-
[center]
GET /v1/fleet/memberscounts what the center stored and what the member says it did.verified.call_recordsis the center's own count for the member.self_reported.callsis the member's:selectedcalls passed the rule and the cap,dropped_by_capdid not pass the cap,left_outwere too large for a delivery, and the member wrotespooledof them for delivery:"intake": {"bytes": 944, "call_records": 1, "removed": 0} "verified": {"call_records": 1, "heartbeats": 3, "peer": "127.0.0.1", "received_at": "2026-11-10T15:44:12Z"} "calls": {"dropped_by_cap": 0, "left_out": 0, "selected": 1, "spooled": 1}The capture held two ended calls. The member sent the one that got a
486. -
[center] The intake is a sealed file. Its first four bytes are
SNFS:head -c 4 /var/lib/sipnab/center.store.intake; echo
SNFS -
[member] After the center acknowledged a delivery, the spool holds no file that ends in
.sealed.
What it covers and what it does not.
- A record the center acknowledged is on the center's disk before the
acknowledgment, and survives a
kill -9of the center. The center counts a record that the member sends again as a duplicate and does not store it twice, also after a restart. - A member keeps no database. A call the rule did not select, or one the cap refused, is at the center only as a count.
- The summary of a call measured over three checked-in captures is 583, 1,283 and 2,592 bytes of JSON. Raw SIP, RTCP and rtpengine control messages are not in the record.
- When the spool is past
[fleet] spool_max_files, the oldest file goes first, a delivery of call records like a heartbeat. - A record's
sent_atand itscallare the member's claim. The member's name comes from its token, which the center verified.
Troubleshoot.
-
needs [fleet] spool_key_file.call_pushasks for records and the member has no key to seal them under. The run stops at its start. Create the key and name it, as in steps 1 and 2:[fleet] call_push "all" needs [fleet] spool_key_file: call records wait on disk only in sealed files. Create a key with `sipnab fleet keygen --out FILE` -
can be read or written by other users, naming the spool key file. The member does not report until you runchmod 600on the file and restart it. The capture goes on. -
sealed layer:, then a file name, in the member's log. A delivery in the spool did not open: something changed the file, or the spool key is not the one that sealed it. The member moves it tofailed/and never sends it. Restore the key that sealed the file to send it, or delete the file. -
verified.call_recordsstays at 0. Check the member's start line. It sayscall records: nonewhen the member has no spool key orcall_pushisnone. Underexceptions, a capture with no failed or degraded call sends nothing. -
dropped_by_capgrows. The member sees more exceptions thanexception_max_per_seclets through. Raise the cap, or read the calls on the member, which still holds them in memory. -
The center logs
call record(s) past --retention-hours. The intake removed its oldest records, by age or because it held 64 MiB. Lower the members' cap, or shorten--retention-hours.
A heartbeat is one JSON record per line, POSTed to /v1/fleet/ingest with
the member's token:
| Field | Meaning |
|---|---|
schema_version |
1. |
kind |
heartbeat. |
stream |
The member's delivery stream: 1 to 64 letters, digits, ., _ or -, chosen once. |
seq |
The record's place in the stream, from 1. |
sent_at |
When the member wrote it, RFC 3339. |
heartbeat_secs |
How often the member sends one, 1 to 3600. |
inventory |
The member's GET /v1/inventory answer. |
The center counts a record once. In the same stream a record is new when its
seq is past the newest one the center accepted. A record of another stream
is new when its sent_at is later: a member that recreates its spool starts
a new stream. The center counts a batch sent twice once, also after it
restarts.
-
[member] Send
edge-a's inventory as a heartbeat:# Run all of these, in order. curl -s -H "Authorization: Bearer $EDGE_A_INVENTORY_TOKEN" http://127.0.0.1:18080/v1/inventory \ | jq -c '{schema_version: 1, kind: "heartbeat", stream: "edge-a-manual", seq: 1, sent_at: (now | todate), heartbeat_secs: 30, inventory: .}' \ > heartbeat.ndjson curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $EDGE_A_REPORT_TOKEN" \ -H 'Content-Type: application/x-ndjson' --data-binary @heartbeat.ndjson \ https://127.0.0.1:8444/v1/fleet/ingest
{"accepted":1,"duplicate":0,"member":"edge-a"}Sent again, the same file is a duplicate:
{"accepted":0,"duplicate":1,"member":"edge-a"}
-
[controller] Read the members:
curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $FLEET_READER_TOKEN" \ https://127.0.0.1:8444/v1/fleet/members | jq 'del(.members[].self_reported.inventory)'
{ "members": [ { "member": "edge-a", "seconds_since_heartbeat": 0, "self_reported": { "heartbeat_secs": 30, "sent_at": "2026-11-07T15:41:36Z", "seq": 1, "stream": "edge-a-manual" }, "silent": false, "verified": { "heartbeats": 1, "peer": "127.0.0.1", "received_at": "2026-11-07T15:41:36Z" } } ], "now": "2026-11-07T15:41:36Z", "retention_hours": 72, "schema_version": 1, "silent_after_missed": 3 }-
memberis the id of the token the heartbeat came with. The center verified it. -
verifiedis what the center saw: when the newest heartbeat arrived, on the center's clock, from which address, and how many heartbeats it accepted from this member. -
self_reportedis what the member says about itself: its stream, place, send time, interval andinventory, the record in GET /v1/inventory. A member that is not honest can put anything there. -
silentistrueonce more thansilent_after_missedof the member's intervals passed with no heartbeat, measured on the center's clock.
-
-
[center] The center logs a member going silent, and coming back. In this run
edge-bsent one heartbeat with a 1 s interval:WARN sipnab::app::fleet_center: fleet member 'edge-b' is silent: no heartbeat for 3 s, more than 3 of its 1 s intervals INFO sipnab::app::fleet_center: fleet member 'edge-b' is heard again
-
[center] Start the center with
--revoked-file, and add the member's token id to that file, one per line. sipnab re-reads the file when it changes, so the center keeps running. This run revokededge-a:echo edge-a >> /etc/sipnab/center.revoked
edge-a's next heartbeat gets401, andedge-b's still gets200. The members list keepsedge-a's last heartbeat until retention removes it.
Purpose. The center's store holds every member's inventory: node names, versions, configuration digests, addresses. The center never writes it in clear. It seals the store file with AES-256-GCM under a key that you keep in a file, and it refuses to start without that key. A center cannot run with encryption off.
The outputs below come from a run on 2026-11-10 on one host, with a development build after 0.5.243 and the files in one working directory.
Before you start.
-
A sipnab after 0.5.243 on the center. Members need no change, and no other system does: OpenSIPS, Kamailio, Homer and any HEP collector keep their configuration.
-
A place to back up one small file, apart from the center's disk. Whoever holds the key file can read the store. Without the key file nobody can.
-
A center from 0.5.237 to 0.5.243 wrote its store in clear and took no key. After the upgrade it does not start until you give it
--store-key-file. Upgrade a center that stored in clear has the steps and the way back. -
A center from 0.5.244 or 0.5.245, or a member from 0.5.245 wrote sealed format 1. A later release reads those files once and writes them again as format 2, and neither 0.5.244 nor 0.5.245 reads format 2. To keep a way back, stop the center and copy its store and its intake before the first start of the new release. A center from 0.5.244 has no intake, so only the first command applies to it:
# Run all of these, in order. cp -p /var/lib/sipnab/center.store /var/lib/sipnab/center.store.format1 cp -p /var/lib/sipnab/center.store.intake /var/lib/sipnab/center.store.intake.format1To go back, stop the center, put the copies back under their own names and start the release that wrote them. It then knows nothing the newer release stored. A member has nothing to copy: let its spool drain before a downgrade, because 0.5.245 moves a format 2 delivery to
failed/and does not send it.
Steps.
-
[center] Create the key. The file gets mode
0600. The id it prints is not secret: the center logs it, and a refusal names it.sipnab fleet keygen --out /etc/sipnab/center.storekey
store.key: key id ac7b02adecf75d31 -
[center] Copy
/etc/sipnab/center.storekeyto your backup, and note the key id beside it. -
[center] Start the center with
--store-key-file, as Step 1 shows, or add the flag to the service unit of a center you upgraded. -
To replace the key, create a second key file, and start the center once with the new file as
--store-key-fileand the old one as--store-old-key-file:# Run all of these, in order. sipnab fleet keygen --out /etc/sipnab/center.storekey.2 sipnab fleet serve --listen 127.0.0.1:8444 --store /var/lib/sipnab/center.store \ --store-key-file /etc/sipnab/center.storekey.2 \ --store-old-key-file /etc/sipnab/center.storekey \ --signing-key-file /etc/sipnab/center.signing \ --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.keyINFO sipnab::app::fleet_center: fleet center store encryption sealed, key id dd70bddbd340fae5, 1 old key(s)The center rewrites the store file under the new key as it starts. It does not rewrite the files of the history store: each history file stays under the key that wrote it until retention removes it. That is
--calls-retention-days(default 90) for call records and--retention-hours(default 72) for heartbeats. Back up the new key file.-
A center without the history store (its
--versiondoes not listfleet-store): from the next start on, leave--store-old-key-fileout and destroy the old key. -
A center with the history store: keep the old key file and
--store-old-key-fileon every start untilsipnab fleet verifywith the new key alone refuses no file (Verify the history store). That takes until retention has removed every history file written under the old key: the longer of the two periods after the rotation, 90 days with the defaults. Only then destroy the old key. A center that starts without the old key runs, and refuses each of those files withwas written under key id <id>, which is not loaded(Keep call history at the center has the entry). Nothing reads a history file after you destroy its key.
-
A center without the history store (its
Releases 0.5.237 to 0.5.243 wrote the store in clear. To upgrade such a center:
-
[center] Stop the center. To keep a way back that loses no member, copy the store aside, and treat the copy as every member's inventory in clear:
cp -p /var/lib/sipnab/center.store /var/lib/sipnab/center.store.clear
-
[center] Install the new sipnab, create the key as in step 1 above and back it up.
-
[center] Add
--store-key-file /etc/sipnab/center.storekeyto the center's command line or service unit, and start it. On this first start the center reads the clear store, writes a sealed copy beside it, waits until that copy is on disk, reads it back, and only then puts it in the store's place. A stop or a failure before that leaves the clear store as it was. The center logs one warning that the storeheld <n> member(s) in clearandis sealed now. The old bytes may stay in the disk's free space until the file system reuses it. -
[center] When the members list is right, delete the copy from step 1.
To go back to 0.5.243 or earlier: stop the center, and put the copy from
step 1 back as the store, or move the sealed store away to start with an
empty one. Do this before the older release starts: it cannot read a sealed
store, and it either stops at start or starts with no members and writes
over the file. Then install the older sipnab and start it without
--store-key-file. With an empty store, the center lists each member again
at its next heartbeat, and does not list a member that is silent then.
Verify.
-
The center's first log line names the key id that
keygenprinted. -
The store is a sealed file. Its first four bytes are
SNFS, and no member's name is in it:# Run all of these, in order. head -c 4 /var/lib/sipnab/center.store; echo grep -c edge-a /var/lib/sipnab/center.store
SNFS 0 -
GET /v1/fleet/memberslists the same members after a restart.
What it covers and what it does not. The center seals the store file and
the file it stages a rewrite in, and writes no other file. It authenticates
each record with the record's position in the file, so a changed byte, a
moved record or a store cut short stops the center at start. That includes
records removed from the end of the file: after each append is on disk the
center writes an authenticated count of the file's entries into the file's
header, and the center refuses a file that holds fewer entries than its
count (was cut at its end). The same holds for the intake beside the store
and for a member's sealed deliveries. Two things it cannot detect: someone
deleting the whole store, and someone putting back an earlier copy of the
whole file, which is a valid file with its own valid count. Releases 0.5.244
and 0.5.245 wrote no count (sealed format 1). This release reads such a file
once at start and writes it again with one (format 2), and an earlier
release does not read a format 2 file. --store-encryption names how the center encrypts
the store. This release has one mechanism, sealed, and refuses the values
modular and both.
Troubleshoot. Each message stops the center with exit 2, and none
shows a key.
-
--store-key-file is required. The center has no key. Create one and pass it:sipnab fleet serve: --store-key-file is required: the center encrypts everything it stores. Create a key with `sipnab fleet keygen --out FILE` and pass --store-key-file FILE -
which is not loaded, after a key id. The store is under another key than the one given. Give the file of the key with that id as--store-key-file, or as--store-old-key-filebeside a new key. The center leaves the store as it found it:sipnab fleet serve: --store 'center.store': sealed layer: the file was written under key id ac7b02adecf75d31, which is not loaded; give that key's file with --store-old-key-file -
did not authenticate,was truncated, orwas cut at its end. Something changed the store or cut it short after the center wrote it. The center leaves it as it found it. Restore the store from a copy, or move it away and start with an empty store: the members report again within one heartbeat interval each, and the list loses only a member that is silent now. When the message names the intake (center.store.intake), the same two choices apply to that file. An empty intake holds none of the call records the old one held.sipnab fleet serve: --store 'center.store': sealed layer: the file has a header that did not authenticate (the file was modified, or the key is wrong)A store with records removed from its end:
sipnab fleet serve: --store 'center.store': sealed layer: the file was cut at its end: it holds 3 of the 5 entries it was written with -
has a record <n> that did not authenticate, on a file no one changed. A center from 0.5.244 to 0.5.247 left this state when an append failed, for example on a full disk, and a later append worked. The center loggedfleet center: the store could not be written:at the time and kept running. The center sealed the record after the failure for a position the file does not reach, so the file does not open at the next start. This release does not leave a file in that state. It does not read one that is in it:sipnab fleet serve: --store 'center.store': sealed layer: the file has a record 4 that did not authenticate (the file was modified, or its records were moved)Which file the message names decides what the center loses.
- The store (
--store '<file>'). Those releases wrote the store again in each minute in which a heartbeat arrived and when the center stopped on a signal, and that removed the state. A store is in it only when the center stopped another way before the next rewrite. Move the store away and start the center. It starts with an empty store. Each member reports again within one heartbeat interval, and the list loses only a member that is silent now. - The intake (
the intake '<file>.intake', from 0.5.245 to 0.5.247). Those releases wrote the intake again only to remove records past the retention or the intake's size, and that rewrite failed on a file in this state, with the warningfleet center: the intake was not rewritten:. Move the intake away and start the center. It starts with an empty intake and holds none of the call records the old file held. The members do not send them again.
Keep the file you moved away until you have checked the center: start it, wait one heartbeat interval, and read
GET /v1/fleet/members. No sipnab command reads a part of a sealed file that does not open. - The store (
-
can be read or written by other users, naming the key file. Runchmod 600on it:sipnab fleet serve: --store-key-file 'store.key' can be read or written by other users (mode 0644); chmod 600 it -
is not a key. The file does not hold 32 bytes in base64. Use the filekeygenwrote, unchanged:sipnab fleet serve: --store-key-file 'bad.key' is not a key: it holds 3 bytes, and a key is 32; create a key with `sipnab fleet keygen --out FILE` -
File exists, fromkeygen.keygennever overwrites a file, because it may be the only copy of a key. Name a new file:sipnab fleet keygen: --out 'store.key': File exists (os error 17) -
this build cannot read or write Parquet modular encryption. Leave--store-encryptionout, or set it tosealed:sipnab fleet serve: --store-encryption modular: this build cannot read or write Parquet modular encryption; `sealed` is the one value it runs under
Purpose. A center with the history store keeps the call records its
members push for --calls-retention-days (default 90), and answers
GET /v1/fleet/calls: the calls of one member, of a time window, or the call
with one Call-ID. Without it, a record stays in the intake for
--retention-hours and no route reads it.
The center files the records into Parquet files under a directory beside its
store, <store>.parquet. It seals each file with the store key before the
file reaches the disk, so no call record is on the center's disk in clear.
The outputs below come from a run on 2026-11-11 on one host, with a
development build after 0.5.247: a center on 127.0.0.1:18447 over HTTP on
loopback, with --calls-flush-secs 5, and two call records sent by hand as
member edge-a. The commands name files as a deployment would, and the log
lines show file names without the run's directory. A second run, on
2026-11-12 with a development build after 0.5.249, gave the lines that
changed in that build: the history store's start line, the history object
of the members report (one call record and two heartbeats filed), and the
answer for a changed file. The answer of step 3 shows files_not_read as
that build answers it. The WARN line of a build without the history store
is the text such a build logs. The test suite starts such a center in each
build without the history store and compares its log line and its 501
answer with this page.
Before you start.
-
A build with the history store.
sipnab --versionlistsfleet-storeamong its features. The gnu tarballs, the.deband.rpmpackages that are not-noaudio, the macOS builds and the container image have it. The static musl tarballs and the-noaudiopackages do not: the Parquet library is larger than the room the static binary has under its size limit. A center from one of those says so when it starts, and the route answers501:WARN sipnab::app::fleet_center: fleet center: this build has no history store (cargo feature fleet-store): call records wait in the intake, which holds at most 64 MiB and --retention-hours of them, no heartbeat history is kept, and GET /v1/fleet/calls and GET /v1/fleet/members/{id}/history answer 501. The gnu tarballs, the .deb and .rpm packages that are not -noaudio, the macOS builds and the container image have the history store; the static musl tarballs and the -noaudio packages do not -
A center that takes call records, and members that push them, as Push call records to the center sets up. The member needs no change for the history store.
-
The store key and its backup, as Encrypt the center's store describes. The center seals the history files under the same key. Without the key no one reads them.
-
A token that reads calls.
GET /v1/fleet/callsneeds scopefull. A call record holds the numbers and addresses of a call, so aninventorytoken does not read it. -
Room on the center's disk. The center compresses the files with zstd. In a run of the tests, 20,000 call records of about 0.8 kB each made one file of 660,654 bytes; releases 0.5.249 to 0.5.253 wrote the same records without compression as 16,493,345 bytes. See Upgrade a center to compressed history files for what the compression gives on other rows.
-
Other systems. Nothing changes on OpenSIPS, Kamailio, Homer or any HEP collector. A backup of the center now includes the directory
<store>.parquet.
Steps.
-
[center] Start the center with the retention you want. Both flags are optional:
sipnab fleet serve --listen 127.0.0.1:8444 --store /var/lib/sipnab/center.store \ --store-key-file /etc/sipnab/center.storekey \ --signing-key-file /etc/sipnab/center.signing \ --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.key \ --calls-retention-days 90 --calls-flush-secs 60
INFO sipnab::app::fleet_center: fleet center history store 'center.store.parquet': call records are kept 90 day(s), heartbeats 72 hour(s)-
--calls-retention-daystakes 1 to 548. The center removes the days past it at start and every minute. A day is the UTC day on which the center received the records. -
--calls-flush-secstakes 1 to 3600. It is the time between two filings of the intake. The center also files when it starts and when it stops on a signal.
-
-
[center] Mint a token that reads calls:
sipnab --mint-token --token-scope full --token-id fleet-ops \ --api-signing-key-file /etc/sipnab/center.signing --api-token-ttl 86400
-
[controller] Ask for calls. With no parameter the answer is the newest 100 records:
curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $OPS_TOKEN" \ 'https://127.0.0.1:8444/v1/fleet/calls?member=edge-a&call_id=busy-1%40192.0.2.10' | jq .
{ "calls": [ { "call": { "call_id": "busy-1@192.0.2.10", "final_status": 486, "from": "sip:+12025550101@example.com", "state": "failed", "to": "sip:+12025550102@example.com" }, "member": "edge-a", "received_at": "2026-11-11T23:54:37.489276093Z", "selected_by": "failed", "sent_at": "2026-11-11T23:54:37Z", "seq": 1, "stream": "edge-a-manual" } ], "complete": true, "count": 1, "files_not_read": 0, "files_read": 1, "more": false, "refused": [] }The parameters, each optional:
Parameter Meaning memberRecords of this member only. fromRecords the center received at this RFC 3339 time or later. toRecords the center received before this RFC 3339 time. call_idThe record of the call with this Call-ID. Percent-encode @as%40.limitThe most records to return, 1 to 1000 ( MAX_LIMIT). Default 100. The newest come first.moreistruewhen the limit cut the answer. Narrow the window or raiselimitto read the rest. A record is in the answer from the moment the center acknowledges it, before the center files it.One question opens at most 4,096 files (
MAX_FILES_PER_QUESTION), those with the latest records first. When a question rules in more, the answer holds what those files gave,completeisfalse, andfiles_not_readcounts the files left. The center writes one file for each filing and day, which holds every member's records of that filing, so 4,096 files are about 2.8 days at the default--calls-flush-secs 60(4,096 minutes), whatever the number of members. To reach further back with one question:- Give
fromandto. They rule files out by the day in the path and the times in the name, so the center answers a window of up to 2.8 days whole, anywhere in the retention. Ask for a longer time one window at a time. - Or start the center with a longer
--calls-flush-secs. At600one question reaches 28 days (4,096 times 10 minutes). Records then wait up to that long in the intake before the center files them; a question reads the waiting records too.
memberdoes not rule a file out: the center opens each file of the window and reads only the part that can hold that member's records. - Give
Verify.
-
[controller]
GET /v1/fleet/membersreports what the history store did since the center started. After one filing of one call record and one of two heartbeats:curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $READER_TOKEN" \ https://127.0.0.1:8444/v1/fleet/members | jq -c '.history, .intake'
{"available":true,"failed_filings":0,"files_written":1,"heartbeat_files_written":1,"heartbeats_filed":2,"heartbeats_removed":0,"heartbeats_waiting":0,"members_restored":0,"retention_days":90,"rows_filed":1} {"bytes":0,"call_records":0,"removed":0}files_writtenandrows_filedcount call records. Theheartbeatfields andmembers_restoredare those of Keep heartbeat history at the center. -
[center] The center logs each filing:
INFO sipnab::app::fleet_center: fleet center: filed 2 call record(s) in 1 file(s) of the history store; 0 were in it already -
[center] The files are under class and day: one file for each filing, which holds the records of every member. Each name holds the first and the last receive time in the file, in milliseconds, and
hwith 16 hex digits that tell the file's records from any other records of the same times:cd /var/lib/sipnab && find center.store.parquet -type f
center.store.parquet/calls/date=2026-11-11/1794355201000-1794355203000-he7b857d7ce5986b7.sealed -
[center] The center sealed each file. Its first four bytes are
SNFS, not thePAR1of a Parquet file in clear:head -c 4 center.store.parquet/calls/date=2026-11-11/*.sealed; echo
SNFSThe directories have mode
0700and the files mode0600.
Troubleshoot.
-
401onGET /v1/fleet/calls. The token's scope is notfull. Aninventory,metricsorfleet-reporttoken does not read call records. Mint afulltoken, as step 2 does. -
400with adetail. The detail names the parameter and what it accepts:{"detail":"limit must be a number from 1 to 1000","status":400,"title":"Bad Request","type":"https://sipnab.com/problems/bad-request"} {"detail":"\"membr\" is not a parameter of this route; it takes member, from, to, call_id and limit","status":400,"title":"Bad Request","type":"https://sipnab.com/problems/bad-request"} -
501andthis build has no history store. The center runs a build without the history store. Install a build that has it, as Before you start lists, and start the center on the same store. The center files the records still in the intake at that start. -
"complete": falseand arefusedlist. The center could not read a file the question needed. The answer holds the records of the other files. Each entry names the file under<store>.parquetand the reason:{ "calls": [], "complete": false, "count": 0, "files_not_read": 0, "files_read": 0, "more": false, "refused": [ { "file": "calls/date=2026-11-11/1794355201000-1794355203000-he7b857d7ce5986b7.sealed", "reason": "sealed layer: the file has a record 2 that did not authenticate (the file was modified, or its records were moved)" } ] }In 0.5.249 and 0.5.250 the reason started with a second copy of the path.
-
did not authenticate,was truncatedorwas cut at its end: the file changed after the center wrote it. Restore the file from a backup, or move it out of the directory. The records in it are then not answered. -
was written under key id <id>, which is not loaded: the file is under an earlier store key. Start the center with that key in--store-old-key-file. -
was written as '<path>' and is not at that path: someone moved or renamed the file. Move it back to the path the message gives.
-
-
"complete": falseandfiles_not_readabove 0. The question ruled in more files than one question opens. Ask again withmember,fromandto, for a part of the time at a time. -
The center logs
the call records were not filed and stay in the intake. The line ends with the cause: a directory the center cannot write, a full disk, or a file of the same day that it cannot read. The records wait in the intake, and the center tries again after--calls-flush-secs.failed_filingsinGET /v1/fleet/memberscounts the tries that failed. Fix the cause before the intake reaches its 64 MiB: past that the oldest waiting records go first. -
The center logs
past --calls-retention-days <n> and not removed. The line names a file in an expired day that the center did not write, or one the system refused to remove. The center leaves such a file and its directory. Remove it by hand. -
What the history store cannot show. A whole file that someone deleted leaves no trace in the store, as for the store file itself. An answer does not include records the intake dropped at its size limit before a filing.
Purpose. Know what a member, a reader and a supervisor get from a center between the time it binds its port and the time it serves, and what a stop does in that time.
Before you start. Nothing to set: every center does this. Nothing changes on a member.
What the center does at start, in order:
-
It reads its keys, binds
--listen, and drops to the user it serves as. -
It installs its SIGTERM and SIGINT handlers.
-
It answers every request, on every route,
503withRetry-After: 5and this problem body, and logs where it does. Both outputs come from a test center with 500 members in 120 heartbeat files and no store file, on a port the system picked:503 {"detail":"the center is rebuilding its member list from the history store","status":503,"title":"Service Unavailable","type":"https://sipnab.com/problems/unavailable"}INFO sipnab::app::fleet_center: fleet center answering 503 on 127.0.0.1:33407 until it serves: the center is rebuilding its member list from the history store -
It opens its store file and its history, rebuilds its member list, files what the intake and the store file held, and applies retention.
-
It logs
fleet center serving HTTP on <address>(orHTTPS) and answers each request from then on.
A start with its store file reads one heartbeat file in step 4. A start
without it reads every heartbeat file of --retention-hours, newest first
(The files of the history store has the
time for a fleet). A build without the history store answers the center is opening its store in step 3.
What a member does with the 503. It keeps the heartbeat or the call
records in its spool and logs heartbeats wait in the spool: the center answered HTTP 503 once. It tries again after 1 s, then 2 s, 4 s and so on,
never more than one [fleet] heartbeat_secs apart, and delivers the spool
once the center serves. It does not read Retry-After.
A stop during the start. A SIGTERM or SIGINT during the rebuild in step 4 ends the rebuild before the next file it would read. The center logs the two lines below and exits 0. It did not rewrite its store file with the rebuilt list and did not open its intake, so the store file holds the members it held before the start, and the next start rebuilds again:
INFO sipnab::app::fleet_center: fleet center stopping: a stop was asked for before it served; it ends at the next file of the rebuild, or once the start's filing has ended
INFO sipnab::app::fleet_center: fleet center: the stop ended the rebuild of the member list at start; the store was not rewritten
A stop after the rebuild, while the start files, waits for that filing to end. The center then rewrites its store as it does at any stop, and exits 0. Releases up to 0.5.258 opened the store before they installed the handlers: a SIGTERM during the rebuild ended the process at once, and a request waited unanswered until the rebuild ended.
Verify. Move a test center's store file aside, start the center with a history that holds many heartbeat files, and ask for the members at once:
curl -s -i -H "Authorization: Bearer $OPS_TOKEN" http://127.0.0.1:8444/v1/fleet/membersThe answer is 503 with Retry-After: 5 until the center logs fleet center serving, and 200 with every member after it.
Troubleshoot.
-
A supervisor marks the center failed during a long start. Its
readiness check gets
503. Wait for200, or for thefleet center servingline, before you mark the center up. -
The
503lasts longer than you expect. The center has no store file, or its store file is older than its newest heartbeat files, so it reads the heartbeat files of the whole window. Keep the store file in your backups with the history.
Purpose. Know which requests the center answers while it files, and which steps of a filing make a request wait.
The center keeps its members, its store file and its intake behind one lock.
A push, GET /v1/fleet/members and the first step of a question each take
that lock for a short time.
A filing of call records has three steps:
- Take. Under the lock, the center opens the intake file and reads its header and its length. It reads no record.
- Read and write. Without the lock, the center reads the records up to that length, parses them and writes the files of the history store. This is the step that takes time. The center answers pushes, status requests and questions during it. A record that arrives now goes to the intake and waits for the next filing.
- End. Under the lock, the center writes the intake again with only the records that arrived during step 2. It kept their lines in memory, so it does not read the intake file for this.
A filing of heartbeats has the same three steps over the rows that wait in memory. The releases 0.5.249 to 0.5.256 held the lock for all three steps of a filing of call records, so nothing got an answer until the filing ended.
Measured in a release build (Cargo feature set full, aarch64, load average
13 to 22, three runs) with a full intake of 54,000 synthetic call records
(67,412,988 bytes before sealing): steps 1 and 3 together held the lock 19
to 22 ms when no record arrived during the filing, and 28 to 30 ms when
2,000 records arrived during it. Step 1 alone took 23 to 30 microseconds.
Step 2 took 495 to 511 ms without the lock. In the same binary, the three
steps as one under the lock took 607 to 633 ms.
What still waits:
- One filing at a time. A second filing, of either class, and the retention pass on the history store wait until the running filing ends. They hold no lock that a request needs while they wait.
- A push waits for another push. The center writes each push to disk under the lock before it answers.
- Steps 1 and 3 of a filing, and the rewrite of the store file and of the intake by retention, hold the lock while they run.
GET /v1/fleet/calls opens the intake under the lock in the same way and
reads the waiting records without it. A question asked between step 2 and
step 3 finds a record both in a file and in the intake, and the answer
shows it once.
A stop during a filing. A center that stops between step 2 and step 3
leaves the records in the files and in the intake. The next filing, at the
next start, finds them in the files and files none of them twice. The line
filed 0 call record(s) in 0 file(s) of the history store; N were in it already reports that.
Why a filing reads the intake safely while pushes arrive. A push adds its records after the length that step 1 took, and step 2 reads no byte past that length. A push also writes one of the two count slots in the file's header, and step 2 does not read the header again: step 1 read it while no push was writing. That matters because the center accepts a file with one count slot that does not check only when exactly one whole addition follows the counted records. A reader that saw a count slot half written, with a second push already started, would refuse a file that is whole. The filing's reader cannot see that state, so the center does not try a refused read again: a refusal is a real one. When retention or a repair writes the intake file again, it writes a new file and renames it into place. Step 2 keeps the file it opened, which nothing writes to.
Purpose. Know what the center writes under <store>.parquet, how many
files that is, what a person who can list the directory learns without the
store key, and what to do with the files release 0.5.249 wrote.
The layout.
<store>.parquet/calls/date=YYYY-MM-DD/<first>-<last>-h<mark>.sealed
<store>.parquet/members/date=YYYY-MM-DD/<first>-<last>-h<mark>.sealed
-
callsholds call records andmembersholds heartbeats. The day is the UTC day on which the center received the rows. - The center writes one file for each filing and day. The file holds the
rows of every member that the center received since the filing before.
The number of files does not depend on the number of members: at the
default
--calls-flush-secs 60a center writes at most 1,440 files a day in each class, for 1 member and for 2,000. A filing with no new rows writes no file. -
<first>and<last>are the earliest and the latest receive time in the file, in milliseconds since 1970.<mark>is 16 hex digits computed from the rows; it keeps two files of the same times apart. - With 90 days of call records and 72 hours of heartbeats at 60 seconds, a center holds at most 129,600 call record files and 5,760 heartbeat files (the heartbeat retention removes whole days, so up to four days are on disk).
What a file holds. Inside the seal, each file is one Parquet file, and
zstd at level 1 compresses each of its columns. Releases 0.5.249 to 0.5.253
wrote the files without compression, and this release reads those in place:
see Upgrade a center to compressed history files.
The column member is the member's id. The rows are in
order of member and then of receive time. A row group ends where the member
changes once it holds 128 rows, and at 10,000 rows at the latest, and each
row group records its least and its greatest member. A question for one
member opens each file of its window and decodes only the row groups that
can hold that member: in a file with one row for each of 2,000 members, 1
row group of 16.
What follows from that layout:
- A question for one member opens every file of its window, and the center unseals each of those files whole before it picks the row group. In the row group it reads the receive time, the member and, for a question that names a call, the Call-ID of every row, and it builds only the rows the question selects: it does not parse the JSON of the other rows.
- One answer holds at most 1,000 rows (
limit). A member's 24 hours at one heartbeat a minute are 1,440 rows: ask twice, withfromandto. - A center that starts without its store file reads the heartbeat files from the newest back, up to 4,096 of them, to rebuild its member list. A center that starts with its store file reads one.
What the reads cost. Measured on 2026-11-14 in a release build (Cargo
feature set full, aarch64, 14 cores, load average about 10) on synthetic
rows: 1,440 heartbeat files, each with one heartbeat of about 300 bytes of
inventory for each of 2,000 members, compressed, 78,860,456 sealed bytes in
the 1,001 files of the first question. Each figure is the range of three
runs.
| What the center did | Time |
|---|---|
One member's newest 1,000 heartbeats, from 1,001 files (GET /v1/fleet/members/{id}/history) |
226 to 230 ms |
| Of that: reading the 1,001 files from disk | 28 to 43 ms |
| Of that: unsealing them | 41 to 42 ms |
| Of that: decoding the one row group of each file that holds the member | 158 to 162 ms |
| The same decoding as releases 0.5.249 to 0.5.254 did it, building all 128 rows of the row group | 586 to 656 ms |
What a start without the store file costs. The center rebuilds its
member list from the heartbeat files, newest first, and builds a row only
when it holds no row of that member yet or the row is newer than the one it
holds. Measured on 2026-11-15 in one release build (Cargo feature set
full, aarch64, 14 cores, load average 23 to 31) on the same kind of
synthetic store, 1,440 files of 113,477,614 sealed bytes, three runs:
| How the rebuild read the 1,440 files | Rows built | Time |
|---|---|---|
| As this release does it | 2,000, one for each member | 1.95 to 1.99 s |
| As releases 0.5.251 to 0.5.258 did it, in the same binary | 2,880,000, every row | 12.4 to 12.8 s |
The center answers 503 for that time
(What the center answers while it starts).
A start with its store file reads one file.
Which reads build every row, and what a damaged row does. The center
seals each file, so a changed file does not open at all. A row whose JSON,
send time or text does not parse can only come from a defect in the release
that wrote the file. A question refuses such a file, and names it under
refused, only when it selects that row: a question for another member, for
another call or for a window that does not hold the row gets its answer
from the same file. Releases 0.5.249 to 0.5.254 refused the file for every
question that read the row's row group. The rebuild of the member list at
start refuses a file only for a damaged row it builds: a damaged row older
than the row it holds of that member is not built, and the start takes the
file's other rows. Releases 0.5.251 to 0.5.258 refused the file at start for
such a row too. One read still builds every row of each file it opens and
refuses the file for one such row anywhere in it: the check a filing makes
of the files of the same day that can already hold its rows. Neither read
opens every file of the store. sipnab fleet verify does: it builds every
row of every file and reports each file it refuses
(Verify the history store). A member id that is not text, or a column
that the center cannot read, refuses the file for every question that reads
its row group, as before.
Reading the files with other tools. No other tool reads a file as it is
on disk: the center seals every file under the store key, and the first
bytes are SNFS, not PAR1. Read the history through the center, which
answers JSON that jq, a spreadsheet import or a script takes:
curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $OPS_TOKEN" \
'https://127.0.0.1:8444/v1/fleet/calls?from=2026-11-11T00:00:00Z&to=2026-11-12T00:00:00Z&limit=1000' \
| jq -c '.calls[] | {member, received_at, call_id: .call.call_id}'sipnab has no command that writes a file's Parquet in clear.
What the directory shows without the key. A person who can list the
directory sees the days that have rows, the time of each filing to the
millisecond, and the size of each file, which grows with the rows of that
filing. The names hold no member id, no count and no part of a record. In
release 0.5.249 each member had a directory member=<id>, so a listing
showed every member's id and the times at which each member had new records.
Files that release 0.5.249 wrote. Releases 0.5.249 and 0.5.250 wrote
call records under calls/date=YYYY-MM-DD/member=<id>/, one file for each
member and filing. What this page says of the files of release 0.5.249 is
true of the files of 0.5.250 too. This release writes no file there and does not move or rewrite the
files that are there:
-
GET /v1/fleet/callsanswers their rows together with the rows of the new files. A question withmemberopens only that member's old files. - The retention removes them with their day, and the log line that counts the removed files counts them.
- A start after 0.5.249 stopped between a filing and the emptying of its intake does not file those records a second time.
- The member ids in those paths stay on disk until the day passes
--calls-retention-days.
Count the files, and check the inodes. A center of release 0.5.249 or 0.5.250 with many members wrote many small files: one for each member with a new call record in each filing, which for 2,000 such members at 60 seconds is 2,880,000 files a day. A file system holds a fixed number of inodes, and a center that runs out of them cannot file.
-
[center] Count the files of the history store:
find /var/lib/sipnab/center.store.parquet -type f | wc -lAnd the files of the 0.5.249 layout among them:
find /var/lib/sipnab/center.store.parquet/calls -path '*/member=*' -type f | wc -l
-
[center] Read how many inodes the file system has left (
IFree, andIUse%):df -i /var/lib/sipnab
-
[center] When
IUse%is near 100, lower the retention so that the center removes the oldest days. The center removes the days past the retention at start and every minute:sipnab fleet serve --listen 127.0.0.1:8444 --store /var/lib/sipnab/center.store \ --store-key-file /etc/sipnab/center.storekey \ --signing-key-file /etc/sipnab/center.signing \ --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.key \ --calls-retention-days 7
The center logs a line of this form with the days and files it removed, and
df -ithen shows the inodes as free:INFO sipnab::app::fleet_center: fleet center: 1 day(s) and 2 file(s) of call records past --calls-retention-days 7 were removedThe center no longer has the call records of the removed days. To keep them, copy the
date=directories to another file system first. The center reads a copied directory again once it is back at its path.
Troubleshoot.
-
The center logs
the call records were not filed and stay in the intakewithNo space left on devicewhiledf -hshows free space. The file system has no free inode. Do step 3. The records wait in the intake, at most 64 MiB of them, and the center files them when it can. -
A listing still shows
member=directories after the upgrade. They are the files of release 0.5.249. The center reads them and removes them when their day passes the retention.
Purpose. sipnab fleet verify reads every row of every file of the
history store and reports each file it refuses. A question to the center
builds only the rows it selects, so a file with a row that does not parse
goes unnoticed until a question selects that row. This command selects
every row of every file. Run it after you restore the store from a backup,
after a disk fault, before you destroy an old store key, and on a schedule
if you want to know about a refused file before a question finds it.
For each file the command makes these checks in this order, and reports the first one that fails:
- It reads the file.
- It opens the sealed layer under a key you gave. The layer must hold every record whole and name the path the file is at.
- It reads the Parquet footer. The schema must be the one this build reads, and this build must read the codec of every column.
- It builds every row: each JSON value, send time and text must parse.
- It compares the rows with the path. Every row's receive time must be on
the UTC day of the
date=directory. The earliest and the latest receive time must be the two times in the name, and the mark in the name must be the mark of the rows. For a file that release 0.5.249 wrote undermember=<id>, every row must be of that member, and the first row must have the stream and the position in the name.
What it does not check. It does not look for a row that two files hold. It does not read the store file or the intake of call records. It reads only files under the names the store writes: another file in one of the directories is not a file of the store, and the center does not read it either.
Prerequisites.
- A build whose
--versionlistsfleet-store: the build the center runs. A build without the history store exits2and says so. - The store key file, and the file of every earlier key that a history file can still be under. A key rotation does not rewrite the history files: each stays under the key that wrote it until retention removes it.
- Read access to
<store>.parquet. Run the command as the center's user. A run that root starts reads the key files as root, so they can stayrootmode 0600. It then lists and reads the history files as--user, elsenobody. "A pass that root starts", below, has the details. - Nothing changes on a member or on the center. You do not stop the center:
the command creates, changes and removes nothing. When retention removes a
file between the command's listing and its read, the command counts the
file as
goneand does not refuse it.
Steps.
-
[center] Run the pass with the center's
--storeand key:sipnab fleet verify --store /var/lib/sipnab/center.store \ --store-key-file /etc/sipnab/center.storekey
calls: 20 files, 20 clean, 0 refused, 0 gone, 1080000 rows members: 1440 files, 1440 clean, 0 refused, 0 gone, 2880000 rowsThe command prints one line for each class when it ends.
filesis the number of files it listed, androwsis the number of rows in the clean files. The outputs on this page are from a test store of synthetic rows for one day: 20 filings of 54,000 call records and one filing a minute of 2,000 members' heartbeats. -
[center] To read less, name a class or a window of UTC days. Both days belong to the window:
sipnab fleet verify --store /var/lib/sipnab/center.store \ --store-key-file /etc/sipnab/center.storekey \ --class calls --from 2026-11-01 --to 2026-11-07
-
[center] After a key rotation, name the earlier key too, as the center does:
sipnab fleet verify --store /var/lib/sipnab/center.store \ --store-key-file /etc/sipnab/center.storekey.2 \ --store-old-key-file /etc/sipnab/center.storekey
Run it once without
--store-old-key-filebefore you destroy the old key. Each file it then refuses withwas written under key idis a file that only the old key opens. -
[center] For a script, add
--json. The command then prints one JSON document on one line and no other line. Here it is throughjq ., for the test store with one file cut in half:{ "classes": [ { "class": "calls", "clean": 19, "files": 20, "gone": 0, "refused": 1, "rows": 1026000 }, { "class": "members", "clean": 1440, "files": 1440, "gone": 0, "refused": 0, "rows": 2880000 } ], "refused": [ { "class": "calls", "file": "calls/date=2026-11-11/1794355200000-1794355253999-h39bb7d2c07e281e0.sealed", "reason": "sealed layer: the file was truncated: it ends before its records do" } ], "result": "refused", "started_at": "2026-11-15T10:36:21Z", "version": "0.5.256" }started_atis when the pass started andversionis the version of the program that ran it.resultiscleanorrefused.
Check a backup copy. The command is also the check of a backup. Restore
or copy the history directory to any place, keep its name as the name of a
store file with .parquet added, and give that store file's path as
--store. The store file itself need not be there, because the command
does not read it. The paths under the history directory must be the ones
the center wrote: the sealed layer of each file names the file's path under
that directory.
sipnab fleet verify --store /mnt/restore/center.store \
--store-key-file /etc/sipnab/center.storekeyA pass that root starts. The command does not stay root.
-
It reads
--store-key-fileand every--store-old-key-fileas root. -
Then it runs as
--user, elsenobody, and lists and reads the history files as that user. The command reads no configuration file, so[privilege] userdoes not name the user. -
That user must read
<store>.parquet, every directory in it and every file in them. When it cannot, the command reads no history file, exits2and names the path and three ways out:sipnab fleet verify: started as root, this command runs as user 'nobody', and 'nobody' cannot read '/var/lib/sipnab/center.store.parquet' (the history store of --store). Nothing was created. Choose one: chown nobody '/var/lib/sipnab/center.store.parquet'; run it with --user USER, naming a user who can; or pass --no-priv-drop to keep rootFor a center's store, take the second way and name the center's user:
--user sipnab. Do not change the owner of a store that a center uses, because the center must keep its write access. -
--no-priv-dropkeeps root for the whole pass and logs one warning line. -
A pass that root did not start changes in nothing, and
--userhas no effect on it.
Verify. The exit status is the verdict:
| Exit status | Meaning |
|---|---|
0 |
The command refused no file. Files that were gone do not change this. |
1 |
The command refused at least one file. Each has one refused: line, or one entry under refused with --json. |
2 |
The pass did not run: the command refused an argument or a key file, it cannot read a directory of the store, the user a run started as root drops to cannot read the store, there is no <store>.parquet directory beside --store, or the build has no history store. |
A store that has filed nothing yet has the directory and no file: the
command prints 0 files for each class and exits 0.
How long a pass takes. Measured on 2026-11-15 with a release build on an NVIDIA Jetson AGX Thor (aarch64, 14 processors), with other jobs running on the host (load average 17 to 28). The pass read the test store above: 583,543,021 bytes of call record files and 441,207,414 bytes of heartbeat files. Each time is one run of the command, from its start to its exit, over three rounds:
| Pass | Files | Rows | Time |
|---|---|---|---|
--class calls |
20 | 1,080,000 | 2.8 to 6.2 s |
--class members |
1,440 | 2,880,000 | 4.2 to 9.0 s |
| Both classes | 1,460 | 3,960,000 | 8.6 to 13.0 s |
The pass reads every file of the classes it checks, whole.
Troubleshoot. A refused file has one line, the path and the first reason:
refused: calls/date=2026-11-11/1794355200000-1794355253999-h39bb7d2c07e281e0.sealed: sealed layer: the file was truncated: it ends before its records do
-
sealed layer: the file was written under key id <id>, which is not loaded. The file is under an earlier store key. Give that key's file with--store-old-key-file. Start the center with it too, or the center refuses the same file for each question that reads it. -
sealed layer:withdid not authenticate,was truncatedorwas cut at its end. The file changed after the center wrote it. Restore the file from a backup and run the command again. -
sealed layer: the file was written as '<path>' and is not at that path. Someone moved or renamed the file. Move it back to the path the message gives. -
parquet:with a row number. A row does not parse. The center seals each file, so only a defect in the release that wrote the file produces such a row. Keep the file and report the line with the version of the release that wrote the file. The center still answers every question that does not select the row. -
rows:. The file opens and every row parses, and the rows are not the rows the path names: a row of another day or member, another earliest or latest time, or another mark. The center seals the path into the file, so only a defect in the release that wrote the file produces this. Keep the file and report the line. -
no history directory '<path>': nothing to verifyand exit2.--storedoes not name the center's store file, or the build that runs the center has no history store. Compare the path with the--storeof the center's service unit. -
is a pipe or FIFO, not a file of the history store. The entry under a history file's name is not a regular file. The message names a character device, a block device or a socket in the same way. The center writes only regular files, and it refuses the same entry for each question that reads it. Neither the command nor the center opens such an entry for reading. Find out who made it, then remove it. -
started as root, this command runs as userand exit2. Root started the pass, and the user it runs as cannot read the path in the message. Run it with--userand the center's user. "A pass that root starts", above, has the other ways. -
A file is
gone. Retention removed it during the pass. Nothing is wrong.
Fleet verify lists each message.
Purpose. From this release the center compresses every history file it writes, call records and heartbeats, with zstd at level 1. Releases 0.5.249 to 0.5.253 wrote the same rows without compression. This section says what an upgrade changes on disk, how to see it, and what a downgrade does.
What the compression gives, measured on synthetic rows (the test
edge_a_filing_is_measured_with_and_without_the_codec, with 2,000 members
and 10 call records for each):
| One sealed file | 0.5.253, bytes | This release, bytes | Share |
|---|---|---|---|
| One heartbeat of each of 2,000 members | 887,305 | 62,048 | 7.0% |
| 20,000 call records | 16,493,345 | 660,654 | 4.0% |
The share depends on the rows. An inventory or a call record repeats most of
its field names and many of its values from row to row, and that is what the
codec removes. The rows of that test repeat more than the rows of a
fleet do. In a second measurement, each row held digests, addresses and
counters of its own, as GET /v1/inventory and GET /v1/dialogs/:call_id
answer them: a heartbeat file was 11.7% of its uncompressed size and a call
record file 10.7%. Text that does not repeat does not shrink.
The cost is time in each filing and each read. In that second measurement, in a release build on aarch64, encoding 2,000 heartbeats took 9.8 ms with zstd and 3.7 ms without, and encoding 20,000 call records took 98 ms and 59 ms. Decoding the 20,000 call records took 17.7 ms and 7.6 ms.
The codec also makes the binary larger. These are the sizes of the stripped binary that each build job of the release workflow printed for 0.5.253 and for 0.5.254, the first release with the codec:
| Release build | 0.5.253, bytes | 0.5.254, bytes | Change |
|---|---|---|---|
| aarch64 gnu | 27,103,912 | 27,693,832 | +589,920 |
| x86_64 gnu | 31,853,960 | 32,532,744 | +678,784 |
| aarch64 musl (static) | 20,944,840 | 20,944,840 | 0 |
| x86_64 musl (static) | 25,385,600 | 25,389,696 | +4,096 |
aarch64 gnu -noaudio
|
23,433,392 | 23,433,392 | 0 |
x86_64 gnu -noaudio
|
27,588,816 | 27,596,304 | +7,488 |
The static musl builds and the -noaudio packages do not link the history
store, so they do not link the codec. Release 0.5.254 also added regular
expressions to the MCP tool search_messages, and all six builds hold that
change. No build measured the two changes apart: the 4,096 and 7,488 bytes
of the x86_64 builds without the store are not the codec's, and the growth
of the two gnu builds holds both changes.
Before you start.
- A center with the history store. The compression is part of the history store and has no flag and no setting. See Keep call history at the center for the builds that carry the store.
-
Other systems. Nothing changes on a member, and a member of any
release that pushes to the center needs no upgrade. Nothing changes on
OpenSIPS, Kamailio, Homer or any HEP collector.
GET /v1/fleet/callsandGET /v1/fleet/members/{id}/historyanswer the same JSON as before, so a script that reads them needs no change. A backup of<store>.parquetcopies smaller files from the day of the upgrade on. - The files on disk stay as they are. The center does not rewrite a file an earlier release wrote. Those files stay uncompressed until the retention removes their day, and the center answers from them and from the new files together.
-
A downgrade does not read the new files. Releases 0.5.249 to 0.5.253
have no zstd reader. A center of one of those releases leaves the
compressed files on disk, answers no row from them, and names each of
them under
refusedin the answer. Upgrade again and the rows are back. Keep this in mind before you move a center back to an earlier release.
Steps.
-
[center] Read the size of the history directory before the upgrade:
du -sb /var/lib/sipnab/center.store.parquet
-
[center] Install this release and start the center with the command line it had. No flag is new.
-
[center] Wait for one filing. The center files the heartbeats and the call records it received every
--calls-flush-secs(60 seconds by default), and writes no file when it received no new row.
Verify.
-
[center] List the heartbeat files of today with their sizes. The files written after the upgrade are smaller than the files of the same fleet written before it:
ls -l /var/lib/sipnab/center.store.parquet/members/date=$(date -u +%F) -
[controller] Ask for the call records of a window that spans the upgrade. The answer holds rows from before and from after it, and
refusedis empty:curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $OPS_TOKEN" \ 'https://127.0.0.1:8444/v1/fleet/calls?from=2026-11-14T00:00:00Z&to=2026-11-15T00:00:00Z&limit=1000' \ | jq '{rows: (.calls | length), refused}'
Troubleshoot.
-
The files written after the upgrade are as large as the files before
it. The center that wrote them is not this release. Run
sipnab --versionon the center's host and check which binary the service starts. -
After a downgrade, an answer lacks the newest rows and
refusednames files with a reason that endsDisabled feature at compile time: zstd. The center is a release from 0.5.249 to 0.5.253 and the files are compressed. The files are intact. Install this release or a later one. -
A reason reads
parquet: column received_at is compressed with snappy; this build reads zstd and uncompressed files, or names another codec. A release newer than this one wrote the file with a codec this release does not link. Install the release that wrote it. See the error reference.
Purpose. A center with the history store keeps every heartbeat it
accepts, not only the newest of each member. GET /v1/fleet/members/{id}/history answers one member's heartbeats: when the
center heard the member, from which address, and what the member's inventory
said at that time. The same rows rebuild the member list for a center that
starts without its store file: the center lists a member that was silent
before the stop as silent after the start, and still counts a record the
member sends again as a duplicate.
The center files the heartbeats into Parquet files under
<store>.parquet/members, sealed with the store key as the call records
are. It keeps them for --retention-hours, the time a member stays listed
after its last heartbeat.
The outputs below come from a run on 2026-11-12 on one host, with a
development build after 0.5.249: a center on 127.0.0.1:18449 over HTTP on
loopback, with --calls-flush-secs 5, and two heartbeats and one call record
sent by hand as member edge-a. The commands name files as a deployment
would.
Before you start.
-
A build with the history store, as
Keep call history at the center lists:
sipnab --versionshowsfleet-store. The static musl tarballs and the-noaudiopackages do not have it. A center from one of those keeps each member's newest heartbeat in its store file, as before, and answers501on the history route. - A center and members that send heartbeats, as Run a fleet center sets up. The member needs no change.
- The store key and its backup. The center seals the heartbeat files under the store key. Without the key no one reads them.
-
A token that reads members. The route needs scope
inventory, the scope ofGET /v1/fleet/members.full,actionsandadminreach it too. Afleet-reporttoken does not. -
Room on the center's disk, and files. The center writes one file for
each filing, which holds the heartbeats of every member: at the default
--calls-flush-secs 60that is 1,440 files a day for 1 member and for 2,000 (The files of the history store). In a run of the tests on 2026-11-13 (a debug build, load average about 30), 2,000 members with one heartbeat of about 300 bytes each made one file of 79,189 bytes in 202 ms, and 1,440 such filings, 24 hours, made 113,491,120 bytes. Releases 0.5.251 to 0.5.253 wrote the same rows without compression as 887,745,307 bytes. In a release build on 2026-11-14 (aarch64, 14 cores, load average about 10), one member's newest 1,000 heartbeats came from 1,001 such files in 226 to 230 ms. In a release build on 2026-11-15 (load average 23 to 31), the center rebuilt the member list without a store file from all 1,440 files in 1.95 to 1.99 s, and answered503meanwhile (The files of the history store has the parts of those times). The center answers members while it writes a file. Heartbeats that wait for a filing are in the store file, and at most 64 MiB of them wait (MAX_WAITING_BYTES). -
Other systems. Nothing changes on a member, on OpenSIPS, Kamailio,
Homer or any HEP collector. A backup of the center includes the directory
<store>.parquet.
Steps.
-
[center] Start the center as before. No flag turns the heartbeat history on: a build with the history store keeps it. Two flags you may already set govern it:
sipnab fleet serve --listen 127.0.0.1:8444 --store /var/lib/sipnab/center.store \ --store-key-file /etc/sipnab/center.storekey \ --signing-key-file /etc/sipnab/center.signing \ --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.key \ --retention-hours 72 --calls-flush-secs 60
INFO sipnab::app::fleet_center: fleet center history store 'center.store.parquet': call records are kept 90 day(s), heartbeats 72 hour(s)-
--retention-hoursis how long a member stays listed after its last heartbeat, and how long the center keeps heartbeats. The center removes a UTC day of heartbeats once every hour of that day is past the retention, at start and every minute. -
--calls-flush-secsis the time between two filings, of call records and of heartbeats. The center also files when it starts and when it stops on a signal.
-
-
[controller] Ask for a member's heartbeats with the token that reads the member list. With no parameter the answer is the newest 100:
curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $READER_TOKEN" \ 'https://127.0.0.1:8444/v1/fleet/members/edge-a/history?limit=1' | jq .
{ "complete": true, "count": 1, "files_not_read": 0, "files_read": 1, "heartbeats": [ { "member": "edge-a", "self_reported": { "calls": null, "heartbeat_secs": 5, "inventory": { "node_name": "edge-a", "version": "0.5.249" }, "sent_at": "2026-11-12T20:59:10Z", "seq": 2, "stream": "edge-a-manual" }, "verified": { "call_records": 0, "heartbeats": 2, "peer": "127.0.0.1", "received_at": "2026-11-12T20:59:10Z" } } ], "member": "edge-a", "more": true, "refused": [] }A row has the fields of the member's row in
GET /v1/fleet/members:verifiedis what the center saw,self_reportedis what the member said.verified.heartbeatscounts the member's heartbeats up to this one. The parameters, each optional:Parameter Meaning fromHeartbeats the center received at this RFC 3339 time or later. toHeartbeats the center received before this RFC 3339 time. limitThe most heartbeats to return, 1 to 1000. Default 100. The newest come first. A heartbeat is in the answer from the moment the center acknowledges it. A member the center has no heartbeat of has an empty answer with status
200. One question opens at most 4,096 files, as for call records: about 2.8 days of heartbeats at the default--calls-flush-secs 60, so one question reaches most of the default--retention-hours 72. Givefromandtofor the oldest hours. One answer holds at most 1,000 heartbeats (limit), which for one member at one heartbeat a minute is 16.7 hours.
Verify.
-
[controller]
GET /v1/fleet/membersreports the filings since the center started:curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $READER_TOKEN" \ https://127.0.0.1:8444/v1/fleet/members | jq -c .history
{"available":true,"failed_filings":0,"files_written":1,"heartbeat_files_written":1,"heartbeats_filed":2,"heartbeats_removed":0,"heartbeats_waiting":0,"members_restored":0,"retention_days":90,"rows_filed":1}heartbeats_waitingcounts the heartbeats not yet in a file,heartbeats_removedthose that left before a filing, andmembers_restoredthe members the last start took from the history store. -
[center] The files are under class
membersand day: one file for each filing, which holds the heartbeats of every member. The names are spelled as those of call record files:cd /var/lib/sipnab && find center.store.parquet/members -type f
center.store.parquet/members/date=2026-11-11/1794355201000-1794355203000-he7b857d7ce5986b7.sealed -
[center] To see the rebuild on a test center, stop the center, move its store file away and start it. In the run, the member had also sent one call record after its last heartbeat:
INFO sipnab::app::fleet_center: fleet center: 1 member(s) the store file held nothing of, or an older heartbeat of, were taken from the history store WARN sipnab::app::fleet_center: fleet member 'edge-a' is silent: no heartbeat for 8 s, more than 1 of its 5 s intervalsGET /v1/fleet/membersthen lists the member with 2 heartbeats, at position 3 of its stream (past the call record) and silent, andhistory.members_restoredis1.
Troubleshoot.
-
401. The token's scope does not read members. Mint aninventorytoken, as step 2 of Run a fleet center does. -
400with adetail. The detail names the parameter, or says the id in the path is no member id. This route takes nomemberand nocall_id:{"detail":"\"call_id\" is not a parameter of this route; it takes from, to and limit","status":400,"title":"Bad Request","type":"https://sipnab.com/problems/bad-request"} -
501andthis build has no history store. Install a build that has it and start the center on the same store. The history starts at that start: a build without the store keeps no earlier heartbeat. -
"complete": false. The center could not read a file (refusednames it and the reason), or the question ruled in more files than one question opens (files_not_read). The reasons and their fixes are those of Keep call history at the center:{"complete":false,"refused":[{"file":"members/date=2026-11-11/1794355201000-1794355203000-he7b857d7ce5986b7.sealed","reason":"sealed layer: the file has a record 2 that did not authenticate (the file was modified, or its records were moved)"}],"files_read":0,"files_not_read":0,"count":0} -
The center logs
the heartbeats were not filed and wait in the store file. The line ends with the cause: a directory the center cannot write, a full disk, or a file of the same day that it cannot read. The center tries again after--calls-flush-secs, andfailed_filingscounts the tries. Past 64 MiB of waiting heartbeats, and past--retention-hours, the oldest leave before a filing andheartbeats_removedcounts them. -
The center logs
the member registry was rebuilt from the newest. The center had no store file, or one older than the history, and more than 4,096 heartbeat files could hold a member's last heartbeat. The center read the newest 4,096, about 2.8 days at--calls-flush-secs 60. It lists a member that went silent before them when that member sends a heartbeat again. -
The center logs
was not read for the member registry. At start, a heartbeat file the center read for its member list did not open. The line names the file and the reason. The center starts and lists a member from an older file or from its store file alone. When the store file is also lost, the member returns to the list with its next heartbeat. -
A member is missing after a start without the store file. The center restores a member from heartbeats it filed. A member whose last heartbeat is older than
--retention-hoursis not listed, with or without the store file. The center no longer has the heartbeats that still waited in the lost store file. -
The center logs
past --retention-hours <n> and not removed. The line names a file in an expired day that the center did not write, or one the system refused to remove. Remove it by hand. -
What the heartbeat history is not. The store file stays the member list's own record in every build: keep it in the backup. The history rebuilds what the file lacks; it does not replace it.
The purpose is a center that starts at boot and never serves as root.
Before you start
- The steps above work: the center starts by hand with your key files.
- The package created the
sipnabuser. Check withid sipnab. - Nothing changes on a member.
Steps
-
[center] Give the key files to the service's user, and to nobody else:
# Run all of these, in order. chown sipnab:sipnab /etc/sipnab/center.signing /etc/sipnab/center.storekey /etc/sipnab/center.key chmod 600 /etc/sipnab/center.signing /etc/sipnab/center.storekey /etc/sipnab/center.key -
[center] Write
/etc/systemd/system/sipnab-fleet-center.service. No package ships this unit:[Unit] Description=sipnab fleet center Documentation=https://sipnab.com After=network-online.target Wants=network-online.target [Service] Type=simple ExecStart=/usr/bin/sipnab fleet serve --listen 127.0.0.1:8444 \ --store /var/lib/sipnab-fleet/center.store \ --store-key-file /etc/sipnab/center.storekey \ --signing-key-file /etc/sipnab/center.signing \ --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.key User=sipnab Group=sipnab NoNewPrivileges=true ProtectSystem=strict ProtectHome=true PrivateTmp=true # /var/lib/sipnab-fleet, private to the sipnab user: the store, its # intake and the history store live there. StateDirectory=sipnab-fleet StateDirectoryMode=0700 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
-
[center] Start it:
# Run all of these, in order. systemctl daemon-reload systemctl enable sipnab-fleet-center systemctl start sipnab-fleet-center
Verify
-
[center] The service runs as
sipnab:systemctl show sipnab-fleet-center -p User -p ActiveState
User=sipnab ActiveState=active
A center that root starts
A unit without User=, or sudo sipnab fleet serve, starts the center as
root. Since 0.5.256 it does not stay root.
-
It reads the signing key, the store key and the TLS key, and binds
--listen, as root. So the key files can stayrootmode 0600, and the center can bind a port below 1024. -
Then it serves as
--user, elsenobody, and says so:INFO sipnab::privilege: Dropped privileges to user 'nobody' (uid=65534, gid=65534) -
The store, its intake, the history store beside it and their directory must be that user's. A center that ran as root before 0.5.256 left them root's, so its first start after the upgrade refuses. It exits 2 and creates nothing:
sipnab fleet serve: started as root, this command runs as user 'nobody', and 'nobody' cannot write in '/var/lib/sipnab' (the directory of --store). Nothing was created. Choose one: chown nobody '/var/lib/sipnab'; run it with --user USER, naming a user who can; or pass --no-priv-drop to keep root -
Fix it once, for the user the center serves as, on the directory that holds your store:
chown -R sipnab /var/lib/sipnab
Then start the center with
--user sipnab, or withUser=sipnabin the unit. -
--no-priv-dropkeeps root, as before 0.5.256, and logs one warning line. -
The center reads a
--revoked-fileagain each time it changes, so the user the center serves as must be able to read it.
Each output below comes from the same run.
-
The center exits 2 with
started as root, this command runs as user. Root started it and the user it serves as cannot use the path the message names. A center that root starts has the three ways out. -
401on the ingest. The token is not afleet-reporttoken this center's key signed, it expired, or its id is in the revocation file. Afull,adminorinventorytoken gets401there too: only a member's token reports. Mint one as in step 2.401 {"status":401,"title":"Unauthorized","type":"https://sipnab.com/problems/unauthorized"} -
401on the members list. The token is a member's token, which reads nothing. Use aninventorytoken. -
403andthe token's id names no member. The token's--token-idhas characters a member name cannot have. This run minted one with the idedge a:403 {"detail":"the token's id names no member: mint it with --token-id of 1 to 64 letters, digits, '.', '_' or '-'","status":403,"title":"Forbidden","type":"https://sipnab.com/problems/forbidden"} -
400and a line number. A record in the batch is not one the center reads, and the center accepted none of the batch. The detail names the line and the field:400 {"detail":"line 2: not a record: expected ident at line 1 column 2","status":400,"title":"Bad Request","type":"https://sipnab.com/problems/bad-request"} -
plain HTTP is refused except on loopback. The command line names an address other than loopback and no--tls-certand--tls-key. It exits2before it creates the store:sipnab fleet serve: --listen 192.0.2.10:8444 without --tls-cert and --tls-key would take members' tokens and inventories in plain HTTP; plain HTTP is refused except on loopback -
can be read or written by other users. Other users can read the store file. Runchmod 600on it. It exits2:sipnab fleet serve: --store 'center.store' can be read or written by other users (mode 0644); chmod 600 it -
503andthe center is rebuilding its member list from the history store. The center has bound its port and not yet opened its store. Ask again after theRetry-Afterseconds (What the center answers while it starts). -
A member stays
silent. The center has had no heartbeat from it for more thansilent_after_missedof its intervals: the member is down, cannot reach the center, or the center refuses its token.seconds_since_heartbeatsays how long ago the last one arrived.
docs/errors.md lists every message the center prints.
State of each instance:
| State | Meaning |
|---|---|
ok |
It answered with an inventory. |
refused |
It answered with an HTTP status other than 200, such as 401 for a wrong, expired or narrower token, or 404 for a build without the inventory. |
unreachable |
No answer: no connection, a timeout, a TLS failure, a token file sipnab would not read, an answer cut short (the reason names the bytes read, as the answer ended after 9 of its 40 bytes), or an answer that is not JSON. |
incompatible |
Its inventory has a schema_version newer than this build reads, or none. Upgrade the controller's sipnab. |
Findings across the fleet:
| Finding | Meaning |
|---|---|
version_mismatch |
The instances that answered do not all run the same build. The finding names each build and the instances on it. |
unlisted_hep_sender |
A HEP collector in the list receives from an address that no listed instance's URL resolves to. |
hep_sender_refused |
A HEP collector in the list refused packets from a listed instance's address for authentication, and has not admitted a packet from it since. |
Every instance reports signature.state as unverified. A process cannot
prove which binary it is, and one that someone had tampered with would report
"verified" too.
-
Download
SHA256SUMS.txtfrom the release the instance claims, and verify it:gh attestation verify SHA256SUMS.txt --repo NormB/sipnab
-
Compare the instance's
binary.sha256with the line for its artifact. On Linux the instance hashes/proc/self/exe, so after an upgrade replaces the file the digest still names the binary that is running.
- Every field is the instance's own claim. An attacker who controls a host can make it report anything.
- For a secret held in a file, the digests cover its path only. Replacing the
contents of such a file, for example rotating the key in
--api-signing-key-file, changes no digest; it changes the file'sinodeormtimeinconfig.secret_filesonce the instance restarts. Two instances that hold different keys at the same path look the same: the answer holds nothing derived from a key. A secret given as a value, such as--api-signing-keyor a${NAME}the configuration expands, changesconfig.effective_sha256. - It reads only the instances in your hosts file. An instance nobody listed is invisible to it, unless it sends HEP to a listed collector.
- It changes nothing. Pushing a configuration, upgrading and restarting are not part of it.
Website · Repository · Issues · Generated from docs/ — edit there, not here.
Getting started
- Installation
- First CLI Triage
- Glossary
- Cookbook
- Runnable Examples
- Real World Captures
- Troubleshooting
- Measure Call Quality
- Run as a Service
- Tuning Capture
- Encapsulations
- Capture With Least Privilege
- Collect and Forward HEP
- Read Captures From Archives
- Rtpengine Relays
- Ask a Media Relay About Itself
- Ask Each Calls rtpengine Engine
- Backers
Using the TUI
CLI & automation
- CLI Reference
- Error Messages
- Capture Queries
- Expectations
- Filter DSL
- Follow A Call
- Output Formats
- Stability
- SIP Response Codes
- SIP Methods
- SIP Header Fields
- SIP Parameters
- SIP Lint Rules
- MOS and Codecs
Configuration
Integrations (API & MCP)
- REST API
- Prometheus Metrics
- Authentication
- Require Client Certificates
- MCP
- MCP Deployment
- MCP Across an Estate
- Manage Many Instances
- MCP Tools
- MCP Tools Core
- MCP Tools Signaling
- MCP Tools Captures
- MCP Tools Security
- MCP Tools Media
- MCP Tools Relay
- MCP Tools TFPS
- MCP Tools Server
- MCP Tools vCon
- MCP Tools TLS
- MCP Protocol
- Uprobe Walkthrough
- Capture SIP over TLS
- WASM Plugins
- Run Commands and Send Alerts
- vCon Export
- vCon Capture Stack
- G729 Decoding
Your SIP server
- Use OpenSIPS as the SIP Server
- Run sipnab beside OpenSIPS
- Use Kamailio as the SIP Server
- Run sipnab beside Kamailio
- Use Asterisk as the PBX
- Run sipnab beside Asterisk
- Use FreeSWITCH in the Voice Stack
- Run sipnab beside FreeSWITCH
Add to your voice stack
- Add a vCon Server
- Send vCons to a vCon Server
- Send vCons to vcon store
- Add TFPS
- Control TFPS from sipnab
- Ban SIP Scanners with fail2ban
- Feed fail2ban from sipnab
- Add rtpengine
- Name rtpengine Media with sipnab
- Add rtpproxy
- Name rtpproxy Media with sipnab
- Add Homer
- Connect sipnab to Homer
- Add Prometheus and Grafana
- Add sipnab Metrics to Prometheus
Development & internals
- Internals Index
- Internals Subsystem Guide
- Internals Invariants
- Internals Testing
- Internals Walkthroughs
- Internals Build CI Release
- Internals Profiling
- Internals Domain Primer
- Library API
- Benchmarks
- Fault Model
- Threat Model
- Assurance Case
- Architecture
- Internals Threading
- Internals TUI Testing
- Internals Zero Copy Payloads
- Internals Uprobe Capture
- Internals Rtpengine Control Plane
- Internals Relay Control Decoding
- Internals vCon Exporter
- Internals Operator Guide Notes
- Internals Postmortems
- Internals Code Map
- Internals Lint Rules
- Internals Encapsulation Decoding
- Internals MCP Design Notes
- Internals Metrics And Output Notes