Skip to content

Manage Many Instances

github-actions[bot] edited this page Nov 21, 2026 · 1 revision

sipnab fleet inventory asks every sipnab instance you list for its inventory, and prints one table of the fleet. Use it to answer "which build and which configuration does each instance run, and which ones are down?" before and after an upgrade or a configuration change.

The table shows, for each instance:

  • the build it runs, and the SHA-256 of its binary,
  • the digest of its effective configuration: the file it loaded and its command line,
  • the user it runs as, and how long it has been up.

Below the table it names the instances that did not answer, the instances on a different build from the rest, the HEP senders a collector receives from that are not in your list, and the listed instances whose HEP a listed collector refuses for authentication. Each instance also reports when each of its key, token and password files was last replaced, so you can check that a rotation took effect. It reports. It changes nothing on any instance.

sipnab fleet serve is the other direction: a fleet center that members report to. Each member pushes a heartbeat, the center keeps the newest one per member, and it says which members went silent, including members the controller cannot reach. See Run a fleet center.

Some upgrades need every sender and its collector on the same build, such as the HEP HMAC version 2 change in Upgrade a fleet that uses HMAC authentication. The report shows which instances are still on the old build.

The commands and output on this page come from a run on 2026-11-03: four instances in containers on the network 192.0.2.0/24, and the controller on the host. The instances ran a development build of 0.5.232 with this feature, and one ran release 0.5.130. The output of A listed sender the collector refuses and Check that a key rotation took effect comes from a run on 2026-11-07 on one host, with a development build of 0.5.236: the collector hub on 127.0.0.2 and the sender edge-a on 127.0.0.1.

How it works

  • Each instance serves GET /v1/inventory on its REST API (--api). See GET /v1/inventory for every field.
  • sipnab fleet inventory runs on one machine, the controller. It reads a hosts file and asks every instance in it at once. The instances never connect to the controller.
  • Each request carries a bearer token with scope inventory. That token reads the inventory and nothing else: no dialogs, no messages, no audio.
  • The answer holds no configuration value and no command-line argument, only their SHA-256 digests.
  • The command reads at most 16 KiB of each answer's status line and headers (MAX_HEAD_BYTES) and at most 4 MiB of its body (MAX_ANSWER_BYTES). The inventories in this page's run were under 2 KB each.

Before you start

  • sipnab on each instance. A build whose sipnab --version lists the api feature, and that serves GET /v1/inventory. An older build answers 404 there, and the report marks it refused.
  • sipnab on the controller. Any build with the api feature. It needs no capture privilege: it opens no capture.
  • TLS for each instance's API. A certificate whose subject alternative name is the address or name in the instance's URL, its key readable by the sipnab user only, and the certificate of the issuing CA on the controller. Plain http:// works too, and sends the token in clear text.
  • A signing key on each instance, in a file only the sipnab user can read. Set up authentication shows how to create one.
  • Network. The controller reaches each instance's API port over TCP, 8443 in this page's examples. Open that port to the controller's address only. The instances need no route to the controller.
  • Other systems. No change. OpenSIPS, Kamailio, Homer and a HEP collector keep their configuration. A collector that runs sipnab joins the list like any other instance.

Step 1: run each instance with the REST API over TLS

  • [each instance] Add --api, the TLS certificate and key, the signing key and a node name to the instance's command line. This is the instance edge-a from the run, which reads a capture file and sends HEP to the collector hub:

    sipnab -N -I /var/lib/sipnab/sample.pcap --config /etc/sipnab/edge-a.toml \
      --hep-send 192.0.2.10:9060 --hep-auth-mode hmac --hep-auth-file /etc/sipnab/hep.key \
      --api 192.0.2.11:8443 --api-tls-cert /etc/sipnab/edge-a.crt --api-tls-key /etc/sipnab/edge-a.key \
      --api-signing-key-file /etc/sipnab/edge-a.signing --node-name edge-a

    The collector ran with the same API flags and its HEP listener:

    sipnab -N --config /etc/sipnab/hub.toml \
      --hep-listen 192.0.2.10:9060 --hep-auth-mode hmac --hep-auth-file /etc/sipnab/hep.key \
      --api 192.0.2.10:8443 --api-tls-cert /etc/sipnab/hub.crt --api-tls-key /etc/sipnab/hub.key \
      --api-signing-key-file /etc/sipnab/hub.signing --node-name hub

    Each instance logs REST API listening on <address> when its API is up.

Step 2: mint an inventory token on each instance

  • [each instance] Mint a token with that instance's signing key:

    sipnab --mint-token --token-scope inventory --token-id fleet-controller \
      --api-signing-key-file /etc/sipnab/edge-a.signing --api-token-ttl 2592000

    It prints one token, starting s2., and exits. --api-token-ttl 2592000 makes it valid for 30 days.

  • [controller] Save each token in its own file, readable by the controller's user only:

    install -m 600 /dev/null /etc/sipnab/fleet/edge-a.token

    Then paste the token into it. sipnab refuses a token file that other users can read.

Step 3: write the hosts file

  • [controller] Write /etc/sipnab/fleet/fleet.toml, one [[host]] table per instance. This is the file from the run:

    [[host]]
    name = "hub"
    url = "https://192.0.2.10:8443"
    token_file = "/etc/sipnab/fleet/hub.token"
    ca_file = "/etc/sipnab/fleet/internal-ca.pem"
    
    [[host]]
    name = "edge-a"
    url = "https://192.0.2.11:8443"
    token_file = "/etc/sipnab/fleet/edge-a.token"
    ca_file = "/etc/sipnab/fleet/internal-ca.pem"
    
    [[host]]
    name = "edge-b"
    url = "https://192.0.2.12:8443"
    token_file = "/etc/sipnab/fleet/edge-b.token"
    ca_file = "/etc/sipnab/fleet/internal-ca.pem"
    • name is the name the report shows. Each name appears once.
    • url is the instance's REST API, https:// or http://.
    • token_file holds the instance's inventory token on one line.
    • ca_file is optional: a PEM CA file to trust instead of the host's CA bundle.

    sipnab refuses an unknown key, an empty name, an empty URL and a name used twice, with exit 2.

Step 4: read the fleet

  • [controller] Run:

    sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml

    Output from the run:

    NAME             STATE        VERSION      COMMIT       BINARY       CONFIG          UID   UPTIME_S
    hub              ok           0.5.232      29204db7     6bba53867ecd ec09522b49b4   1000         54
    edge-a           ok           0.5.232      29204db7     6bba53867ecd 6a434df0c5e2   1000         40
    edge-b           ok           0.5.232      29204db7     6bba53867ecd f63dfdce2c9d   1000         39
    

    It exited 0.

    • BINARY and CONFIG are the first 12 hex digits of binary.sha256 and config.effective_sha256. The three instances run one binary and three configurations: each has its own config file and node name.
    • --json prints the whole report, with every instance's full inventory.
    • --timeout <SECS> sets how long each instance has to answer. Default 10.

Verify

  • Every instance is ok, and there is no finding line. That is exit 0.

  • Two instances you expect to match show one CONFIG. Two instances with the same CONFIG loaded the same configuration values and run the same command line. A different --node-name alone gives a different CONFIG.

  • The exit status says the same for a script:

    Code Meaning
    0 Every instance answered and there is no finding.
    1 An instance did not answer, refused the token, or sent an inventory this build cannot read; or there is a finding.
    2 sipnab cannot read or use the hosts file.

Troubleshoot

Each output below comes from the same run, with one thing changed.

  • unreachable and connection timed out. Nothing answered at that address within --timeout: the instance is down, the URL names the wrong address or port, or a firewall drops the controller's packets. The other instances are still listed:

    edge-c           unreachable  192.0.2.14:8443: connection timed out
    

    Connection refused instead means the host answered and nothing listens on that port: check --api on the instance.

  • unreachable and invalid peer certificate: UnknownIssuer. The controller does not trust the CA that issued the instance's certificate. Set ca_file for that host:

    hub              unreachable  sending: invalid peer certificate: UnknownIssuer
    
  • refused with HTTP 401. The instance did not accept the token: it was minted with another instance's signing key, it expired, someone revoked it, or its scope is not inventory, full, actions or admin. Mint a new one on that instance (step 2). This run gave edge-b the hub's token:

    edge-b           refused      the instance answered HTTP 401
    
  • refused with HTTP 404. The instance runs a build without GET /v1/inventory. Upgrade it. Release 0.5.130 answered:

    edge-old         refused      the instance answered HTTP 404
    

    Release 0.5.130 also refuses --token-scope inventory at mint time, with invalid value 'inventory' for '--token-scope <SCOPE>'.

  • unreachable and can be read or written by other users. The token file's mode lets other users read it. Run chmod 600 on it:

    edge-a           unreachable  token_file '/etc/sipnab/fleet/open.token' can be read or written by other users (mode 0644); chmod 600 it
    
  • finding hep_sender_refused. A collector in the list refuses the HEP of an instance in the list for authentication. See A listed sender the collector refuses.

  • finding unlisted_hep_sender. A collector in the list receives HEP from an address that no listed instance's URL resolves to. Add that sender to the hosts file, or find out why it sends. With edge-b left out of the file:

    finding unlisted_hep_sender: hub receives HEP from 192.0.2.12, which is no instance in the hosts file
    

    The check compares addresses only. A sender whose packets leave from a different address than its API URL names shows up as unlisted.

  • incompatible and a schema_version. The instance answered an inventory format newer than the controller reads. Upgrade the controller's sipnab. No release answers a newer format yet, so this output comes from a test server that answered schema_version 2:

    edge-next        incompatible the instance answered inventory schema_version 2; this sipnab reads up to 1
    

A listed sender the collector refuses

A collector that refuses a sender's HEP drops every packet it refuses. The sender counts them as sent, because UDP reports no delivery, so neither end fails. The report names that sender, and exits 1, while the collector refused the latest packet it received from it.

  • [controller] Run the report. In this run edge-a sent with a key the collector hub does not hold:

    sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml
    NAME             STATE        VERSION      COMMIT       BINARY       CONFIG          UID   UPTIME_S
    hub              ok           0.5.236      fd7d466e     d7cd2b7e5769 a5dc546e4913   1000          5
    edge-a           ok           0.5.236      fd7d466e     d7cd2b7e5769 79480fa13223   1000          3
    finding hep_sender_refused: hub refused 7 HEP packets from edge-a (127.0.0.1): refused: authentication (wrong key or HMAC version); the collector's reasons: hmac_bad_mac 7
    

    It exited 1.

    • The finding names the collector, every listed instance whose URL resolves to the refused address, the address, and the authentication refusals only.
    • refused: authentication (wrong key or HMAC version) names both causes and claims neither. A sender with the wrong key, and a sender on a different HEP HMAC version, can both arrive as a MAC that does not verify, depending on which build each end runs.
    • the collector's reasons are the collector's own refusal reasons, with the names GET /v1/hep/senders uses: hmac_bad_mac for a MAC that does not verify, hmac_unsupported_version for a version-1 token on a collector on 0.5.131 or later, auth_mismatch for a wrong plain key, and auth_missing, hmac_bad_format, hmac_timestamp_out_of_window and hmac_replay. Refusals for other reasons, such as the rate limit or the allowlist, are not this finding.
  • [controller] See the collector's refusals, from the same report:

    sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml --json \
      | jq -c '.hosts[] | {name, refused: .inventory.hep_listener.refused_sources}'
    {"name":"hub","refused":[{"by_reason":{"hmac_bad_mac":7},"last_seen":"2026-11-07T12:52:30.675Z","packets":7,"peer":"127.0.0.1"}]}
    {"name":"edge-a","refused":[]}
    
  • [sender] Fix the sender. Give it the collector's key, or upgrade it to the collector's build when the reason is hmac_unsupported_version, as Upgrade a fleet that uses HMAC authentication describes. Restart it.

  • [controller] Verify. The collector keeps counting the old refusals for as long as it runs, but it has admitted the sender since its last refusal, so the report has no finding and exits 0:

    sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml --json \
      | jq -c '.hosts[0].inventory.hep_listener | {senders: [.senders[] | {peer, last_seen}], refused: [.refused_sources[] | {peer, packets, last_seen}]}'
    {"senders":[{"peer":"127.0.0.1","last_seen":"2026-11-07T12:52:43.016Z"}],"refused":[{"peer":"127.0.0.1","packets":7,"last_seen":"2026-11-07T12:52:30.675Z"}]}
    

Troubleshoot:

  • The collector refuses the sender, and there is no finding. The refused address is no listed instance's: the check compares addresses, as unlisted_hep_sender does. A sender whose packets leave from another address than its API URL names is not matched. Read the collector's refused_sources as above.
  • No refused_sources at all. A collector on a build before this check does not report them. Its packets_refused still counts every refusal.
  • The finding stays after the fix. The collector has not admitted a packet from that address since its last refusal: the sender is not sending yet, or is still refused. Its last_seen in senders must be later than in refused_sources.

Check that a key rotation took effect

The configuration digests cover a key file's path only, so replacing the key in it changes no digest. Each instance therefore reports, for every secret or credential file it named, the file's inode and modification time when it started, under the setting that named it. The answer holds nothing derived from the key. See GET /v1/inventory for the fields.

  • [each instance] Rotate the key the usual way: write the new key to a new file, rename it over the old one, and restart sipnab. This run fixed edge-a's HEP key:

    install -m 600 hep.key edge-hep.key.new && mv edge-hep.key.new edge-hep.key
  • [controller] Before and after the restart, read each instance's stamps:

    sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml --json \
      | jq -c '.hosts[] | {name, secrets: [.inventory.config.secret_files[] | {setting, inode, mtime}]}'

    Before:

    {"name":"hub","secrets":[{"setting":"--api-signing-key-file","inode":6849703,"mtime":"2026-11-07T12:41:19.472148552Z"},{"setting":"--hep-auth-file","inode":6849697,"mtime":"2026-11-07T12:41:19.472148552Z"}]}
    {"name":"edge-a","secrets":[{"setting":"--api-signing-key-file","inode":6849705,"mtime":"2026-11-07T12:41:19.476148542Z"},{"setting":"--hep-auth-file","inode":6852341,"mtime":"2026-11-07T12:52:28.642390890Z"}]}
    

    After:

    {"name":"hub","secrets":[{"setting":"--api-signing-key-file","inode":6849703,"mtime":"2026-11-07T12:41:19.472148552Z"},{"setting":"--hep-auth-file","inode":6849697,"mtime":"2026-11-07T12:41:19.472148552Z"}]}
    {"name":"edge-a","secrets":[{"setting":"--api-signing-key-file","inode":6849705,"mtime":"2026-11-07T12:41:19.476148542Z"},{"setting":"--hep-auth-file","inode":6817841,"mtime":"2026-11-07T12:52:41.958355975Z"}]}
    

Verify: the rotated file's inode or mtime changed on the instance you rotated, and on no other. CONFIG did not change: the path is the same.

Troubleshoot:

  • Nothing changed. The instance has not restarted: sipnab takes the stamp at start, and it describes the file the run read. Restart it.
  • Only mtime changed. The key was rewritten in place rather than replaced. That is a rotation too.
  • error holds a reason. The instance could not stat the file at start, for example No such file or directory (os error 2). The reason never names the path.

Run a fleet center

A fleet center is a sipnab, sipnab fleet serve, that fleet members push heartbeats to. Each heartbeat carries the member's inventory, the record GET /v1/inventory answers. The center keeps the newest heartbeat of each member, and answers "which members are there, what do they run, and which went silent" on GET /v1/fleet/members, without asking any member.

Use it when the controller cannot reach every member, such as members behind NAT, or to learn that a member stopped without polling it. A member can also push a record of each call that ended badly, as Push call records to the center shows. A capture run becomes a member with --fleet-report-url (or [fleet] report_url), as Make an instance a fleet member shows. Step 3 sends one heartbeat by hand, in the format a member sends.

The output in this section comes from a run on 2026-11-07 on one host, with a development build of 0.5.236: the center on 127.0.0.1:8444 over HTTPS, and one instance, edge-a, on 127.0.0.1:18080. The run kept its files in one working directory. The commands name them as a deployment would, under /etc/sipnab and /var/lib/sipnab, and the log lines show the names the run used.

Before you start a center

  • sipnab on the center. A build with the api feature. It opens no capture and needs no capture privilege.
  • A TLS certificate for the center, whose subject alternative name is the address or name members use, its key readable by the center's user only, and the certificate of the issuing CA on each member. The center refuses plain HTTP on any address but loopback, because members send their token and their inventory.
  • A signing key for the center, in a file only the center's user can read. The center mints and verifies member tokens with it. It is not any member's REST API key.
  • A store key for the center, in a file only the center's user can read. The center encrypts its store under it, and does not start without it. Step 1 creates it. Encrypt the center's store covers backup and rotation.
  • Network. Each member reaches the center's port over TCP, 8444 in this section. The center needs no route to the members.
  • Disk. The store file holds the newest heartbeat of each member: 1,884 and 1,883 bytes of JSON for the two members of this run. Encryption adds 21 bytes to each record and 120 bytes to the file.
  • Bounds. The center reads at most 4 MiB of one request (MAX_INGEST_BYTES) and answers 413 past it. Its rate limit counts at most 4096 addresses in one second (MAX_TRACKED_PEERS); a further address in that second gets 503, with --rate-limit-per-peer above 0.
  • Other systems. No change. OpenSIPS, Kamailio, Homer and any HEP collector keep their configuration.

Step 1: start the center

  • [center] Create the signing key, readable by the center's user only:

    # Run all of these, in order.
    install -m 600 /dev/null /etc/sipnab/center.signing
    head -c 32 /dev/urandom | base64 > /etc/sipnab/center.signing
  • [center] Create the store key, and copy it to your backup. The command writes the file with mode 0600 and prints the key's id:

    sipnab fleet keygen --out /etc/sipnab/center.storekey
    store.key: key id ac7b02adecf75d31
    
  • [center] Start it:

    sipnab fleet serve --listen 127.0.0.1:8444 --store /var/lib/sipnab/center.store \
      --store-key-file /etc/sipnab/center.storekey \
      --signing-key-file /etc/sipnab/center.signing \
      --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.key
    INFO sipnab::app::fleet_center: fleet center store 'center.store': 0 member(s), 0 past --retention-hours 72 removed
    INFO sipnab::app::fleet_center: fleet center serving HTTPS on 127.0.0.1:8444
    INFO sipnab::output::api: REST API serves HTTPS only (TLS 1.2/1.3, ALPN http/1.1)
    INFO sipnab::output::api: REST API listening on 127.0.0.1:8444
    

    A center after 0.5.243 also logs the key it encrypts the store under, before the store line. This line is from a run on 2026-11-10:

    INFO sipnab::app::fleet_center: fleet center store encryption sealed, key id ac7b02adecf75d31, 0 old key(s)
    

    It creates the store with mode 0600, encrypted. It removes a member it has not heard from for --retention-hours (default 72), at start and every minute. --silent-after-missed (default 3) sets how many of a member's heartbeat intervals may pass before it is silent. Members reaching the center by a name need that name in --allowed-host. The CLI reference lists every flag.

  • [center] Start it as the user that owns /var/lib/sipnab, not as root. Since 0.5.256 a center that root starts reads its key files and binds, then serves as --user, else nobody. Run the center as a service has the unit and what a root start needs.

  • [center] To refuse any member without a certificate from your own CA, as well as its token, add --tls-client-ca /etc/sipnab/members-ca.pem. Require member certificates on a fleet center makes the CA and the member certificates, and says how a member presents one.

Step 2: mint a token for each member and for each reader

  • [center] A member's token has scope fleet-report, and its id is the member's name. The center vouches for that name, and for nothing else the member says:

    sipnab --mint-token --token-scope fleet-report --token-id edge-a \
      --api-signing-key-file /etc/sipnab/center.signing --api-token-ttl 2592000

    It prints one token, starting s2.. A fleet-report token reaches the center's ingest and nothing else, on the center and on every other sipnab. Give each member its own token, so revoking one leaves the others working.

  • [center] A token that reads the members list has scope inventory, as for sipnab fleet inventory:

    sipnab --mint-token --token-scope inventory --token-id fleet-reader \
      --api-signing-key-file /etc/sipnab/center.signing --api-token-ttl 2592000

Make an instance a fleet member

A member writes its inventory as a heartbeat every [fleet] heartbeat_secs (default 30) into its spool directory, and sends the spool to the center, oldest first. While the center is away the heartbeats wait, and the capture goes on: the member runs on a thread of its own and the capture never waits on it. The spool keeps at most [fleet] spool_max_files (default 2880, a day at 30 s) and drops the oldest past that. A heartbeat the center refuses for good (400, 413) moves to failed/ in the spool.

The output below comes from a second run on 2026-11-07 on one host: the center on 127.0.0.1:8444 over HTTPS, and edge-a reading a capture file with its REST API on 127.0.0.1:18080, reporting every 5 seconds.

  • Other systems. The member opens TCP connections to the center's port: open it outbound from each member. Nothing else changes.

  • [member] Put the token minted in step 2 in a file only the sipnab user can read, then add a [fleet] section to the member's configuration:

    [fleet]
    report_url = "https://127.0.0.1:8444"
    report_token_file = "/etc/sipnab/fleet/edge-a.report.token"
    report_ca_file = "/etc/sipnab/fleet/internal-ca.pem"
    spool_dir = "/var/lib/sipnab/fleet-spool"
    heartbeat_secs = 5

    The flags --fleet-report-url, --fleet-report-token-file, --fleet-report-ca-file and --fleet-spool-dir set the same and override the keys. report_url takes http:// only to loopback: sipnab refuses plain HTTP to another machine and does not start (exit 2 for a flag, 1 for a key).

  • [member] Start the run as usual. It logs where it reports:

    INFO sipnab::app::fleet_member: fleet member reporting to https://127.0.0.1:8444/ every 5 s, spool 'fleet-spool'
    
  • [controller] The center lists the member, with the member's own inventory under self_reported:

    {"member":"edge-a","silent":false,"seconds_since_heartbeat":3,"verified":{"heartbeats":2,"peer":"127.0.0.1","received_at":"2026-11-07T18:31:28Z"},"node":"edge-a","interval":5,"seq":2}
    

    That line comes from the members list through jq -c '.members[] | {member, silent, seconds_since_heartbeat, verified, node: .self_reported.inventory.node_name, interval: .self_reported.heartbeat_secs, seq: .self_reported.seq}'.

Verify the member:

  • The center goes away. This run stopped the center for 12 seconds. The member logged it once, its REST API still answered 200, and two heartbeats waited in the spool (00000000000000000005.json and 00000000000000000006.json):

    WARN sipnab::app::fleet_member: fleet member: heartbeats wait in the spool: the center did not answer: 127.0.0.1:8444: Connection refused (os error 111)
    
  • The center returns. The spool drained, and the center counted every heartbeat once: heartbeats 8 for seq 8.

    INFO sipnab::app::fleet_member: fleet member: the center takes heartbeats again (was: the center did not answer: 127.0.0.1:8444: Connection refused (os error 111))
    

    The returned center first reported edge-a silent, from the time of its last heartbeat before the outage, then heard again once the spool drained.

  • The member stops. This run killed edge-a. The center reported it silent three 5 s intervals later:

    WARN sipnab::app::fleet_center: fleet member 'edge-a' is silent: no heartbeat for 15 s, more than 3 of its 5 s intervals
    

Troubleshoot the member:

  • the center refused this member's token (HTTP 401). The token was not minted with this center's key, it expired, or the center revoked its id. The heartbeats wait in the spool. Mint a new token on the center (step 2), put it in the token file, and restart the member: the member reads the token once, at start.
  • the center did not answer. The center is down, or a firewall drops the member's connections. The heartbeats wait; nothing needs doing on the member once the center is back.
  • the center answered HTTP 503. The center is starting and rebuilds its member list, or its rate limit refused the member. The heartbeats wait, and the member tries again after 1 s, 2 s, 4 s and so on, at most one heartbeat_secs apart (What the center answers while it starts).
  • the center did not answer: sending: invalid peer certificate: UnknownIssuer. The member does not trust the center's certificate. Set report_ca_file to the certificate of the issuing CA.
  • the spool holds [fleet] spool_max_files. The center was away long enough to fill the spool, and the member dropped the oldest heartbeats. A newer heartbeat says everything an older one did.

Push call records to the center

Purpose. A member can send the center one record for each call that ended badly, so the center holds the calls an operator asks about without anyone opening a session on the member. The record's call is the member's own answer to GET /v1/dialogs/{call_id}, built by the same code, so the center and the member cannot disagree about a call. The record leaves out only the capture's identity, which the REST API stamps on each answer.

The outputs below come from a run on 2026-11-10 on one host, with a development build after 0.5.243: a center on 127.0.0.1:18444 and edge-a reading a capture file that holds one call that got a 486 and one call that the callee answered and that then ended.

Before you start.

  • A center that takes call records. Upgrade the center to this release first. An older center refuses a call record with 400, and the member moves that delivery to failed/ in its spool.
  • A fleet member that already reports heartbeats, as Make an instance a fleet member sets up.
  • A key file on the member, apart from every other key. Call records hold numbers and URIs. While they wait for the center they are on the member's disk only in sealed files, under this key.
  • Other systems. Nothing changes on OpenSIPS, Kamailio, Homer or any HEP collector. The member uses the connection to the center that its heartbeats already use.
  • Room at the center. The center keeps the records in one sealed file beside its store, <store>.intake, for --retention-hours, and holds at most 64 MiB there (MAX_INTAKE_BYTES); past that the oldest records go first. A center with the history store moves them from there into files it keeps for days, as Keep call history at the center shows.

Steps.

  1. [member] Create the spool key. The file gets mode 0600:

    sipnab fleet keygen --out /etc/sipnab/fleet/spool.key
    spool.key: key id 0728832701dd1d86
    
  2. [member] Name the key in the member's configuration and restart the member. With a spool key and no call_push, the member pushes exceptions:

    [fleet]
    spool_key_file = "/etc/sipnab/fleet/spool.key"
    call_push = "exceptions"
    exception_max_per_sec = 10
    • call_push = "exceptions" sends a call that ended on a final error response, or whose media diagnosis found one-way audio, no media, a NAT mismatch, an asymmetry between the two directions, late media or any other finding. The record says which in selected_by.
    • call_push = "all" sends every ended call. call_push = "none" sends none.
    • exception_max_per_sec (default 10, DEFAULT_EXCEPTION_MAX_PER_SEC; at most 10000, MAX_EXCEPTION_MAX_PER_SEC) caps the exceptions. The member keeps the unused allowance of at most 10 seconds (CAP_BURST_SECS), counts every call the cap refused, and does not send that call later. all is not capped.

    The member's start line names the rule:

    INFO sipnab::app::fleet_member: fleet member reporting to http://127.0.0.1:18444/ every 5 s, spool '/var/lib/sipnab/fleet-spool', call records: exceptions
    

The member looks at its dialogs every 5 seconds. It sends a call once the call has ended and has stood unchanged for 5 seconds, so the record is the answer the call settled on. When a call changes after the member sent it, such as one whose BYE arrives late, the member sends it again as it then stands. One delivery holds at most 500 records (MAX_DELIVERY_RECORDS) and 2 MiB (MAX_DELIVERY_BYTES). The member deletes a delivery when the center acknowledges it.

Verify.

  • [center] GET /v1/fleet/members counts what the center stored and what the member says it did. verified.call_records is the center's own count for the member. self_reported.calls is the member's: selected calls passed the rule and the cap, dropped_by_cap did not pass the cap, left_out were too large for a delivery, and the member wrote spooled of them for delivery:

    "intake": {"bytes": 944, "call_records": 1, "removed": 0}
    "verified": {"call_records": 1, "heartbeats": 3, "peer": "127.0.0.1", "received_at": "2026-11-10T15:44:12Z"}
    "calls": {"dropped_by_cap": 0, "left_out": 0, "selected": 1, "spooled": 1}
    

    The capture held two ended calls. The member sent the one that got a 486.

  • [center] The intake is a sealed file. Its first four bytes are SNFS:

    head -c 4 /var/lib/sipnab/center.store.intake; echo
    SNFS
    
  • [member] After the center acknowledged a delivery, the spool holds no file that ends in .sealed.

What it covers and what it does not.

  • A record the center acknowledged is on the center's disk before the acknowledgment, and survives a kill -9 of the center. The center counts a record that the member sends again as a duplicate and does not store it twice, also after a restart.
  • A member keeps no database. A call the rule did not select, or one the cap refused, is at the center only as a count.
  • The summary of a call measured over three checked-in captures is 583, 1,283 and 2,592 bytes of JSON. Raw SIP, RTCP and rtpengine control messages are not in the record.
  • When the spool is past [fleet] spool_max_files, the oldest file goes first, a delivery of call records like a heartbeat.
  • A record's sent_at and its call are the member's claim. The member's name comes from its token, which the center verified.

Troubleshoot.

  • needs [fleet] spool_key_file. call_push asks for records and the member has no key to seal them under. The run stops at its start. Create the key and name it, as in steps 1 and 2:

    [fleet] call_push "all" needs [fleet] spool_key_file: call records wait on disk only in sealed files. Create a key with `sipnab fleet keygen --out FILE`
    
  • can be read or written by other users, naming the spool key file. The member does not report until you run chmod 600 on the file and restart it. The capture goes on.

  • sealed layer:, then a file name, in the member's log. A delivery in the spool did not open: something changed the file, or the spool key is not the one that sealed it. The member moves it to failed/ and never sends it. Restore the key that sealed the file to send it, or delete the file.

  • verified.call_records stays at 0. Check the member's start line. It says call records: none when the member has no spool key or call_push is none. Under exceptions, a capture with no failed or degraded call sends nothing.

  • dropped_by_cap grows. The member sees more exceptions than exception_max_per_sec lets through. Raise the cap, or read the calls on the member, which still holds them in memory.

  • The center logs call record(s) past --retention-hours. The intake removed its oldest records, by age or because it held 64 MiB. Lower the members' cap, or shorten --retention-hours.

Step 3: send a heartbeat by hand

A heartbeat is one JSON record per line, POSTed to /v1/fleet/ingest with the member's token:

Field Meaning
schema_version 1.
kind heartbeat.
stream The member's delivery stream: 1 to 64 letters, digits, ., _ or -, chosen once.
seq The record's place in the stream, from 1.
sent_at When the member wrote it, RFC 3339.
heartbeat_secs How often the member sends one, 1 to 3600.
inventory The member's GET /v1/inventory answer.

The center counts a record once. In the same stream a record is new when its seq is past the newest one the center accepted. A record of another stream is new when its sent_at is later: a member that recreates its spool starts a new stream. The center counts a batch sent twice once, also after it restarts.

  • [member] Send edge-a's inventory as a heartbeat:

    # Run all of these, in order.
    curl -s -H "Authorization: Bearer $EDGE_A_INVENTORY_TOKEN" http://127.0.0.1:18080/v1/inventory \
      | jq -c '{schema_version: 1, kind: "heartbeat", stream: "edge-a-manual", seq: 1, sent_at: (now | todate), heartbeat_secs: 30, inventory: .}' \
      > heartbeat.ndjson
    curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $EDGE_A_REPORT_TOKEN" \
      -H 'Content-Type: application/x-ndjson' --data-binary @heartbeat.ndjson \
      https://127.0.0.1:8444/v1/fleet/ingest
    {"accepted":1,"duplicate":0,"member":"edge-a"}
    

    Sent again, the same file is a duplicate:

    {"accepted":0,"duplicate":1,"member":"edge-a"}
    

Verify the center

  • [controller] Read the members:

    curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $FLEET_READER_TOKEN" \
      https://127.0.0.1:8444/v1/fleet/members | jq 'del(.members[].self_reported.inventory)'
    {
      "members": [
        {
          "member": "edge-a",
          "seconds_since_heartbeat": 0,
          "self_reported": {
            "heartbeat_secs": 30,
            "sent_at": "2026-11-07T15:41:36Z",
            "seq": 1,
            "stream": "edge-a-manual"
          },
          "silent": false,
          "verified": {
            "heartbeats": 1,
            "peer": "127.0.0.1",
            "received_at": "2026-11-07T15:41:36Z"
          }
        }
      ],
      "now": "2026-11-07T15:41:36Z",
      "retention_hours": 72,
      "schema_version": 1,
      "silent_after_missed": 3
    }
    • member is the id of the token the heartbeat came with. The center verified it.
    • verified is what the center saw: when the newest heartbeat arrived, on the center's clock, from which address, and how many heartbeats it accepted from this member.
    • self_reported is what the member says about itself: its stream, place, send time, interval and inventory, the record in GET /v1/inventory. A member that is not honest can put anything there.
    • silent is true once more than silent_after_missed of the member's intervals passed with no heartbeat, measured on the center's clock.
  • [center] The center logs a member going silent, and coming back. In this run edge-b sent one heartbeat with a 1 s interval:

    WARN sipnab::app::fleet_center: fleet member 'edge-b' is silent: no heartbeat for 3 s, more than 3 of its 1 s intervals
    INFO sipnab::app::fleet_center: fleet member 'edge-b' is heard again
    

Revoke a member

  • [center] Start the center with --revoked-file, and add the member's token id to that file, one per line. sipnab re-reads the file when it changes, so the center keeps running. This run revoked edge-a:

    echo edge-a >> /etc/sipnab/center.revoked

    edge-a's next heartbeat gets 401, and edge-b's still gets 200. The members list keeps edge-a's last heartbeat until retention removes it.

Encrypt the center's store

Purpose. The center's store holds every member's inventory: node names, versions, configuration digests, addresses. The center never writes it in clear. It seals the store file with AES-256-GCM under a key that you keep in a file, and it refuses to start without that key. A center cannot run with encryption off.

The outputs below come from a run on 2026-11-10 on one host, with a development build after 0.5.243 and the files in one working directory.

Before you start.

  • A sipnab after 0.5.243 on the center. Members need no change, and no other system does: OpenSIPS, Kamailio, Homer and any HEP collector keep their configuration.

  • A place to back up one small file, apart from the center's disk. Whoever holds the key file can read the store. Without the key file nobody can.

  • A center from 0.5.237 to 0.5.243 wrote its store in clear and took no key. After the upgrade it does not start until you give it --store-key-file. Upgrade a center that stored in clear has the steps and the way back.

  • A center from 0.5.244 or 0.5.245, or a member from 0.5.245 wrote sealed format 1. A later release reads those files once and writes them again as format 2, and neither 0.5.244 nor 0.5.245 reads format 2. To keep a way back, stop the center and copy its store and its intake before the first start of the new release. A center from 0.5.244 has no intake, so only the first command applies to it:

    # Run all of these, in order.
    cp -p /var/lib/sipnab/center.store /var/lib/sipnab/center.store.format1
    cp -p /var/lib/sipnab/center.store.intake /var/lib/sipnab/center.store.intake.format1

    To go back, stop the center, put the copies back under their own names and start the release that wrote them. It then knows nothing the newer release stored. A member has nothing to copy: let its spool drain before a downgrade, because 0.5.245 moves a format 2 delivery to failed/ and does not send it.

Steps.

  1. [center] Create the key. The file gets mode 0600. The id it prints is not secret: the center logs it, and a refusal names it.

    sipnab fleet keygen --out /etc/sipnab/center.storekey
    store.key: key id ac7b02adecf75d31
    
  2. [center] Copy /etc/sipnab/center.storekey to your backup, and note the key id beside it.

  3. [center] Start the center with --store-key-file, as Step 1 shows, or add the flag to the service unit of a center you upgraded.

  4. To replace the key, create a second key file, and start the center once with the new file as --store-key-file and the old one as --store-old-key-file:

    # Run all of these, in order.
    sipnab fleet keygen --out /etc/sipnab/center.storekey.2
    sipnab fleet serve --listen 127.0.0.1:8444 --store /var/lib/sipnab/center.store \
      --store-key-file /etc/sipnab/center.storekey.2 \
      --store-old-key-file /etc/sipnab/center.storekey \
      --signing-key-file /etc/sipnab/center.signing \
      --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.key
    INFO sipnab::app::fleet_center: fleet center store encryption sealed, key id dd70bddbd340fae5, 1 old key(s)
    

    The center rewrites the store file under the new key as it starts. It does not rewrite the files of the history store: each history file stays under the key that wrote it until retention removes it. That is --calls-retention-days (default 90) for call records and --retention-hours (default 72) for heartbeats. Back up the new key file.

    • A center without the history store (its --version does not list fleet-store): from the next start on, leave --store-old-key-file out and destroy the old key.
    • A center with the history store: keep the old key file and --store-old-key-file on every start until sipnab fleet verify with the new key alone refuses no file (Verify the history store). That takes until retention has removed every history file written under the old key: the longer of the two periods after the rotation, 90 days with the defaults. Only then destroy the old key. A center that starts without the old key runs, and refuses each of those files with was written under key id <id>, which is not loaded (Keep call history at the center has the entry). Nothing reads a history file after you destroy its key.

Upgrade a center that stored in clear

Releases 0.5.237 to 0.5.243 wrote the store in clear. To upgrade such a center:

  1. [center] Stop the center. To keep a way back that loses no member, copy the store aside, and treat the copy as every member's inventory in clear:

    cp -p /var/lib/sipnab/center.store /var/lib/sipnab/center.store.clear
  2. [center] Install the new sipnab, create the key as in step 1 above and back it up.

  3. [center] Add --store-key-file /etc/sipnab/center.storekey to the center's command line or service unit, and start it. On this first start the center reads the clear store, writes a sealed copy beside it, waits until that copy is on disk, reads it back, and only then puts it in the store's place. A stop or a failure before that leaves the clear store as it was. The center logs one warning that the store held <n> member(s) in clear and is sealed now. The old bytes may stay in the disk's free space until the file system reuses it.

  4. [center] When the members list is right, delete the copy from step 1.

To go back to 0.5.243 or earlier: stop the center, and put the copy from step 1 back as the store, or move the sealed store away to start with an empty one. Do this before the older release starts: it cannot read a sealed store, and it either stops at start or starts with no members and writes over the file. Then install the older sipnab and start it without --store-key-file. With an empty store, the center lists each member again at its next heartbeat, and does not list a member that is silent then.

Verify.

  • The center's first log line names the key id that keygen printed.

  • The store is a sealed file. Its first four bytes are SNFS, and no member's name is in it:

    # Run all of these, in order.
    head -c 4 /var/lib/sipnab/center.store; echo
    grep -c edge-a /var/lib/sipnab/center.store
    SNFS
    0
    
  • GET /v1/fleet/members lists the same members after a restart.

What it covers and what it does not. The center seals the store file and the file it stages a rewrite in, and writes no other file. It authenticates each record with the record's position in the file, so a changed byte, a moved record or a store cut short stops the center at start. That includes records removed from the end of the file: after each append is on disk the center writes an authenticated count of the file's entries into the file's header, and the center refuses a file that holds fewer entries than its count (was cut at its end). The same holds for the intake beside the store and for a member's sealed deliveries. Two things it cannot detect: someone deleting the whole store, and someone putting back an earlier copy of the whole file, which is a valid file with its own valid count. Releases 0.5.244 and 0.5.245 wrote no count (sealed format 1). This release reads such a file once at start and writes it again with one (format 2), and an earlier release does not read a format 2 file. --store-encryption names how the center encrypts the store. This release has one mechanism, sealed, and refuses the values modular and both.

Troubleshoot. Each message stops the center with exit 2, and none shows a key.

  • --store-key-file is required. The center has no key. Create one and pass it:

    sipnab fleet serve: --store-key-file is required: the center encrypts everything it stores. Create a key with `sipnab fleet keygen --out FILE` and pass --store-key-file FILE
    
  • which is not loaded, after a key id. The store is under another key than the one given. Give the file of the key with that id as --store-key-file, or as --store-old-key-file beside a new key. The center leaves the store as it found it:

    sipnab fleet serve: --store 'center.store': sealed layer: the file was written under key id ac7b02adecf75d31, which is not loaded; give that key's file with --store-old-key-file
    
  • did not authenticate, was truncated, or was cut at its end. Something changed the store or cut it short after the center wrote it. The center leaves it as it found it. Restore the store from a copy, or move it away and start with an empty store: the members report again within one heartbeat interval each, and the list loses only a member that is silent now. When the message names the intake (center.store.intake), the same two choices apply to that file. An empty intake holds none of the call records the old one held.

    sipnab fleet serve: --store 'center.store': sealed layer: the file has a header that did not authenticate (the file was modified, or the key is wrong)
    

    A store with records removed from its end:

    sipnab fleet serve: --store 'center.store': sealed layer: the file was cut at its end: it holds 3 of the 5 entries it was written with
    
  • has a record <n> that did not authenticate, on a file no one changed. A center from 0.5.244 to 0.5.247 left this state when an append failed, for example on a full disk, and a later append worked. The center logged fleet center: the store could not be written: at the time and kept running. The center sealed the record after the failure for a position the file does not reach, so the file does not open at the next start. This release does not leave a file in that state. It does not read one that is in it:

    sipnab fleet serve: --store 'center.store': sealed layer: the file has a record 4 that did not authenticate (the file was modified, or its records were moved)
    

    Which file the message names decides what the center loses.

    • The store (--store '<file>'). Those releases wrote the store again in each minute in which a heartbeat arrived and when the center stopped on a signal, and that removed the state. A store is in it only when the center stopped another way before the next rewrite. Move the store away and start the center. It starts with an empty store. Each member reports again within one heartbeat interval, and the list loses only a member that is silent now.
    • The intake (the intake '<file>.intake', from 0.5.245 to 0.5.247). Those releases wrote the intake again only to remove records past the retention or the intake's size, and that rewrite failed on a file in this state, with the warning fleet center: the intake was not rewritten:. Move the intake away and start the center. It starts with an empty intake and holds none of the call records the old file held. The members do not send them again.

    Keep the file you moved away until you have checked the center: start it, wait one heartbeat interval, and read GET /v1/fleet/members. No sipnab command reads a part of a sealed file that does not open.

  • can be read or written by other users, naming the key file. Run chmod 600 on it:

    sipnab fleet serve: --store-key-file 'store.key' can be read or written by other users (mode 0644); chmod 600 it
    
  • is not a key. The file does not hold 32 bytes in base64. Use the file keygen wrote, unchanged:

    sipnab fleet serve: --store-key-file 'bad.key' is not a key: it holds 3 bytes, and a key is 32; create a key with `sipnab fleet keygen --out FILE`
    
  • File exists, from keygen. keygen never overwrites a file, because it may be the only copy of a key. Name a new file:

    sipnab fleet keygen: --out 'store.key': File exists (os error 17)
    
  • this build cannot read or write Parquet modular encryption. Leave --store-encryption out, or set it to sealed:

    sipnab fleet serve: --store-encryption modular: this build cannot read or write Parquet modular encryption; `sealed` is the one value it runs under
    

Keep call history at the center

Purpose. A center with the history store keeps the call records its members push for --calls-retention-days (default 90), and answers GET /v1/fleet/calls: the calls of one member, of a time window, or the call with one Call-ID. Without it, a record stays in the intake for --retention-hours and no route reads it.

The center files the records into Parquet files under a directory beside its store, <store>.parquet. It seals each file with the store key before the file reaches the disk, so no call record is on the center's disk in clear.

The outputs below come from a run on 2026-11-11 on one host, with a development build after 0.5.247: a center on 127.0.0.1:18447 over HTTP on loopback, with --calls-flush-secs 5, and two call records sent by hand as member edge-a. The commands name files as a deployment would, and the log lines show file names without the run's directory. A second run, on 2026-11-12 with a development build after 0.5.249, gave the lines that changed in that build: the history store's start line, the history object of the members report (one call record and two heartbeats filed), and the answer for a changed file. The answer of step 3 shows files_not_read as that build answers it. The WARN line of a build without the history store is the text such a build logs. The test suite starts such a center in each build without the history store and compares its log line and its 501 answer with this page.

Before you start.

  • A build with the history store. sipnab --version lists fleet-store among its features. The gnu tarballs, the .deb and .rpm packages that are not -noaudio, the macOS builds and the container image have it. The static musl tarballs and the -noaudio packages do not: the Parquet library is larger than the room the static binary has under its size limit. A center from one of those says so when it starts, and the route answers 501:

    WARN sipnab::app::fleet_center: fleet center: this build has no history store (cargo feature fleet-store): call records wait in the intake, which holds at most 64 MiB and --retention-hours of them, no heartbeat history is kept, and GET /v1/fleet/calls and GET /v1/fleet/members/{id}/history answer 501. The gnu tarballs, the .deb and .rpm packages that are not -noaudio, the macOS builds and the container image have the history store; the static musl tarballs and the -noaudio packages do not
    
  • A center that takes call records, and members that push them, as Push call records to the center sets up. The member needs no change for the history store.

  • The store key and its backup, as Encrypt the center's store describes. The center seals the history files under the same key. Without the key no one reads them.

  • A token that reads calls. GET /v1/fleet/calls needs scope full. A call record holds the numbers and addresses of a call, so an inventory token does not read it.

  • Room on the center's disk. The center compresses the files with zstd. In a run of the tests, 20,000 call records of about 0.8 kB each made one file of 660,654 bytes; releases 0.5.249 to 0.5.253 wrote the same records without compression as 16,493,345 bytes. See Upgrade a center to compressed history files for what the compression gives on other rows.

  • Other systems. Nothing changes on OpenSIPS, Kamailio, Homer or any HEP collector. A backup of the center now includes the directory <store>.parquet.

Steps.

  1. [center] Start the center with the retention you want. Both flags are optional:

    sipnab fleet serve --listen 127.0.0.1:8444 --store /var/lib/sipnab/center.store \
      --store-key-file /etc/sipnab/center.storekey \
      --signing-key-file /etc/sipnab/center.signing \
      --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.key \
      --calls-retention-days 90 --calls-flush-secs 60
    INFO sipnab::app::fleet_center: fleet center history store 'center.store.parquet': call records are kept 90 day(s), heartbeats 72 hour(s)
    
    • --calls-retention-days takes 1 to 548. The center removes the days past it at start and every minute. A day is the UTC day on which the center received the records.
    • --calls-flush-secs takes 1 to 3600. It is the time between two filings of the intake. The center also files when it starts and when it stops on a signal.
  2. [center] Mint a token that reads calls:

    sipnab --mint-token --token-scope full --token-id fleet-ops \
      --api-signing-key-file /etc/sipnab/center.signing --api-token-ttl 86400
  3. [controller] Ask for calls. With no parameter the answer is the newest 100 records:

    curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $OPS_TOKEN" \
      'https://127.0.0.1:8444/v1/fleet/calls?member=edge-a&call_id=busy-1%40192.0.2.10' | jq .
    {
      "calls": [
        {
          "call": {
            "call_id": "busy-1@192.0.2.10",
            "final_status": 486,
            "from": "sip:+12025550101@example.com",
            "state": "failed",
            "to": "sip:+12025550102@example.com"
          },
          "member": "edge-a",
          "received_at": "2026-11-11T23:54:37.489276093Z",
          "selected_by": "failed",
          "sent_at": "2026-11-11T23:54:37Z",
          "seq": 1,
          "stream": "edge-a-manual"
        }
      ],
      "complete": true,
      "count": 1,
      "files_not_read": 0,
      "files_read": 1,
      "more": false,
      "refused": []
    }

    The parameters, each optional:

    Parameter Meaning
    member Records of this member only.
    from Records the center received at this RFC 3339 time or later.
    to Records the center received before this RFC 3339 time.
    call_id The record of the call with this Call-ID. Percent-encode @ as %40.
    limit The most records to return, 1 to 1000 (MAX_LIMIT). Default 100. The newest come first.

    more is true when the limit cut the answer. Narrow the window or raise limit to read the rest. A record is in the answer from the moment the center acknowledges it, before the center files it.

    One question opens at most 4,096 files (MAX_FILES_PER_QUESTION), those with the latest records first. When a question rules in more, the answer holds what those files gave, complete is false, and files_not_read counts the files left. The center writes one file for each filing and day, which holds every member's records of that filing, so 4,096 files are about 2.8 days at the default --calls-flush-secs 60 (4,096 minutes), whatever the number of members. To reach further back with one question:

    • Give from and to. They rule files out by the day in the path and the times in the name, so the center answers a window of up to 2.8 days whole, anywhere in the retention. Ask for a longer time one window at a time.
    • Or start the center with a longer --calls-flush-secs. At 600 one question reaches 28 days (4,096 times 10 minutes). Records then wait up to that long in the intake before the center files them; a question reads the waiting records too.

    member does not rule a file out: the center opens each file of the window and reads only the part that can hold that member's records.

Verify.

  • [controller] GET /v1/fleet/members reports what the history store did since the center started. After one filing of one call record and one of two heartbeats:

    curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $READER_TOKEN" \
      https://127.0.0.1:8444/v1/fleet/members | jq -c '.history, .intake'
    {"available":true,"failed_filings":0,"files_written":1,"heartbeat_files_written":1,"heartbeats_filed":2,"heartbeats_removed":0,"heartbeats_waiting":0,"members_restored":0,"retention_days":90,"rows_filed":1}
    {"bytes":0,"call_records":0,"removed":0}

    files_written and rows_filed count call records. The heartbeat fields and members_restored are those of Keep heartbeat history at the center.

  • [center] The center logs each filing:

    INFO sipnab::app::fleet_center: fleet center: filed 2 call record(s) in 1 file(s) of the history store; 0 were in it already
    
  • [center] The files are under class and day: one file for each filing, which holds the records of every member. Each name holds the first and the last receive time in the file, in milliseconds, and h with 16 hex digits that tell the file's records from any other records of the same times:

    cd /var/lib/sipnab && find center.store.parquet -type f
    center.store.parquet/calls/date=2026-11-11/1794355201000-1794355203000-he7b857d7ce5986b7.sealed
    
  • [center] The center sealed each file. Its first four bytes are SNFS, not the PAR1 of a Parquet file in clear:

    head -c 4 center.store.parquet/calls/date=2026-11-11/*.sealed; echo
    SNFS
    

    The directories have mode 0700 and the files mode 0600.

Troubleshoot.

  • 401 on GET /v1/fleet/calls. The token's scope is not full. An inventory, metrics or fleet-report token does not read call records. Mint a full token, as step 2 does.

  • 400 with a detail. The detail names the parameter and what it accepts:

    {"detail":"limit must be a number from 1 to 1000","status":400,"title":"Bad Request","type":"https://sipnab.com/problems/bad-request"}
    {"detail":"\"membr\" is not a parameter of this route; it takes member, from, to, call_id and limit","status":400,"title":"Bad Request","type":"https://sipnab.com/problems/bad-request"}
    
  • 501 and this build has no history store. The center runs a build without the history store. Install a build that has it, as Before you start lists, and start the center on the same store. The center files the records still in the intake at that start.

  • "complete": false and a refused list. The center could not read a file the question needed. The answer holds the records of the other files. Each entry names the file under <store>.parquet and the reason:

    {
      "calls": [],
      "complete": false,
      "count": 0,
      "files_not_read": 0,
      "files_read": 0,
      "more": false,
      "refused": [
        {
          "file": "calls/date=2026-11-11/1794355201000-1794355203000-he7b857d7ce5986b7.sealed",
          "reason": "sealed layer: the file has a record 2 that did not authenticate (the file was modified, or its records were moved)"
        }
      ]
    }

    In 0.5.249 and 0.5.250 the reason started with a second copy of the path.

    • did not authenticate, was truncated or was cut at its end: the file changed after the center wrote it. Restore the file from a backup, or move it out of the directory. The records in it are then not answered.
    • was written under key id <id>, which is not loaded: the file is under an earlier store key. Start the center with that key in --store-old-key-file.
    • was written as '<path>' and is not at that path: someone moved or renamed the file. Move it back to the path the message gives.
  • "complete": false and files_not_read above 0. The question ruled in more files than one question opens. Ask again with member, from and to, for a part of the time at a time.

  • The center logs the call records were not filed and stay in the intake. The line ends with the cause: a directory the center cannot write, a full disk, or a file of the same day that it cannot read. The records wait in the intake, and the center tries again after --calls-flush-secs. failed_filings in GET /v1/fleet/members counts the tries that failed. Fix the cause before the intake reaches its 64 MiB: past that the oldest waiting records go first.

  • The center logs past --calls-retention-days <n> and not removed. The line names a file in an expired day that the center did not write, or one the system refused to remove. The center leaves such a file and its directory. Remove it by hand.

  • What the history store cannot show. A whole file that someone deleted leaves no trace in the store, as for the store file itself. An answer does not include records the intake dropped at its size limit before a filing.

What the center answers while it starts

Purpose. Know what a member, a reader and a supervisor get from a center between the time it binds its port and the time it serves, and what a stop does in that time.

Before you start. Nothing to set: every center does this. Nothing changes on a member.

What the center does at start, in order:

  1. It reads its keys, binds --listen, and drops to the user it serves as.

  2. It installs its SIGTERM and SIGINT handlers.

  3. It answers every request, on every route, 503 with Retry-After: 5 and this problem body, and logs where it does. Both outputs come from a test center with 500 members in 120 heartbeat files and no store file, on a port the system picked:

    503 {"detail":"the center is rebuilding its member list from the history store","status":503,"title":"Service Unavailable","type":"https://sipnab.com/problems/unavailable"}
    
    INFO sipnab::app::fleet_center: fleet center answering 503 on 127.0.0.1:33407 until it serves: the center is rebuilding its member list from the history store
    
  4. It opens its store file and its history, rebuilds its member list, files what the intake and the store file held, and applies retention.

  5. It logs fleet center serving HTTP on <address> (or HTTPS) and answers each request from then on.

A start with its store file reads one heartbeat file in step 4. A start without it reads every heartbeat file of --retention-hours, newest first (The files of the history store has the time for a fleet). A build without the history store answers the center is opening its store in step 3.

What a member does with the 503. It keeps the heartbeat or the call records in its spool and logs heartbeats wait in the spool: the center answered HTTP 503 once. It tries again after 1 s, then 2 s, 4 s and so on, never more than one [fleet] heartbeat_secs apart, and delivers the spool once the center serves. It does not read Retry-After.

A stop during the start. A SIGTERM or SIGINT during the rebuild in step 4 ends the rebuild before the next file it would read. The center logs the two lines below and exits 0. It did not rewrite its store file with the rebuilt list and did not open its intake, so the store file holds the members it held before the start, and the next start rebuilds again:

INFO sipnab::app::fleet_center: fleet center stopping: a stop was asked for before it served; it ends at the next file of the rebuild, or once the start's filing has ended
INFO sipnab::app::fleet_center: fleet center: the stop ended the rebuild of the member list at start; the store was not rewritten

A stop after the rebuild, while the start files, waits for that filing to end. The center then rewrites its store as it does at any stop, and exits 0. Releases up to 0.5.258 opened the store before they installed the handlers: a SIGTERM during the rebuild ended the process at once, and a request waited unanswered until the rebuild ended.

Verify. Move a test center's store file aside, start the center with a history that holds many heartbeat files, and ask for the members at once:

curl -s -i -H "Authorization: Bearer $OPS_TOKEN" http://127.0.0.1:8444/v1/fleet/members

The answer is 503 with Retry-After: 5 until the center logs fleet center serving, and 200 with every member after it.

Troubleshoot.

  • A supervisor marks the center failed during a long start. Its readiness check gets 503. Wait for 200, or for the fleet center serving line, before you mark the center up.
  • The 503 lasts longer than you expect. The center has no store file, or its store file is older than its newest heartbeat files, so it reads the heartbeat files of the whole window. Keep the store file in your backups with the history.

What waits during a filing

Purpose. Know which requests the center answers while it files, and which steps of a filing make a request wait.

The center keeps its members, its store file and its intake behind one lock. A push, GET /v1/fleet/members and the first step of a question each take that lock for a short time.

A filing of call records has three steps:

  1. Take. Under the lock, the center opens the intake file and reads its header and its length. It reads no record.
  2. Read and write. Without the lock, the center reads the records up to that length, parses them and writes the files of the history store. This is the step that takes time. The center answers pushes, status requests and questions during it. A record that arrives now goes to the intake and waits for the next filing.
  3. End. Under the lock, the center writes the intake again with only the records that arrived during step 2. It kept their lines in memory, so it does not read the intake file for this.

A filing of heartbeats has the same three steps over the rows that wait in memory. The releases 0.5.249 to 0.5.256 held the lock for all three steps of a filing of call records, so nothing got an answer until the filing ended.

Measured in a release build (Cargo feature set full, aarch64, load average 13 to 22, three runs) with a full intake of 54,000 synthetic call records (67,412,988 bytes before sealing): steps 1 and 3 together held the lock 19 to 22 ms when no record arrived during the filing, and 28 to 30 ms when 2,000 records arrived during it. Step 1 alone took 23 to 30 microseconds. Step 2 took 495 to 511 ms without the lock. In the same binary, the three steps as one under the lock took 607 to 633 ms.

What still waits:

  • One filing at a time. A second filing, of either class, and the retention pass on the history store wait until the running filing ends. They hold no lock that a request needs while they wait.
  • A push waits for another push. The center writes each push to disk under the lock before it answers.
  • Steps 1 and 3 of a filing, and the rewrite of the store file and of the intake by retention, hold the lock while they run.

GET /v1/fleet/calls opens the intake under the lock in the same way and reads the waiting records without it. A question asked between step 2 and step 3 finds a record both in a file and in the intake, and the answer shows it once.

A stop during a filing. A center that stops between step 2 and step 3 leaves the records in the files and in the intake. The next filing, at the next start, finds them in the files and files none of them twice. The line filed 0 call record(s) in 0 file(s) of the history store; N were in it already reports that.

Why a filing reads the intake safely while pushes arrive. A push adds its records after the length that step 1 took, and step 2 reads no byte past that length. A push also writes one of the two count slots in the file's header, and step 2 does not read the header again: step 1 read it while no push was writing. That matters because the center accepts a file with one count slot that does not check only when exactly one whole addition follows the counted records. A reader that saw a count slot half written, with a second push already started, would refuse a file that is whole. The filing's reader cannot see that state, so the center does not try a refused read again: a refusal is a real one. When retention or a repair writes the intake file again, it writes a new file and renames it into place. Step 2 keeps the file it opened, which nothing writes to.

The files of the history store

Purpose. Know what the center writes under <store>.parquet, how many files that is, what a person who can list the directory learns without the store key, and what to do with the files release 0.5.249 wrote.

The layout.

<store>.parquet/calls/date=YYYY-MM-DD/<first>-<last>-h<mark>.sealed
<store>.parquet/members/date=YYYY-MM-DD/<first>-<last>-h<mark>.sealed
  • calls holds call records and members holds heartbeats. The day is the UTC day on which the center received the rows.
  • The center writes one file for each filing and day. The file holds the rows of every member that the center received since the filing before. The number of files does not depend on the number of members: at the default --calls-flush-secs 60 a center writes at most 1,440 files a day in each class, for 1 member and for 2,000. A filing with no new rows writes no file.
  • <first> and <last> are the earliest and the latest receive time in the file, in milliseconds since 1970. <mark> is 16 hex digits computed from the rows; it keeps two files of the same times apart.
  • With 90 days of call records and 72 hours of heartbeats at 60 seconds, a center holds at most 129,600 call record files and 5,760 heartbeat files (the heartbeat retention removes whole days, so up to four days are on disk).

What a file holds. Inside the seal, each file is one Parquet file, and zstd at level 1 compresses each of its columns. Releases 0.5.249 to 0.5.253 wrote the files without compression, and this release reads those in place: see Upgrade a center to compressed history files. The column member is the member's id. The rows are in order of member and then of receive time. A row group ends where the member changes once it holds 128 rows, and at 10,000 rows at the latest, and each row group records its least and its greatest member. A question for one member opens each file of its window and decodes only the row groups that can hold that member: in a file with one row for each of 2,000 members, 1 row group of 16.

What follows from that layout:

  • A question for one member opens every file of its window, and the center unseals each of those files whole before it picks the row group. In the row group it reads the receive time, the member and, for a question that names a call, the Call-ID of every row, and it builds only the rows the question selects: it does not parse the JSON of the other rows.
  • One answer holds at most 1,000 rows (limit). A member's 24 hours at one heartbeat a minute are 1,440 rows: ask twice, with from and to.
  • A center that starts without its store file reads the heartbeat files from the newest back, up to 4,096 of them, to rebuild its member list. A center that starts with its store file reads one.

What the reads cost. Measured on 2026-11-14 in a release build (Cargo feature set full, aarch64, 14 cores, load average about 10) on synthetic rows: 1,440 heartbeat files, each with one heartbeat of about 300 bytes of inventory for each of 2,000 members, compressed, 78,860,456 sealed bytes in the 1,001 files of the first question. Each figure is the range of three runs.

What the center did Time
One member's newest 1,000 heartbeats, from 1,001 files (GET /v1/fleet/members/{id}/history) 226 to 230 ms
Of that: reading the 1,001 files from disk 28 to 43 ms
Of that: unsealing them 41 to 42 ms
Of that: decoding the one row group of each file that holds the member 158 to 162 ms
The same decoding as releases 0.5.249 to 0.5.254 did it, building all 128 rows of the row group 586 to 656 ms

What a start without the store file costs. The center rebuilds its member list from the heartbeat files, newest first, and builds a row only when it holds no row of that member yet or the row is newer than the one it holds. Measured on 2026-11-15 in one release build (Cargo feature set full, aarch64, 14 cores, load average 23 to 31) on the same kind of synthetic store, 1,440 files of 113,477,614 sealed bytes, three runs:

How the rebuild read the 1,440 files Rows built Time
As this release does it 2,000, one for each member 1.95 to 1.99 s
As releases 0.5.251 to 0.5.258 did it, in the same binary 2,880,000, every row 12.4 to 12.8 s

The center answers 503 for that time (What the center answers while it starts). A start with its store file reads one file.

Which reads build every row, and what a damaged row does. The center seals each file, so a changed file does not open at all. A row whose JSON, send time or text does not parse can only come from a defect in the release that wrote the file. A question refuses such a file, and names it under refused, only when it selects that row: a question for another member, for another call or for a window that does not hold the row gets its answer from the same file. Releases 0.5.249 to 0.5.254 refused the file for every question that read the row's row group. The rebuild of the member list at start refuses a file only for a damaged row it builds: a damaged row older than the row it holds of that member is not built, and the start takes the file's other rows. Releases 0.5.251 to 0.5.258 refused the file at start for such a row too. One read still builds every row of each file it opens and refuses the file for one such row anywhere in it: the check a filing makes of the files of the same day that can already hold its rows. Neither read opens every file of the store. sipnab fleet verify does: it builds every row of every file and reports each file it refuses (Verify the history store). A member id that is not text, or a column that the center cannot read, refuses the file for every question that reads its row group, as before.

Reading the files with other tools. No other tool reads a file as it is on disk: the center seals every file under the store key, and the first bytes are SNFS, not PAR1. Read the history through the center, which answers JSON that jq, a spreadsheet import or a script takes:

curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $OPS_TOKEN" \
  'https://127.0.0.1:8444/v1/fleet/calls?from=2026-11-11T00:00:00Z&to=2026-11-12T00:00:00Z&limit=1000' \
  | jq -c '.calls[] | {member, received_at, call_id: .call.call_id}'

sipnab has no command that writes a file's Parquet in clear.

What the directory shows without the key. A person who can list the directory sees the days that have rows, the time of each filing to the millisecond, and the size of each file, which grows with the rows of that filing. The names hold no member id, no count and no part of a record. In release 0.5.249 each member had a directory member=<id>, so a listing showed every member's id and the times at which each member had new records.

Files that release 0.5.249 wrote. Releases 0.5.249 and 0.5.250 wrote call records under calls/date=YYYY-MM-DD/member=<id>/, one file for each member and filing. What this page says of the files of release 0.5.249 is true of the files of 0.5.250 too. This release writes no file there and does not move or rewrite the files that are there:

  • GET /v1/fleet/calls answers their rows together with the rows of the new files. A question with member opens only that member's old files.
  • The retention removes them with their day, and the log line that counts the removed files counts them.
  • A start after 0.5.249 stopped between a filing and the emptying of its intake does not file those records a second time.
  • The member ids in those paths stay on disk until the day passes --calls-retention-days.

Count the files, and check the inodes. A center of release 0.5.249 or 0.5.250 with many members wrote many small files: one for each member with a new call record in each filing, which for 2,000 such members at 60 seconds is 2,880,000 files a day. A file system holds a fixed number of inodes, and a center that runs out of them cannot file.

  1. [center] Count the files of the history store:

    find /var/lib/sipnab/center.store.parquet -type f | wc -l

    And the files of the 0.5.249 layout among them:

    find /var/lib/sipnab/center.store.parquet/calls -path '*/member=*' -type f | wc -l
  2. [center] Read how many inodes the file system has left (IFree, and IUse%):

    df -i /var/lib/sipnab
  3. [center] When IUse% is near 100, lower the retention so that the center removes the oldest days. The center removes the days past the retention at start and every minute:

    sipnab fleet serve --listen 127.0.0.1:8444 --store /var/lib/sipnab/center.store \
      --store-key-file /etc/sipnab/center.storekey \
      --signing-key-file /etc/sipnab/center.signing \
      --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.key \
      --calls-retention-days 7

    The center logs a line of this form with the days and files it removed, and df -i then shows the inodes as free:

    INFO sipnab::app::fleet_center: fleet center: 1 day(s) and 2 file(s) of call records past --calls-retention-days 7 were removed
    

    The center no longer has the call records of the removed days. To keep them, copy the date= directories to another file system first. The center reads a copied directory again once it is back at its path.

Troubleshoot.

  • The center logs the call records were not filed and stay in the intake with No space left on device while df -h shows free space. The file system has no free inode. Do step 3. The records wait in the intake, at most 64 MiB of them, and the center files them when it can.
  • A listing still shows member= directories after the upgrade. They are the files of release 0.5.249. The center reads them and removes them when their day passes the retention.

Verify the history store

Purpose. sipnab fleet verify reads every row of every file of the history store and reports each file it refuses. A question to the center builds only the rows it selects, so a file with a row that does not parse goes unnoticed until a question selects that row. This command selects every row of every file. Run it after you restore the store from a backup, after a disk fault, before you destroy an old store key, and on a schedule if you want to know about a refused file before a question finds it.

For each file the command makes these checks in this order, and reports the first one that fails:

  1. It reads the file.
  2. It opens the sealed layer under a key you gave. The layer must hold every record whole and name the path the file is at.
  3. It reads the Parquet footer. The schema must be the one this build reads, and this build must read the codec of every column.
  4. It builds every row: each JSON value, send time and text must parse.
  5. It compares the rows with the path. Every row's receive time must be on the UTC day of the date= directory. The earliest and the latest receive time must be the two times in the name, and the mark in the name must be the mark of the rows. For a file that release 0.5.249 wrote under member=<id>, every row must be of that member, and the first row must have the stream and the position in the name.

What it does not check. It does not look for a row that two files hold. It does not read the store file or the intake of call records. It reads only files under the names the store writes: another file in one of the directories is not a file of the store, and the center does not read it either.

Prerequisites.

  • A build whose --version lists fleet-store: the build the center runs. A build without the history store exits 2 and says so.
  • The store key file, and the file of every earlier key that a history file can still be under. A key rotation does not rewrite the history files: each stays under the key that wrote it until retention removes it.
  • Read access to <store>.parquet. Run the command as the center's user. A run that root starts reads the key files as root, so they can stay root mode 0600. It then lists and reads the history files as --user, else nobody. "A pass that root starts", below, has the details.
  • Nothing changes on a member or on the center. You do not stop the center: the command creates, changes and removes nothing. When retention removes a file between the command's listing and its read, the command counts the file as gone and does not refuse it.

Steps.

  1. [center] Run the pass with the center's --store and key:

    sipnab fleet verify --store /var/lib/sipnab/center.store \
      --store-key-file /etc/sipnab/center.storekey
    calls: 20 files, 20 clean, 0 refused, 0 gone, 1080000 rows
    members: 1440 files, 1440 clean, 0 refused, 0 gone, 2880000 rows
    

    The command prints one line for each class when it ends. files is the number of files it listed, and rows is the number of rows in the clean files. The outputs on this page are from a test store of synthetic rows for one day: 20 filings of 54,000 call records and one filing a minute of 2,000 members' heartbeats.

  2. [center] To read less, name a class or a window of UTC days. Both days belong to the window:

    sipnab fleet verify --store /var/lib/sipnab/center.store \
      --store-key-file /etc/sipnab/center.storekey \
      --class calls --from 2026-11-01 --to 2026-11-07
  3. [center] After a key rotation, name the earlier key too, as the center does:

    sipnab fleet verify --store /var/lib/sipnab/center.store \
      --store-key-file /etc/sipnab/center.storekey.2 \
      --store-old-key-file /etc/sipnab/center.storekey

    Run it once without --store-old-key-file before you destroy the old key. Each file it then refuses with was written under key id is a file that only the old key opens.

  4. [center] For a script, add --json. The command then prints one JSON document on one line and no other line. Here it is through jq ., for the test store with one file cut in half:

    {
      "classes": [
        {
          "class": "calls",
          "clean": 19,
          "files": 20,
          "gone": 0,
          "refused": 1,
          "rows": 1026000
        },
        {
          "class": "members",
          "clean": 1440,
          "files": 1440,
          "gone": 0,
          "refused": 0,
          "rows": 2880000
        }
      ],
      "refused": [
        {
          "class": "calls",
          "file": "calls/date=2026-11-11/1794355200000-1794355253999-h39bb7d2c07e281e0.sealed",
          "reason": "sealed layer: the file was truncated: it ends before its records do"
        }
      ],
      "result": "refused",
      "started_at": "2026-11-15T10:36:21Z",
      "version": "0.5.256"
    }

    started_at is when the pass started and version is the version of the program that ran it. result is clean or refused.

Check a backup copy. The command is also the check of a backup. Restore or copy the history directory to any place, keep its name as the name of a store file with .parquet added, and give that store file's path as --store. The store file itself need not be there, because the command does not read it. The paths under the history directory must be the ones the center wrote: the sealed layer of each file names the file's path under that directory.

sipnab fleet verify --store /mnt/restore/center.store \
  --store-key-file /etc/sipnab/center.storekey

A pass that root starts. The command does not stay root.

  • It reads --store-key-file and every --store-old-key-file as root.

  • Then it runs as --user, else nobody, and lists and reads the history files as that user. The command reads no configuration file, so [privilege] user does not name the user.

  • That user must read <store>.parquet, every directory in it and every file in them. When it cannot, the command reads no history file, exits 2 and names the path and three ways out:

    sipnab fleet verify: started as root, this command runs as user 'nobody', and 'nobody' cannot read '/var/lib/sipnab/center.store.parquet' (the history store of --store). Nothing was created. Choose one: chown nobody '/var/lib/sipnab/center.store.parquet'; run it with --user USER, naming a user who can; or pass --no-priv-drop to keep root
    

    For a center's store, take the second way and name the center's user: --user sipnab. Do not change the owner of a store that a center uses, because the center must keep its write access.

  • --no-priv-drop keeps root for the whole pass and logs one warning line.

  • A pass that root did not start changes in nothing, and --user has no effect on it.

Verify. The exit status is the verdict:

Exit status Meaning
0 The command refused no file. Files that were gone do not change this.
1 The command refused at least one file. Each has one refused: line, or one entry under refused with --json.
2 The pass did not run: the command refused an argument or a key file, it cannot read a directory of the store, the user a run started as root drops to cannot read the store, there is no <store>.parquet directory beside --store, or the build has no history store.

A store that has filed nothing yet has the directory and no file: the command prints 0 files for each class and exits 0.

How long a pass takes. Measured on 2026-11-15 with a release build on an NVIDIA Jetson AGX Thor (aarch64, 14 processors), with other jobs running on the host (load average 17 to 28). The pass read the test store above: 583,543,021 bytes of call record files and 441,207,414 bytes of heartbeat files. Each time is one run of the command, from its start to its exit, over three rounds:

Pass Files Rows Time
--class calls 20 1,080,000 2.8 to 6.2 s
--class members 1,440 2,880,000 4.2 to 9.0 s
Both classes 1,460 3,960,000 8.6 to 13.0 s

The pass reads every file of the classes it checks, whole.

Troubleshoot. A refused file has one line, the path and the first reason:

refused: calls/date=2026-11-11/1794355200000-1794355253999-h39bb7d2c07e281e0.sealed: sealed layer: the file was truncated: it ends before its records do
  • sealed layer: the file was written under key id <id>, which is not loaded. The file is under an earlier store key. Give that key's file with --store-old-key-file. Start the center with it too, or the center refuses the same file for each question that reads it.
  • sealed layer: with did not authenticate, was truncated or was cut at its end. The file changed after the center wrote it. Restore the file from a backup and run the command again.
  • sealed layer: the file was written as '<path>' and is not at that path. Someone moved or renamed the file. Move it back to the path the message gives.
  • parquet: with a row number. A row does not parse. The center seals each file, so only a defect in the release that wrote the file produces such a row. Keep the file and report the line with the version of the release that wrote the file. The center still answers every question that does not select the row.
  • rows:. The file opens and every row parses, and the rows are not the rows the path names: a row of another day or member, another earliest or latest time, or another mark. The center seals the path into the file, so only a defect in the release that wrote the file produces this. Keep the file and report the line.
  • no history directory '<path>': nothing to verify and exit 2. --store does not name the center's store file, or the build that runs the center has no history store. Compare the path with the --store of the center's service unit.
  • is a pipe or FIFO, not a file of the history store. The entry under a history file's name is not a regular file. The message names a character device, a block device or a socket in the same way. The center writes only regular files, and it refuses the same entry for each question that reads it. Neither the command nor the center opens such an entry for reading. Find out who made it, then remove it.
  • started as root, this command runs as user and exit 2. Root started the pass, and the user it runs as cannot read the path in the message. Run it with --user and the center's user. "A pass that root starts", above, has the other ways.
  • A file is gone. Retention removed it during the pass. Nothing is wrong.

Fleet verify lists each message.

Upgrade a center to compressed history files

Purpose. From this release the center compresses every history file it writes, call records and heartbeats, with zstd at level 1. Releases 0.5.249 to 0.5.253 wrote the same rows without compression. This section says what an upgrade changes on disk, how to see it, and what a downgrade does.

What the compression gives, measured on synthetic rows (the test edge_a_filing_is_measured_with_and_without_the_codec, with 2,000 members and 10 call records for each):

One sealed file 0.5.253, bytes This release, bytes Share
One heartbeat of each of 2,000 members 887,305 62,048 7.0%
20,000 call records 16,493,345 660,654 4.0%

The share depends on the rows. An inventory or a call record repeats most of its field names and many of its values from row to row, and that is what the codec removes. The rows of that test repeat more than the rows of a fleet do. In a second measurement, each row held digests, addresses and counters of its own, as GET /v1/inventory and GET /v1/dialogs/:call_id answer them: a heartbeat file was 11.7% of its uncompressed size and a call record file 10.7%. Text that does not repeat does not shrink.

The cost is time in each filing and each read. In that second measurement, in a release build on aarch64, encoding 2,000 heartbeats took 9.8 ms with zstd and 3.7 ms without, and encoding 20,000 call records took 98 ms and 59 ms. Decoding the 20,000 call records took 17.7 ms and 7.6 ms.

The codec also makes the binary larger. These are the sizes of the stripped binary that each build job of the release workflow printed for 0.5.253 and for 0.5.254, the first release with the codec:

Release build 0.5.253, bytes 0.5.254, bytes Change
aarch64 gnu 27,103,912 27,693,832 +589,920
x86_64 gnu 31,853,960 32,532,744 +678,784
aarch64 musl (static) 20,944,840 20,944,840 0
x86_64 musl (static) 25,385,600 25,389,696 +4,096
aarch64 gnu -noaudio 23,433,392 23,433,392 0
x86_64 gnu -noaudio 27,588,816 27,596,304 +7,488

The static musl builds and the -noaudio packages do not link the history store, so they do not link the codec. Release 0.5.254 also added regular expressions to the MCP tool search_messages, and all six builds hold that change. No build measured the two changes apart: the 4,096 and 7,488 bytes of the x86_64 builds without the store are not the codec's, and the growth of the two gnu builds holds both changes.

Before you start.

  • A center with the history store. The compression is part of the history store and has no flag and no setting. See Keep call history at the center for the builds that carry the store.
  • Other systems. Nothing changes on a member, and a member of any release that pushes to the center needs no upgrade. Nothing changes on OpenSIPS, Kamailio, Homer or any HEP collector. GET /v1/fleet/calls and GET /v1/fleet/members/{id}/history answer the same JSON as before, so a script that reads them needs no change. A backup of <store>.parquet copies smaller files from the day of the upgrade on.
  • The files on disk stay as they are. The center does not rewrite a file an earlier release wrote. Those files stay uncompressed until the retention removes their day, and the center answers from them and from the new files together.
  • A downgrade does not read the new files. Releases 0.5.249 to 0.5.253 have no zstd reader. A center of one of those releases leaves the compressed files on disk, answers no row from them, and names each of them under refused in the answer. Upgrade again and the rows are back. Keep this in mind before you move a center back to an earlier release.

Steps.

  1. [center] Read the size of the history directory before the upgrade:

    du -sb /var/lib/sipnab/center.store.parquet
  2. [center] Install this release and start the center with the command line it had. No flag is new.

  3. [center] Wait for one filing. The center files the heartbeats and the call records it received every --calls-flush-secs (60 seconds by default), and writes no file when it received no new row.

Verify.

  1. [center] List the heartbeat files of today with their sizes. The files written after the upgrade are smaller than the files of the same fleet written before it:

    ls -l /var/lib/sipnab/center.store.parquet/members/date=$(date -u +%F)
  2. [controller] Ask for the call records of a window that spans the upgrade. The answer holds rows from before and from after it, and refused is empty:

    curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $OPS_TOKEN" \
      'https://127.0.0.1:8444/v1/fleet/calls?from=2026-11-14T00:00:00Z&to=2026-11-15T00:00:00Z&limit=1000' \
      | jq '{rows: (.calls | length), refused}'

Troubleshoot.

  • The files written after the upgrade are as large as the files before it. The center that wrote them is not this release. Run sipnab --version on the center's host and check which binary the service starts.
  • After a downgrade, an answer lacks the newest rows and refused names files with a reason that ends Disabled feature at compile time: zstd. The center is a release from 0.5.249 to 0.5.253 and the files are compressed. The files are intact. Install this release or a later one.
  • A reason reads parquet: column received_at is compressed with snappy; this build reads zstd and uncompressed files, or names another codec. A release newer than this one wrote the file with a codec this release does not link. Install the release that wrote it. See the error reference.

Keep heartbeat history at the center

Purpose. A center with the history store keeps every heartbeat it accepts, not only the newest of each member. GET /v1/fleet/members/{id}/history answers one member's heartbeats: when the center heard the member, from which address, and what the member's inventory said at that time. The same rows rebuild the member list for a center that starts without its store file: the center lists a member that was silent before the stop as silent after the start, and still counts a record the member sends again as a duplicate.

The center files the heartbeats into Parquet files under <store>.parquet/members, sealed with the store key as the call records are. It keeps them for --retention-hours, the time a member stays listed after its last heartbeat.

The outputs below come from a run on 2026-11-12 on one host, with a development build after 0.5.249: a center on 127.0.0.1:18449 over HTTP on loopback, with --calls-flush-secs 5, and two heartbeats and one call record sent by hand as member edge-a. The commands name files as a deployment would.

Before you start.

  • A build with the history store, as Keep call history at the center lists: sipnab --version shows fleet-store. The static musl tarballs and the -noaudio packages do not have it. A center from one of those keeps each member's newest heartbeat in its store file, as before, and answers 501 on the history route.
  • A center and members that send heartbeats, as Run a fleet center sets up. The member needs no change.
  • The store key and its backup. The center seals the heartbeat files under the store key. Without the key no one reads them.
  • A token that reads members. The route needs scope inventory, the scope of GET /v1/fleet/members. full, actions and admin reach it too. A fleet-report token does not.
  • Room on the center's disk, and files. The center writes one file for each filing, which holds the heartbeats of every member: at the default --calls-flush-secs 60 that is 1,440 files a day for 1 member and for 2,000 (The files of the history store). In a run of the tests on 2026-11-13 (a debug build, load average about 30), 2,000 members with one heartbeat of about 300 bytes each made one file of 79,189 bytes in 202 ms, and 1,440 such filings, 24 hours, made 113,491,120 bytes. Releases 0.5.251 to 0.5.253 wrote the same rows without compression as 887,745,307 bytes. In a release build on 2026-11-14 (aarch64, 14 cores, load average about 10), one member's newest 1,000 heartbeats came from 1,001 such files in 226 to 230 ms. In a release build on 2026-11-15 (load average 23 to 31), the center rebuilt the member list without a store file from all 1,440 files in 1.95 to 1.99 s, and answered 503 meanwhile (The files of the history store has the parts of those times). The center answers members while it writes a file. Heartbeats that wait for a filing are in the store file, and at most 64 MiB of them wait (MAX_WAITING_BYTES).
  • Other systems. Nothing changes on a member, on OpenSIPS, Kamailio, Homer or any HEP collector. A backup of the center includes the directory <store>.parquet.

Steps.

  1. [center] Start the center as before. No flag turns the heartbeat history on: a build with the history store keeps it. Two flags you may already set govern it:

    sipnab fleet serve --listen 127.0.0.1:8444 --store /var/lib/sipnab/center.store \
      --store-key-file /etc/sipnab/center.storekey \
      --signing-key-file /etc/sipnab/center.signing \
      --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.key \
      --retention-hours 72 --calls-flush-secs 60
    INFO sipnab::app::fleet_center: fleet center history store 'center.store.parquet': call records are kept 90 day(s), heartbeats 72 hour(s)
    
    • --retention-hours is how long a member stays listed after its last heartbeat, and how long the center keeps heartbeats. The center removes a UTC day of heartbeats once every hour of that day is past the retention, at start and every minute.
    • --calls-flush-secs is the time between two filings, of call records and of heartbeats. The center also files when it starts and when it stops on a signal.
  2. [controller] Ask for a member's heartbeats with the token that reads the member list. With no parameter the answer is the newest 100:

    curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $READER_TOKEN" \
      'https://127.0.0.1:8444/v1/fleet/members/edge-a/history?limit=1' | jq .
    {
      "complete": true,
      "count": 1,
      "files_not_read": 0,
      "files_read": 1,
      "heartbeats": [
        {
          "member": "edge-a",
          "self_reported": {
            "calls": null,
            "heartbeat_secs": 5,
            "inventory": {
              "node_name": "edge-a",
              "version": "0.5.249"
            },
            "sent_at": "2026-11-12T20:59:10Z",
            "seq": 2,
            "stream": "edge-a-manual"
          },
          "verified": {
            "call_records": 0,
            "heartbeats": 2,
            "peer": "127.0.0.1",
            "received_at": "2026-11-12T20:59:10Z"
          }
        }
      ],
      "member": "edge-a",
      "more": true,
      "refused": []
    }

    A row has the fields of the member's row in GET /v1/fleet/members: verified is what the center saw, self_reported is what the member said. verified.heartbeats counts the member's heartbeats up to this one. The parameters, each optional:

    Parameter Meaning
    from Heartbeats the center received at this RFC 3339 time or later.
    to Heartbeats the center received before this RFC 3339 time.
    limit The most heartbeats to return, 1 to 1000. Default 100. The newest come first.

    A heartbeat is in the answer from the moment the center acknowledges it. A member the center has no heartbeat of has an empty answer with status 200. One question opens at most 4,096 files, as for call records: about 2.8 days of heartbeats at the default --calls-flush-secs 60, so one question reaches most of the default --retention-hours 72. Give from and to for the oldest hours. One answer holds at most 1,000 heartbeats (limit), which for one member at one heartbeat a minute is 16.7 hours.

Verify.

  • [controller] GET /v1/fleet/members reports the filings since the center started:

    curl -s --cacert /etc/sipnab/fleet/internal-ca.pem -H "Authorization: Bearer $READER_TOKEN" \
      https://127.0.0.1:8444/v1/fleet/members | jq -c .history
    {"available":true,"failed_filings":0,"files_written":1,"heartbeat_files_written":1,"heartbeats_filed":2,"heartbeats_removed":0,"heartbeats_waiting":0,"members_restored":0,"retention_days":90,"rows_filed":1}

    heartbeats_waiting counts the heartbeats not yet in a file, heartbeats_removed those that left before a filing, and members_restored the members the last start took from the history store.

  • [center] The files are under class members and day: one file for each filing, which holds the heartbeats of every member. The names are spelled as those of call record files:

    cd /var/lib/sipnab && find center.store.parquet/members -type f
    center.store.parquet/members/date=2026-11-11/1794355201000-1794355203000-he7b857d7ce5986b7.sealed
    
  • [center] To see the rebuild on a test center, stop the center, move its store file away and start it. In the run, the member had also sent one call record after its last heartbeat:

    INFO sipnab::app::fleet_center: fleet center: 1 member(s) the store file held nothing of, or an older heartbeat of, were taken from the history store
    WARN sipnab::app::fleet_center: fleet member 'edge-a' is silent: no heartbeat for 8 s, more than 1 of its 5 s intervals
    

    GET /v1/fleet/members then lists the member with 2 heartbeats, at position 3 of its stream (past the call record) and silent, and history.members_restored is 1.

Troubleshoot.

  • 401. The token's scope does not read members. Mint an inventory token, as step 2 of Run a fleet center does.

  • 400 with a detail. The detail names the parameter, or says the id in the path is no member id. This route takes no member and no call_id:

    {"detail":"\"call_id\" is not a parameter of this route; it takes from, to and limit","status":400,"title":"Bad Request","type":"https://sipnab.com/problems/bad-request"}
    
  • 501 and this build has no history store. Install a build that has it and start the center on the same store. The history starts at that start: a build without the store keeps no earlier heartbeat.

  • "complete": false. The center could not read a file (refused names it and the reason), or the question ruled in more files than one question opens (files_not_read). The reasons and their fixes are those of Keep call history at the center:

    {"complete":false,"refused":[{"file":"members/date=2026-11-11/1794355201000-1794355203000-he7b857d7ce5986b7.sealed","reason":"sealed layer: the file has a record 2 that did not authenticate (the file was modified, or its records were moved)"}],"files_read":0,"files_not_read":0,"count":0}
  • The center logs the heartbeats were not filed and wait in the store file. The line ends with the cause: a directory the center cannot write, a full disk, or a file of the same day that it cannot read. The center tries again after --calls-flush-secs, and failed_filings counts the tries. Past 64 MiB of waiting heartbeats, and past --retention-hours, the oldest leave before a filing and heartbeats_removed counts them.

  • The center logs the member registry was rebuilt from the newest. The center had no store file, or one older than the history, and more than 4,096 heartbeat files could hold a member's last heartbeat. The center read the newest 4,096, about 2.8 days at --calls-flush-secs 60. It lists a member that went silent before them when that member sends a heartbeat again.

  • The center logs was not read for the member registry. At start, a heartbeat file the center read for its member list did not open. The line names the file and the reason. The center starts and lists a member from an older file or from its store file alone. When the store file is also lost, the member returns to the list with its next heartbeat.

  • A member is missing after a start without the store file. The center restores a member from heartbeats it filed. A member whose last heartbeat is older than --retention-hours is not listed, with or without the store file. The center no longer has the heartbeats that still waited in the lost store file.

  • The center logs past --retention-hours <n> and not removed. The line names a file in an expired day that the center did not write, or one the system refused to remove. Remove it by hand.

  • What the heartbeat history is not. The store file stays the member list's own record in every build: keep it in the backup. The history rebuilds what the file lacks; it does not replace it.

Run the center as a service

The purpose is a center that starts at boot and never serves as root.

Before you start

  • The steps above work: the center starts by hand with your key files.
  • The package created the sipnab user. Check with id sipnab.
  • Nothing changes on a member.

Steps

  • [center] Give the key files to the service's user, and to nobody else:

    # Run all of these, in order.
    chown sipnab:sipnab /etc/sipnab/center.signing /etc/sipnab/center.storekey /etc/sipnab/center.key
    chmod 600 /etc/sipnab/center.signing /etc/sipnab/center.storekey /etc/sipnab/center.key
  • [center] Write /etc/systemd/system/sipnab-fleet-center.service. No package ships this unit:

    [Unit]
    Description=sipnab fleet center
    Documentation=https://sipnab.com
    After=network-online.target
    Wants=network-online.target
    
    [Service]
    Type=simple
    ExecStart=/usr/bin/sipnab fleet serve --listen 127.0.0.1:8444 \
      --store /var/lib/sipnab-fleet/center.store \
      --store-key-file /etc/sipnab/center.storekey \
      --signing-key-file /etc/sipnab/center.signing \
      --tls-cert /etc/sipnab/center.crt --tls-key /etc/sipnab/center.key
    User=sipnab
    Group=sipnab
    NoNewPrivileges=true
    ProtectSystem=strict
    ProtectHome=true
    PrivateTmp=true
    # /var/lib/sipnab-fleet, private to the sipnab user: the store, its
    # intake and the history store live there.
    StateDirectory=sipnab-fleet
    StateDirectoryMode=0700
    Restart=on-failure
    RestartSec=5
    
    [Install]
    WantedBy=multi-user.target
  • [center] Start it:

    # Run all of these, in order.
    systemctl daemon-reload
    systemctl enable sipnab-fleet-center
    systemctl start sipnab-fleet-center

Verify

  • [center] The service runs as sipnab:

    systemctl show sipnab-fleet-center -p User -p ActiveState
    User=sipnab
    ActiveState=active
    

A center that root starts

A unit without User=, or sudo sipnab fleet serve, starts the center as root. Since 0.5.256 it does not stay root.

  • It reads the signing key, the store key and the TLS key, and binds --listen, as root. So the key files can stay root mode 0600, and the center can bind a port below 1024.

  • Then it serves as --user, else nobody, and says so:

    INFO sipnab::privilege: Dropped privileges to user 'nobody' (uid=65534, gid=65534)
    
  • The store, its intake, the history store beside it and their directory must be that user's. A center that ran as root before 0.5.256 left them root's, so its first start after the upgrade refuses. It exits 2 and creates nothing:

    sipnab fleet serve: started as root, this command runs as user 'nobody', and 'nobody' cannot write in '/var/lib/sipnab' (the directory of --store). Nothing was created. Choose one: chown nobody '/var/lib/sipnab'; run it with --user USER, naming a user who can; or pass --no-priv-drop to keep root
    
  • Fix it once, for the user the center serves as, on the directory that holds your store:

    chown -R sipnab /var/lib/sipnab

    Then start the center with --user sipnab, or with User=sipnab in the unit.

  • --no-priv-drop keeps root, as before 0.5.256, and logs one warning line.

  • The center reads a --revoked-file again each time it changes, so the user the center serves as must be able to read it.

Troubleshoot the center

Each output below comes from the same run.

  • The center exits 2 with started as root, this command runs as user. Root started it and the user it serves as cannot use the path the message names. A center that root starts has the three ways out.

  • 401 on the ingest. The token is not a fleet-report token this center's key signed, it expired, or its id is in the revocation file. A full, admin or inventory token gets 401 there too: only a member's token reports. Mint one as in step 2.

    401 {"status":401,"title":"Unauthorized","type":"https://sipnab.com/problems/unauthorized"}
    
  • 401 on the members list. The token is a member's token, which reads nothing. Use an inventory token.

  • 403 and the token's id names no member. The token's --token-id has characters a member name cannot have. This run minted one with the id edge a:

    403 {"detail":"the token's id names no member: mint it with --token-id of 1 to 64 letters, digits, '.', '_' or '-'","status":403,"title":"Forbidden","type":"https://sipnab.com/problems/forbidden"}
    
  • 400 and a line number. A record in the batch is not one the center reads, and the center accepted none of the batch. The detail names the line and the field:

    400 {"detail":"line 2: not a record: expected ident at line 1 column 2","status":400,"title":"Bad Request","type":"https://sipnab.com/problems/bad-request"}
    
  • plain HTTP is refused except on loopback. The command line names an address other than loopback and no --tls-cert and --tls-key. It exits 2 before it creates the store:

    sipnab fleet serve: --listen 192.0.2.10:8444 without --tls-cert and --tls-key would take members' tokens and inventories in plain HTTP; plain HTTP is refused except on loopback
    
  • can be read or written by other users. Other users can read the store file. Run chmod 600 on it. It exits 2:

    sipnab fleet serve: --store 'center.store' can be read or written by other users (mode 0644); chmod 600 it
    
  • 503 and the center is rebuilding its member list from the history store. The center has bound its port and not yet opened its store. Ask again after the Retry-After seconds (What the center answers while it starts).

  • A member stays silent. The center has had no heartbeat from it for more than silent_after_missed of its intervals: the member is down, cannot reach the center, or the center refuses its token. seconds_since_heartbeat says how long ago the last one arrived.

docs/errors.md lists every message the center prints.

States and findings

State of each instance:

State Meaning
ok It answered with an inventory.
refused It answered with an HTTP status other than 200, such as 401 for a wrong, expired or narrower token, or 404 for a build without the inventory.
unreachable No answer: no connection, a timeout, a TLS failure, a token file sipnab would not read, an answer cut short (the reason names the bytes read, as the answer ended after 9 of its 40 bytes), or an answer that is not JSON.
incompatible Its inventory has a schema_version newer than this build reads, or none. Upgrade the controller's sipnab.

Findings across the fleet:

Finding Meaning
version_mismatch The instances that answered do not all run the same build. The finding names each build and the instances on it.
unlisted_hep_sender A HEP collector in the list receives from an address that no listed instance's URL resolves to.
hep_sender_refused A HEP collector in the list refused packets from a listed instance's address for authentication, and has not admitted a packet from it since.

Check a binary against its release

Every instance reports signature.state as unverified. A process cannot prove which binary it is, and one that someone had tampered with would report "verified" too.

  • Download SHA256SUMS.txt from the release the instance claims, and verify it:

    gh attestation verify SHA256SUMS.txt --repo NormB/sipnab
  • Compare the instance's binary.sha256 with the line for its artifact. On Linux the instance hashes /proc/self/exe, so after an upgrade replaces the file the digest still names the binary that is running.

What it cannot tell you

  • Every field is the instance's own claim. An attacker who controls a host can make it report anything.
  • For a secret held in a file, the digests cover its path only. Replacing the contents of such a file, for example rotating the key in --api-signing-key-file, changes no digest; it changes the file's inode or mtime in config.secret_files once the instance restarts. Two instances that hold different keys at the same path look the same: the answer holds nothing derived from a key. A secret given as a value, such as --api-signing-key or a ${NAME} the configuration expands, changes config.effective_sha256.
  • It reads only the instances in your hosts file. An instance nobody listed is invisible to it, unless it sends HEP to a listed collector.
  • It changes nothing. Pushing a configuration, upgrading and restarting are not part of it.

Home

Getting started

Using the TUI

CLI & automation

Configuration

Integrations (API & MCP)

Your SIP server

Add to your voice stack

Development & internals

Clone this wiki locally