RGS sync 404s after the first sync: persisted latest_rgs_snapshot_timestamp isn't aligned to the RGS server's snapshot cadence

[vincenzopalazzo]

Summary

After a successful RGS sync, update_rgs_snapshot() (in ldk-node src/gossip.rs) persists latest_rgs_snapshot_timestamp from the value returned by update_network_graph() — which is the snapshot's internal latest_seen_timestamp (e.g. mid-day UTC). The reference RGS server (https://rapidsync.lightningdevkit.org) only serves snapshots at 24h-aligned timestamps (00:00 UTC). On the next periodic sync, LDK requests /snapshot/v2/<mid-day-ts> and gets HTTP 404, so every subsequent sync fails until the persisted value is wiped.

Effect: a fresh node syncs once (works), then Background sync of RGS gossip data failed: Failed to update gossip data repeats every interval and across all restarts.

Filing here in ldk-server because that's the layer I'm running, but the fix is upstream in ldk-node.

Reproduction

1. Run ldk-server with rgs_server_url = "https://rapidsync.lightningdevkit.org/snapshot/v2/" (the contrib default).
2. First sync succeeds.
3. get_node_info().latest_rgs_snapshot_timestamp is e.g. 1778068800 = 2026-05-06 12:00:00 UTC.
4. On any subsequent sync (or restart), the failure repeats.

Server behavior, confirmed manually:

for ts in 1778025600 1777939200 1777852800 1778068800; do
  code=$(curl -sS -o /dev/null -w "%{http_code}" "https://rapidsync.lightningdevkit.org/snapshot/v2/$ts")
  echo "ts=$ts ($(date -u -d @$ts '+%F %T')): $code"
done
ts=1778025600 (2026-05-06 00:00:00): 200
ts=1777939200 (2026-05-05 00:00:00): 200
ts=1777852800 (2026-05-04 00:00:00): 200
ts=1778068800 (2026-05-06 12:00:00): 404

Root cause

In ldk-node src/gossip.rs update_rgs_snapshot():

200 => {
    let new_latest_sync_timestamp =
        gossip_sync.update_network_graph(response.as_bytes()).map_err(...)?;
    latest_sync_timestamp.store(new_latest_sync_timestamp, Ordering::Release);
    Ok(new_latest_sync_timestamp)
}

update_network_graph() reads latest_seen_timestamp from the snapshot binary header (rust-lightning lightning-rapid-gossip-sync/src/processing.rs:90) and returns it — that's the most recent gossip-message timestamp inside the snapshot, not the snapshot's URL/cadence timestamp.

Workaround

Delete the node_metrics row from ldk_node_data.sqlite before each daemon start; forces a fresh /snapshot/v2/0. Functional but ugly:

DELETE FROM ldk_node_data WHERE primary_namespace='' AND secondary_namespace='' AND key='node_metrics';

Suggested fix

Round new_latest_sync_timestamp down to the nearest 24h boundary before persisting (matches the reference RGS server's cadence and keeps deltas small). Could be a const or made configurable.

Alternatively, persist the request URL's query_timestamp instead of the snapshot's internal one — but then deltas always start from the previously-requested boundary, which means full re-fetch when starting from 0.

If the canonical contract is that any client-supplied timestamp must be honored, this is a server bug instead — but updating deployed clients is the more tractable fix.

Environment

• ldk-server @ 50fe752
• ldk-node 0.8.0+git @ 21eea8c
• lightning-rapid-gossip-sync 0.3.0+git @ 38a62c3
• Network: bitcoin (mainnet)
• Bitcoin backend: [esplora] server_url = "https://mempool.space/api"
• Linux x86_64

Closest existing ldk-node issue: #615 (different — shutdown race). This one is steady-state.

[TheBlueMatt]

This shouldn't be the case since lightningdevkit/rapid-gossip-sync-server#35 can you share more details about when you saw this?

[vincenzopalazzo]

Re-checked with the help of claude the live server today (2026-05-07 ~10:00 UTC):

ts=1778122800 (2026-05-07 03:00:00, 3h-aligned) → 200
ts=1778133600 (2026-05-07 06:00:00, 3h-aligned) → 200
ts=1778144400 (2026-05-07 09:00:00, 3h-aligned) → 404   ← latest 3h boundary
ts=1778148000 (2026-05-07 10:00:00)              → 404
ts=1778144401 (off by 1s)                        → 404

So even with perfect 3h alignment, anything after the deployed server's latest fully-regenerated reference_timestamp 404s. That points to the production server running with RAPID_GOSSIP_SYNC_SERVER_SNAPSHOT_INTERVAL larger than the symlink granularity (e.g. 86400) — symlinks only refresh once a day, so any latest_seen_timestamp from intra-day gossip is "in the future" until tomorrow's regen.

Client side: update_network_graph returns the snapshot's binary-header latest_seen_timestamp (lightning-rapid-gossip-sync/src/processing.rs:332):

let latest_seen_timestamp: u32 = Readable::read(read_cursor)?;
...
return Ok(latest_seen_timestamp);

That value is the timestamp of the latest gossip message embedded in the snapshot — derived from channel_updates, not from the snapshot's URL/cadence — and ldk-node persists it verbatim (gossip.rs::update_rgs_snapshot). On the next sync the client requests /snapshot/v2/<that-message-ts>, and unless the server's latest reference_timestamp has caught up past it, we get 404.

Repro on my running ldk-server right now:

• Persisted: latest_rgs_snapshot_timestamp = 1778144400 (returned by a successful 09:49 UTC sync)
• GET /snapshot/v2/1778144400 → 404
• RGS sync failures since: 6 / 10 attempts today (Failed to update gossip data)

Two ways out client-side:

1. Persist the timestamp ldk-node sent in the request (the boundary the server demonstrably had a symlink for) rather than the snapshot's latest_seen_timestamp. On a fresh /snapshot/v2/0 we'd start from 0, but every subsequent boundary advances cleanly.
2. Round latest_seen_timestamp down — but the client doesn't know the server's snapshot interval, so this needs config or a header.

Option 1 is the cheapest correct behavior. Happy to put up a PR if that direction sounds right.

Env: ldk-server 50fe752, ldk-node 0.8.0+git @21eea8c, RGS server https://rapidsync.lightningdevkit.org/snapshot/v2/

P.S: I asked claude to solve the 404 now lets see if it came out with some kind of solution that fix the 404

[vincenzopalazzo]

@TheBlueMatt looks like that with this patch vincenzopalazzo/ldk-node#3. I do not see the error anymore that I had before.

Wondering if this is correct tho

[TheBlueMatt]

anything after the deployed server's latest fully-regenerated reference_timestamp 404s.

Yes, that's the expected behavior. The clients should be requesting the URL for "last time I synced" which should never go past the server's latest timestamp (which it reports in all its files).

That points to the production server running with RAPID_GOSSIP_SYNC_SERVER_SNAPSHOT_INTERVAL larger than the symlink granularity (e.g. 86400) — symlinks only refresh once a day, so any latest_seen_timestamp from intra-day gossip is "in the future" until tomorrow's regen.

No, the server is refreshing every few hours and symlinks appropriately for it.

That value is the timestamp of the latest gossip message embedded in the snapshot — derived from channel_updates, not from the snapshot's URL/cadence

This certainly isn't supposed to be true. The LDK Node code is correct, if we're returning something other than the aligned timestamp from the sync method that's the bug in RL.

[vincenzopalazzo]

Ah ok so I misunderstood how rgs works, thanks for clarifying :)

[TheBlueMatt]

We shouldn't ever hit a 404 during normal operation, though, if we are there's still a bug that needs fixing.

[vincenzopalazzo]

Mh! so on my fork this was stopping to generate the vincenzopalazzo/ldk-node#3 404 but not deep enough in rgs to see if the bug is ldk node or rgs itself?