ADAMANT IPFS Node v0.1.0: bounded storage, lifecycle-aware GC, and deterministic replication

[massivedev0]

We have completed one of the largest infrastructure changes in the ADAMANT IPFS Node: Issue #22 is closed and PR #26 has been merged into dev.

The node now has a production-oriented storage lifecycle. It bounds disk growth, cleans up failed uploads, distinguishes durable content from reclaimable cache, places replicas deterministically across the ADAMANT node set, repairs missing copies, and preserves every pre-existing CID during an upgrade.

Why this was necessary

The previous implementation could stream blocks into the blockstore before all request limits were known. An interrupted or rejected upload could leave blocks behind, successful uploads remained pinned without an expiration policy, and there was no explicit replication quorum or repair process.

That made several operational questions impossible to answer reliably:

• How much disk space can an upload consume?
• Which files are durable, temporary, released, or safe to reclaim?
• What happens if a request disconnects after importing only part of a file?
• Which nodes are responsible for a CID?
• Can a node reclaim space without deleting confirmed content?
• Will existing files remain available after the cluster is upgraded?

The new design answers those questions in code, configuration, metrics, tests, and an operational runbook.

1. Admission happens before storage

Uploads are rejected before they can consume unbounded disk space:

Boundary	Configuration	Response
Concurrent uploads	storage.maxConcurrentUploads	429
Aggregate request size	storage.maxRequestSizeBytes	413
Disk reserve	storage.diskReserveBytes	507
Files per request	maxFileCount	400
Individual file size	uploadLimitSizeBytes	400

The aggregate limit is enforced both against Content-Length and against the bytes actually streamed, because chunked requests do not declare their final size. Concurrent requests reserve disk atomically, so several uploads cannot all spend the same free-space headroom.

Each request also owns an upload session that tracks the blocks it created. A parser rejection, import failure, route error, strict-quorum failure, or client disconnect removes only those new blocks. Pre-existing blocks, blocks retained by another concurrent upload, and pinned blocks are preserved.

2. Files now have an explicit lifecycle

A datastore-backed registry under /adm/files records the lifecycle and storage accounting of every known CID.

• temporary represents an upload waiting for confirmation or transactional settlement
• confirmed represents durable content protected by policy
• expired represents content that has been released and may be reclaimed under pressure
• pinned and heldLocally are tracked separately from the logical state

Lifecycle transitions, pin operations, registry writes, upload cleanup, replica settlement, repair, and garbage collection are coordinated with per-CID locks and a storage-wide collection lease. Failure compensation restores both the pin and the registry record to their observed baseline instead of leaving them in contradictory states.

The default storage.confirmationRequired: false keeps the existing API contract: uploads become durable immediately. Deployments that enable confirmation receive a configurable TTL for abandoned uploads and must call the authenticated confirmation endpoint.

3. Garbage collection is pressure-driven and lifecycle-aware

Releasing a pin and deleting blocks are intentionally separate decisions.

A released file remains in the blockstore and can continue serving reads for free. Blocks are deleted only when the blockstore exceeds the configured high watermark or the filesystem falls into the disk reserve. This avoids throwing away useful cache only to fetch it again later.

The collector has several safety properties:

• Confirmed content held by this node is never selected for eviction
• Missing protection on a confirmed file is repaired before any deletion starts
• A run that cannot verify durable content aborts before its first destructive action
• Partial GC failures retain registry records so the next pass can retry safely
• Dry-run mode reports the exact release and retention plan without changing pins or blocks
• Scheduled sweeps are bounded and advance instead of repeatedly scanning the entire registry

The documented defaults are a 50 GiB high watermark, a 40 GiB low watermark, a 5 GiB free-space reserve, and a scheduled pass every 15 minutes. All values are configurable. Scheduled GC is enabled by default, but it performs no deletion while space remains above the safety thresholds.

Operators can inspect the plan with:

curl --fail-with-body \
  -X POST \
  -H "x-api-key: $ADMIN_API_KEY" \
  "https://ipfs.example.org/api/storage/gc?dryRun=true"

4. Replication uses the existing libp2p network

Replication runs over /adamant/replication/1.0.0, not over an additional HTTP service. The libp2p handshake proves the remote peer identity, so replication needs no shared API secret, second public port, or separate cluster daemon.

Operations that make this node responsible for content are accepted only from peers listed in nodes. Control messages are length-framed and bounded. Replica transactions record their originating peer, and only that peer can settle them.

Holders are selected with rendezvous hashing over the CID. Every node with the same membership list independently computes the same holder set, without a central coordinator. The default placement policy keeps:

• Four copies for fresh content
• Three copies after 180 days
• Two copies after one year

The count is capped by the actual network size, so a three-node network asked for four copies places one copy on every available node. Placement shrinks by file age rather than last-access time: tracking reads would create metadata about when users retrieve files.

Strict upload durability is optional. When replication.requireQuorumOnUpload is enabled, local admission and remote replicas form one rollback-capable transaction: peers stage copies, the origin verifies the configured acknowledgement quorum, and then commits or aborts every prepared replica. A strict configuration requires ackQuorum >= 2, ensuring that success proves at least one remote copy.

5. Repair, handover, and retrieval are part of the model

The repair job asks a peer whether it already has a CID and whether it has room before transferring data. Intake is bounded by concurrency, request size, disk reservation, timeout, and per-peer budget.

A node outside the current holder set hands its durable copy to the designated holders and releases its own pin only after those holders confirm they have the file. If every remote holder later disappears while the blocks are still local, the node takes responsibility again instead of allowing the last recoverable copy to vanish.

Reads also use placement information. Before serving a CID, a node connects directly to the peers expected to hold it rather than relying on a useful Bitswap peer already being connected. Periodic peering keeps the configured mesh available after startup.

This matters for ADAMANT Messenger: a sender and receiver normally use different infrastructure nodes, so the receiver's first read commonly lands on a node that is not a designated holder.

6. Existing files and CIDs are preserved

The upgrade does not re-import, rewrite, or rename stored content. CID generation remains compatible with the previous stack, so existing message links continue to address the same files.

At startup, pins that predate the lifecycle registry are backfilled as confirmed records. Their DAG sizes are measured offline, and incomplete content is reported rather than silently registered as durable. The API can start while the backfill continues in the background.

One capacity implication is important: the original upload time of a legacy pin cannot be recovered, so backfilled files are initially treated as fresh and enter the widest placement tier. Operators should plan cluster capacity for the existing corpus, not only for future uploads. Repair processes that corpus in bounded advancing batches rather than attempting to replicate everything in one pass.

Only /adamant/replication/1.0.0 is currently offered, so this release is intended for a coordinated cluster-wide upgrade. GET /api/storage/metrics exposes the active protocol version, making a mixed deployment visible.

7. Operational visibility and access boundaries

Public read-only routes expose capacity and lifecycle state without filenames, CID inventories, or peer topology:

• GET /api/file/:cid/status
• GET /api/storage/metrics
• GET /api/storage/policy

Administrative mutations such as confirmation, release, on-demand GC, repair, pin management, and libp2p topology operations require the configured x-api-key.

The storage report includes pinned and reclaimable bytes, filesystem availability, reserved and usable capacity, lifecycle counts, staged replica transactions, job status, and replication health. It provides enough information to validate an upgrade and monitor subsequent collection and repair passes without exposing private operational details.

Defaults operators should review

Setting	Default
Aggregate upload size	512 MiB
Concurrent uploads	32
Disk reserve	5 GiB
Temporary TTL	24 hours
GC schedule	Every 15 minutes
Repair schedule	Every 30 minutes
Fresh placement	4 copies
Upload acknowledgement quorum	1, best-effort replication
Strict upload quorum	Disabled

The defaults suit a dedicated storage volume and preserve the current immediate-upload behavior, but every operator should review capacity, watermarks, membership lists, and placement tiers before deployment.

Verification

The merged implementation passed the complete repository validation:

• 232 unit tests
• 102 integration tests
• Production TypeScript build
• ESLint and Prettier checks
• Production dependency audit
• Semgrep SAST and Semgrep OSS scans

It was also exercised on a four-node network. Sixteen files were placed on three holders while fresh, converged to exactly two holders after ageing into the next tier, and then read back byte-identically from all four nodes: 64 successful cross-node reads.

No new runtime dependency was introduced.

Deliberate follow-up work

This release establishes bounded storage and node-to-node durability, but it does not claim to solve every ownership or network-membership problem:

• #27 tracks deletion authorized by the original uploader's signature
• #28 tracks decentralized node discovery and Sybil resistance
• #29 tracks traffic accounting, backoff, and monthly limits

Content encryption remains the responsibility of the ADAMANT client protocol. The storage node manages encrypted content by CID and does not need plaintext access.

References

• Bounded storage issue #22
• Merged implementation PR #26
• Storage lifecycle and recovery guide
• IPFS Node README and deployment configuration

This is a substantial step toward an ADAMANT storage layer that is predictable under pressure, durable across node failures, observable in production, and compatible with every file already stored in the cluster.