Release N.2.9.7-alpha1-dev-sync-performance-20260909

Arweave 2.9.7-alpha1-dev-sync-performance-20260909 Release Notes

This is a development prerelease containing changes under active
development. It may not work correctly, and some changes may not be included
in a regular release.

Based on Arweave 2.9.7-alpha1, this release introduces adaptive per-peer sync
concurrency, better balancing across storage modules, localized backpressure
for slow peers and disks, and lower sync-record snapshot memory use.

Configuration changes

• sync_jobs and [sync, jobs] are removed. To limit your sync rate, use
  [sync, max_download_rate] or sync_max_download_rate in bytes per second,
  or infinity (the default). A value of 0
  disables syncing.
• [sync, max_concurrent_peer_scans] and the legacy
  data_discovery_max_concurrent_peer_scans option have been removed without
  replacement. Discovery concurrency is automatic.
• [sync, cache_size_limit] is now [sync, cache_size], measured in MiB.
• [network, client, http, connections_per_peer] is new and defaults to 8.
  Most operators should leave it unchanged.

Other option changes are below. Legacy flat names remain supported and unchanged.

Old option	New option
[disk_pool, jobs]	[disk_pool, workers]
[gossip, header_sync_jobs]	[gossip, header, workers]
[gossip, header_cache_size]	[gossip, header, cache_size]
[network, server, tcp, ...] socket settings	[network, server, socket, ...]
[network, client, tcp, ...]	[network, client, socket, ...]
Server listener, connection, and idle settings	[network, server, http, ...]
Server shutdown and socket-backend settings	Nested shutdown and socket paths

Replica 2.9 guidance

Preparing entropy first remains recommended: start with
[sync, max_download_rate] set to 0, wait for preparation, then restart with
a positive rate or infinity.

Metrics and compatibility

Custom dashboards may need updating. Key replacements are
chunk_write_rate_bytes_per_second → store_drain_rate_bytes_per_second,
peer_interval_cache_* → chunk_interval_cache_*, and data_discovery →
sync_discovery_peers. New sync_peer_*, sync_tasks_by_*, sync_claimed_*,
and sync_sweep_offset metrics describe scheduler behavior.

If downgrading to 2.9.7-alpha1 after running this
release, stop the node and remove the peers file from data_dir; the older
release cannot read the new peer-performance cache format, and the cache will
be rebuilt automatically.

Release N.2.9.6

Arweave 2.9.6 Release Notes

This Arweave node implementation proposes a soft fork that activates at height 2,000,000, approximately 2026-09-13 12:00 UTC. This software was prepared by the Digital History Association, in cooperation with the wider Arweave ecosystem.

This release terminates the grace period for accepting format: 1 transactions which were deprecated in April 2020. format: 1 present issues for clients that use them and have long been deprioritized for distribution by nodes in the network. format: 1 transactions now represent ~0.007% of all Arweave base layer interactions, and all known libraries have long-since upgraded. Continuing to preserve format: 1 TX support presents complications in ensuring robustness and reliability of the many transaction management/caching subsystems of the node -- one such latent issue is also fixed in this release. Subsequently, as of height 2,000,000 the grace period for acceptance of format: 1 transactions is scheduled to cease.

While this release itself does not create an immediate forking incompatibility at the activation height, all operators are encouraged to upgrade ahead of time in case their node does attempt to place a Format: 1 transaction after the activation height. Such blocks will not be accepted by other peers.

Release N.2.9.7-alpha1

Arweave 2.9.7-alpha1 Release Notes

This is an alpha update and may not be ready for production use.

This Arweave node implementation proposes soft fork 2.9.6 that activates at height 2,000,000, approximately 2026-09-13 12:00 UTC. This software was prepared by the Digital History Association, in cooperation with the wider Arweave ecosystem.

This release terminates the grace period for accepting format: 1 transactions which were deprecated in April 2020. format: 1 present issues for clients that use them and have long been deprioritized for distribution by nodes in the network. format: 1 transactions now represent ~0.007% of all Arweave base layer interactions, and all known libraries have long-since upgraded. Continuing to preserve format: 1 TX support presents complications in ensuring robustness and reliability of the many transaction management/caching subsystems of the node -- one such latent issue is also fixed in this release. Subsequently, as of height 2,000,000 the grace period for acceptance of format: 1 transactions is scheduled to cease.

While this release itself does not create an immediate forking incompatibility at the activation height, all operators are encouraged to upgrade ahead of time in case their node does attempt to place a Format: 1 transaction after the activation height. Such blocks will not be accepted by other peers.

Configuration

Arweave node configuration has been completely rewritten. For details please
see the update docs: https://docs.arweave.org/developers/mining/readme/configuration

New Option Names

All configuration options have been renamed to allow grouping, and reflect their hierarchy.
Legacy options are still supported.

The new option names and their descriptions can be seen through an updated arweave config help.

New Option Formats

New ways of supplying configuration to arweave have been implemented.

• New, hierachic JSON and YAML config files
• --config_param Command line flags with double-dash allow access to the renamed
  configuration parameters.
• AR_CONFIG_PARAM=VALUE Every configuration option can be set through an environment variable.
• Legacy command line flags and config.json format are still supported.

Dynamic Set/Get

All options can now be queried directly from a running node, and many options can be changed
without needing to restart the node.

• arweave config get [option] — print the current value of an option (or the
  whole config). Values are returned as flattened strings, and the output of
  config get can be fed straight back into config set.
• arweave config set <option> <value> — set an option, including list options
  such as peers.local.

Configuration migration tool

A CLI tool has been added to arweave to convert old style configuration files
to new JSON or YAML configuration files.

./bin/arweave convert_config yaml config.json config.yaml

or

./bin/arweave convert_config json config.json config-new.json

Please, see the following link for detailed information on migrating your configuration:
https://docs.arweave.org/developers/mining/readme/configuration/migrating-config

Storage modules defined by byte range

Storage modules are now defined by an explicit range_start / range_end byte
range rather than the older bucket_size / index (bucket) notation. Any range
with range_end > range_start is valid, giving operators arbitrary byte-range
storage modules. A module is defined by either a partition or an explicit
range_start + range_end (mutually exclusive). The pre-2.9.6 bucket notation
is still accepted through the legacy config parser for backward compatibility.

https://docs.arweave.org/developers/mining/readme/configuration/migrating-config

Rate Limiter and Throttling

The rate limiter is still the sliding window limiter integrated with leaky bucket
tokens limiter. However several improvements and fixes have been applied.

Please, see the following link for detailed information on rate-limiting and configuring
your node's rate-limiting: https://docs.arweave.org/developers/mining/operations/rate-limiting

Headers

It is using new naming convention for the different limiting groups in the HTTP headers.
There is an additional header that is not part of the Polli IETF draft.
https://www.ietf.org/archive/id/draft-polli-ratelimit-headers-02.html

Throttling

The client throttling mechanism has been replace with a fully-adaptive throttling
logic. Throttling doesn't need configuration parameters, and it will
acquire the remote peer's settings after the first request sent to a compatible
host. Traffic to incompatible peers is not throttled.

The new throttling solution relies on appropriately named limiting pools that
have been introduced in this release.

Peer configured as local_peers are not throttled.

Account Tree Performance Improvements

The account (wallet) tree has been re-implemented to improve tree initialization and
update times by an order of magnitude. These improvements are visible now via slightly
faster node launch and block processing times, and ensure efficient scalability as
the number of accounts grow in the future.

Faster metrics endpoint

GET /metrics now serves from a cache. The endpoint stays fast
regardless of node load or the number of metrics.

Additional Fixes

• Several input validation steps could crash on invalid values, in some cases halting the arweave node.
  The patch includes graceful validation of certain inputs and defensive deserialization of local binaries.
• Previously repack-in-place from replica.2.9 to unpacked would occasionally stall with only a few
  GB left in the storage module. This has been fixed.
• In order to simplify repack-in-place we've removed the repack_cache_size_mb option. Repack-in-place
  can now be tuned almost entirely via the [packing, entropy, cache_size] option
  (legacy: replica_2_9_entropy_cache_size_mb). Allocating more memory to the entropy cache should speed up
  repack-in-place performance.
• A collection of sync performance improvements have been implemented (including a fix for a stall that
  could occur with take_one_timeout warnings).
• Performance improvements for GET /sync_buckets

Community involvement

A huge thank you to all the Mining community members who contributed to this release by identifying and investigating bugs, sharing debug logs and node metrics, and providing guidance on performance tuning!

Discord users (alphabetical order):

• Butcher_
• Evalcast
• JF
• lawso2517
• smash
• timothynode

And a further huge thank you to the following researchers who identified and helped to patch issues addressed in this release!

• bbl4de (https://github.com/bbl4de)

Release N.2.9.6-alpha2

This is an alpha update and may not be ready for production use. This software was prepared by the Digital History Association, in cooperation from the wider Arweave ecosystem.

Data Sync

General data sync performance improvements resulting in more stable download throughput.

The problem

Set of candidate peers periodically drops low or to zero while syncing

The fix

Optimize peer discovery and sync_task queue:

One process per peer rather than one process per storage module. This prevents a lot of rate limiting that used to happen before when multiple storage module processes would hit the same peer concurrently. Now there's one process that can manage backpressure when making GET /data_sync_record to a peer

Several bug fixes on peer expiration, and "leak" bugs preventing data sync being completed.

Rate-limiting

Scalability

Server-side rate limiter has been improved to handle requests better under higher loads. Now, multiple rate-limiter processes handle the requests per rate-limiting-groups.

Rate-limiting timeout has been lowered to 1000ms, if this is breached requests will be responded with status 503.

Quota headers

Additional headers are present in the responses of the HTTP API. These headers provide information on the current quota for the calling endpoint, and will be used to optimise the client throttling in future releases.

Please see this link for documentation and further information on the headers.

Further improvements

• Optimised data table use for ar_data_discovery, as it previously could get bogged down removing empty peers.
• Several logging optimisations have been made to reduce repetive messages.

Community involvement

A huge thank you to all the Mining community members who contributed to this release by testing the alpha releases, providing feedback, and helping us debug issues!

Discord users (alphabetical order):

• Butcher_
• doesn't stay up late
• Evalcast
• lawso2517
• timothynode

Release N.2.9.6-alpha1

This is an alpha update and may not be ready for production use. This software was prepared by the Digital History Association, in cooperation from the wider Arweave ecosystem.

NASA Integration

This release includes several updates required by the Network Availability Stacking Alpha (NASA). Please visit the #nasa channel on the Arweave Miners discord for more information. If you are running a NASA node we recommend you upgrade to this release.

absolute_end_offset included in GET /chunk and GET /chunk2 responses

Both GET /chunk and GET /chunk2 now include a arweave-absolute-end-offset HTTP header in their response. GET /chunk also includes the absolute_end_offset JSON key in its HTTP body.

GET /unconfirmed_chunk/{TXID}/{offset} endpoint

The new GET /unconfirmed_chunk/{TXID}/{offset} endpoint provides access to chunk data that has been recently received by the node, exists in its disk pool, but may not yet have been confirmed.

Footprint-based replica 2.9 syncing

This release introduces footprint-based syncing for replica 2.9 data. Instead of syncing chunk-by-chunk (which caused redundant entropy generation), the node now groups chunks by their entropy footprint and syncs them together.

How it works

Replica 2.9 chunks share entropy in groups called footprints. A full footprint is 1024 chunks spread evenly across a partition. In prior releases, syncing chunk-by-chunk meant the same entropy was regenerated many times. This release alternates between two syncing phases:

• Normal phase: syncs non-replica-2.9 data (unpacked, spora_2_6, etc.) in the traditional left-to-right manner. Also handles disk pool syncing.
• Footprint phase: syncs replica 2.9 data footprint-by-footprint, so all chunks sharing the same entropy are fetched together.

The node limits the number of concurrently active footprints based on the configured replica_2_9_entropy_cache_size_mb to avoid entropy cache overload.

New endpoints

• GET /footprint_buckets: returns a compact representation of footprint availability, similar to sync buckets.
• GET /footprints/{partition}/{footprint}: returns the detailed footprint intervals for a given partition and footprint number. Peers use this for footprint-based data discovery. Note that footprint records are maintained for all packing types (not just replica 2.9), making it convenient for any client to query data availability using the footprint-based layout.

Both endpoints require release 91 or later. Footprint syncing gracefully falls back when communicating with older peers.

Sync record to footprint record migration

On first startup after upgrade, the node automatically migrates its existing sync records into the new footprint record format. This is a one-time, non-destructive migration.

Migration details

• The migration runs in the background, processing 200 chunks per batch with a 1-second delay between batches.
• Progress is checkpointed, so if the node restarts, migration resumes from where it left off.
• Footprint-based syncing is disabled until the migration completes. Normal syncing proceeds as usual during the migration.

Implications for operators

• No new configuration is required. The migration starts automatically.
• Depending on the amount of synced data, the migration may take some time. Progress is logged (initializing_footprint_record / footprint_record_initialized events).
• Once migration is complete, nodes begin participating in footprint-based syncing.

Data root syncing

This release introduces a background data root syncing process. Each storage module now runs a process that periodically scans its range (every 10 minutes) and ensures data roots are recorded for all blocks in that range. Missing data roots are fetched from peers via GET /data_roots/{offset}. You can disable this background process by setting enable_data_roots_syncing false (it is enabled by default).

POST /data_roots/{offset}

A new endpoint, POST /data_roots/{offset}, allows pushing data roots to a node. The endpoint:

1. Validates the offset is within the disk pool threshold and maps to a known block.
2. Checks whether data roots are already synced for that block (returns 200 immediately if so).
3. Validates the submitted data roots: verifies the block size matches, recomputes the TX root from the entries, and verifies all Merkle paths.
4. Stores the validated data roots.

The POST /chunk endpoint requires the receiving node to have the data root for the chunk being submitted. A node that has not synced the data root index for a given block will reject chunks from that block. By pushing data roots via POST /data_roots/{offset}, you can ensure the receiving node is ready to accept chunks from arbitrary historical blocks. This may be useful e.g. for pushing historical chunks via POST /chunk.

Packing benchmark rewrite

The benchmark packing command (formerly benchmark 2_9) has been rewritten to provide realistic performance measurements that account for disk I/O, not just CPU.

What changed

The old benchmark only measured CPU-bound entropy generation and reported inflated rates (often >1 GiB/s) because it ignored disk I/O entirely. The new benchmark has two phases:

• Phase 1 (Entropy Preparation): measures entropy generation rate and disk write rate separately, identifies which is the bottleneck, and extrapolates the preparation time for a full partition.
• Phase 2 (Packing): measures the complete packing cycle — unpack entropy generation, decipher (XOR unpack), disk read (pack entropy from disk), encipher (XOR pack), and disk write (packed chunk). Identifies the bottleneck among all 5 operations.

Realistic disk I/O simulation

• Uses the same file layout as ar_chunk_storage (spread-out writes matching the production entropy storage pattern).
• Simulates background disk contention with configurable read load threads performing random 4-64KB reads (typical RocksDB access pattern).
• Filters out cached writes using the rated_speed parameter — samples faster than the rated disk speed (with a 10% margin) are excluded.
• Syncs files and drops page cache between phases to ensure reads hit disk.

Usage

benchmark packing [options]

Options:

• threads — number of threads (default: CPU cores)
• samples — number of samples to average (default: 20)
• large_pages — use large pages for RandomX, 0=off, 1=on (default: 1)
• rated_speed — expected disk write speed in MB/s, used to exclude cached samples (default: 250)
• read_load — background read threads simulating other disk activity (default: 2)
• read_file_gb — size of read load file in GB (default: 4)
• dir — directory to write to; if omitted, runs CPU-only benchmark

Examples:

benchmark packing threads 8 dir /mnt/storage1
benchmark packing rated_speed 246 dir /tmp/bench

Mining cache bug fixes

This release fixes a mining cache management bug where entries could become orphaned when chunks fail to read from disk.

The problem

When a chunk read failed during mining (e.g., the chunk was missing from the recall range tail), the cache reservation for that chunk was not released. Over time, orphaned entries accumulated and consumed all reserved cache space, preventing new compute_h0 tasks from making progress. This inefficiency was typically reset by new VDF sessions.

The fix

• When a chunk read or store operation fails, the cache reservation for that sub-chunk is now properly released.
• If both chunks have failed for a given nonce, the entry is dropped from the cache entirely — preventing orphaned entries from filling the cache.

Rate-limiting reworked

A new rate limiting model has been introduce that allows finer tuning of allowed load. It combines a sliding window and a leaky bucket token limiter. It also allows limiting concurrent requests per limiter groups.

Precedence of rate limiting:

• Concurrency
• Sliding Window
• Leaky bucket token

If configured concurrency limit is breached for a limiter group, requests are rejected.
If configured Sliding Window rate limit is breached, it falls back to Leaky bucket token limiter. If the group burned all the leaky tokens, the request is rejected.

The current version keeps default config values to preserve previous default behaviour. Further documentation is available on the website.

Please see:
https://docs.arweave.org/developers/mining/operations/rate-limiting

P3 Removed

All Permaweb Payment Protocol (P3) functionality has been removed. Future support for decentralized payments will make use of HyperBEAM integrations.

EPMD leak improvement

Symptoms

• When restarting, or stopping Arweave nodes epmd might have had terminated arweave Erlang node still registered, and stuck

Solution

Provided code change for graceful shutdown of Arweave to help epmd deregister Erlang node.

Config validation

On startup, config validation now won't fail if peers (member of any configured peer list - trusted_peer, local_peer, peer) can't be validated. A warning message will be emmitted and the peer will be ignored.

arweave-denomination header

All price endpoints now include an arweave-denomination HTTP header that states the denomination of the price value. To date and for the foreseeable future the Arweave denomination is 1. This is unlikely to change for many years or perhaps decades, if ever. For more information please see the Denomination guide.

This release introduces various stability and validation enhancements.

Several input va...

Release N.2.9.5.1

Arweave 2.9.5.1 Patch Release Notes

This release introduces various stability and validation enhancements.

Several input validation steps could crash on invalid values, in some cases halting the arweave node.

The patch includes graceful validation of certain inputs and defensive deserialization of local binaries.

We recommend you to install this update as soon as you're able, to address potential node stability issues.

NOTE: No funds are at risk.

Community involvement

A huge thank you to the following researchers who identified and helped to patch issues addressed in this release!

• Cantina's AppSec agent, Apex (https://www.cantina.security/)
• bbl4de (https://github.com/bbl4de)
• 0xJ3an (https://github.com/0xJ3an)
• windhustler from Burra Security (https://x.com/windhustler)

Release N.2.9.5

This is a substantial update. This software was prepared by the Digital History Association, in cooperation from the wider Arweave ecosystem.

This release is primarily a bug fix, stability, and performance release. It includes all changes from all of the 2.9.5 alpha releases. Full details can be found in the release notes for each alpha:

• alpha1
• alpha2
• alpha3
• alpha4
• alpha5
• alpha6

Some of the changes described above were to address regressions introduced in a prior alpha. The full set of changes you can expect when upgrading from 2.9.4.1 are described below.

A special call out to the mining community members who installed and tested each of the alpha releases. Their help was critical in addressing regressions, fixing bugs, and implementing imrprovements. Thank you! Full list of contributors in the Community involvement section.

New Binaries

This release includes an updated set of pre-built binaries:

• Ubuntu 22:04, erlang R26
• Ubuntu 24:04, erlang R26
• rocky9, erlang R26
• MacOS, erlang R26

The default linux release refers to Ubuntu 22:04, erlang R26.

Going forward we recommend Arweave be built with erlang R26 rather than erlang R24.

The MacOS binaries are intended to be used for VDF Servers. Packing and mining on MacOS is still unsupported.

Changes to miner config

• Several changes to options related to repack-in-place. See Support for repack-in-place from the replica.2.9 format.
• vdf: see Optimized VDF.
• Several changes to options related to the verify tool. See verify Tool Improvements.
• disable_replica_2_9_device_limit: Disable the device limit for the replica.2.9 format. By default, at most one worker will be active per physical disk at a time, setting this flag removes this limit allowing multiple workers to be active on a given physical disk.
• Several options to manually configure low level network performance. See help for options starting with network., http_client. and http_api..
• mining_cache_size_mb: the default is set to 100MiB per partition being mined (e.g. if you leave mining_cache_size_mb unset while mining 64 partitions, your mining cache will be set to 6,400 MiB).
• The process for running multiple nodes on a single server has changed. Each instance will need to set distinct/unique values for the ARNODE and ARCOOKIE environment variables. Here is an example script to launch 2 nodes one named exit and one named miner:

#!/usr/bin/env bash
ARNODE=exit@127.0.0.1 \
ARCOOKIE=exit \
screen -dmSL arweave.exit -Logfile ./screenlog.exit \
    ./bin/start config_file config.exit.json;

ARNODE=miner@127.0.0.1 \
ARCOOKIE=miner \
screen -dmSL arweave.miner -Logfile ./screenlog.miner \
    ./bin/start config_file config.miner.json

Optimized VDF

This release includes the optimized VDF algorithm developed by Discord user hihui.

To use this optimized VDF algorithm set the vdf hiopt_m4 config option. By default the node will run with the legacy openssl implementation.

Support for repack-in-place from the replica.2.9 format

This release introduces support for repack-in-place from replica.2.9 to unpacked or to a different replica.2.9 address. In addition we've made several performance improvements and fixed a number of edge case bugs which may previously have caused some chunks to be skipped by the repack process.

Performance

Due to how replica.2.9 chunks are processed, the parameters for tuning the repack-in-place performance have changed. There are 4 main considerations:

• Repack footprint size: replica.2.9 chunks are grouped in footprints of chunks. A full footprint is 1024 chunks distributed evenly across a partition.
• Repack batch size: The repack-in-place process reads some number of chunks, repacks them, and then writes them back to disk. The batch size controls how many contiguous chunks are read at once. Previously a batch size of 10 would mean that 10 chunks would be read, repacked, and written. However in order to handle replica.2.9 data efficiently, a batch size indicates the number of footprints to process at once. So a batch size of 10 means that 10 footprints will be read, repacked, and written. Since a full footprint is 1024 chunks, the amount of memory required to process a batch size of 10 is now 10,240 chunks or roughly 2.5 GiB.
• Available RAM: The footprint size and batch size drive how much RAM is required by the repack in place process. And if you're repacking multiple partitions at once, the RAM requirements can grow quickly.
• Disk IO: If you determine that disk IO is your bottleneck, you'd want to increase the batch size as much as you can as reading contiguous chunks are generally much faster than reading non-contiguous chunks.
• CPU: However in some cases you may find that CPU is your bottleneck - this can happen when repacking from a legacy format like spora_2_6, or can happen when repacking many partitions between 2 replica.2.9 addresses. The saving grace here is that if CPU is your bottleneck, you can reduce your batch size or footprint size to ease off on your memory utilization.

To control all these factors, repack-in-place has 2 config options:

• repack_batch_size: controls the batch size - i.e. the number of footprints processed at once
• repack_cache_size_mb: sets the total amount of memory to allocate to the repack-in-place process per partition. So if you set repack_cache_size_mb to 2000 and are repacking 4 partitions, you can expect the repack-in-place process to consume roughly 8 GiB of memory. Note: the node will automatically set the footprint size based on your configured batch and cache sizes - this typically means that it will reduce the footprint size as much as needed. A smaller footprint size will increase your CPU load as it will result in your node generating the same entropy multiple times. For example, if your footprint size is 256 the node will need to generate teh same entropy 4 times in order to process all 1024 chunks in the full footprint.

Debugging

This release also includes a new option on the data-doctor inspect tool that may help with debugging packing issues.

/bin/data-doctor inspect bitmap <data_dir> <storage_module>

Example: /bin/data-doctor inspect bitmap /opt/data 36,En2eqsVJARnTVOSh723PBXAKGmKgrGSjQ2YIGwE_ZRI.replica.2.9

Will generate a bitmap where every pixel represents the packing state of a specific chunk. The bitmap is laid out so that each vertical column of pixels is a complete entropy footprint. Here is an example of bitmap:

This bitmap shows the state of one node's partition 5 that has been repacked to replica.2.9. The green pixels are chunks that are in the expected replica.2.9 format, the black pixels are chunks that are missing from the miner's dataset, and the pink pixels are chunks that are too small to be packed (prior to partition ~9, users were allowed to pay for chunks that were smaller than 256KiB - these chunks are stored unpacked and can't be packed).

Performance Improvements

• Improvements to both syncing speed and memory use while syncing
• In our tests using solo as well as coordinated miners configured to mine while syncing many partitions, we observed steady memory use and full expected hashrate. This improves on 2.9.4.1 performance. Notably: the same tests run on 2.9.4.1 showed growing memory use, ultimately causing an OOM.
• Reduce the volume of unnecessary network traffic due to a flood of 404 requests when trying to sync chunks from a node which only serves replica.2.9 data. Note: the benefit of this change will only be seen when most of the nodes in the network upgrade.
• Performance improvements to HTTP handling that should improve performance more generally.
• Optimization to speed up the collection of peer intervals when syncing. This can improve syncing performance in some situations.
• Fix a bug which could cause syncing to occasionally stall out.
• Optimize the shutdown process. This should help with, but not fully address, the slow node shutdown issues.
• Fix a bug where a VDF client might get pinned to a slow or stalled VDF server.
• Several updates to the mining cache logic. These changes address a number of edge case performance and memory bloat issues that can occur while mining.
• Improve the transaction validation performance, this should reduce the frequency of "desyncs". I.e. nodes should now be able to handle a higher network transaction volume without stalling
  • Do not delay ready_for_mining on validator nodes
  • Make sure identical tx-status pairs do not cause extra mempool updates
  • Cache the owner address once computed for every TX
• Reduce the time it takes for a node to join the network:
  • Do not re-download local blocks on join
  • Do not re-write written txs on join
  • Reduce per peer retry budget on join 10 -> 5
• Fix edge case that could occasionally cause a mining pool to reject a replica.2.9 solution.
• Fix edge case crash that occurred when a coordinated miner timed out while fetching partitions from peers
• Fix bug where storage module crossing weave end may cause syncing stall
• Fix bug where crash during peer interval collection may cause syncing stall
• Fix race condition where we may no...

Release N.2.9.5-alpha6

This is an alpha update and may not be ready for production use. This software was prepared by the Digital History Association, in cooperation from the wider Arweave ecosystem.

This release addresses several of the mining performance issues that had been reported on previous alphas. It passes all automated tests and has undergone a base level of internal testing, but is not considered production ready. We only recommend upgrading if you wish to take advantage of the new performance improvements.

Fix: Crash during coordinated mining when a solution is found

(Reported by discord user Vidiot)

Symptoms

• After mining well for some time, hashrate dropped to 0
• Logs had messages like: Generic server ar_mining_server terminating. Reason: {badarg,[{ar_block,compute_h1,3

Fix: session_not_found error during coordinated mining

(Reported by discord user Qwinn)

Symptoms

• Hahrate lower than expected
• Logs had mining_worker_failed_to_add_chunk_to_cache errors, with reason set to session_not_found

Guidance: cache_limit_exceeded warning during solo and coordinated mining

(Reported by discord users BerryCZ, mousetu, radion_nizametdinov, qq87237850, Qwinn, Vidiot)

Symptoms

• Logs show mining_worker_failed_to_reserve_cache_space warnings, with reason set to cache_limit_exceeded

Resolution

The warning, if seen periodically, is expected and safe to ignore.

Root Cause

All VDF servers - even those with the exact same VDF time - will be on slightly different steps. This is because new VDF epochs are opened roughly every 20 minutes and are opened when a block is added to the chain. Depending on when your VDF server receives that block it may start calculating the new VDF chain earlier or later than other VDF servers.

This can cause there to be a gap in the VDF steps generated by two different servers even if they are able to compute new VDF steps at the exact same speed.

When a VDF server receives a block that is ahead of it in the VDF chain, it is able to quickly validate and use all the new VDF steps. This can cause the associated miners to receive a batch of VDF steps all at once. In these situations, the miner may exceed its mining cache causing the cache_limit_exceeded warning.

However this ultimately does not materially impact the miner's true hashrate. A miner will process VDF steps in reverse order (latest steps first) as those are the most valuable steps. The steps being dropped from the cache will be the oldest steps. Old steps may still be useful, but there is a far greater chance that any solution mined off an old step will be orphaned. The older the VDF step, the less useful it is.

TLDR: the warning, if seen periodically, is expected and safe to ignore.

Exception: If you are continually seeing the warning (i.e. not in periodic batches, but constantly and all the time) it may indicate that your miner is not able to keep up with its workload. This can indicate a hardware configuration issue (e.g. disk read rates are too slow), or perhaps a hardware capacity issue (E.g. CPU not fast enough to run hashes on all attached storage module), or some other performance-related issue.

Guidance

• This alpha increases the default cache size from 4 steps to 20 VDF steps. This should noticeably reduce (but not eliminate) the frequency of the cache_limit_exceeded warning
• If you want to increase it further you can set the mining_cache_size_mb option.

Guidance: 2.9.5-alphaX hashrate appears to be slower than 2.9.4.1

(Reported by discord users EvM, Lawso2517, Qwinn)

Symptoms

• 2.9.4.1 hashrate is higher than 2.9.5-alphaX
• 2.9.4.1 hashrate when solo mining might even be higher than the "ideal" hashrate listed in the mining report or grafana metrics

Resolution

The 2.9.4.1 hashrate included invalid hashes and the 2.9.5-alpha6 hashrate, although lower, includes only valid hashes.

Root Cause

2.9.4.1 (and earlier releases) had a bug which caused miners to generate hashes off of entropy in addition to valid packed data. The replica.2.9 data format lays down a full covering of entropy in each storage module before adding packed chunks. The result that is that for any storage module with less than 3.6TB of packed data, there is some amount of data on disk that is just entropy. A bug in the 2.9.4.1 mining algorithm generated hashes off of this entropy causing an inflated hashrate. Often the 2.9.4.1 hashrate is above the estimated "ideal" hashrate even when solo mining.

Another symptom of this bug is the chunk_not_found error occasionally reported by miners. This occurs under 2.9.4.1 (and earlier releases) when the miner hashes a range of entropy and generates a hash that exceeds the network difficulty. The miner believes this to be a valid solution and begins to build a block. At some point in the block building process the miner has to validate and include the packed chunk data. However since no packed chunk data exists (only entropy), the solution fails and the error is printed.

2.9.5-alpha2 fixed this bug so that miners correctly exclude entropy data when mining. This means that under 2.9.5-alpha2 and later releases miners spend fewer resources hashing entropy data, and generate fewer failed solution errors. The reported hashrate on 2.9.5-alpha2 is lower than 2.9.4.1 because the invalid hashes are no longer being counted.

Community involvement

A huge thank you to all the Mining community members who contributed to this release by identifying and investigating bugs, sharing debug logs and node metrics, and providing guidance on performance tuning!

Discord users (alphabetical order):

• BerryCZ
• Evalcast
• EvM
• JanP
• lawso2517
• mousetu
• qq87237850
• Qwinn
• radion_nizametdinov
• smash
• T777
• Vidiot

Release N.2.9.5-alpha5

This is an alpha update and may not be ready for production use. This software was prepared by the Digital History Association, in cooperation from the wider Arweave ecosystem.

This release includes several syncing and mining performance improvements. It passes all automated tests and has undergone a base level of internal testing, but is not considered production ready. We only recommend upgrading if you wish to take advantage of the new performance improvements.

Note regarding the release binaries:
We have changed the naming of the tar balls, dropped ubuntu20 and added ubuntu24.

All of the provided tarballs are built with erlang26. Going forward (and barring any serious issues that show up), arweave will be built with r26. We've been running r26 internally for a while now with no issues (and with some small stability and performance improvements)

Performance improvements

In all cases we ran tests on a full-weave solo miner, as well as a full-weave coordinated mining cluster. We believe the observed performance improvements are generalizable to other miners, but, as always, the performance observed by a given miner is often influenced by many factors that we are not able to test for. TLDR: your mileage may vary.

Syncing

Improvements to both syncing speed and memory use while syncing. The improvements address some regressions that were reported in the 2.9.5 alphas, but also improve on 2.9.4.1 performance.

Mining

This release addresses the significant hashrate loss that was observed during Coordinated Mining on the 2.9.5 alphas.

Syncing + Mining

In our tests using solo as well as coordinated miners configured to mine while syncing many partitions, we observed steady memory use and full expected hashrate. This addresses some regressions that were reported in the 2.9.5 alphas, but also improves on 2.9.4.1 performance. Notably: the same tests run on 2.9.4.1 showed growing memory use, ultimately causing an OOM.

Community involvement

A huge thank you to all the Mining community members who contributed to this release by identifying and investigating bugs, sharing debug logs and node metrics, and providing guidance on performance tuning!

Discord users (alphabetical order):

• BerryCZ
• Butcher_
• edzo
• Evalcast
• EvM
• JF
• lawso2517
• MaSTeRMinD
• qq87237850
• Qwinn
• radion_nizametdinov
• RedMOoN
• smash
• T777
• Vidiot

Release 2.9.5-alpha4

This is an alpha update and may not be ready for production use. This software was prepared by the Digital History Association, in cooperation from the wider Arweave ecosystem.

This release includes the VDF optimization as well as several bug fixes. It passes all automated tests and has undergone a base level of internal testing, but is not considered production ready. We only recommend upgrading if you wish to use the new VDF optimization or if you believe one of the listed bug fixes will improve your mining experience.

New Binaries

This release includes an updated set of pre-built binaries:

• Ubuntu 20:04, erlang r24 (arweave-2.9.5-alpha4.ubuntu20.r24-x86_64.tar.gz)
• Ubuntu 20:04, erlang r26 (arweave-2.9.5-alpha4.ubuntu20.r26-x86_64.tar.gz)
• Ubuntu 22:04, erlang r24 (arweave-2.9.5-alpha4.ubuntu22.r24-x86_64.tar.gz)
• Ubuntu 22:04, erlang r26 (arweave-2.9.5-alpha4.ubuntu22.r26-x86_64.tar.gz)
• MacOS, erlang r24 (N.2.9.5-alpha4-Darwin-arm64-R24.tar.gz)
• MacOS, erlang r26 (N.2.9.5-alpha4-Darwin-arm64-R26.tar.gz)

The default linux release refers to Ubuntu 22:04, erlang r26

We recommend trying the appropriate "erlang r26" binary first. Internal testing shows it to be more stable and slightly more performant.

The MacOS binaries are intended to be used for VDF Servers. Packing and mining on MacOS is still unsupported.

Optimized VDF

This release includes the optimized VDF algorithm developed by Discord user hihui.

To use this optimized VDF algorithm set the vdf hiopt_m4 config option. By default the node will run with the legacy openssl implementation.

Mining Fixes

This release fixes a number of performance and memory issues that were observed while mining on previous 2.9.5 alpha releases.

Other Fixes and Improvements

• Several fixes to address the issue of nodes occasionally "desyncing" (i.e. appearing to stall and refusing to validate new blocks as they are received)
• Several fixes to improve the Arweave shutdown process
  • Close hanging HTTP connections
  • Prevent new HTTP connections from being created during shutdown
  • Fix a timeout issue
• Add a verify flag to the benchmark-vdf script
  • When running benchmark-vdf you can specify the verify true flag to have the script verify the VDF output against a slower "debug" VDF algorithm.
• Support CMake 4 on MacOS

Full Changelog: N.2.9.5-alpha3...N.2.9.5-alpha4

Community involvement

A huge thank you to all the Mining community members who contributed to this release by identifying and investigating bugs, sharing debug logs and node metrics, and providing guidance on performance tuning!

Discord users (alphabetical order):

• BerryCZ
• bigbang
• BloodHunter
• Butcher_
• doesn't stay up late
• edzo
• Evalcast
• EvM
• hihui
• Iba Shinu
• JamsJun
• JF
• jimmyjoe7768
• lawso2517
• MaSTeRMinD
• Merdi Kim
• Niiiko
• qq87237850
• Qwinn
• RedMOoN
• sk
• smash
• sumimi
• T777
• tashilo
• U genius
• Vidiot