Releases: near/nearcore
1.36.0
CODE_COLOR: CODE_YELLOW_MAINNET
RELEASE_VERSION: 1.36.0
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: TRUE
Protocol Changes
- The support for fixed shards in shard layout was removed. #9219
Protocol Upgrade Voting
This release contains a protocol upgrade. Voting for upgrading to protocol version 63 will start on 2023-11-13 15:00:00 UTC.
Database Upgrade
This release contains a database upgrade. STATE_SYNC_DUMP_KEY and STATE_SNAPSHOT_KEY was added to the BlockMisc column for support of state sync.
Non-protocol Changes
- New option
transaction_pool_size_limitinconfig.jsonallows to limit the size of the node's transaction pool.
By default the limit is set to 100 MB. #3284 - Database snapshots at the end of an epoch. This lets a node obtain state parts using flat storage. #9090
- Number of transactions included in a chunk will be lowered if there is a congestion of more than 20000 delayed receipts in a shard. #9222
- Our more efficient and scalable V2 routing protocol is implemented. It shadows the V1 protocol for now while we verify its performance. #9187
- The default config now enables TIER1 outbound connections by default. #9349
- State Sync from GCS is available for experimental use. #9398
Note around node performance
During internal testing of 1.36 we noticed nodes were impacted by a performance degradation manifesting as slowness in producing blocks. This impact was statistical, and the investigation we performed did not reveal any clear indicator of the root cause of this degradation.
In order to better assess this issue, we decided to test the release on testnet. The testnet release so far did not reveal any performance degradation, so we are releasing to mainnet.
After upgrading on mainnet to 1.36, if you get kicked out or notice any performance issue, please reach out to us on the NEAR Validators Telegram channel with data regarding node configuration and performance.
1.36.0-rc.2
CODE_COLOR: CODE_GREEN_TESTNET
RELEASE_VERSION: 1.36.0-rc.2
PROTOCOL_UPGRADE: FALSE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: FALSE
Protocol Changes
Non-protocol Changes
1.36.0-rc.1
CODE_COLOR: CODE_YELLOW_TESTNET
RELEASE_VERSION: 1.36.0-rc.1
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE
Protocol Changes
- The support for fixed shards in shard layout was removed. #9219
Protocol Upgrade Voting
This release contains a protocol upgrade. Voting for upgrading to protocol version 63 will start on 2023-09-26 15:00:00 UTC.
Database Upgrade
This release contains a database upgrade. STATE_SYNC_DUMP_KEY and STATE_SNAPSHOT_KEY was added to the BlockMisc column for support of state sync.
Non-protocol Changes
- New option
transaction_pool_size_limitinconfig.jsonallows to limit the size of the node's transaction pool.
By default the limit is set to 100 MB. #3284 - Database snapshots at the end of an epoch. This lets a node obtain state parts using flat storage. #9090
- Number of transactions included in a chunk will be lowered if there is a congestion of more than 20000 delayed receipts in a shard. #9222
- Our more efficient and scalable V2 routing protocol is implemented. It shadows the V1 protocol for now while we verify its performance. #9187
- The default config now enables TIER1 outbound connections by default. #9349
- State Sync from GCS is available for experimental use. #9398
Note around node performance
During internal testing of 1.36 we noticed nodes were impacted by a performance degradation manifesting as slowness in producing blocks. This impact was statistical, and the investigation we performed did not reveal any clear indicator of the rootcause of this degradation. In order to better assess this issue, we need to determine the behaviour of the 1.36 nodes in a real-life environment, such as testnet. After upgrading on testnet to 1.36, if you get kicked out or notice any performance issue, please reach out to us on the NEAR Validators Telegram channel with data regarding node configuration and performance.
1.35.0
CODE_COLOR: CODE_YELLOW_MAINNET
RELEASE_VERSION: 1.35.0
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE
Protocol Changes
- Upgrade the contract preparation code to use finite-wasm, which guarantees deterministic limits on execution time and space of compiled contracts
Non-protocol Changes
- Dump state by multiple nodes, each node will refer to s3 for which parts need to be dumped. #9049
- Small values in the flat storage trie are inlined for faster accesses #9029
- A current protocol version metric is added to the prometheus metrics under near_current_protocol_version #9030
- The transaction pool size is tracked, and if the
transaction_pool_size_limitconfig option is set, we now avoid storing more than the specified size of transactions in each shard's transaction pool #8970 and #9036
Split Storage Feature
Split storage allows nodes to improve storage costs and block production performance on archival nodes by splitting storage into hot and cold databases. Hot database stores all of the data from the past several epochs, and cold database stores all historical data from a limited list of columns.
In split storage mode Client only works with a smaller hot database to produce blocks, which results in increased performance.
Cold database is not accessed for reads during normal block production and is only read when historical data is specifically requested. Thus, we recommend keeping the cold database on HDD and only optimize speed for the hot database, which is 10 times smaller.
Split storage can be enabled with a config change and a migration that requires manual steps. You can learn more about the migration process here: https://near-nodes.io/archival/split-storage-archival
Protocol Upgrade Voting
This release contains a protocol upgrade. Voting for upgrading to protocol version 62 will start on 2023-08-01 15:00:00 UTC.
Database Upgrade
This release includes a db upgrade, which migrates entries in the FlatStateChanges, the size of which should not exceed several dozens of entries
1.35.0-rc.1
CODE_COLOR: CODE_YELLOW_TESTNET
RELEASE_VERSION: 1.35.0-rc.1
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE
Protocol Changes
- Upgrade the contract preparation code to use finite-wasm, which guarantees deterministic limits on execution time and space of compiled contracts
Non-protocol Changes
- Dump state by multiple nodes, each node will refer to s3 for which parts need to be dumped. #9049
- Small values in the flat storage trie are inlined for faster accesses #9029
- A current protocol version metric is added to the prometheus metrics under near_current_protocol_version #9030
- The transaction pool size is tracked, and if the
transaction_pool_size_limitconfig option is set, we now avoid storing more than the specified size of transactions in each shard's transaction pool #8970 and #9036
Split Storage Feature
Split storage allows nodes to improve storage costs and block production performance on archival nodes by splitting storage into hot and cold databases. Hot database stores all of the data from the past several epochs, and cold database stores all historical data from a limited list of columns.
In split storage mode Client only works with a smaller hot database to produce blocks, which results in increased performance.
Cold database is not accessed for reads during normal block production and is only read when historical data is specifically requested. Thus, we recommend keeping the cold database on HDD and only optimize speed for the hot database, which is 10 times smaller.
Split storage can be enabled with a config change and a migration that requires manual steps. We have several choices for the migration:
https://docs.google.com/document/d/1PVuETy4Ogf_8Hl0nFRrIvBs-94OhSC1mrULjsQcH-Bc/edit?usp=sharing
Protocol Upgrade Voting
This release contains a protocol upgrade. Voting for upgrading to protocol version 62 will start on 2023-06-27 15:00:00 UTC.
Database Upgrade
This release includes a db upgrade, which migrates entries in the FlatStateChanges, the size of which should not exceed several dozens of entries
1.34.0
CODE_COLOR: CODE_YELLOW_MAINNET
RELEASE_VERSION: 1.34.0
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: TRUE
Protocol Changes
- Flat Storage for reads. Reduces the number of DB accesses for state reads from
2 * key.len()in the worst case to2. #8761, NEP-399 - Compute Costs are implemented and stabilized. Compute usage of the chunk is now limited according to the compute costs. #8915, NEP-455.
- Write-related storage compute costs are increased which means they fill a chunk sooner but gas costs are unaffected. #8924
Flat Storage Migration
This is a very important migration that needs special attention.
The migration starts automatically after switching to the 1.34.0 binary and runs in the background.
The migration completes approximately in:
- RPC nodes: 11-15 hours
- Archival nodes: 30-40 hours
The migration increases disk usage by approximately 10 GB.
Please upgrade early to make sure there is enough time for the migration to complete.
Note that the protocol voting date is set further into the future than usually, to make sure that all RPC and Archival nodes have an opportunity to upgrade.
Failure to finish the migration before the protocol upgrade will result in the node getting stuck.
The migration cannot be started if the node has already missed the protocol upgrade.
To check that the migration has completed, check the metrics page, e.g. http://127.0.0.1:3030/metrics .
If flat_storage_creation_status has value of 2 for each of the shards, then the migration is complete.
If flat_storage_creation_status is missing and flat_storage_distance_to_head has a very low value for each of the shards, then the migration is complete.
Reduce Memory Usage
If you are running a node on minimal hardware specifications, you can save 2GB of RAM by decreasing store.trie_cache.per_shard_max_bytes.s3.v1 from 3000000000 (default value) to 1000000000.
All our testing shows that things work stable with that and block processing stays below 1 second.
However, keeping store.trie_cache.per_shard_max_bytes.s3.v1=3000000000 has a positive impact on reducing the processing time and therefore gives an extra safety buffer.
Protocol Upgrade Voting
Voting for upgrading to protocol version 61 will start on 2023-06-12 15:00:00 UTC. Please upgrade early because of a special Flat Storage migration.
Database Upgrade
This release includes a trivial DB upgrade. It changes no data, but informs about the Flat Storage migration. The upgrade completes instantly and requires no extra disk space.
Several new columns will be created by the Flat Storage migration in the background.
Non-Protocol Changes
undo-blocktool to reset the chain head from current head to its prev block. Use the tool by running:./target/release/neard undo-block. #8681- Add prometheus metrics for expected number of blocks/chunks at the end of the epoch. #8759
- Node can sync State from S3. #8789
- Node can sync State from local filesystem. #8913
- Add per shard granularity for chunks in validator info metric. #8934
Fixes
- Extra validation of
DeleteAccountAction
1.34.0-rc.2
CODE_COLOR: CODE_GREEN_TESTNET
RELEASE_VERSION: 1.34.0-rc.2
PROTOCOL_UPGRADE: FALSE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: TRUE
Fixes
- Extra validation of
DeleteAccountAction
1.34.0-rc.1
CODE_COLOR: CODE_YELLOW_TESTNET
RELEASE_VERSION: 1.34.0-rc.1
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE
Protocol Changes
- Flat Storage for reads. Reduces the number of DB accesses for state reads from
2 * key.len()in the worst case to2. #8761, NEP-399 - Compute Costs are implemented and stabilized. Compute usage of the chunk is now limited according to the compute costs. #8915, NEP-455.
- Write-related storage compute costs are increased which means they fill a chunk sooner but gas costs are unaffected. #8924
Flat Storage Migration
This is a very important migration that needs special attention.
The migration starts automatically after switching to the 1.34.0-rc.1 binary and runs in the background.
The migration completes approximately in:
- RPC nodes: 11-15 hours
- Archival nodes: 30-40 hours
The migration increases disk usage by approximately 10 GB.
Please upgrade early to make sure there is enough time for the migration to complete.
Note that the protocol voting date is set further into the future than usually, to make sure that all RPC and Archival nodes have an opportunity to upgrade.
Failure to finish the migration before the protocol upgrade will result in the node getting stuck.
The migration cannot be started if the node has already missed the protocol upgrade.
To check that the migration has completed, check the metrics page, e.g. http://127.0.0.1:3030/metrics .
If flat_storage_creation_status has value of 2 for each of the shards, then the migration is complete.
If flat_storage_creation_status is missing and flat_storage_distance_to_head has a very low value for each of the shards, then the migration is complete.
Reduce Memory Usage
If you are running a node on minimal hardware specifications, you can save 2GB of RAM by decreasing store.trie_cache.per_shard_max_bytes.s3.v1 from 3000000000 (default value) to 1000000000.
All our testing shows that things work stable with that and block processing stays below 1 second.
However, keeping store.trie_cache.per_shard_max_bytes.s3.v1=3000000000 has a positive impact on reducing the processing time and therefore gives an extra safety buffer.
Protocol Upgrade Voting
Voting for upgrading to protocol version 61 will start on 2023-05-22 15:00:00 UTC. Please upgrade early because of a special Flat Storage migration.
Database Upgrade
This release includes a trivial DB upgrade. It changes no data, but informs about the Flat Storage migration. The upgrade completes instantly and requires no extra disk space.
Several new columns will be created by the Flat Storage migration in the background.
Non-Protocol Changes
undo-blocktool to reset the chain head from current head to its prev block. Use the tool by running:./target/release/neard undo-block. #8681- Add prometheus metrics for expected number of blocks/chunks at the end of the epoch. #8759
- Node can sync State from S3. #8789
- Node can sync State from local filesystem. #8913
- Add per shard granularity for chunks in validator info metric. #8934
1.33.0
CODE_COLOR: CODE_YELLOW_MAINNET
RELEASE_VERSION: 1.33.0
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE
#Protocol Changes
This release includes no protocol changes, but it has an empty protocol upgrade (59->60). Thus, as usual, failure to upgrade the node in time will result in validator kick-out.
Protocol Upgrade Voting
This release contains a protocol upgrade. Voting for upgrading to protocol version 60 will start on 2023-05-02 15:00:00 UTC.
Database Upgrade
This release includes a db upgrade. It is a very fast upgrade -- it deletes the deprecated column _Peers, and because of the nature of the RocksDB it shouldn't stall the node startup almost at all (<3 seconds on archival node).
State Sync Notice
This release is disabling state sync by default. You can still enable it via config, but a more reliable way to sync from a very old state faster than waiting for block sync to finish is to download a db backup from s3 and restart the node.
Non-Protocol Changes
- State-viewer tool to dump and apply state changes from/to a range of blocks. #8628
- Experimental option to dump state of every epoch to external storage. #8661
- Add prometheus metrics for tracked shards, block height within epoch, if is block/chunk producer. #8728
- State sync is disabled by default #8730
- Node can restart if State Sync gets interrupted. #8732
- Merged two neard view-state commands: apply-state-parts and dump-state-parts into a single state-parts command. #8739
- Fix: rosetta zero balance accounts #8830
- Fix two bugs that regarding verifying shard_id #8977
1.33.0-rc.1
CODE_COLOR: CODE_YELLOW_TESTNET
RELEASE_VERSION: 1.33.0-rc.1
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE
#Protocol Changes
This release includes no protocol changes, but it has an empty protocol upgrade (59->60). Thus, as usual, failure to upgrade the node in time will result in validator kick-out.
Protocol Upgrade Voting
This release contains a protocol upgrade. Voting for upgrading to protocol version 60 will start on 2023-04-04 15:00:00 UTC.
Database Upgrade
This release includes a db upgrade. It is a very fast upgrade -- it deletes the deprecated column _Peers, and because of the nature of the RocksDB it shouldn't stall the node startup almost at all (<3 seconds on archival node).
State Sync Notice
This release is disabling state sync by default. You can still enable it via config, but a more reliable way to sync from a very old state faster than waiting for block sync to finish is to download a db backup from s3 and restart the node.
Non-Protocol Changes
- State-viewer tool to dump and apply state changes from/to a range of blocks. #8628
- Experimental option to dump state of every epoch to external storage. #8661
- Add prometheus metrics for tracked shards, block height within epoch, if is block/chunk producer. #8728
- State sync is disabled by default #8730
- Node can restart if State Sync gets interrupted. #8732
- Merged two neard view-state commands: apply-state-parts and dump-state-parts into a single state-parts command. #8739
- Fix: rosetta zero balance accounts #8830