Prepare the codebase to use base gas price during block production #1642
Assignees
Labels
epic
An epic is a high-level master issue for large pieces of work.
Comments
xgreenx
added
the
epic
xpecxanum
pushed a commit
to xpecxanum/fuel-core
that referenced
this issue
Feb 16, 2024
## Version 0.21.0 This release focuses on preparing `fuel-core` for the mainnet environment. - Most of the changes improved the security and stability of the node. - The gas model was reworked to cover all aspects of execution. - The benchmarking system was significantly enhanced, covering worst scenarios. - A new set of benchmarks was added to track the accuracy of gas prices. - Optimized heavy operations and removed/replaced exploitable functionality. Besides that, there are more concrete changes: - Unified naming conventions for all CLI arguments. Added dependencies between related fields to avoid misconfiguration in case of missing arguments. Added `--debug` flag that enables additional functionality like a debugger. - Improved telemetry to cover the internal work of services and added support for the Pyroscope, allowing it to generate real-time flamegraphs to track performance. - Improved stability of the P2P layer and adjusted the updating of reputation. The speed of block synchronization was significantly increased. - The node is more stable and resilient. Improved DoS resistance and resource management. Fixed critical bugs during state transition. - Reworked the `Mint` transaction to accumulate the fee from block production inside the contract defined by the block producer. FuelVM received a lot of safety and stability improvements: - The audit helped identify some bugs and errors that have been successfully fixed. - Updated the gas price model to charge for resources used during the transaction lifecycle. - Added `no_std` and 32 bit system support. This opens doors for fraud proving in the future. - Removed the `ChainId` from the `PredicateId` calculation, allowing the use of predicates cross-chain. - Improvements in the performance of some storage-related opcodes. - Support the `ECAL` instruction that allows adding custom functionality to the VM. It can be used to create unique rollups or advanced indexers in the future. - Support of [transaction policies](https://github.com/FuelLabs/fuel-vm/blob/master/CHANGELOG.md#version-0420) provides additional safety for the user. It also allows the implementation of a multi-dimensional price model in the future, making the transaction execution cheaper and allowing more transactions that don't affect storage. - Refactored errors, returning more detailed errors to the user, simplifying debugging. ### Added - [FuelLabs#1503](FuelLabs#1503): Add `gtf` opcode sanity check. - [FuelLabs#1502](FuelLabs#1502): Added price benchmark for `vm_initialization`. - [FuelLabs#1501](FuelLabs#1501): Add a CLI command for generating a fee collection contract. - [FuelLabs#1492](FuelLabs#1492): Support backward iteration in the RocksDB. It allows backward queries that were not allowed before. - [FuelLabs#1490](FuelLabs#1490): Add push and pop benchmarks. - [FuelLabs#1485](FuelLabs#1485): Prepare rc release of fuel core v0.21 - [FuelLabs#1476](FuelLabs#1453): Add the majority of the "other" benchmarks for contract opcodes. - [FuelLabs#1473](FuelLabs#1473): Expose fuel-core version as a constant - [FuelLabs#1469](FuelLabs#1469): Added support of bloom filter for RocksDB tables and increased the block cache. - [FuelLabs#1465](FuelLabs#1465): Improvements for keygen cli and crates - [FuelLabs#1642](FuelLabs#1462): Added benchmark to measure the performance of contract state and contract ID calculation; use for gas costing. - [FuelLabs#1457](FuelLabs#1457): Fixing incorrect measurement for fast(µs) opcodes. - [FuelLabs#1456](FuelLabs#1456): Added flushing of the RocksDB during a graceful shutdown. - [FuelLabs#1456](FuelLabs#1456): Added more logs to track the service lifecycle. - [FuelLabs#1453](FuelLabs#1453): Add the majority of the "sanity" benchmarks for contract opcodes. - [FuelLabs#1452](FuelLabs#1452): Added benchmark to measure the performance of contract root calculation when utilizing the maximum contract size; used for gas costing of contract root during predicate owner validation. - [FuelLabs#1449](FuelLabs#1449): Fix coin pagination in e2e test client. - [FuelLabs#1447](FuelLabs#1447): Add timeout for continuous e2e tests - [FuelLabs#1444](FuelLabs#1444): Add "sanity" benchmarks for memory opcodes. - [FuelLabs#1437](FuelLabs#1437): Add some transaction throughput tests for basic transfers. - [FuelLabs#1436](FuelLabs#1436): Add a github action to continuously test beta-4. - [FuelLabs#1433](FuelLabs#1433): Add "sanity" benchmarks for flow opcodes. - [FuelLabs#1432](FuelLabs#1432): Add a new `--api-request-timeout` argument to control TTL for GraphQL requests. - [FuelLabs#1430](FuelLabs#1430): Add "sanity" benchmarks for crypto opcodes. - [FuelLabs#1426](FuelLabs#1426) Split keygen into a create and a binary. - [FuelLabs#1419](FuelLabs#1419): Add additional "sanity" benchmarks for arithmetic op code instructions. - [FuelLabs#1411](FuelLabs#1411): Added WASM and `no_std` compatibility. - [FuelLabs#1405](FuelLabs#1405): Use correct names for service metrics. - [FuelLabs#1400](FuelLabs#1400): Add releasy beta to fuel-core so that new commits to fuel-core master triggers fuels-rs. - [FuelLabs#1371](FuelLabs#1371): Add new client function for querying the `MessageStatus` for a specific message (by `Nonce`). - [FuelLabs#1356](FuelLabs#1356): Add peer reputation reporting to heartbeat code. - [FuelLabs#1355](FuelLabs#1355): Added new metrics related to block importing, such as tps, sync delays etc. - [FuelLabs#1339](FuelLabs#1339): Adds `baseAssetId` to `FeeParameters` in the GraphQL API. - [FuelLabs#1331](FuelLabs#1331): Add peer reputation reporting to block import code. - [FuelLabs#1324](FuelLabs#1324): Added pyroscope profiling to fuel-core, intended to be used by a secondary docker image that has debug symbols enabled. - [FuelLabs#1309](FuelLabs#1309): Add documentation for running debug builds with CLion and Visual Studio Code. - [FuelLabs#1308](FuelLabs#1308): Add support for loading .env files when compiling with the `env` feature. This allows users to conveniently supply CLI arguments in a secure and IDE-agnostic way. - [FuelLabs#1304](FuelLabs#1304): Implemented `submit_and_await_commit_with_receipts` method for `FuelClient`. - [FuelLabs#1286](FuelLabs#1286): Include readable names for test cases where missing. - [FuelLabs#1274](FuelLabs#1274): Added tests to benchmark block synchronization. - [FuelLabs#1263](FuelLabs#1263): Add gas benchmarks for `ED19` and `ECR1` instructions. ### Changed - [FuelLabs#1512](FuelLabs#1512): Internally simplify merkle_contract_state_range. - [FuelLabs#1507](FuelLabs#1507): Updated chain configuration to be ready for beta 5 network. It includes opcode prices from the latest benchmark and contract for the block producer. - [FuelLabs#1477](FuelLabs#1477): Upgraded the Rust version used in CI and containers to 1.73.0. Also includes associated Clippy changes. - [FuelLabs#1469](FuelLabs#1469): Replaced usage of `MemoryTransactionView` by `Checkpoint` database in the benchmarks. - [FuelLabs#1468](FuelLabs#1468): Bumped version of the `fuel-vm` to `v0.40.0`. It brings some breaking changes into consensus parameters API because of changes in the underlying types. - [FuelLabs#1466](FuelLabs#1466): Handling overflows during arithmetic operations. - [FuelLabs#1460](FuelLabs#1460): Change tracking branch from main to master for releasy tests. - [FuelLabs#1454](FuelLabs#1454): Update gas benchmarks for opcodes that append receipts. - [FuelLabs#1440](FuelLabs#1440): Don't report reserved nodes that send invalid transactions. - [FuelLabs#1439](FuelLabs#1439): Reduced memory BMT consumption during creation of the header. - [FuelLabs#1434](FuelLabs#1434): Continue gossiping transactions to reserved peers regardless of gossiping reputation score. - [FuelLabs#1408](FuelLabs#1408): Update gas benchmarks for storage opcodes to use a pre-populated database to get more accurate worst-case costs. - [FuelLabs#1399](FuelLabs#1399): The Relayer now queries Ethereum for its latest finalized block instead of using a configurable "finalization period" to presume finality. - [FuelLabs#1397](FuelLabs#1397): Improved keygen. Created a crate to be included from forc plugins and upgraded internal library to drop requirement of protoc to build - [FuelLabs#1395](FuelLabs#1395): Add DependentCost benchmarks for `k256`, `s256` and `mcpi` instructions. - [FuelLabs#1393](FuelLabs#1393): Increase heartbeat timeout from `2` to `60` seconds, as suggested in [this issue](FuelLabs#1330). - [FuelLabs#1392](FuelLabs#1392): Fixed an overflow in `message_proof`. - [FuelLabs#1390](FuelLabs#1390): Up the `ethers` version to `2` to fix an issue with `tungstenite`. - [FuelLabs#1383](FuelLabs#1383): Disallow usage of `log` crate internally in favor of `tracing` crate. - [FuelLabs#1380](FuelLabs#1380): Add preliminary, hard-coded config values for heartbeat peer reputation, removing `todo`. - [FuelLabs#1377](FuelLabs#1377): Remove `DiscoveryEvent` and use `KademliaEvent` directly in `DiscoveryBehavior`. - [FuelLabs#1366](FuelLabs#1366): Improve caching during docker builds in CI by replacing gha - [FuelLabs#1358](FuelLabs#1358): Upgraded the Rust version used in CI to 1.72.0. Also includes associated Clippy changes. - [FuelLabs#1349](FuelLabs#1349): Updated peer-to-peer transactions API to support multiple blocks in a single request, and updated block synchronization to request multiple blocks based on the configured range of headers. - [FuelLabs#1342](FuelLabs#1342): Add error handling for P2P requests to return `None` to requester and log error. - [FuelLabs#1318](FuelLabs#1318): Modified block synchronization to use asynchronous task execution when retrieving block headers. - [FuelLabs#1314](FuelLabs#1314): Removed `types::ConsensusParameters` in favour of `fuel_tx:ConsensusParameters`. - [FuelLabs#1302](FuelLabs#1302): Removed the usage of flake and building of the bridge contract ABI. It simplifies the maintenance and updating of the events, requiring only putting the event definition into the codebase of the relayer. - [FuelLabs#1293](FuelLabs#1293): Parallelized the `estimate_predicates` endpoint to utilize all available threads. - [FuelLabs#1270](FuelLabs#1270): Modify the way block headers are retrieved from peers to be done in batches. #### Breaking - [FuelLabs#1506](FuelLabs#1506): Added validation of the coin's fields during block production and validation. Before, it was possible to submit a transaction that didn't match the coin's values in the database, allowing printing/using unavailable assets. - [FuelLabs#1491](FuelLabs#1491): Removed unused request and response variants from the Gossipsub implementation, as well as related definitions and tests. Specifically, this removes gossiping of `ConsensusVote` and `NewBlock` events. - [FuelLabs#1472](FuelLabs#1472): Upgraded `fuel-vm` to `v0.42.0`. It introduces transaction policies that changes layout of the transaction. FOr more information check the [v0.42.0](FuelLabs/fuel-vm#635) release. - [FuelLabs#1470](FuelLabs#1470): Divide `DependentCost` into "light" and "heavy" operations. - [FuelLabs#1464](FuelLabs#1464): Avoid possible truncation of higher bits. It may invalidate the code that truncated higher bits causing different behavior on 32-bit vs. 64-bit systems. The change affects some endpoints that now require lesser integers. - [FuelLabs#1432](FuelLabs#1432): All subscriptions and requests have a TTL now. So each subscription lifecycle is limited in time. If the subscription is closed because of TTL, it means that you subscribed after your transaction had been dropped by the network. - [FuelLabs#1407](FuelLabs#1407): The recipient is a `ContractId` instead of `Address`. The block producer should deploy its contract to receive the transaction fee. The collected fee is zero until the recipient contract is set. - [FuelLabs#1407](FuelLabs#1407): The `Mint` transaction is reworked with new fields to support the account-base model. It affects serialization and deserialization of the transaction and also affects GraphQL schema. - [FuelLabs#1407](FuelLabs#1407): The `Mint` transaction is the last transaction in the block instead of the first. - [FuelLabs#1374](FuelLabs#1374): Renamed `base_chain_height` to `da_height` and return current relayer height instead of latest Fuel block height. - [FuelLabs#1367](FuelLabs#1367): Update to the latest version of fuel-vm. - [FuelLabs#1363](FuelLabs#1363): Change message_proof api to take `nonce` instead of `message_id` - [FuelLabs#1355](FuelLabs#1355): Removed the `metrics` feature flag from the fuel-core crate, and metrics are now included by default. - [FuelLabs#1339](FuelLabs#1339): Added a new required field called `base_asset_id` to the `FeeParameters` definition in `ConsensusParameters`, as well as default values for `base_asset_id` in the `beta` and `dev` chain specifications. - [FuelLabs#1322](FuelLabs#1322): The `debug` flag is added to the CLI. The flag should be used for local development only. Enabling debug mode: - Allows GraphQL Endpoints to arbitrarily advance blocks. - Enables debugger GraphQL Endpoints. - Allows setting `utxo_validation` to `false`. - [FuelLabs#1318](FuelLabs#1318): Removed the `--sync-max-header-batch-requests` CLI argument, and renamed `--sync-max-get-txns` to `--sync-block-stream-buffer-size` to better represent the current behavior in the import. - [FuelLabs#1290](FuelLabs#1290): Standardize CLI args to use `-` instead of `_`. - [FuelLabs#1279](FuelLabs#1279): Added a new CLI flag to enable the Relayer service `--enable-relayer`, and disabled the Relayer service by default. When supplying the `--enable-relayer` flag, the `--relayer` argument becomes mandatory, and omitting it is an error. Similarly, providing a `--relayer` argument without the `--enable-relayer` flag is an error. Lastly, providing the `--keypair` or `--network` arguments will also produce an error if the `--enable-p2p` flag is not set. - [FuelLabs#1262](FuelLabs#1262): The `ConsensusParameters` aggregates all configuration data related to the consensus. It contains many fields that are segregated by the usage. The API of some functions was affected to use lesser types instead the whole `ConsensusParameters`. It is a huge breaking change requiring repetitively monotonically updating all places that use the `ConsensusParameters`. But during updating, consider that maybe you can use lesser types. Usage of them may simplify signatures of methods and make them more user-friendly and transparent. ### Removed #### Breaking - [FuelLabs#1484](FuelLabs#1484): Removed `--network` CLI argument. Now the name of the network is fetched form chain configuration. - [FuelLabs#1399](FuelLabs#1399): Removed `relayer-da-finalization` parameter from the relayer CLI. - [FuelLabs#1338](FuelLabs#1338): Updated GraphQL client to use `DependentCost` for `k256`, `mcpi`, `s256`, `scwq`, `swwq` opcodes. - [FuelLabs#1322](FuelLabs#1322): The `manual_blocks_enabled` flag is removed from the CLI. The analog is a `debug` flag. ## What's Changed * Added changelog in the same way as we did for `fuel-vm` by @xgreenx in FuelLabs#1287 * Decompose consensus params struct by @MitchTurner in FuelLabs#1262 * Replace all instances of `_` in CLI `long` arguments with `-` by @MitchTurner in FuelLabs#1290 * [rpc] Parallelise estimate_predicates endpoint by @YusongWang in FuelLabs#1293 * Add readable test names to test cases by @MitchTurner in FuelLabs#1286 * ED19 and ECR1 benchmarks by @Dentosal in FuelLabs#1263 * feat: Add `enable-relayer` flag to `fuel-core` CLI by @bvrooman in FuelLabs#1279 * Small nits found during deploying and benchmarking by @xgreenx in FuelLabs#1303 * Implemented `submit_and_await_commit_with_receipts` method by @xgreenx in FuelLabs#1304 * Removed the usage of flake and building of the bridge contract ABI by @xgreenx in FuelLabs#1302 * Download range of headers during syncing by @MitchTurner in FuelLabs#1270 * feat: Enable .env file loading using `env` feature by @bvrooman in FuelLabs#1308 * chore: Remove types::ConsensusParameters by @bvrooman in FuelLabs#1314 * docs: Debugging instructions using IDEs by @bvrooman in FuelLabs#1309 * test: Block Import benchmarks and test helpers by @bvrooman in FuelLabs#1274 * Fix audit CI for `webpki` by @xgreenx in FuelLabs#1319 * chore: Single buffer block sync by @bvrooman in FuelLabs#1318 * Run unit tests without default features only for specific crates by @xgreenx in FuelLabs#1321 * Added support for `debug` CLI flag. by @xgreenx in FuelLabs#1322 * Report Peers that give bad Block Info by @MitchTurner in FuelLabs#1331 * Pyroscope profiling by @Voxelot in FuelLabs#1324 * Add error handling for P2P requests to return `None` to requester and log error by @MitchTurner in FuelLabs#1342 * Importer Metrics by @Voxelot in FuelLabs#1355 * chore: Upgrade to Rust 1.72.0 in CI by @bvrooman in FuelLabs#1358 * chore: Upgrade `fuel-vm` to 0.37.0 by @bvrooman in FuelLabs#1338 * docs: remove stale schema by @calldelegation in FuelLabs#1362 * Replace message_id with nonce in `MessageProof` query by @MitchTurner in FuelLabs#1363 * feat: Configurable base asset by @bvrooman in FuelLabs#1339 * avoid gha caching for docker builds by @Voxelot in FuelLabs#1366 * Add tracking for peer heartbeats by @MitchTurner in FuelLabs#1356 * docs: fix broken link in README by @PaulRBerg in FuelLabs#1357 * Use right level of error for `.env` file and duplicated service by @xgreenx in FuelLabs#1372 * Add query and handling for `MessageStatus` by @MitchTurner in FuelLabs#1371 * Renamed `base_chain_height` to `da_height` and return current relayer height instead of latest Fuel block height by @xgreenx in FuelLabs#1374 * Remove `DiscoveryEvent` and use `KademliaEvent` instead by @MitchTurner in FuelLabs#1377 * Remove todo from code and provide some preliminary peer review config values by @MitchTurner in FuelLabs#1380 * Clippy lint to disallow logging crate macros in favor of tracing by @Dentosal in FuelLabs#1383 * fuel-vm nostd update by @Dentosal in FuelLabs#1367 * Up the `ethers` version to `2` to fix an issue with `tungstenite` by @xgreenx in FuelLabs#1390 * Increase heartbeat timeout to one minute by @Dentosal in FuelLabs#1393 * feat: batch transactions import by @bvrooman in FuelLabs#1349 * Fix an interger overflow in message_proof when looking beyond genesis by @Dentosal in FuelLabs#1392 * Improve keygen by @cr-fuel in FuelLabs#1397 * Benchmark {k256, s256, mcpi} opcodes using DependentCost by @Dentosal in FuelLabs#1395 * Use correct names for service metrics by @xgreenx in FuelLabs#1405 * feat: Retrieve finalized blocks from Ethereum by @bvrooman in FuelLabs#1399 * Updated CI to run `cargo update` weekly by @xgreenx in FuelLabs#1415 * CI `cargo update` weekly by @xgreenx in FuelLabs#1416 * Weekly `cargo update` by @github-actions in FuelLabs#1417 * Fix typos by @GoodDaisy in FuelLabs#1424 * feat: Support no_std and WASM compilation for `fuel-core` crates by @bvrooman in FuelLabs#1411 * Create sanity benchmark checks for all the arithmetic op codes by @MitchTurner in FuelLabs#1419 * Update benchmarking and collecting for `scwq`, `swwq` and `srwq` opcodes by @xgreenx in FuelLabs#1427 * Account-base fee collection by @xgreenx in FuelLabs#1407 * Rollback modification of the chain specification by @xgreenx in FuelLabs#1429 * Tx subscription cleanup (FuelLabs#1422) by @xgreenx in FuelLabs#1432 * Mark reserved peers as explicit for gossipsub to avoid reputation decreasing (FuelLabs#1423) by @xgreenx in FuelLabs#1434 * Weekly `cargo update` by @github-actions in FuelLabs#1438 * ci: add beta releasy to fuel-core by @kayagokalp in FuelLabs#1400 * setup a cron job to test beta-4 continuously by @Voxelot in FuelLabs#1436 * chore: Use MerkleRootCalculator when only BMT root is needed by @bvrooman in FuelLabs#1439 * Crypto op code sanity checks by @MitchTurner in FuelLabs#1430 * Update gas benchmarks for some storage opcodes by @Dentosal in FuelLabs#1408 * Split keygen into a create and a binary by @cr-fuel in FuelLabs#1426 * Basic Transfer TPS Benchmarking by @Voxelot in FuelLabs#1437 * "flow" op code sanity benchmarks by @MitchTurner in FuelLabs#1433 * Applying `FuelLabs#1435` to `master` by @xgreenx in FuelLabs#1440 * "memory" opcode sanity checks by @MitchTurner in FuelLabs#1444 * Add timeout to E2E test by @Voxelot in FuelLabs#1447 * fix owns_coins to properly paginate by @Voxelot in FuelLabs#1449 * Fixing incorrect measurement for fast(µs) opcodes by @xgreenx in FuelLabs#1457 * Call `flush` during end of the `FuelService` by @xgreenx in FuelLabs#1456 * chore: fix releasy test tracking branch from `main` to `master` by @kayagokalp in FuelLabs#1460 * Weekly `cargo update` by @github-actions in FuelLabs#1461 * `DB::repair` breaks the table for checkpoints by @xgreenx in FuelLabs#1463 * Avoid possible truncation of higher bits by @xgreenx in FuelLabs#1464 * Bumping `fuel-vm` to `v0.40.0` by @xgreenx in FuelLabs#1468 * Improvements for keygen cli and crates by @cr-fuel in FuelLabs#1465 * Handling overflows during arithmetic operations by @xgreenx in FuelLabs#1466 * test: Add contract code root benchmark by @bvrooman in FuelLabs#1452 * test: Add state root and contract id benchmarks by @bvrooman in FuelLabs#1462 * Avoid storage caching in benchmarks by @xgreenx in FuelLabs#1469 * Weekly `cargo update` by @github-actions in FuelLabs#1471 * Expose fuel-core version as a constant by @Br1ght0ne in FuelLabs#1473 * Upgrade ed25519_dalek 1.0.1 -> 2.0.0 by @Dentosal in FuelLabs#1475 * chore: Upgrade Rust 1.73 by @bvrooman in FuelLabs#1477 * Decrease number of iterations for sequential benchmarks by @xgreenx in FuelLabs#1480 * Add mldv bench by @Dentosal in FuelLabs#1481 * feat: Separate light and heavy gas costs (core) by @bvrooman in FuelLabs#1483 * chore: Remove `network` parameter from CLI by @bvrooman in FuelLabs#1484 * Transaction policies support by @xgreenx in FuelLabs#1472 * stage release candidate for beta-5 by @Voxelot in FuelLabs#1485 * setup docker auth for binary publishing by @Voxelot in FuelLabs#1488 * Fix macos binary publishing by @xgreenx in FuelLabs#1489 * Add documentation for our calculation of a _reasonable_ gas fee for storage by @MitchTurner in FuelLabs#1478 * "contract" op code sanity checks by @MitchTurner in FuelLabs#1453 * "other" op code sanity check by @MitchTurner in FuelLabs#1476 * Add push and pop benches by @MitchTurner in FuelLabs#1490 * chore: Remove unused gossipsub items by @bvrooman in FuelLabs#1491 * Adjusting of the prices benchmarks after sanity check benchmarks by @xgreenx in FuelLabs#1494 * Weekly `cargo update` by @github-actions in FuelLabs#1498 * Update benches that are affected by receipts by @Dentosal in FuelLabs#1454 * Support backward iteration in the RocksDB by @xgreenx in FuelLabs#1492 * Added validation of the coin's fields during block production and validation by @xgreenx in FuelLabs#1506 * Added price benchmark for `vm_initialization` by @xgreenx in FuelLabs#1502 * Sanity-check `gtf` opcode by @MitchTurner in FuelLabs#1503 * Tool for generating fee collection contract by @Voxelot in FuelLabs#1501 * Chain config for beta 5 network by @xgreenx in FuelLabs#1507 * Remove usage of iter_all_filtered from merkle_contract_state_range by @Dentosal in FuelLabs#1512 ## New Contributors * @YusongWang made their first contribution in FuelLabs#1293 * @calldelegation made their first contribution in FuelLabs#1362 * @PaulRBerg made their first contribution in FuelLabs#1357 * @cr-fuel made their first contribution in FuelLabs#1397 * @github-actions made their first contribution in FuelLabs#1417 * @GoodDaisy made their first contribution in FuelLabs#1424 **Full Changelog**: FuelLabs/fuel-core@v0.20.8...v0.21.0 --------- Co-authored-by: Brandon Vrooman <brandon.vrooman@fuel.sh>
Merged
xgreenx
added a commit
that referenced
this issue
Mar 30, 2024
## [Version 0.24.0] ### Added - [#1786](#1786): Regenesis now includes off-chain tables. - [#1716](#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](#1770): Add the new L1 event type for forced transactions. - [#1767](#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](#1747): The DA block height is now included in the genesis state. - [#1740](#1740): Remove optional fields from genesis configs - [#1737](#1737): Remove temporary tables for calculating roots during genesis. - [#1731](#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](#1779): Modify Relayer service to order Events from L1 by block index - [#1783](#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after #1664. - [#1693](#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in #1731 * Regenesis support by @MujkicA in #1693 * Remove genesis temp tables by @MujkicA in #1737 * Remove optional fields from configs by @MujkicA in #1740 * Weekly `cargo update` by @github-actions in #1745 * Regenesis should also store da block height by @xgreenx in #1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in #1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in #1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in #1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in #1757 * Weekly `cargo update` by @github-actions in #1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in #1760 * Adjustments to the upcoming testnet configs by @xgreenx in #1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in #1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in #1768 * Keep record of events from L1 in Block Header by @MitchTurner in #1769 * Feature/new fti event by @Voxelot in #1770 * Forkless state transition with upgradable WASM executor by @xgreenx in #1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in #1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in #1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in #1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in #1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in #1786 **Full Changelog**: v0.23.0...v0.24.0
Merged
xgreenx
added a commit
that referenced
this issue
Mar 30, 2024
## Version v0.24.0 ### Added - [#1786](#1786): Regenesis now includes off-chain tables. - [#1716](#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](#1770): Add the new L1 event type for forced transactions. - [#1767](#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](#1747): The DA block height is now included in the genesis state. - [#1740](#1740): Remove optional fields from genesis configs - [#1737](#1737): Remove temporary tables for calculating roots during genesis. - [#1731](#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](#1779): Modify Relayer service to order Events from L1 by block index - [#1783](#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after #1664. - [#1693](#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in #1731 * Regenesis support by @MujkicA in #1693 * Remove genesis temp tables by @MujkicA in #1737 * Remove optional fields from configs by @MujkicA in #1740 * Weekly `cargo update` by @github-actions in #1745 * Regenesis should also store da block height by @xgreenx in #1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in #1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in #1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in #1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in #1757 * Weekly `cargo update` by @github-actions in #1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in #1760 * Adjustments to the upcoming testnet configs by @xgreenx in #1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in #1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in #1768 * Keep record of events from L1 in Block Header by @MitchTurner in #1769 * Feature/new fti event by @Voxelot in #1770 * Forkless state transition with upgradable WASM executor by @xgreenx in #1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in #1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in #1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in #1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in #1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in #1786 * Release v0.24.0 by @xgreenx in #1791 * Moved chain specification into `fuel-core-bin` crate by @xgreenx in #1792 **Full Changelog**: v0.23.0...v0.24.1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It can be done by a new service or can be part of the block producer logic.
The text was updated successfully, but these errors were encountered: