Create sanity benchmark checks for all the arithmetic op codes #1419
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
8 tasks
xgreenx
reviewed
Oct 13, 2023
benches/benches/block_target_gas.rs
Outdated
| @@ -29,6 +38,27 @@ use fuel_core_types::{ | |||
| fuel_types::AssetId, | |||
| }; | |||
|
|
|||
| /// Allocates a byte array from heap and initializes it. Then points `reg` to it. | |||
| fn aloc_bytearray<const S: usize>(reg: u8, v: [u8; S]) -> Vec<Instruction> { | |||
There was a problem hiding this comment.
It would be nice to move these functions to utils and reuse them instead of duplicating them.
benches/benches/block_target_gas.rs
Outdated
Comment on lines
393
to
405
| { | ||
| let data = vec![0u8; 32]; | ||
| run( | ||
| "gtf opcode", | ||
| group, | ||
| [ | ||
| op::gtf_args(0x10, RegId::ZERO, GTFArgs::ScriptData), | ||
| op::jmpb(RegId::ZERO, 0), | ||
| ] | ||
| .to_vec(), | ||
| data, | ||
| ); | ||
| } |
There was a problem hiding this comment.
Moving into a separate section during this PR would be nice because we need to test with any possible GTFArgs.
You don't need to test with all GTFArgs. But could you create a dedicated follow-up issue only for the gtf opcode, please?
We need to find the worst possible getter and use it as the primary price for the GTF opcode.
There was a problem hiding this comment.
I added it as a sub-requirement of this issue. It looks like a lot of them, so, yeah, I agree it should be done in a separate PR :)
There was a problem hiding this comment.
It looks like you're putting GTF tests in the run_alu benches, but GTF is not part of the arithmetic ops - For this PR, would it make sense to move GTF out of run_alu and into a separate function?
There was a problem hiding this comment.
@bvrooman We intend to add thorough coverage for gtf with another PR.
bvrooman
reviewed
Oct 16, 2023
benches/benches/block_target_gas.rs
Outdated
Comment on lines
393
to
405
| { | ||
| let data = vec![0u8; 32]; | ||
| run( | ||
| "gtf opcode", | ||
| group, | ||
| [ | ||
| op::gtf_args(0x10, RegId::ZERO, GTFArgs::ScriptData), | ||
| op::jmpb(RegId::ZERO, 0), | ||
| ] | ||
| .to_vec(), | ||
| data, | ||
| ); | ||
| } |
There was a problem hiding this comment.
It looks like you're putting GTF tests in the run_alu benches, but GTF is not part of the arithmetic ops - For this PR, would it make sense to move GTF out of run_alu and into a separate function?
benches/benches/block_target_gas.rs
Outdated
| .to_vec(), | ||
| vec![], | ||
| ); | ||
|
|
There was a problem hiding this comment.
It looks like you're missing the MLDV op (but maybe that was intentional?). I just realized that the ops spec isn't really in alphabetical order, but this is where it would go relative to the rest of the arithmetic ops.
benches/benches/block_target_gas.rs
Outdated
Comment on lines
674
to
676
| MathArgs { | ||
| op: MathOp::SUB, | ||
| indirect_rhs: true, |
There was a problem hiding this comment.
Would it be valuable to test any of the other misc wide ops as well?
(Same question for WQ ops)
|
@bvrooman It might have been misleading to call this That might be an oversight by the writers of the |
If I had to guess, Now, your |
xgreenx
reviewed
Oct 17, 2023
There was a problem hiding this comment.
After running these, I've found a number of divergent op codes that could possibly require updated gas costs:
Yeah, it seems these opcodes are a little bit underpriced. But not too much. Because you used another hardware to run them than we used to define prices. I've run benchmarks for prices first, and the block_gas_limit benchmark with new prices produces better results.
block target estimation/sub opcode
time: [9.6523 ms 9.6840 ms 9.7187 ms]
change: [-3.7896% -3.2630% -2.7580%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/subi opcode
time: [9.4285 ms 9.4505 ms 9.4767 ms]
change: [-4.1387% -3.7634% -3.4060%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/xor opcode
time: [9.4393 ms 9.4714 ms 9.5094 ms]
change: [-3.9255% -3.4993% -3.0863%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/xori opcode
time: [9.3637 ms 9.4041 ms 9.4483 ms]
change: [-3.2782% -2.7571% -2.1994%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wdcm opcode
time: [11.603 ms 11.630 ms 11.663 ms]
change: [-4.2311% -3.8511% -3.4542%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wdop opcode
time: [13.161 ms 13.205 ms 13.257 ms]
change: [-3.6999% -3.3480% -2.9338%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wdml opcode
time: [13.354 ms 13.393 ms 13.436 ms]
change: [-4.4740% -4.0350% -3.5724%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wddv opcode
time: [15.602 ms 15.626 ms 15.656 ms]
change: [-35.788% -35.616% -35.449%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wdmd opcode
time: [19.051 ms 19.090 ms 19.138 ms]
change: [-22.502% -22.294% -22.054%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wdam opcode
time: [17.373 ms 17.429 ms 17.489 ms]
change: [-39.283% -32.188% -26.632%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wdmm opcode
time: [18.882 ms 18.901 ms 18.922 ms]
change: [-22.358% -22.208% -22.065%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wqcm opcode
time: [12.041 ms 12.085 ms 12.142 ms]
change: [-5.0285% -3.5735% -2.5372%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wqop opcode
time: [15.092 ms 15.136 ms 15.184 ms]
change: [-3.1202% -2.7670% -2.4070%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wqml opcode
time: [15.082 ms 15.123 ms 15.172 ms]
change: [-3.6045% -3.2474% -2.8672%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wqdv opcode
time: [19.316 ms 19.375 ms 19.458 ms]
change: [-3.7534% -3.3422% -2.8591%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wqmd opcode
time: [19.568 ms 19.633 ms 19.706 ms]
change: [-30.202% -29.907% -29.640%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wqam opcode
time: [15.648 ms 15.671 ms 15.700 ms]
change: [-22.636% -22.422% -22.217%] (p = 0.00 < 0.05)
Performance has improved.
block target estimation/wqmm opcode
time: [26.896 ms 26.946 ms 27.006 ms]
change: [-22.056% -21.430% -20.983%] (p = 0.00 < 0.05)
Performance has improved.While average execution time for 1M gas is 10ms.
benches/benches/block_target_gas.rs
Outdated
| group.finish(); | ||
| } | ||
|
|
||
| fn run_alu(group: &mut BenchmarkGroup<WallTime>) { | ||
| run( | ||
| "add opcode", |
There was a problem hiding this comment.
It would be nice to add an alu/ prefix to each benchmark. In this case, we can easily run only all alu benchmarks later.
bvrooman
previously approved these changes
Oct 17, 2023
xgreenx
reviewed
Oct 17, 2023
benches/benches/block_target_gas.rs
Outdated
| #[path = "utils.rs"] | ||
| mod utils; | ||
|
|
||
| #[path = "block_target_gas/alu.rs"] |
There was a problem hiding this comment.
Hmm, it already should work without path
There was a problem hiding this comment.
I moved the block_target_gas code into a new block_target_gas/main.rs and now it does (apparently main.rs is the convention for benches).
benches/benches/block_target_gas.rs
Outdated
| @@ -29,6 +39,17 @@ use fuel_core_types::{ | |||
| fuel_types::AssetId, | |||
| }; | |||
|
|
|||
| #[path = "utils.rs"] | |||
There was a problem hiding this comment.
It should work without path
There was a problem hiding this comment.
Now that I've moved block_target_gas into its own module thing, it won't. But it's true in the old structure that I could have.
error[E0583]: file not found for module `utils`
--> benches/benches/block_target_gas/main.rs:43:1
|
43 | mod utils;
| ^^^^^^^^^^
|
= help: to create the module `utils`, create file "benches/benches/block_target_gas/utils.rs" or "benches/benches/block_target_gas/utils/mod.rs"
benches/benches/vm.rs
Outdated
| @@ -1,4 +1,4 @@ | |||
| mod set; | |||
| pub mod set; | |||
There was a problem hiding this comment.
Why do we need to make it public?
benches/benches/set/alu.rs
Outdated
| fn make_u128(reg: u8, v: u128) -> Vec<Instruction> { | ||
| aloc_bytearray(reg, v.to_be_bytes()) | ||
| } | ||
| #[path = "../utils.rs"] |
There was a problem hiding this comment.
I think you should be able to import utils on the set module level and be able to use them here from super
There was a problem hiding this comment.
Yes. From the vm.rs because that is the parent to set.
Created `block_gas_limit_set` similar to `vm_set` folder. Removed `#[path = ...]`.
xgreenx
approved these changes
Oct 17, 2023
bvrooman
pushed a commit
that referenced
this pull request
Nov 24, 2023
## Version 0.21.0 This release focuses on preparing `fuel-core` for the mainnet environment. - Most of the changes improved the security and stability of the node. - The gas model was reworked to cover all aspects of execution. - The benchmarking system was significantly enhanced, covering worst scenarios. - A new set of benchmarks was added to track the accuracy of gas prices. - Optimized heavy operations and removed/replaced exploitable functionality. Besides that, there are more concrete changes: - Unified naming conventions for all CLI arguments. Added dependencies between related fields to avoid misconfiguration in case of missing arguments. Added `--debug` flag that enables additional functionality like a debugger. - Improved telemetry to cover the internal work of services and added support for the Pyroscope, allowing it to generate real-time flamegraphs to track performance. - Improved stability of the P2P layer and adjusted the updating of reputation. The speed of block synchronization was significantly increased. - The node is more stable and resilient. Improved DoS resistance and resource management. Fixed critical bugs during state transition. - Reworked the `Mint` transaction to accumulate the fee from block production inside the contract defined by the block producer. FuelVM received a lot of safety and stability improvements: - The audit helped identify some bugs and errors that have been successfully fixed. - Updated the gas price model to charge for resources used during the transaction lifecycle. - Added `no_std` and 32 bit system support. This opens doors for fraud proving in the future. - Removed the `ChainId` from the `PredicateId` calculation, allowing the use of predicates cross-chain. - Improvements in the performance of some storage-related opcodes. - Support the `ECAL` instruction that allows adding custom functionality to the VM. It can be used to create unique rollups or advanced indexers in the future. - Support of [transaction policies](https://github.com/FuelLabs/fuel-vm/blob/master/CHANGELOG.md#version-0420) provides additional safety for the user. It also allows the implementation of a multi-dimensional price model in the future, making the transaction execution cheaper and allowing more transactions that don't affect storage. - Refactored errors, returning more detailed errors to the user, simplifying debugging. ### Added - [#1503](#1503): Add `gtf` opcode sanity check. - [#1502](#1502): Added price benchmark for `vm_initialization`. - [#1501](#1501): Add a CLI command for generating a fee collection contract. - [#1492](#1492): Support backward iteration in the RocksDB. It allows backward queries that were not allowed before. - [#1490](#1490): Add push and pop benchmarks. - [#1485](#1485): Prepare rc release of fuel core v0.21 - [#1476](#1453): Add the majority of the "other" benchmarks for contract opcodes. - [#1473](#1473): Expose fuel-core version as a constant - [#1469](#1469): Added support of bloom filter for RocksDB tables and increased the block cache. - [#1465](#1465): Improvements for keygen cli and crates - [#1642](#1462): Added benchmark to measure the performance of contract state and contract ID calculation; use for gas costing. - [#1457](#1457): Fixing incorrect measurement for fast(µs) opcodes. - [#1456](#1456): Added flushing of the RocksDB during a graceful shutdown. - [#1456](#1456): Added more logs to track the service lifecycle. - [#1453](#1453): Add the majority of the "sanity" benchmarks for contract opcodes. - [#1452](#1452): Added benchmark to measure the performance of contract root calculation when utilizing the maximum contract size; used for gas costing of contract root during predicate owner validation. - [#1449](#1449): Fix coin pagination in e2e test client. - [#1447](#1447): Add timeout for continuous e2e tests - [#1444](#1444): Add "sanity" benchmarks for memory opcodes. - [#1437](#1437): Add some transaction throughput tests for basic transfers. - [#1436](#1436): Add a github action to continuously test beta-4. - [#1433](#1433): Add "sanity" benchmarks for flow opcodes. - [#1432](#1432): Add a new `--api-request-timeout` argument to control TTL for GraphQL requests. - [#1430](#1430): Add "sanity" benchmarks for crypto opcodes. - [#1426](#1426) Split keygen into a create and a binary. - [#1419](#1419): Add additional "sanity" benchmarks for arithmetic op code instructions. - [#1411](#1411): Added WASM and `no_std` compatibility. - [#1405](#1405): Use correct names for service metrics. - [#1400](#1400): Add releasy beta to fuel-core so that new commits to fuel-core master triggers fuels-rs. - [#1371](#1371): Add new client function for querying the `MessageStatus` for a specific message (by `Nonce`). - [#1356](#1356): Add peer reputation reporting to heartbeat code. - [#1355](#1355): Added new metrics related to block importing, such as tps, sync delays etc. - [#1339](#1339): Adds `baseAssetId` to `FeeParameters` in the GraphQL API. - [#1331](#1331): Add peer reputation reporting to block import code. - [#1324](#1324): Added pyroscope profiling to fuel-core, intended to be used by a secondary docker image that has debug symbols enabled. - [#1309](#1309): Add documentation for running debug builds with CLion and Visual Studio Code. - [#1308](#1308): Add support for loading .env files when compiling with the `env` feature. This allows users to conveniently supply CLI arguments in a secure and IDE-agnostic way. - [#1304](#1304): Implemented `submit_and_await_commit_with_receipts` method for `FuelClient`. - [#1286](#1286): Include readable names for test cases where missing. - [#1274](#1274): Added tests to benchmark block synchronization. - [#1263](#1263): Add gas benchmarks for `ED19` and `ECR1` instructions. ### Changed - [#1512](#1512): Internally simplify merkle_contract_state_range. - [#1507](#1507): Updated chain configuration to be ready for beta 5 network. It includes opcode prices from the latest benchmark and contract for the block producer. - [#1477](#1477): Upgraded the Rust version used in CI and containers to 1.73.0. Also includes associated Clippy changes. - [#1469](#1469): Replaced usage of `MemoryTransactionView` by `Checkpoint` database in the benchmarks. - [#1468](#1468): Bumped version of the `fuel-vm` to `v0.40.0`. It brings some breaking changes into consensus parameters API because of changes in the underlying types. - [#1466](#1466): Handling overflows during arithmetic operations. - [#1460](#1460): Change tracking branch from main to master for releasy tests. - [#1454](#1454): Update gas benchmarks for opcodes that append receipts. - [#1440](#1440): Don't report reserved nodes that send invalid transactions. - [#1439](#1439): Reduced memory BMT consumption during creation of the header. - [#1434](#1434): Continue gossiping transactions to reserved peers regardless of gossiping reputation score. - [#1408](#1408): Update gas benchmarks for storage opcodes to use a pre-populated database to get more accurate worst-case costs. - [#1399](#1399): The Relayer now queries Ethereum for its latest finalized block instead of using a configurable "finalization period" to presume finality. - [#1397](#1397): Improved keygen. Created a crate to be included from forc plugins and upgraded internal library to drop requirement of protoc to build - [#1395](#1395): Add DependentCost benchmarks for `k256`, `s256` and `mcpi` instructions. - [#1393](#1393): Increase heartbeat timeout from `2` to `60` seconds, as suggested in [this issue](#1330). - [#1392](#1392): Fixed an overflow in `message_proof`. - [#1390](#1390): Up the `ethers` version to `2` to fix an issue with `tungstenite`. - [#1383](#1383): Disallow usage of `log` crate internally in favor of `tracing` crate. - [#1380](#1380): Add preliminary, hard-coded config values for heartbeat peer reputation, removing `todo`. - [#1377](#1377): Remove `DiscoveryEvent` and use `KademliaEvent` directly in `DiscoveryBehavior`. - [#1366](#1366): Improve caching during docker builds in CI by replacing gha - [#1358](#1358): Upgraded the Rust version used in CI to 1.72.0. Also includes associated Clippy changes. - [#1349](#1349): Updated peer-to-peer transactions API to support multiple blocks in a single request, and updated block synchronization to request multiple blocks based on the configured range of headers. - [#1342](#1342): Add error handling for P2P requests to return `None` to requester and log error. - [#1318](#1318): Modified block synchronization to use asynchronous task execution when retrieving block headers. - [#1314](#1314): Removed `types::ConsensusParameters` in favour of `fuel_tx:ConsensusParameters`. - [#1302](#1302): Removed the usage of flake and building of the bridge contract ABI. It simplifies the maintenance and updating of the events, requiring only putting the event definition into the codebase of the relayer. - [#1293](#1293): Parallelized the `estimate_predicates` endpoint to utilize all available threads. - [#1270](#1270): Modify the way block headers are retrieved from peers to be done in batches. #### Breaking - [#1506](#1506): Added validation of the coin's fields during block production and validation. Before, it was possible to submit a transaction that didn't match the coin's values in the database, allowing printing/using unavailable assets. - [#1491](#1491): Removed unused request and response variants from the Gossipsub implementation, as well as related definitions and tests. Specifically, this removes gossiping of `ConsensusVote` and `NewBlock` events. - [#1472](#1472): Upgraded `fuel-vm` to `v0.42.0`. It introduces transaction policies that changes layout of the transaction. FOr more information check the [v0.42.0](FuelLabs/fuel-vm#635) release. - [#1470](#1470): Divide `DependentCost` into "light" and "heavy" operations. - [#1464](#1464): Avoid possible truncation of higher bits. It may invalidate the code that truncated higher bits causing different behavior on 32-bit vs. 64-bit systems. The change affects some endpoints that now require lesser integers. - [#1432](#1432): All subscriptions and requests have a TTL now. So each subscription lifecycle is limited in time. If the subscription is closed because of TTL, it means that you subscribed after your transaction had been dropped by the network. - [#1407](#1407): The recipient is a `ContractId` instead of `Address`. The block producer should deploy its contract to receive the transaction fee. The collected fee is zero until the recipient contract is set. - [#1407](#1407): The `Mint` transaction is reworked with new fields to support the account-base model. It affects serialization and deserialization of the transaction and also affects GraphQL schema. - [#1407](#1407): The `Mint` transaction is the last transaction in the block instead of the first. - [#1374](#1374): Renamed `base_chain_height` to `da_height` and return current relayer height instead of latest Fuel block height. - [#1367](#1367): Update to the latest version of fuel-vm. - [#1363](#1363): Change message_proof api to take `nonce` instead of `message_id` - [#1355](#1355): Removed the `metrics` feature flag from the fuel-core crate, and metrics are now included by default. - [#1339](#1339): Added a new required field called `base_asset_id` to the `FeeParameters` definition in `ConsensusParameters`, as well as default values for `base_asset_id` in the `beta` and `dev` chain specifications. - [#1322](#1322): The `debug` flag is added to the CLI. The flag should be used for local development only. Enabling debug mode: - Allows GraphQL Endpoints to arbitrarily advance blocks. - Enables debugger GraphQL Endpoints. - Allows setting `utxo_validation` to `false`. - [#1318](#1318): Removed the `--sync-max-header-batch-requests` CLI argument, and renamed `--sync-max-get-txns` to `--sync-block-stream-buffer-size` to better represent the current behavior in the import. - [#1290](#1290): Standardize CLI args to use `-` instead of `_`. - [#1279](#1279): Added a new CLI flag to enable the Relayer service `--enable-relayer`, and disabled the Relayer service by default. When supplying the `--enable-relayer` flag, the `--relayer` argument becomes mandatory, and omitting it is an error. Similarly, providing a `--relayer` argument without the `--enable-relayer` flag is an error. Lastly, providing the `--keypair` or `--network` arguments will also produce an error if the `--enable-p2p` flag is not set. - [#1262](#1262): The `ConsensusParameters` aggregates all configuration data related to the consensus. It contains many fields that are segregated by the usage. The API of some functions was affected to use lesser types instead the whole `ConsensusParameters`. It is a huge breaking change requiring repetitively monotonically updating all places that use the `ConsensusParameters`. But during updating, consider that maybe you can use lesser types. Usage of them may simplify signatures of methods and make them more user-friendly and transparent. ### Removed #### Breaking - [#1484](#1484): Removed `--network` CLI argument. Now the name of the network is fetched form chain configuration. - [#1399](#1399): Removed `relayer-da-finalization` parameter from the relayer CLI. - [#1338](#1338): Updated GraphQL client to use `DependentCost` for `k256`, `mcpi`, `s256`, `scwq`, `swwq` opcodes. - [#1322](#1322): The `manual_blocks_enabled` flag is removed from the CLI. The analog is a `debug` flag. ## What's Changed * Added changelog in the same way as we did for `fuel-vm` by @xgreenx in #1287 * Decompose consensus params struct by @MitchTurner in #1262 * Replace all instances of `_` in CLI `long` arguments with `-` by @MitchTurner in #1290 * [rpc] Parallelise estimate_predicates endpoint by @YusongWang in #1293 * Add readable test names to test cases by @MitchTurner in #1286 * ED19 and ECR1 benchmarks by @Dentosal in #1263 * feat: Add `enable-relayer` flag to `fuel-core` CLI by @bvrooman in #1279 * Small nits found during deploying and benchmarking by @xgreenx in #1303 * Implemented `submit_and_await_commit_with_receipts` method by @xgreenx in #1304 * Removed the usage of flake and building of the bridge contract ABI by @xgreenx in #1302 * Download range of headers during syncing by @MitchTurner in #1270 * feat: Enable .env file loading using `env` feature by @bvrooman in #1308 * chore: Remove types::ConsensusParameters by @bvrooman in #1314 * docs: Debugging instructions using IDEs by @bvrooman in #1309 * test: Block Import benchmarks and test helpers by @bvrooman in #1274 * Fix audit CI for `webpki` by @xgreenx in #1319 * chore: Single buffer block sync by @bvrooman in #1318 * Run unit tests without default features only for specific crates by @xgreenx in #1321 * Added support for `debug` CLI flag. by @xgreenx in #1322 * Report Peers that give bad Block Info by @MitchTurner in #1331 * Pyroscope profiling by @Voxelot in #1324 * Add error handling for P2P requests to return `None` to requester and log error by @MitchTurner in #1342 * Importer Metrics by @Voxelot in #1355 * chore: Upgrade to Rust 1.72.0 in CI by @bvrooman in #1358 * chore: Upgrade `fuel-vm` to 0.37.0 by @bvrooman in #1338 * docs: remove stale schema by @calldelegation in #1362 * Replace message_id with nonce in `MessageProof` query by @MitchTurner in #1363 * feat: Configurable base asset by @bvrooman in #1339 * avoid gha caching for docker builds by @Voxelot in #1366 * Add tracking for peer heartbeats by @MitchTurner in #1356 * docs: fix broken link in README by @PaulRBerg in #1357 * Use right level of error for `.env` file and duplicated service by @xgreenx in #1372 * Add query and handling for `MessageStatus` by @MitchTurner in #1371 * Renamed `base_chain_height` to `da_height` and return current relayer height instead of latest Fuel block height by @xgreenx in #1374 * Remove `DiscoveryEvent` and use `KademliaEvent` instead by @MitchTurner in #1377 * Remove todo from code and provide some preliminary peer review config values by @MitchTurner in #1380 * Clippy lint to disallow logging crate macros in favor of tracing by @Dentosal in #1383 * fuel-vm nostd update by @Dentosal in #1367 * Up the `ethers` version to `2` to fix an issue with `tungstenite` by @xgreenx in #1390 * Increase heartbeat timeout to one minute by @Dentosal in #1393 * feat: batch transactions import by @bvrooman in #1349 * Fix an interger overflow in message_proof when looking beyond genesis by @Dentosal in #1392 * Improve keygen by @cr-fuel in #1397 * Benchmark {k256, s256, mcpi} opcodes using DependentCost by @Dentosal in #1395 * Use correct names for service metrics by @xgreenx in #1405 * feat: Retrieve finalized blocks from Ethereum by @bvrooman in #1399 * Updated CI to run `cargo update` weekly by @xgreenx in #1415 * CI `cargo update` weekly by @xgreenx in #1416 * Weekly `cargo update` by @github-actions in #1417 * Fix typos by @GoodDaisy in #1424 * feat: Support no_std and WASM compilation for `fuel-core` crates by @bvrooman in #1411 * Create sanity benchmark checks for all the arithmetic op codes by @MitchTurner in #1419 * Update benchmarking and collecting for `scwq`, `swwq` and `srwq` opcodes by @xgreenx in #1427 * Account-base fee collection by @xgreenx in #1407 * Rollback modification of the chain specification by @xgreenx in #1429 * Tx subscription cleanup (#1422) by @xgreenx in #1432 * Mark reserved peers as explicit for gossipsub to avoid reputation decreasing (#1423) by @xgreenx in #1434 * Weekly `cargo update` by @github-actions in #1438 * ci: add beta releasy to fuel-core by @kayagokalp in #1400 * setup a cron job to test beta-4 continuously by @Voxelot in #1436 * chore: Use MerkleRootCalculator when only BMT root is needed by @bvrooman in #1439 * Crypto op code sanity checks by @MitchTurner in #1430 * Update gas benchmarks for some storage opcodes by @Dentosal in #1408 * Split keygen into a create and a binary by @cr-fuel in #1426 * Basic Transfer TPS Benchmarking by @Voxelot in #1437 * "flow" op code sanity benchmarks by @MitchTurner in #1433 * Applying `#1435` to `master` by @xgreenx in #1440 * "memory" opcode sanity checks by @MitchTurner in #1444 * Add timeout to E2E test by @Voxelot in #1447 * fix owns_coins to properly paginate by @Voxelot in #1449 * Fixing incorrect measurement for fast(µs) opcodes by @xgreenx in #1457 * Call `flush` during end of the `FuelService` by @xgreenx in #1456 * chore: fix releasy test tracking branch from `main` to `master` by @kayagokalp in #1460 * Weekly `cargo update` by @github-actions in #1461 * `DB::repair` breaks the table for checkpoints by @xgreenx in #1463 * Avoid possible truncation of higher bits by @xgreenx in #1464 * Bumping `fuel-vm` to `v0.40.0` by @xgreenx in #1468 * Improvements for keygen cli and crates by @cr-fuel in #1465 * Handling overflows during arithmetic operations by @xgreenx in #1466 * test: Add contract code root benchmark by @bvrooman in #1452 * test: Add state root and contract id benchmarks by @bvrooman in #1462 * Avoid storage caching in benchmarks by @xgreenx in #1469 * Weekly `cargo update` by @github-actions in #1471 * Expose fuel-core version as a constant by @Br1ght0ne in #1473 * Upgrade ed25519_dalek 1.0.1 -> 2.0.0 by @Dentosal in #1475 * chore: Upgrade Rust 1.73 by @bvrooman in #1477 * Decrease number of iterations for sequential benchmarks by @xgreenx in #1480 * Add mldv bench by @Dentosal in #1481 * feat: Separate light and heavy gas costs (core) by @bvrooman in #1483 * chore: Remove `network` parameter from CLI by @bvrooman in #1484 * Transaction policies support by @xgreenx in #1472 * stage release candidate for beta-5 by @Voxelot in #1485 * setup docker auth for binary publishing by @Voxelot in #1488 * Fix macos binary publishing by @xgreenx in #1489 * Add documentation for our calculation of a _reasonable_ gas fee for storage by @MitchTurner in #1478 * "contract" op code sanity checks by @MitchTurner in #1453 * "other" op code sanity check by @MitchTurner in #1476 * Add push and pop benches by @MitchTurner in #1490 * chore: Remove unused gossipsub items by @bvrooman in #1491 * Adjusting of the prices benchmarks after sanity check benchmarks by @xgreenx in #1494 * Weekly `cargo update` by @github-actions in #1498 * Update benches that are affected by receipts by @Dentosal in #1454 * Support backward iteration in the RocksDB by @xgreenx in #1492 * Added validation of the coin's fields during block production and validation by @xgreenx in #1506 * Added price benchmark for `vm_initialization` by @xgreenx in #1502 * Sanity-check `gtf` opcode by @MitchTurner in #1503 * Tool for generating fee collection contract by @Voxelot in #1501 * Chain config for beta 5 network by @xgreenx in #1507 * Remove usage of iter_all_filtered from merkle_contract_state_range by @Dentosal in #1512 ## New Contributors * @YusongWang made their first contribution in #1293 * @calldelegation made their first contribution in #1362 * @PaulRBerg made their first contribution in #1357 * @cr-fuel made their first contribution in #1397 * @github-actions made their first contribution in #1417 * @GoodDaisy made their first contribution in #1424 **Full Changelog**: v0.20.8...v0.21.0 --------- Co-authored-by: Brandon Vrooman <brandon.vrooman@fuel.sh>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#1386
This PR adds benchmark tests for all the arithmetic op codes. The actual purpose being running those tests to see if the costs actually line up for all the codes.
These tests work by starting a vm with a set amount of gas, and running an "infinite" loop of each op code until the vm runs out of gas. This is timed and the time is reported.
After running these, I've found a number of divergent op codes that could possibly require updated gas costs:
So, that's
wddv,wdmd,wdam,wdmm,wqdv,wqmd,wqam,wqmm.Whereas all the other op codes take close to 2ms to run out of gas, implying that these divergent op codes could be priced higher. These "wide" op codes all share similar setup, and not all of the "wide" op codes fail the test, so it's probably not the setup that is causing these specific op code benchmarks to fail.
Here are all of the results: