Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proper bare-metal instances for benchmarking workflows #505

Merged
merged 3 commits into from
May 2, 2022

Conversation

ghzlatarev
Copy link
Contributor

@ghzlatarev ghzlatarev commented Apr 18, 2022

Description

closes: #298

  • Switch to c5d.metal for the benchmarking workflows, which is actually a bare-metal instance with the following specification:
Instance Name | Logical Processors | Memory  | Local Storage        | EBS-Optimized Bandwidth | Network Bandwidth
c5d.metal     | 96                 | 192 GiB | 4 x 900 GB NVMe SSD  | 14 Gbps                 | 25 Gbps

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

  • Linked to Github issue with discussion and accepted design OR have an explanation in the PR that describes this work.
  • Wrote unit tests.
  • Updated relevant documentation in the code.
  • Added one line describing your change in <branch>/CHANGELOG.md
  • Re-reviewed Files changed in the Github PR explorer.
  • If runtime changes, need to update the version numbers properly:
    • authoring_version: The version of the authorship interface. An authoring node will not attempt to author blocks unless this is equal to its native runtime.
    • spec_version: The version of the runtime specification. A full node will not attempt to use its native runtime in substitute for the on-chain Wasm runtime unless all of spec_name, spec_version, and authoring_version are the same between Wasm and native.
    • impl_version: The version of the implementation of the specification. Nodes are free to ignore this; it serves only as an indication that the code is different; as long as the other two versions are the same then while the actual code may be different, it is nonetheless required to do the same thing. Non-consensus-breaking optimizations are about the only changes that could be made which would result in only the impl_version changing.
    • transaction_version: The version of the extrinsics interface. This number must be updated in the following circumstances: extrinsic parameters (number, order, or types) have been changed; extrinsics or pallets have been removed; or the pallet order in the construct_runtime! macro or extrinsic order in a pallet has been changed. If this number is updated, then the spec_version must also be updated
  • Verify benchmarks & weights have been updated for any modified runtime logics
  • If import a new pallet, choose a proper module index for it, and allow it in BaseFilter. Ensure every extrinsic works from front-end. If there's corresponding tool, ensure both work for each other.
  • If needed, update our Javascript/Typescript APIs. These APIs are offcially used by exchanges or community developers.
  • If modifying existing runtime storage items, make sure to implement storage migrations for the runtime and test them with try-runtime. This includes migrations inhreited from upstream changes, and you can search the diffs for modifications of #[pallet::storage] items to check for any.

@ghzlatarev ghzlatarev self-assigned this Apr 18, 2022
@ghzlatarev ghzlatarev added the A-weights Area: Issues and PRs related to Substrate Weights label Apr 18, 2022
@ghzlatarev ghzlatarev marked this pull request as ready for review April 18, 2022 21:10
@ghzlatarev ghzlatarev added the A-ci Area: Continuous Integration label Apr 21, 2022
@ghzlatarev
Copy link
Contributor Author

ghzlatarev commented Apr 21, 2022

@Dengjianping @stechu @Garandor
you can find the benchmarking results from https://cockpit.expralite.pelagos.systems/system/terminal (Rob's office) in the following commit c4f9de1
The configuration is Ubuntu 20.04; Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz; 32GB RAM, HDD storage 15k spinning disks in hardware raid 0.

Overall the results are roughly 3 times worse than c5d.metal in the PR.
MantaPay is roughly 2x worse than c5d.metal

Copy link
Contributor

@Dengjianping Dengjianping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ghzlatarev
Copy link
Contributor Author

You can see the results from the Hetzner machine in this commit f71d350
That configuration is exactly like Polkadot's required hardware:
AMD Ryzen 7 3700X 8-Core Processor 8 cores, 64GB ECC, NVMe SSD
And the results are actually better than the c5d.metal machine, which is expected as the RAM doesn't really matter and Polkadot's required CPU is actually better than the c5d.metal one.

@ghzlatarev ghzlatarev merged commit aa0b10d into manta May 2, 2022
@ghzlatarev
Copy link
Contributor Author

We're going with the slightly more conservative c4d.metal weights.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-ci Area: Continuous Integration A-weights Area: Issues and PRs related to Substrate Weights
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Benchmark unfiltered pallets by baremetal machine
3 participants