Skip to content
This repository has been archived by the owner on Jan 19, 2021. It is now read-only.

Add benchmark.js and github-action-benchmark #130

Merged
merged 1 commit into from
Dec 8, 2020
Merged

Conversation

ryanio
Copy link
Contributor

@ryanio ryanio commented Sep 7, 2020

This PR formats our existing benchmarks into a benchmark.js suite to ensure statistical significance of benchmark runs, and adds github-action-benchmark to compare and keep track of results.

@coveralls
Copy link

coveralls commented Sep 7, 2020

Coverage Status

Coverage remained the same at 95.29% when pulling 3947fdf on benchmarkjs into aa9e47c on master.

@ryanio
Copy link
Contributor Author

ryanio commented Sep 7, 2020

Sample results from run:

1k-3-32-ran x 20,597 ops/sec ±23.17% (75 runs sampled)
1k-5-32-ran x 24,254 ops/sec ±2.16% (81 runs sampled)
1k-9-32-ran x 22,278 ops/sec ±2.52% (75 runs sampled)
1k-1k-32-ran x 9,052 ops/sec ±44.07% (33 runs sampled)
1k-1k-32-mir x 22,115 ops/sec ±5.47% (67 runs sampled)
Checkpointing: 100 iterations x 1,066 ops/sec ±6.10% (69 runs sampled)
Checkpointing: 500 iterations x 107 ops/sec ±107.49% (47 runs sampled)
Checkpointing: 1000 iterations x 110 ops/sec ±8.79% (58 runs sampled)
Checkpointing: 5000 iterations x 23.07 ops/sec ±8.41% (32 runs sampled)

Some interesting results here:

  • You can see how quickly the checkpointing slows down with larger iterations.
  • 1k-1k-32-ran slowdown above seems out of place with 44% amplitude, might be an outlier that normalizes with future runs.
    • Prior run results on my local machine:
      • 1k-1k-32-ran x 11,990 ops/sec ±11.70% (47 runs sampled)
      • 1k-1k-32-ran x 13,377 ops/sec ±12.03% (57 runs sampled)

i++
for (let i = 0; i < numOfIter; i++) {
trie.checkpoint()
await trie.put(keys[i], vals[i])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something else that'd be interesting to see is how checkpointing performance changes with the size of data to be commited. Here that size is one value (each value is committed separately).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something else that'd be interesting for finding the slowdown is a benchmark that does checkpoint, inserts a bunch of stuff and then does revert instead of commit.

Comparing this bench on v4 and v3 should tell us whether the slowdown is happening in commit (which is my suspicion) or not.

@holgerd77
Copy link
Member

Rebased this.

Copy link
Member

@holgerd77 holgerd77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Coverage is not working right now. Will merge nevertheless since the library has to be switched to codecov anyhow to prepare for the monorepo transition.

@holgerd77 holgerd77 merged commit 6d15e2b into master Dec 8, 2020
@holgerd77 holgerd77 deleted the benchmarkjs branch December 8, 2020 09:00
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants