Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Benchmark tracking #1

Open
porcuquine opened this issue Jan 8, 2020 · 6 comments
Open

Benchmark tracking #1

porcuquine opened this issue Jan 8, 2020 · 6 comments

Comments

@porcuquine
Copy link
Collaborator

Issue for tracking benchmarks over time.
@porcuquine
Copy link
Collaborator Author

porcuquine commented Jan 8, 2020 •
edited
Loading

3e92fa9f1be80575f1e5a0b1617a75b04805f152 on my development machine

(base) ➜  neptune git:(master) cargo bench
    …
    Finished bench [optimized] target(s) in 28.58s
     Running target/release/deps/neptune-6aa7ba28772916f6

running 3 tests
test poseidon::tests::hash_det ... ignored
test poseidon::tests::reset ... ignored
test tests::constants_consistency ... ignored

test result: ok. 0 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out

     Running target/release/deps/hash-6e8d97b43ee19a2b
Gnuplot not found or not usable, disabling plotting
`gnuplot --version` failed with error message:
dyld: Library not loaded: /usr/local/opt/readline/lib/libreadline.7.dylib
  Referenced from: /usr/local/bin/gnuplot
  Reason: image not found

hash/Sha2 256/Generated scalars
                        time:   [2.7975 us 2.8167 us 2.8364 us]
                        change: [-2.2134% -1.5500% -0.9210%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high mild
hash/Sha2 512/Generated scalars
                        time:   [2.6885 us 2.7036 us 2.7203 us]
                        change: [+1.8016% +2.7368% +3.8340%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
  5 (5.00%) high mild
  6 (6.00%) high severe
hash/Poseidon hash/Generated scalars
                        time:   [63.715 us 64.102 us 64.499 us]
                        change: [-1.6427% -0.9227% -0.2130%] (p = 0.01 < 0.05)
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild

Gnuplot not found or not usable, disabling plotting
`gnuplot --version` failed with error message:
dyld: Library not loaded: /usr/local/opt/readline/lib/libreadline.7.dylib
  Referenced from: /usr/local/bin/gnuplot
  Reason: image not found

@porcuquine
Copy link
Collaborator Author

615ec19

Binary hash (width 3):

hash-64/Sha2 256/Generated scalars
                        time:   [1.4690 us 1.4713 us 1.4738 us]
                        change: [-1.7410% -1.3525% -0.8979%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) high mild
  6 (6.00%) high severe
hash-64/Sha2 512/Generated scalars
                        time:   [1.2767 us 1.2783 us 1.2800 us]
                        change: [-1.4560% -1.0486% -0.6472%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
  4 (4.00%) high mild
  6 (6.00%) high severe
hash-64/Poseidon hash/Generated scalars
                        time:   [29.887 us 29.937 us 29.991 us]
                        change: [-1.8042% -1.3567% -0.9145%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 11 outliers among 100 measurements (11.00%)
  7 (7.00%) high mild
  4 (4.00%) high severe

Circuit constraints (ARITY = 2): https://github.com/filecoin-project/neptune/blob/615ec19c3570a4c4e268273115bc8535db2b20ca/src/circuit.rs#L282

@porcuquine
Copy link
Collaborator Author

7acbdcf

https://github.com/filecoin-project/neptune/blob/7acbdcfa70d1f3f11d47e2c38ed79ff7da84559b/src/circuit.rs#L416

@porcuquine
Copy link
Collaborator Author

porcuquine commented Jan 27, 2020 •
edited
Loading

e483a51

https://github.com/filecoin-project/neptune/blob/e483a514666ab828ab38051413e132fc49dbe867/src/circuit.rs#L445

Perf regression might be related to benchmark. We're now computing (rather than just loading) constants on first access, and benchmark may be including this time.

[UPDATE: benchmark was actually run with arity 2, although the commit shown is arity 8. That setting change for comparison wasn't committed.]

hash-64/Sha2 256/Generated scalars
                        time:   [1.4922 us 1.4979 us 1.5042 us]
                        change: [-0.6302% -0.2113% +0.2502%] (p = 0.36 > 0.05)
                        No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
hash-64/Sha2 512/Generated scalars
                        time:   [1.3212 us 1.3259 us 1.3303 us]
                        change: [-0.4008% +0.1080% +0.6053%] (p = 0.68 > 0.05)
                        No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
hash-64/Poseidon hash/Generated scalars
                        time:   [32.169 us 32.284 us 32.406 us]
                        change: [+2.5868% +3.2309% +3.8328%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  7 (7.00%) high mild

@porcuquine
Copy link
Collaborator Author

8bc8794

(base) ➜  neptune git:(master) git rev-parse HEAD
8bc879426d11212797ada4edb9c3010fd92618e2

(base) ➜  neptune git:(master) cargo bench
    Finished bench [optimized] target(s) in 0.13s
     Running target/release/deps/neptune-678d721d93608f16

hash-64/Sha2 256/Generated scalars
                        time:   [1.6942 us 1.6991 us 1.7046 us]
                        change: [-1.7510% -1.1445% -0.5614%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
  7 (7.00%) high mild
hash-64/Sha2 512/Generated scalars
                        time:   [1.5103 us 1.5197 us 1.5322 us]
                        change: [-0.8553% -0.3516% +0.2200%] (p = 0.21 > 0.05)
                        No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe
hash-64/Poseidon hash/Generated scalars
                        time:   [23.866 us 23.994 us 24.123 us]
                        change: [-2.4276% -1.8550% -1.3182%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
hash-64/Poseidon hash optimized/Generated scalars
                        time:   [17.558 us 17.621 us 17.704 us]
                        change: [-4.1792% -3.7571% -3.3465%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

hash-128/Sha2 256/Generated scalars
                        time:   [3.1106 us 3.1175 us 3.1244 us]
                        change: [-1.8971% -1.5546% -1.1891%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  5 (5.00%) high mild
  2 (2.00%) high severe
hash-128/Sha2 512/Generated scalars
                        time:   [2.9772 us 2.9877 us 2.9996 us]
                        change: [-1.7984% -1.2300% -0.6251%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe
hash-128/Poseidon hash/Generated scalars
                        time:   [58.596 us 58.887 us 59.241 us]
                        change: [+1.2014% +1.6733% +2.1535%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
hash-128/Poseidon hash optimized/Generated scalars
                        time:   [31.912 us 32.007 us 32.105 us]
                        change: [-0.8927% -0.4021% +0.0908%] (p = 0.13 > 0.05)
                        No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild

hash-256/Sha2 256/Generated scalars
                        time:   [6.0731 us 6.0949 us 6.1204 us]
                        change: [-1.2142% -0.7038% -0.1072%] (p = 0.01 < 0.05)
                        Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe
hash-256/Sha2 512/Generated scalars
                        time:   [5.6254 us 5.6399 us 5.6546 us]
                        change: [-1.9873% -1.5588% -1.1264%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild
hash-256/Poseidon hash/Generated scalars
                        time:   [169.73 us 170.14 us 170.58 us]
                        change: [-0.3360% -0.0222% +0.3106%] (p = 0.90 > 0.05)
                        No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
hash-256/Poseidon hash optimized/Generated scalars
                        time:   [63.139 us 63.286 us 63.442 us]
                        change: [-1.1950% -0.7944% -0.3897%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  1 (1.00%) high severe

hash-352/Sha2 256/Generated scalars
                        time:   [7.9285 us 7.9429 us 7.9579 us]
                        change: [-2.4148% -1.9924% -1.5447%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  6 (6.00%) high mild
  4 (4.00%) high severe
hash-352/Sha2 512/Generated scalars
                        time:   [7.3065 us 7.3227 us 7.3410 us]
                        change: [-2.9885% -2.5386% -2.1119%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low mild
  7 (7.00%) high mild
  1 (1.00%) high severe
hash-352/Poseidon hash/Generated scalars
                        time:   [287.88 us 288.84 us 289.99 us]
                        change: [-2.3777% -1.8167% -1.3036%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  4 (4.00%) high severe
hash-352/Poseidon hash optimized/Generated scalars
                        time:   [91.337 us 91.649 us 91.969 us]
                        change: [-1.9678% -1.5258% -1.0567%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

GPU:

➜  gbench git:(master) RUST_LOG=info cargo run --release
    Finished release [optimized] target(s) in 0.05s
     Running `target/release/gbench`
[2020-05-02T01:27:06Z INFO  gbench] KiB: 4194304
[2020-05-02T01:27:06Z INFO  gbench] leaves: 134217728
[2020-05-02T01:27:06Z INFO  gbench] max column batch size: 400000
[2020-05-02T01:27:06Z INFO  gbench] max tree batch size: 700000
[2020-05-02T01:27:06Z INFO  gbench] Creating ColumnTreeBuilder
[2020-05-02T01:27:10Z INFO  gbench] ColumnTreeBuilder created
[2020-05-02T01:27:10Z INFO  gbench] Using effective batch size 400000 to build columns
[2020-05-02T01:27:10Z INFO  gbench] adding column batches
[2020-05-02T01:27:10Z INFO  gbench] start commitment
...............................................................................................................................................................................................................................................................................................................................................
[2020-05-02T01:29:23Z INFO  gbench] adding final column batch and building tree
[2020-05-02T01:29:39Z INFO  gbench] end commitment
[2020-05-02T01:29:39Z INFO  gbench] commitment time: 149.518970373s
[2020-05-02T01:29:39Z INFO  gbench] end

@porcuquine
Copy link
Collaborator Author

Latest number with pure OpenCL implementation are reported here: #78 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@porcuquine