Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AST] [Evaluation] Use 'SmallArray' in 'Case' #5865

Closed

Conversation

effectfully
Copy link
Contributor

Another competitor. Previous in the series: #5818, #5813, #5816.

@effectfully effectfully added Evaluation AST Performance EXPERIMENT Experiments that we probably don't want to merge labels Mar 28, 2024
@effectfully effectfully force-pushed the effectfully/ast/evaluation/use-SmallArray-in-Case branch from 0721158 to 512301c Compare March 28, 2024 14:35
@effectfully
Copy link
Contributor Author

/benchmark nofib

Copy link
Contributor

Click here to check the status of your benchmark.

Copy link
Contributor

Comparing benchmark results of 'nofib' on 'fe9876c4b' (base) and '512301cd0' (PR)

Results table
Script fe9876c 512301c Change
clausify/formula1 3.796 ms 2.886 ms -24.0%
clausify/formula2 5.000 ms 3.844 ms -23.1%
clausify/formula3 13.56 ms 10.43 ms -23.1%
clausify/formula4 28.11 ms 23.33 ms -17.0%
clausify/formula5 65.47 ms 50.51 ms -22.9%
knights/4x4 17.97 ms 15.96 ms -11.2%
knights/6x6 47.01 ms 42.09 ms -10.5%
knights/8x8 82.80 ms 73.66 ms -11.0%
primetest/05digits 8.965 ms 8.963 ms -0.0%
primetest/08digits 14.36 ms 14.43 ms +0.5%
primetest/10digits 17.24 ms 17.28 ms +0.2%
primetest/20digits 34.42 ms 34.44 ms +0.1%
primetest/30digits 52.38 ms 53.16 ms +1.5%
primetest/40digits 71.17 ms 71.78 ms +0.9%
primetest/50digits 86.68 ms 87.66 ms +1.1%
queens4x4/bt 5.547 ms 5.009 ms -9.7%
queens4x4/bm 7.302 ms 6.542 ms -10.4%
queens4x4/bjbt1 6.879 ms 6.201 ms -9.9%
queens4x4/bjbt2 6.355 ms 5.749 ms -9.5%
queens4x4/fc 15.04 ms 13.36 ms -11.2%
queens5x5/bt 74.50 ms 67.37 ms -9.6%
queens5x5/bm 81.69 ms 73.70 ms -9.8%
queens5x5/bjbt1 87.65 ms 79.59 ms -9.2%
queens5x5/bjbt2 84.95 ms 76.90 ms -9.5%
queens5x5/fc 189.6 ms 169.0 ms -10.9%
TOTAL 1.108 s 1.014 s -8.5%

@michaelpj
Copy link
Contributor

Looks like Vector is faster? Could be measurement error, but the difference looks big and consistent enough to be real?

@effectfully
Copy link
Contributor Author

/benchmark nofib

Copy link
Contributor

github-actions bot commented Apr 2, 2024

Click here to check the status of your benchmark.

@effectfully
Copy link
Contributor Author

Looks like Vector is faster? Could be measurement error, but the difference looks big and consistent enough to be real?

Yes, let me rerun and check the diff again, but otherwise Vector is clearly better, which I find really surprising. Maybe it's fusion that's kicking in somehow.

@michaelpj
Copy link
Contributor

I'll rebase my branch and rerun the benchmarks in case something has changed

Copy link
Contributor

github-actions bot commented Apr 2, 2024

Comparing benchmark results of 'nofib' on 'fe9876c4b' (base) and '512301cd0' (PR)

Results table
Script fe9876c 512301c Change
clausify/formula1 3.778 ms 2.870 ms -24.0%
clausify/formula2 4.977 ms 3.845 ms -22.7%
clausify/formula3 13.49 ms 10.46 ms -22.5%
clausify/formula4 27.99 ms 23.38 ms -16.5%
clausify/formula5 65.24 ms 50.62 ms -22.4%
knights/4x4 17.84 ms 15.90 ms -10.9%
knights/6x6 46.80 ms 42.19 ms -9.9%
knights/8x8 82.05 ms 73.60 ms -10.3%
primetest/05digits 8.834 ms 8.973 ms +1.6%
primetest/08digits 14.19 ms 14.39 ms +1.4%
primetest/10digits 17.05 ms 17.23 ms +1.1%
primetest/20digits 34.19 ms 34.49 ms +0.9%
primetest/30digits 52.10 ms 52.96 ms +1.7%
primetest/40digits 70.34 ms 71.93 ms +2.3%
primetest/50digits 85.98 ms 87.84 ms +2.2%
queens4x4/bt 5.525 ms 4.998 ms -9.5%
queens4x4/bm 7.263 ms 6.527 ms -10.1%
queens4x4/bjbt1 6.815 ms 6.169 ms -9.5%
queens4x4/bjbt2 6.317 ms 5.729 ms -9.3%
queens4x4/fc 14.84 ms 13.28 ms -10.5%
queens5x5/bt 73.88 ms 67.07 ms -9.2%
queens5x5/bm 80.99 ms 73.33 ms -9.5%
queens5x5/bjbt1 87.07 ms 79.23 ms -9.0%
queens5x5/bjbt2 84.56 ms 76.43 ms -9.6%
queens5x5/fc 188.8 ms 167.8 ms -11.1%
TOTAL 1.101 s 1.011 s -8.1%

@effectfully
Copy link
Contributor Author

CPU performance isn't that big of a deal anymore and what we can get here should be fairly insignificant, so I added it to the epic and let's close it for now.

@effectfully effectfully deleted the effectfully/ast/evaluation/use-SmallArray-in-Case branch May 30, 2024 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AST Evaluation EXPERIMENT Experiments that we probably don't want to merge Performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants