Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Costing] Provide support for multiple 'CostModel's #5851

Conversation

effectfully
Copy link
Contributor

@effectfully effectfully commented Mar 22, 2024

This makes cost models configurable, so that the cost model can depend on the protocol version. All commentary is in the diff.

@effectfully effectfully changed the title [Builtins] Store 'BuiltinRuntime' lazily explicitly [Costing] Provide support for multiple 'CostModel's Mar 25, 2024
@effectfully effectfully added the Costing Anything relating to costs, fees, gas, etc. label Mar 25, 2024
@effectfully effectfully force-pushed the effectfully/builtins/both-MajorProtocolVersion-and-multiple-CostModels branch from c8abb84 to a9a71bd Compare March 25, 2024 08:56
@effectfully
Copy link
Contributor Author

/benchmark validation

Copy link
Contributor

Click here to check the status of your benchmark.

@effectfully effectfully force-pushed the effectfully/builtins/both-MajorProtocolVersion-and-multiple-CostModels branch 3 times, most recently from 18f2db1 to 9758d10 Compare March 26, 2024 15:06
@effectfully effectfully force-pushed the effectfully/builtins/both-MajorProtocolVersion-and-multiple-CostModels branch from 9758d10 to b97e9f1 Compare March 26, 2024 22:03
@effectfully
Copy link
Contributor Author

/benchmark validation

Copy link
Contributor

Click here to check the status of your benchmark.

Copy link
Contributor

Comparing benchmark results of 'validation' on '381172295' (base) and 'b97e9f11e' (PR)

Results table
Script 3811722 b97e9f1 Change
auction_1-1 169.6 μs 169.2 μs -0.2%
auction_1-2 619.7 μs 609.5 μs -1.6%
auction_1-3 615.6 μs 607.0 μs -1.4%
auction_1-4 221.7 μs 215.3 μs -2.9%
auction_2-1 170.4 μs 169.5 μs -0.5%
auction_2-2 620.8 μs 611.2 μs -1.5%
auction_2-3 815.9 μs 797.2 μs -2.3%
auction_2-4 614.1 μs 605.9 μs -1.3%
auction_2-5 221.5 μs 214.6 μs -3.1%
crowdfunding-success-1 199.4 μs 196.3 μs -1.6%
crowdfunding-success-2 199.3 μs 196.5 μs -1.4%
crowdfunding-success-3 199.5 μs 196.2 μs -1.7%
currency-1 242.8 μs 237.0 μs -2.4%
escrow-redeem_1-1 337.6 μs 332.1 μs -1.6%
escrow-redeem_1-2 338.7 μs 332.0 μs -2.0%
escrow-redeem_2-1 389.3 μs 383.8 μs -1.4%
escrow-redeem_2-2 390.5 μs 385.3 μs -1.3%
escrow-redeem_2-3 391.2 μs 385.3 μs -1.5%
escrow-refund-1 149.2 μs 146.6 μs -1.7%
future-increase-margin-1 244.6 μs 237.0 μs -3.1%
future-increase-margin-2 525.1 μs 513.7 μs -2.2%
future-increase-margin-3 525.7 μs 515.5 μs -1.9%
future-increase-margin-4 477.0 μs 468.1 μs -1.9%
future-increase-margin-5 791.6 μs 792.6 μs +0.1%
future-pay-out-1 243.4 μs 236.5 μs -2.8%
future-pay-out-2 522.9 μs 513.8 μs -1.7%
future-pay-out-3 525.5 μs 514.0 μs -2.2%
future-pay-out-4 796.0 μs 789.5 μs -0.8%
future-settle-early-1 244.3 μs 236.9 μs -3.0%
future-settle-early-2 524.7 μs 513.8 μs -2.1%
future-settle-early-3 523.3 μs 514.4 μs -1.7%
future-settle-early-4 592.7 μs 590.9 μs -0.3%
game-sm-success_1-1 371.5 μs 370.3 μs -0.3%
game-sm-success_1-2 192.0 μs 187.4 μs -2.4%
game-sm-success_1-3 623.7 μs 616.5 μs -1.2%
game-sm-success_1-4 217.8 μs 216.1 μs -0.8%
game-sm-success_2-1 372.1 μs 371.7 μs -0.1%
game-sm-success_2-2 191.2 μs 186.8 μs -2.3%
game-sm-success_2-3 622.7 μs 615.4 μs -1.2%
game-sm-success_2-4 217.2 μs 215.9 μs -0.6%
game-sm-success_2-5 623.9 μs 617.3 μs -1.1%
game-sm-success_2-6 217.7 μs 216.7 μs -0.5%
multisig-sm-1 381.6 μs 378.0 μs -0.9%
multisig-sm-2 372.0 μs 366.0 μs -1.6%
multisig-sm-3 376.1 μs 370.4 μs -1.5%
multisig-sm-4 380.2 μs 378.4 μs -0.5%
multisig-sm-5 543.5 μs 530.2 μs -2.4%
multisig-sm-6 382.4 μs 377.7 μs -1.2%
multisig-sm-7 371.4 μs 366.6 μs -1.3%
multisig-sm-8 376.6 μs 375.7 μs -0.2%
multisig-sm-9 380.8 μs 377.1 μs -1.0%
multisig-sm-10 540.9 μs 532.1 μs -1.6%
ping-pong-1 317.0 μs 312.3 μs -1.5%
ping-pong-2 316.6 μs 311.3 μs -1.7%
ping-pong_2-1 193.1 μs 186.4 μs -3.5%
prism-1 161.0 μs 156.2 μs -3.0%
prism-2 397.3 μs 390.8 μs -1.6%
prism-3 356.3 μs 348.1 μs -2.3%
pubkey-1 136.7 μs 134.3 μs -1.8%
stablecoin_1-1 883.9 μs 885.7 μs +0.2%
stablecoin_1-2 187.3 μs 182.0 μs -2.8%
stablecoin_1-3 1.014 ms 1.015 ms +0.1%
stablecoin_1-4 198.0 μs 194.6 μs -1.7%
stablecoin_1-5 1.306 ms 1.305 ms -0.1%
stablecoin_1-6 245.0 μs 239.7 μs -2.2%
stablecoin_2-1 881.7 μs 885.0 μs +0.4%
stablecoin_2-2 186.8 μs 182.0 μs -2.6%
stablecoin_2-3 1.015 ms 1.018 ms +0.3%
stablecoin_2-4 198.0 μs 194.6 μs -1.7%
token-account-1 182.7 μs 181.8 μs -0.5%
token-account-2 321.0 μs 318.8 μs -0.7%
uniswap-1 402.8 μs 395.3 μs -1.9%
uniswap-2 211.8 μs 209.1 μs -1.3%
uniswap-3 1.689 ms 1.678 ms -0.7%
uniswap-4 322.5 μs 313.6 μs -2.8%
uniswap-5 1.115 ms 1.100 ms -1.3%
uniswap-6 306.5 μs 303.0 μs -1.1%
vesting-1 336.6 μs 332.1 μs -1.3%

@effectfully
Copy link
Contributor Author

/benchmark nofib

Copy link
Contributor

Click here to check the status of your benchmark.

…to effectfully/builtins/both-MajorProtocolVersion-and-multiple-CostModels
@effectfully effectfully force-pushed the effectfully/builtins/both-MajorProtocolVersion-and-multiple-CostModels branch from b97e9f1 to 0a1af53 Compare March 27, 2024 09:13
Copy link
Contributor

Click here to check the status of your benchmark.

@effectfully
Copy link
Contributor Author

/benchmark lists

1 similar comment
@effectfully
Copy link
Contributor Author

/benchmark lists

Copy link
Contributor

Comparing benchmark results of 'validation' on '16a986fd8' (base) and '9d8942da1' (PR)

Results table
Script 16a986f 9d8942d Change
auction_1-1 168.6 μs 171.2 μs +1.5%
auction_1-2 613.6 μs 622.8 μs +1.5%
auction_1-3 611.6 μs 622.0 μs +1.7%
auction_1-4 218.3 μs 222.4 μs +1.9%
auction_2-1 169.5 μs 172.0 μs +1.5%
auction_2-2 613.9 μs 626.3 μs +2.0%
auction_2-3 803.3 μs 816.1 μs +1.6%
auction_2-4 608.9 μs 621.6 μs +2.1%
auction_2-5 218.6 μs 222.3 μs +1.7%
crowdfunding-success-1 198.8 μs 201.2 μs +1.2%
crowdfunding-success-2 198.3 μs 201.3 μs +1.5%
crowdfunding-success-3 198.6 μs 201.0 μs +1.2%
currency-1 240.6 μs 241.5 μs +0.4%
escrow-redeem_1-1 338.9 μs 337.2 μs -0.5%
escrow-redeem_1-2 340.2 μs 338.8 μs -0.4%
escrow-redeem_2-1 392.1 μs 391.5 μs -0.2%
escrow-redeem_2-2 393.5 μs 391.9 μs -0.4%
escrow-redeem_2-3 393.7 μs 391.8 μs -0.5%
escrow-refund-1 150.5 μs 147.7 μs -1.9%
future-increase-margin-1 240.1 μs 242.7 μs +1.1%
future-increase-margin-2 517.0 μs 523.2 μs +1.2%
future-increase-margin-3 517.3 μs 524.6 μs +1.4%
future-increase-margin-4 471.9 μs 476.6 μs +1.0%
future-increase-margin-5 782.3 μs 792.0 μs +1.2%
future-pay-out-1 239.5 μs 242.7 μs +1.3%
future-pay-out-2 515.8 μs 523.2 μs +1.4%
future-pay-out-3 518.0 μs 525.3 μs +1.4%
future-pay-out-4 781.9 μs 797.4 μs +2.0%
future-settle-early-1 239.3 μs 243.1 μs +1.6%
future-settle-early-2 515.5 μs 523.9 μs +1.6%
future-settle-early-3 516.5 μs 525.6 μs +1.8%
future-settle-early-4 585.5 μs 602.0 μs +2.8%
game-sm-success_1-1 366.9 μs 373.9 μs +1.9%
game-sm-success_1-2 187.1 μs 191.4 μs +2.3%
game-sm-success_1-3 614.8 μs 625.5 μs +1.7%
game-sm-success_1-4 212.2 μs 216.1 μs +1.8%
game-sm-success_2-1 367.2 μs 373.4 μs +1.7%
game-sm-success_2-2 187.0 μs 191.1 μs +2.2%
game-sm-success_2-3 615.6 μs 626.1 μs +1.7%
game-sm-success_2-4 212.0 μs 216.2 μs +2.0%
game-sm-success_2-5 615.7 μs 625.0 μs +1.5%
game-sm-success_2-6 212.5 μs 216.8 μs +2.0%
multisig-sm-1 376.3 μs 380.9 μs +1.2%
multisig-sm-2 366.1 μs 369.9 μs +1.0%
multisig-sm-3 371.5 μs 375.6 μs +1.1%
multisig-sm-4 375.3 μs 381.0 μs +1.5%
multisig-sm-5 533.4 μs 543.1 μs +1.8%
multisig-sm-6 377.0 μs 381.1 μs +1.1%
multisig-sm-7 364.8 μs 371.4 μs +1.8%
multisig-sm-8 370.7 μs 378.0 μs +2.0%
multisig-sm-9 375.9 μs 382.1 μs +1.6%
multisig-sm-10 533.3 μs 542.4 μs +1.7%
ping-pong-1 313.0 μs 316.7 μs +1.2%
ping-pong-2 312.8 μs 317.6 μs +1.5%
ping-pong_2-1 190.2 μs 193.2 μs +1.6%
prism-1 158.2 μs 161.1 μs +1.8%
prism-2 393.1 μs 398.3 μs +1.3%
prism-3 350.5 μs 359.2 μs +2.5%
pubkey-1 133.4 μs 135.8 μs +1.8%
stablecoin_1-1 872.4 μs 899.0 μs +3.0%
stablecoin_1-2 183.0 μs 186.9 μs +2.1%
stablecoin_1-3 1.005 ms 1.032 ms +2.7%
stablecoin_1-4 193.2 μs 198.3 μs +2.6%
stablecoin_1-5 1.289 ms 1.321 ms +2.5%
stablecoin_1-6 240.4 μs 244.9 μs +1.9%
stablecoin_2-1 875.2 μs 897.9 μs +2.6%
stablecoin_2-2 183.1 μs 186.8 μs +2.0%
stablecoin_2-3 1.004 ms 1.030 ms +2.6%
stablecoin_2-4 192.5 μs 197.8 μs +2.8%
token-account-1 179.9 μs 183.0 μs +1.7%
token-account-2 318.9 μs 322.5 μs +1.1%
uniswap-1 396.2 μs 400.8 μs +1.2%
uniswap-2 209.0 μs 213.4 μs +2.1%
uniswap-3 1.681 ms 1.713 ms +1.9%
uniswap-4 314.8 μs 322.7 μs +2.5%
uniswap-5 1.104 ms 1.127 ms +2.1%
uniswap-6 300.3 μs 309.6 μs +3.1%
vesting-1 331.5 μs 336.8 μs +1.6%
TOTAL 33.67 ms 34.25 ms +1.7%

Copy link
Contributor

Click here to check the status of your benchmark.

Copy link
Contributor

Comparing benchmark results of 'validation' on '16a986fd8' (base) and '9d8942da1' (PR)

Results table
Script 16a986f 9d8942d Change
auction_1-1 169.4 μs 171.4 μs +1.2%
auction_1-2 615.3 μs 627.8 μs +2.0%
auction_1-3 613.3 μs 620.6 μs +1.2%
auction_1-4 219.7 μs 222.7 μs +1.4%
auction_2-1 169.8 μs 172.1 μs +1.4%
auction_2-2 616.5 μs 629.1 μs +2.0%
auction_2-3 807.5 μs 817.1 μs +1.2%
auction_2-4 613.2 μs 620.2 μs +1.1%
auction_2-5 219.7 μs 222.8 μs +1.4%
crowdfunding-success-1 199.1 μs 202.0 μs +1.5%
crowdfunding-success-2 199.0 μs 201.8 μs +1.4%
crowdfunding-success-3 198.8 μs 201.9 μs +1.6%
currency-1 240.3 μs 242.4 μs +0.9%
escrow-redeem_1-1 333.9 μs 339.6 μs +1.7%
escrow-redeem_1-2 334.6 μs 339.6 μs +1.5%
escrow-redeem_2-1 386.7 μs 393.2 μs +1.7%
escrow-redeem_2-2 386.8 μs 393.3 μs +1.7%
escrow-redeem_2-3 387.1 μs 392.8 μs +1.5%
escrow-refund-1 146.4 μs 148.6 μs +1.5%
future-increase-margin-1 240.3 μs 243.0 μs +1.1%
future-increase-margin-2 516.8 μs 524.4 μs +1.5%
future-increase-margin-3 517.1 μs 526.2 μs +1.8%
future-increase-margin-4 472.4 μs 477.0 μs +1.0%
future-increase-margin-5 788.1 μs 799.3 μs +1.4%
future-pay-out-1 240.9 μs 242.8 μs +0.8%
future-pay-out-2 517.6 μs 524.1 μs +1.3%
future-pay-out-3 519.5 μs 524.9 μs +1.0%
future-pay-out-4 789.5 μs 799.5 μs +1.3%
future-settle-early-1 240.1 μs 243.5 μs +1.4%
future-settle-early-2 517.9 μs 526.7 μs +1.7%
future-settle-early-3 518.6 μs 526.4 μs +1.5%
future-settle-early-4 588.6 μs 601.4 μs +2.2%
game-sm-success_1-1 368.6 μs 373.4 μs +1.3%
game-sm-success_1-2 187.4 μs 191.2 μs +2.0%
game-sm-success_1-3 616.7 μs 623.5 μs +1.1%
game-sm-success_1-4 213.1 μs 216.6 μs +1.6%
game-sm-success_2-1 369.3 μs 373.2 μs +1.1%
game-sm-success_2-2 186.9 μs 191.1 μs +2.2%
game-sm-success_2-3 615.3 μs 622.2 μs +1.1%
game-sm-success_2-4 212.5 μs 217.2 μs +2.2%
game-sm-success_2-5 616.0 μs 624.5 μs +1.4%
game-sm-success_2-6 212.5 μs 217.0 μs +2.1%
multisig-sm-1 376.9 μs 382.3 μs +1.4%
multisig-sm-2 366.4 μs 372.3 μs +1.6%
multisig-sm-3 371.7 μs 375.4 μs +1.0%
multisig-sm-4 377.7 μs 380.3 μs +0.7%
multisig-sm-5 533.1 μs 539.3 μs +1.2%
multisig-sm-6 377.8 μs 381.2 μs +0.9%
multisig-sm-7 366.3 μs 372.0 μs +1.6%
multisig-sm-8 371.7 μs 380.8 μs +2.4%
multisig-sm-9 375.0 μs 382.0 μs +1.9%
multisig-sm-10 534.0 μs 540.4 μs +1.2%
ping-pong-1 312.7 μs 317.0 μs +1.4%
ping-pong-2 312.7 μs 318.3 μs +1.8%
ping-pong_2-1 190.1 μs 193.3 μs +1.7%
prism-1 158.6 μs 161.7 μs +2.0%
prism-2 392.8 μs 397.8 μs +1.3%
prism-3 351.1 μs 359.6 μs +2.4%
pubkey-1 133.6 μs 136.0 μs +1.8%
stablecoin_1-1 874.7 μs 898.0 μs +2.7%
stablecoin_1-2 183.7 μs 187.0 μs +1.8%
stablecoin_1-3 1.006 ms 1.029 ms +2.3%
stablecoin_1-4 193.6 μs 197.6 μs +2.1%
stablecoin_1-5 1.289 ms 1.318 ms +2.2%
stablecoin_1-6 240.5 μs 245.2 μs +2.0%
stablecoin_2-1 876.1 μs 896.1 μs +2.3%
stablecoin_2-2 183.5 μs 187.0 μs +1.9%
stablecoin_2-3 1.005 ms 1.030 ms +2.5%
stablecoin_2-4 193.1 μs 198.0 μs +2.5%
token-account-1 180.3 μs 183.0 μs +1.5%
token-account-2 318.7 μs 323.1 μs +1.4%
uniswap-1 396.8 μs 401.3 μs +1.1%
uniswap-2 209.1 μs 213.5 μs +2.1%
uniswap-3 1.684 ms 1.712 ms +1.7%
uniswap-4 315.9 μs 323.4 μs +2.4%
uniswap-5 1.106 ms 1.126 ms +1.8%
uniswap-6 300.9 μs 310.8 μs +3.3%
vesting-1 333.0 μs 337.4 μs +1.3%
TOTAL 33.71 ms 34.27 ms +1.7%

Copy link
Contributor

Click here to check the status of your benchmark.

Copy link
Contributor

Comparing benchmark results of 'nofib' on '16a986fd8' (base) and '9d8942da1' (PR)

Results table
Script 16a986f 9d8942d Change
clausify/formula1 2.944 ms 3.025 ms +2.8%
clausify/formula2 3.949 ms 4.046 ms +2.5%
clausify/formula3 10.75 ms 11.07 ms +3.0%
clausify/formula4 24.23 ms 24.98 ms +3.1%
clausify/formula5 51.93 ms 53.82 ms +3.6%
knights/4x4 16.53 ms 17.28 ms +4.5%
knights/6x6 43.22 ms 44.49 ms +2.9%
knights/8x8 75.85 ms 78.23 ms +3.1%
primetest/05digits 9.871 ms 10.09 ms +2.2%
primetest/08digits 16.16 ms 16.77 ms +3.8%
primetest/10digits 19.20 ms 20.15 ms +4.9%
primetest/20digits 38.16 ms 38.49 ms +0.9%
primetest/30digits 58.52 ms 59.79 ms +2.2%
primetest/40digits 78.83 ms 80.71 ms +2.4%
primetest/50digits 96.67 ms 97.48 ms +0.8%
queens4x4/bt 5.153 ms 5.295 ms +2.8%
queens4x4/bm 6.659 ms 6.860 ms +3.0%
queens4x4/bjbt1 6.318 ms 6.493 ms +2.8%
queens4x4/bjbt2 5.877 ms 6.065 ms +3.2%
queens4x4/fc 13.53 ms 13.97 ms +3.3%
queens5x5/bt 68.97 ms 71.02 ms +3.0%
queens5x5/bm 74.34 ms 76.61 ms +3.1%
queens5x5/bjbt1 81.16 ms 83.43 ms +2.8%
queens5x5/bjbt2 78.64 ms 80.88 ms +2.8%
queens5x5/fc 170.9 ms 175.6 ms +2.8%
TOTAL 1.058 s 1.087 s +2.7%

Copy link
Contributor

Click here to check the status of your benchmark.

Copy link
Member

@zliu41 zliu41 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks totally legit

@@ -1287,7 +1291,8 @@ instance uni ~ DefaultUni => ToBuiltinMeaning uni DefaultFun where
:: BS.ByteString -> BS.ByteString -> BS.ByteString -> BuiltinResult Bool
verifyEd25519SignatureDenotation =
case semvar of
DefaultFunSemanticsVariant1 -> verifyEd25519Signature_V1
DefaultFunSemanticsVariant0 -> verifyEd25519Signature_V1
DefaultFunSemanticsVariant1 -> verifyEd25519Signature_V2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mapping is potentially easy to get wrong. Is there any test for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're just replacing one implementation with another for maintenance reasons, so doing this matching here is only for the purpose of being paranoid about not breaking anything in the past in case of some unknown unknown. There's nothing additional to test therefore beyond what we already test, which is how we use ed25519_Variant0Prop to test all variants of this builtin.

Note that consByteString is untestable in the same way, but for a different reason: Variant0 and Variant1 versions of that builtin are completely identical (unlike with verifyEd25519Signature, whose variants behave identically but come from different libraries, i.e. are nominally different), so again there's nothing to test.

So for both the builtins we only make sure that whatever their variant is, it behaves as expected, we cannot test that Variant0 behaves differently from Variant1, because it doesn't.

Copy link
Contributor

Comparing benchmark results of 'nofib' on '16a986fd8' (base) and '9d8942da1' (PR)

Results table
Script 16a986f 9d8942d Change
clausify/formula1 2.941 ms 3.036 ms +3.2%
clausify/formula2 3.945 ms 4.063 ms +3.0%
clausify/formula3 10.76 ms 11.06 ms +2.8%
clausify/formula4 24.23 ms 24.94 ms +2.9%
clausify/formula5 51.83 ms 53.57 ms +3.4%
knights/4x4 16.50 ms 17.17 ms +4.1%
knights/6x6 43.08 ms 44.65 ms +3.6%
knights/8x8 75.49 ms 77.69 ms +2.9%
primetest/05digits 9.810 ms 10.07 ms +2.7%
primetest/08digits 15.88 ms 16.29 ms +2.6%
primetest/10digits 19.57 ms 19.65 ms +0.4%
primetest/20digits 38.09 ms 38.42 ms +0.9%
primetest/30digits 58.60 ms 59.13 ms +0.9%
primetest/40digits 80.12 ms 83.12 ms +3.7%
primetest/50digits 96.94 ms 97.84 ms +0.9%
queens4x4/bt 5.151 ms 5.286 ms +2.6%
queens4x4/bm 6.648 ms 6.827 ms +2.7%
queens4x4/bjbt1 6.309 ms 6.471 ms +2.6%
queens4x4/bjbt2 5.880 ms 6.049 ms +2.9%
queens4x4/fc 13.51 ms 13.87 ms +2.7%
queens5x5/bt 69.03 ms 71.17 ms +3.1%
queens5x5/bm 74.49 ms 76.45 ms +2.6%
queens5x5/bjbt1 81.31 ms 83.78 ms +3.0%
queens5x5/bjbt2 78.59 ms 80.70 ms +2.7%
queens5x5/fc 170.8 ms 176.2 ms +3.2%
TOTAL 1.060 s 1.088 s +2.6%

Copy link
Contributor

Click here to check the status of your benchmark.

Copy link
Contributor

Comparing benchmark results of 'lists' on '16a986fd8' (base) and '9d8942da1' (PR)

Results table
Script 16a986f 9d8942d Change
sort/ghcSort/10 34.36 μs 33.99 μs -1.1%
sort/ghcSort/20 80.47 μs 78.81 μs -2.1%
sort/ghcSort/30 124.1 μs 121.5 μs -2.1%
sort/ghcSort/40 184.3 μs 179.8 μs -2.4%
sort/ghcSort/50 227.2 μs 222.2 μs -2.2%
sort/ghcSort/60 284.2 μs 279.3 μs -1.7%
sort/ghcSort/70 372.2 μs 362.1 μs -2.7%
sort/ghcSort/80 422.6 μs 413.6 μs -2.1%
sort/ghcSort/90 470.5 μs 457.0 μs -2.9%
sort/ghcSort/100 528.7 μs 516.5 μs -2.3%
sort/ghcSort/110 576.9 μs 567.2 μs -1.7%
sort/ghcSort/120 649.5 μs 636.0 μs -2.1%
sort/ghcSort/130 803.8 μs 788.2 μs -1.9%
sort/ghcSort/140 849.6 μs 831.6 μs -2.1%
sort/ghcSort/150 910.2 μs 890.9 μs -2.1%
sort/ghcSort/160 975.6 μs 953.3 μs -2.3%
sort/ghcSort/170 1.037 ms 1.001 ms -3.5%
sort/ghcSort/180 1.079 ms 1.056 ms -2.1%
sort/ghcSort/190 1.138 ms 1.117 ms -1.8%
sort/ghcSort/200 1.227 ms 1.199 ms -2.3%
sort/ghcSort/210 1.277 ms 1.250 ms -2.1%
sort/ghcSort/220 1.337 ms 1.308 ms -2.2%
sort/ghcSort/230 1.426 ms 1.397 ms -2.0%
sort/ghcSort/240 1.512 ms 1.478 ms -2.2%
sort/ghcSort/250 1.587 ms 1.552 ms -2.2%
sort/ghcSort/260 1.826 ms 1.788 ms -2.1%
sort/ghcSort/270 1.892 ms 1.849 ms -2.3%
sort/ghcSort/280 1.937 ms 1.898 ms -2.0%
sort/ghcSort/290 2.009 ms 1.964 ms -2.2%
sort/ghcSort/300 2.090 ms 2.037 ms -2.5%
sort/ghcSort/310 2.154 ms 2.109 ms -2.1%
sort/ghcSort/320 2.228 ms 2.179 ms -2.2%
sort/ghcSort/330 2.287 ms 2.239 ms -2.1%
sort/ghcSort/340 2.346 ms 2.300 ms -2.0%
sort/ghcSort/350 2.412 ms 2.358 ms -2.2%
sort/ghcSort/360 2.475 ms 2.423 ms -2.1%
sort/ghcSort/370 2.552 ms 2.495 ms -2.2%
sort/ghcSort/380 2.626 ms 2.568 ms -2.2%
sort/ghcSort/390 2.744 ms 2.689 ms -2.0%
sort/ghcSort/400 2.819 ms 2.761 ms -2.1%
sort/ghcSort/410 2.859 ms 2.797 ms -2.2%
sort/ghcSort/420 2.953 ms 2.890 ms -2.1%
sort/ghcSort/430 2.993 ms 2.938 ms -1.8%
sort/ghcSort/440 3.086 ms 3.029 ms -1.8%
sort/ghcSort/450 3.249 ms 3.188 ms -1.9%
sort/ghcSort/460 3.305 ms 3.226 ms -2.4%
sort/ghcSort/470 3.368 ms 3.297 ms -2.1%
sort/ghcSort/480 3.487 ms 3.405 ms -2.4%
sort/ghcSort/490 3.554 ms 3.482 ms -2.0%
sort/ghcSort/500 3.647 ms 3.576 ms -1.9%
sort/insertionSort/10 33.83 μs 33.02 μs -2.4%
sort/insertionSort/20 124.3 μs 120.9 μs -2.7%
sort/insertionSort/30 272.6 μs 264.5 μs -3.0%
sort/insertionSort/40 480.3 μs 466.7 μs -2.8%
sort/insertionSort/50 745.9 μs 724.1 μs -2.9%
sort/insertionSort/60 1.072 ms 1.040 ms -3.0%
sort/insertionSort/70 1.458 ms 1.414 ms -3.0%
sort/insertionSort/80 1.909 ms 1.848 ms -3.2%
sort/insertionSort/90 2.400 ms 2.331 ms -2.9%
sort/insertionSort/100 2.966 ms 2.877 ms -3.0%
sort/insertionSort/110 3.583 ms 3.475 ms -3.0%
sort/insertionSort/120 4.270 ms 4.145 ms -2.9%
sort/insertionSort/130 5.011 ms 4.849 ms -3.2%
sort/insertionSort/140 5.814 ms 5.638 ms -3.0%
sort/insertionSort/150 6.665 ms 6.452 ms -3.2%
sort/insertionSort/160 7.589 ms 7.368 ms -2.9%
sort/insertionSort/170 8.574 ms 8.307 ms -3.1%
sort/insertionSort/180 9.606 ms 9.323 ms -2.9%
sort/insertionSort/190 10.71 ms 10.40 ms -2.9%
sort/insertionSort/200 11.90 ms 11.54 ms -3.0%
sort/insertionSort/210 13.18 ms 12.71 ms -3.6%
sort/insertionSort/220 14.41 ms 13.97 ms -3.1%
sort/insertionSort/230 15.80 ms 15.34 ms -2.9%
sort/insertionSort/240 17.17 ms 16.64 ms -3.1%
sort/insertionSort/250 18.69 ms 18.06 ms -3.4%
sort/insertionSort/260 20.19 ms 19.63 ms -2.8%
sort/insertionSort/270 21.89 ms 21.15 ms -3.4%
sort/insertionSort/280 23.58 ms 22.77 ms -3.4%
sort/insertionSort/290 25.19 ms 24.43 ms -3.0%
sort/insertionSort/300 27.13 ms 26.15 ms -3.6%
sort/insertionSort/310 28.91 ms 27.94 ms -3.4%
sort/insertionSort/320 30.75 ms 29.82 ms -3.0%
sort/insertionSort/330 32.82 ms 31.76 ms -3.2%
sort/insertionSort/340 34.79 ms 33.72 ms -3.1%
sort/insertionSort/350 37.02 ms 35.80 ms -3.3%
sort/insertionSort/360 39.13 ms 37.92 ms -3.1%
sort/insertionSort/370 41.57 ms 40.07 ms -3.6%
sort/insertionSort/380 43.79 ms 42.32 ms -3.4%
sort/insertionSort/390 46.07 ms 44.68 ms -3.0%
sort/insertionSort/400 48.62 ms 47.05 ms -3.2%
sort/insertionSort/410 51.09 ms 49.57 ms -3.0%
sort/insertionSort/420 53.73 ms 52.08 ms -3.1%
sort/insertionSort/430 56.36 ms 54.58 ms -3.2%
sort/insertionSort/440 58.87 ms 57.19 ms -2.9%
sort/insertionSort/450 61.52 ms 59.61 ms -3.1%
sort/insertionSort/460 64.47 ms 62.60 ms -2.9%
sort/insertionSort/470 67.66 ms 65.41 ms -3.3%
sort/insertionSort/480 70.56 ms 68.44 ms -3.0%
sort/insertionSort/490 73.52 ms 71.31 ms -3.0%
sort/insertionSort/500 76.68 ms 74.44 ms -2.9%
sort/mergeSort/10 90.80 μs 89.41 μs -1.5%
sort/mergeSort/20 219.4 μs 214.3 μs -2.3%
sort/mergeSort/30 362.4 μs 354.8 μs -2.1%
sort/mergeSort/40 516.4 μs 503.6 μs -2.5%
sort/mergeSort/50 674.7 μs 659.4 μs -2.3%
sort/mergeSort/60 846.7 μs 821.2 μs -3.0%
sort/mergeSort/70 1.018 ms 986.5 μs -3.1%
sort/mergeSort/80 1.195 ms 1.164 ms -2.6%
sort/mergeSort/90 1.370 ms 1.335 ms -2.6%
sort/mergeSort/100 1.546 ms 1.507 ms -2.5%
sort/mergeSort/110 1.733 ms 1.681 ms -3.0%
sort/mergeSort/120 1.928 ms 1.869 ms -3.1%
sort/mergeSort/130 2.148 ms 2.091 ms -2.7%
sort/mergeSort/140 2.316 ms 2.251 ms -2.8%
sort/mergeSort/150 2.501 ms 2.426 ms -3.0%
sort/mergeSort/160 2.714 ms 2.632 ms -3.0%
sort/mergeSort/170 2.884 ms 2.808 ms -2.6%
sort/mergeSort/180 3.101 ms 3.025 ms -2.5%
sort/mergeSort/190 3.312 ms 3.224 ms -2.7%
sort/mergeSort/200 3.503 ms 3.396 ms -3.1%
sort/mergeSort/210 3.723 ms 3.608 ms -3.1%
sort/mergeSort/220 3.916 ms 3.798 ms -3.0%
sort/mergeSort/230 4.136 ms 4.021 ms -2.8%
sort/mergeSort/240 4.349 ms 4.208 ms -3.2%
sort/mergeSort/250 4.577 ms 4.460 ms -2.6%
sort/mergeSort/260 4.819 ms 4.675 ms -3.0%
sort/mergeSort/270 4.987 ms 4.831 ms -3.1%
sort/mergeSort/280 5.171 ms 5.032 ms -2.7%
sort/mergeSort/290 5.368 ms 5.221 ms -2.7%
sort/mergeSort/300 5.580 ms 5.426 ms -2.8%
sort/mergeSort/310 5.809 ms 5.628 ms -3.1%
sort/mergeSort/320 6.061 ms 5.998 ms -1.0%
sort/mergeSort/330 6.251 ms 6.052 ms -3.2%
sort/mergeSort/340 6.478 ms 6.271 ms -3.2%
sort/mergeSort/350 6.731 ms 6.505 ms -3.4%
sort/mergeSort/360 6.941 ms 6.718 ms -3.2%
sort/mergeSort/370 7.137 ms 6.956 ms -2.5%
sort/mergeSort/380 7.394 ms 7.191 ms -2.7%
sort/mergeSort/390 7.645 ms 7.421 ms -2.9%
sort/mergeSort/400 7.804 ms 7.586 ms -2.8%
sort/mergeSort/410 8.073 ms 7.821 ms -3.1%
sort/mergeSort/420 8.300 ms 8.034 ms -3.2%
sort/mergeSort/430 8.531 ms 8.266 ms -3.1%
sort/mergeSort/440 8.720 ms 8.427 ms -3.4%
sort/mergeSort/450 8.952 ms 8.839 ms -1.3%
sort/mergeSort/460 9.178 ms 8.924 ms -2.8%
sort/mergeSort/470 9.444 ms 9.153 ms -3.1%
sort/mergeSort/480 9.664 ms 9.362 ms -3.1%
sort/mergeSort/490 9.939 ms 9.644 ms -3.0%
sort/mergeSort/500 10.23 ms 9.864 ms -3.6%
sort/quickSort/10 78.11 μs 76.55 μs -2.0%
sort/quickSort/20 291.8 μs 286.6 μs -1.8%
sort/quickSort/30 643.5 μs 632.9 μs -1.6%
sort/quickSort/40 1.152 ms 1.135 ms -1.5%
sort/quickSort/50 1.826 ms 1.790 ms -2.0%
sort/quickSort/60 2.627 ms 2.591 ms -1.4%
sort/quickSort/70 3.600 ms 3.533 ms -1.9%
sort/quickSort/80 4.742 ms 4.646 ms -2.0%
sort/quickSort/90 5.970 ms 5.891 ms -1.3%
sort/quickSort/100 7.355 ms 7.209 ms -2.0%
sort/quickSort/110 8.919 ms 8.741 ms -2.0%
sort/quickSort/120 10.64 ms 10.41 ms -2.2%
sort/quickSort/130 12.44 ms 12.24 ms -1.6%
sort/quickSort/140 14.37 ms 14.09 ms -1.9%
sort/quickSort/150 16.52 ms 16.22 ms -1.8%
sort/quickSort/160 18.78 ms 18.40 ms -2.0%
sort/quickSort/170 21.19 ms 20.74 ms -2.1%
sort/quickSort/180 23.78 ms 23.26 ms -2.2%
sort/quickSort/190 26.45 ms 25.95 ms -1.9%
sort/quickSort/200 29.19 ms 28.67 ms -1.8%
sort/quickSort/210 32.36 ms 31.65 ms -2.2%
sort/quickSort/220 35.40 ms 34.77 ms -1.8%
sort/quickSort/230 38.78 ms 37.98 ms -2.1%
sort/quickSort/240 42.11 ms 41.40 ms -1.7%
sort/quickSort/250 45.66 ms 44.95 ms -1.6%
sort/quickSort/260 49.39 ms 48.59 ms -1.6%
sort/quickSort/270 53.29 ms 52.35 ms -1.8%
sort/quickSort/280 57.35 ms 56.25 ms -1.9%
sort/quickSort/290 61.63 ms 60.32 ms -2.1%
sort/quickSort/300 65.81 ms 64.50 ms -2.0%
sort/quickSort/310 70.52 ms 69.02 ms -2.1%
sort/quickSort/320 74.96 ms 73.71 ms -1.7%
sort/quickSort/330 79.99 ms 78.28 ms -2.1%
sort/quickSort/340 84.93 ms 83.27 ms -2.0%
sort/quickSort/350 90.09 ms 88.48 ms -1.8%
sort/quickSort/360 95.43 ms 93.48 ms -2.0%
sort/quickSort/370 100.7 ms 99.06 ms -1.6%
sort/quickSort/380 106.3 ms 104.5 ms -1.7%
sort/quickSort/390 112.8 ms 110.4 ms -2.1%
sort/quickSort/400 118.3 ms 115.8 ms -2.1%
sort/quickSort/410 124.3 ms 122.4 ms -1.5%
sort/quickSort/420 130.6 ms 128.0 ms -2.0%
sort/quickSort/430 136.5 ms 134.6 ms -1.4%
sort/quickSort/440 143.5 ms 141.0 ms -1.7%
sort/quickSort/450 150.1 ms 147.6 ms -1.7%
sort/quickSort/460 156.9 ms 153.8 ms -2.0%
sort/quickSort/470 164.1 ms 161.1 ms -1.8%
sort/quickSort/480 171.3 ms 168.1 ms -1.9%
sort/quickSort/490 178.6 ms 175.3 ms -1.8%
sort/quickSort/500 185.7 ms 182.9 ms -1.5%
sum/compiled-from-Haskell/sum-right-builtin/10 10.11 μs 10.01 μs -1.0%
sum/compiled-from-Haskell/sum-right-builtin/50 47.82 μs 47.44 μs -0.8%
sum/compiled-from-Haskell/sum-right-builtin/100 96.44 μs 95.93 μs -0.5%
sum/compiled-from-Haskell/sum-right-builtin/500 521.3 μs 514.9 μs -1.2%
sum/compiled-from-Haskell/sum-right-builtin/1000 1.145 ms 1.133 ms -1.0%
sum/compiled-from-Haskell/sum-right-builtin/5000 7.812 ms 7.717 ms -1.2%
sum/compiled-from-Haskell/sum-right-builtin/10000 17.15 ms 17.00 ms -0.9%
sum/compiled-from-Haskell/sum-right-Scott/10 6.890 μs 6.777 μs -1.6%
sum/compiled-from-Haskell/sum-right-Scott/50 31.75 μs 31.61 μs -0.4%
sum/compiled-from-Haskell/sum-right-Scott/100 62.99 μs 62.82 μs -0.3%
sum/compiled-from-Haskell/sum-right-Scott/500 338.5 μs 338.9 μs +0.1%
sum/compiled-from-Haskell/sum-right-Scott/1000 745.5 μs 743.4 μs -0.3%
sum/compiled-from-Haskell/sum-right-Scott/5000 5.997 ms 5.969 ms -0.5%
sum/compiled-from-Haskell/sum-right-Scott/10000 13.95 ms 13.75 ms -1.4%
sum/compiled-from-Haskell/sum-right-data/10 24.54 μs 24.91 μs +1.5%
sum/compiled-from-Haskell/sum-right-data/50 118.4 μs 118.2 μs -0.2%
sum/compiled-from-Haskell/sum-right-data/100 236.1 μs 235.5 μs -0.3%
sum/compiled-from-Haskell/sum-right-data/500 1.322 ms 1.318 ms -0.3%
sum/compiled-from-Haskell/sum-right-data/1000 3.004 ms 2.989 ms -0.5%
sum/compiled-from-Haskell/sum-right-data/5000 17.13 ms 17.11 ms -0.1%
sum/compiled-from-Haskell/sum-right-data/10000 35.84 ms 35.75 ms -0.3%
sum/compiled-from-Haskell/sum-left-builtin/10 9.642 μs 9.779 μs +1.4%
sum/compiled-from-Haskell/sum-left-builtin/50 45.70 μs 46.11 μs +0.9%
sum/compiled-from-Haskell/sum-left-builtin/100 92.98 μs 92.92 μs -0.1%
sum/compiled-from-Haskell/sum-left-builtin/500 497.7 μs 493.1 μs -0.9%
sum/compiled-from-Haskell/sum-left-builtin/1000 1.095 ms 1.095 ms 0.0%
sum/compiled-from-Haskell/sum-left-builtin/5000 7.564 ms 7.578 ms +0.2%
sum/compiled-from-Haskell/sum-left-builtin/10000 16.33 ms 16.23 ms -0.6%
sum/compiled-from-Haskell/sum-left-Scott/10 6.789 μs 6.541 μs -3.7%
sum/compiled-from-Haskell/sum-left-Scott/50 31.07 μs 30.41 μs -2.1%
sum/compiled-from-Haskell/sum-left-Scott/100 61.99 μs 60.54 μs -2.3%
sum/compiled-from-Haskell/sum-left-Scott/500 325.2 μs 319.9 μs -1.6%
sum/compiled-from-Haskell/sum-left-Scott/1000 711.3 μs 704.8 μs -0.9%
sum/compiled-from-Haskell/sum-left-Scott/5000 5.819 ms 5.787 ms -0.5%
sum/compiled-from-Haskell/sum-left-Scott/10000 13.48 ms 13.38 ms -0.7%
sum/compiled-from-Haskell/sum-left-data/10 24.62 μs 24.00 μs -2.5%
sum/compiled-from-Haskell/sum-left-data/50 121.3 μs 119.1 μs -1.8%
sum/compiled-from-Haskell/sum-left-data/100 244.0 μs 240.2 μs -1.6%
sum/compiled-from-Haskell/sum-left-data/500 1.355 ms 1.338 ms -1.3%
sum/compiled-from-Haskell/sum-left-data/1000 3.086 ms 3.064 ms -0.7%
sum/compiled-from-Haskell/sum-left-data/5000 17.44 ms 17.31 ms -0.7%
sum/compiled-from-Haskell/sum-left-data/10000 35.86 ms 35.64 ms -0.6%
sum/hand-written-PLC/sum-right-builtin/10 9.876 μs 9.788 μs -0.9%
sum/hand-written-PLC/sum-right-builtin/50 46.05 μs 44.85 μs -2.6%
sum/hand-written-PLC/sum-right-builtin/100 91.58 μs 88.25 μs -3.6%
sum/hand-written-PLC/sum-right-builtin/500 461.1 μs 450.7 μs -2.3%
sum/hand-written-PLC/sum-right-builtin/1000 956.3 μs 932.8 μs -2.5%
sum/hand-written-PLC/sum-right-builtin/5000 5.562 ms 5.435 ms -2.3%
sum/hand-written-PLC/sum-right-builtin/10000 11.35 ms 11.16 ms -1.7%
sum/hand-written-PLC/sum-right-Scott/10 5.779 μs 5.731 μs -0.8%
sum/hand-written-PLC/sum-right-Scott/50 25.73 μs 24.93 μs -3.1%
sum/hand-written-PLC/sum-right-Scott/100 50.11 μs 47.48 μs -5.2%
sum/hand-written-PLC/sum-right-Scott/500 255.1 μs 247.9 μs -2.8%
sum/hand-written-PLC/sum-right-Scott/1000 532.5 μs 519.9 μs -2.4%
sum/hand-written-PLC/sum-right-Scott/5000 3.816 ms 3.724 ms -2.4%
sum/hand-written-PLC/sum-right-Scott/10000 8.478 ms 8.288 ms -2.2%
sum/hand-written-PLC/sum-left-builtin/10 10.29 μs 10.31 μs +0.2%
sum/hand-written-PLC/sum-left-builtin/50 47.48 μs 47.01 μs -1.0%
sum/hand-written-PLC/sum-left-builtin/100 93.59 μs 93.26 μs -0.4%
sum/hand-written-PLC/sum-left-builtin/500 459.6 μs 457.1 μs -0.5%
sum/hand-written-PLC/sum-left-builtin/1000 915.6 μs 913.3 μs -0.3%
sum/hand-written-PLC/sum-left-builtin/5000 4.542 ms 4.515 ms -0.6%
sum/hand-written-PLC/sum-left-builtin/10000 9.070 ms 9.041 ms -0.3%
sum/hand-written-PLC/sum-left-Scott/10 6.189 μs 5.746 μs -7.2%
sum/hand-written-PLC/sum-left-Scott/50 28.66 μs 26.39 μs -7.9%
sum/hand-written-PLC/sum-left-Scott/100 56.50 μs 53.08 μs -6.1%
sum/hand-written-PLC/sum-left-Scott/500 281.9 μs 259.8 μs -7.8%
sum/hand-written-PLC/sum-left-Scott/1000 572.4 μs 537.9 μs -6.0%
sum/hand-written-PLC/sum-left-Scott/5000 3.490 ms 3.289 ms -5.8%
sum/hand-written-PLC/sum-left-Scott/10000 7.223 ms 6.835 ms -5.4%
TOTAL 5.062 s 4.953 s -2.2%

Copy link
Contributor

Click here to check the status of your benchmark.

Copy link
Contributor

Comparing benchmark results of 'lists' on '16a986fd8' (base) and '9d8942da1' (PR)

Results table
Script 16a986f 9d8942d Change
sort/ghcSort/10 34.23 μs 33.94 μs -0.8%
sort/ghcSort/20 79.63 μs 79.27 μs -0.5%
sort/ghcSort/30 122.8 μs 121.7 μs -0.9%
sort/ghcSort/40 183.3 μs 180.2 μs -1.7%
sort/ghcSort/50 224.1 μs 223.7 μs -0.2%
sort/ghcSort/60 281.7 μs 280.6 μs -0.4%
sort/ghcSort/70 366.7 μs 363.9 μs -0.8%
sort/ghcSort/80 417.7 μs 415.7 μs -0.5%
sort/ghcSort/90 470.3 μs 458.8 μs -2.4%
sort/ghcSort/100 521.0 μs 517.3 μs -0.7%
sort/ghcSort/110 570.9 μs 569.4 μs -0.3%
sort/ghcSort/120 643.5 μs 638.5 μs -0.8%
sort/ghcSort/130 794.8 μs 789.6 μs -0.7%
sort/ghcSort/140 839.2 μs 834.5 μs -0.6%
sort/ghcSort/150 905.8 μs 894.0 μs -1.3%
sort/ghcSort/160 969.0 μs 955.1 μs -1.4%
sort/ghcSort/170 1.025 ms 1.004 ms -2.0%
sort/ghcSort/180 1.069 ms 1.058 ms -1.0%
sort/ghcSort/190 1.132 ms 1.119 ms -1.1%
sort/ghcSort/200 1.215 ms 1.201 ms -1.2%
sort/ghcSort/210 1.271 ms 1.254 ms -1.3%
sort/ghcSort/220 1.331 ms 1.313 ms -1.4%
sort/ghcSort/230 1.417 ms 1.398 ms -1.3%
sort/ghcSort/240 1.501 ms 1.480 ms -1.4%
sort/ghcSort/250 1.576 ms 1.555 ms -1.3%
sort/ghcSort/260 1.815 ms 1.794 ms -1.2%
sort/ghcSort/270 1.870 ms 1.855 ms -0.8%
sort/ghcSort/280 1.924 ms 1.903 ms -1.1%
sort/ghcSort/290 1.990 ms 1.970 ms -1.0%
sort/ghcSort/300 2.067 ms 2.044 ms -1.1%
sort/ghcSort/310 2.134 ms 2.113 ms -1.0%
sort/ghcSort/320 2.216 ms 2.185 ms -1.4%
sort/ghcSort/330 2.270 ms 2.247 ms -1.0%
sort/ghcSort/340 2.333 ms 2.306 ms -1.2%
sort/ghcSort/350 2.385 ms 2.362 ms -1.0%
sort/ghcSort/360 2.453 ms 2.427 ms -1.1%
sort/ghcSort/370 2.524 ms 2.500 ms -1.0%
sort/ghcSort/380 2.600 ms 2.573 ms -1.0%
sort/ghcSort/390 2.727 ms 2.695 ms -1.2%
sort/ghcSort/400 2.793 ms 2.762 ms -1.1%
sort/ghcSort/410 2.833 ms 2.802 ms -1.1%
sort/ghcSort/420 2.936 ms 2.899 ms -1.3%
sort/ghcSort/430 2.986 ms 2.946 ms -1.3%
sort/ghcSort/440 3.070 ms 3.031 ms -1.3%
sort/ghcSort/450 3.237 ms 3.191 ms -1.4%
sort/ghcSort/460 3.297 ms 3.236 ms -1.9%
sort/ghcSort/470 3.345 ms 3.304 ms -1.2%
sort/ghcSort/480 3.464 ms 3.417 ms -1.4%
sort/ghcSort/490 3.532 ms 3.493 ms -1.1%
sort/ghcSort/500 3.631 ms 3.589 ms -1.2%
sort/insertionSort/10 33.40 μs 33.01 μs -1.2%
sort/insertionSort/20 123.2 μs 121.1 μs -1.7%
sort/insertionSort/30 270.1 μs 265.6 μs -1.7%
sort/insertionSort/40 474.8 μs 466.7 μs -1.7%
sort/insertionSort/50 736.8 μs 726.6 μs -1.4%
sort/insertionSort/60 1.060 ms 1.043 ms -1.6%
sort/insertionSort/70 1.439 ms 1.416 ms -1.6%
sort/insertionSort/80 1.879 ms 1.852 ms -1.4%
sort/insertionSort/90 2.372 ms 2.334 ms -1.6%
sort/insertionSort/100 2.932 ms 2.882 ms -1.7%
sort/insertionSort/110 3.541 ms 3.478 ms -1.8%
sort/insertionSort/120 4.216 ms 4.147 ms -1.6%
sort/insertionSort/130 4.953 ms 4.862 ms -1.8%
sort/insertionSort/140 5.722 ms 5.652 ms -1.2%
sort/insertionSort/150 6.591 ms 6.478 ms -1.7%
sort/insertionSort/160 7.499 ms 7.391 ms -1.4%
sort/insertionSort/170 8.501 ms 8.333 ms -2.0%
sort/insertionSort/180 9.512 ms 9.345 ms -1.8%
sort/insertionSort/190 10.59 ms 10.44 ms -1.4%
sort/insertionSort/200 11.76 ms 11.57 ms -1.6%
sort/insertionSort/210 13.00 ms 12.76 ms -1.8%
sort/insertionSort/220 14.26 ms 14.03 ms -1.6%
sort/insertionSort/230 15.61 ms 15.29 ms -2.0%
sort/insertionSort/240 17.01 ms 16.70 ms -1.8%
sort/insertionSort/250 18.51 ms 18.11 ms -2.2%
sort/insertionSort/260 19.93 ms 19.69 ms -1.2%
sort/insertionSort/270 21.59 ms 21.21 ms -1.8%
sort/insertionSort/280 23.26 ms 22.89 ms -1.6%
sort/insertionSort/290 25.07 ms 24.56 ms -2.0%
sort/insertionSort/300 26.69 ms 26.32 ms -1.4%
sort/insertionSort/310 28.58 ms 28.07 ms -1.8%
sort/insertionSort/320 30.42 ms 29.96 ms -1.5%
sort/insertionSort/330 32.42 ms 31.86 ms -1.7%
sort/insertionSort/340 34.51 ms 33.84 ms -1.9%
sort/insertionSort/350 36.55 ms 35.94 ms -1.7%
sort/insertionSort/360 38.78 ms 38.02 ms -2.0%
sort/insertionSort/370 40.97 ms 40.22 ms -1.8%
sort/insertionSort/380 43.17 ms 42.58 ms -1.4%
sort/insertionSort/390 45.46 ms 44.76 ms -1.5%
sort/insertionSort/400 47.69 ms 47.18 ms -1.1%
sort/insertionSort/410 50.19 ms 49.61 ms -1.2%
sort/insertionSort/420 52.74 ms 52.07 ms -1.3%
sort/insertionSort/430 55.43 ms 54.64 ms -1.4%
sort/insertionSort/440 57.77 ms 57.31 ms -0.8%
sort/insertionSort/450 60.65 ms 59.87 ms -1.3%
sort/insertionSort/460 64.28 ms 62.79 ms -2.3%
sort/insertionSort/470 66.60 ms 65.48 ms -1.7%
sort/insertionSort/480 69.71 ms 68.70 ms -1.4%
sort/insertionSort/490 72.77 ms 71.75 ms -1.4%
sort/insertionSort/500 75.89 ms 74.92 ms -1.3%
sort/mergeSort/10 90.41 μs 89.33 μs -1.2%
sort/mergeSort/20 218.1 μs 214.1 μs -1.8%
sort/mergeSort/30 361.9 μs 355.0 μs -1.9%
sort/mergeSort/40 513.7 μs 505.3 μs -1.6%
sort/mergeSort/50 669.7 μs 657.8 μs -1.8%
sort/mergeSort/60 836.4 μs 821.6 μs -1.8%
sort/mergeSort/70 1.010 ms 990.7 μs -1.9%
sort/mergeSort/80 1.188 ms 1.165 ms -1.9%
sort/mergeSort/90 1.359 ms 1.334 ms -1.8%
sort/mergeSort/100 1.541 ms 1.510 ms -2.0%
sort/mergeSort/110 1.718 ms 1.688 ms -1.7%
sort/mergeSort/120 1.918 ms 1.876 ms -2.2%
sort/mergeSort/130 2.129 ms 2.095 ms -1.6%
sort/mergeSort/140 2.298 ms 2.252 ms -2.0%
sort/mergeSort/150 2.480 ms 2.428 ms -2.1%
sort/mergeSort/160 2.694 ms 2.638 ms -2.1%
sort/mergeSort/170 2.870 ms 2.823 ms -1.6%
sort/mergeSort/180 3.080 ms 3.026 ms -1.8%
sort/mergeSort/190 3.298 ms 3.228 ms -2.1%
sort/mergeSort/200 3.487 ms 3.402 ms -2.4%
sort/mergeSort/210 3.692 ms 3.617 ms -2.0%
sort/mergeSort/220 3.874 ms 3.802 ms -1.9%
sort/mergeSort/230 4.095 ms 4.023 ms -1.8%
sort/mergeSort/240 4.322 ms 4.233 ms -2.1%
sort/mergeSort/250 4.559 ms 4.461 ms -2.1%
sort/mergeSort/260 4.803 ms 4.679 ms -2.6%
sort/mergeSort/270 4.957 ms 4.837 ms -2.4%
sort/mergeSort/280 5.148 ms 5.048 ms -1.9%
sort/mergeSort/290 5.345 ms 5.226 ms -2.2%
sort/mergeSort/300 5.550 ms 5.439 ms -2.0%
sort/mergeSort/310 5.783 ms 5.654 ms -2.2%
sort/mergeSort/320 6.045 ms 5.900 ms -2.4%
sort/mergeSort/330 6.206 ms 6.082 ms -2.0%
sort/mergeSort/340 6.433 ms 6.269 ms -2.5%
sort/mergeSort/350 6.662 ms 6.622 ms -0.6%
sort/mergeSort/360 6.897 ms 6.731 ms -2.4%
sort/mergeSort/370 7.106 ms 6.934 ms -2.4%
sort/mergeSort/380 7.331 ms 7.184 ms -2.0%
sort/mergeSort/390 7.631 ms 7.447 ms -2.4%
sort/mergeSort/400 7.795 ms 7.595 ms -2.6%
sort/mergeSort/410 7.991 ms 7.862 ms -1.6%
sort/mergeSort/420 8.239 ms 8.048 ms -2.3%
sort/mergeSort/430 8.458 ms 8.298 ms -1.9%
sort/mergeSort/440 8.669 ms 8.479 ms -2.2%
sort/mergeSort/450 8.930 ms 8.708 ms -2.5%
sort/mergeSort/460 9.129 ms 8.944 ms -2.0%
sort/mergeSort/470 9.399 ms 9.188 ms -2.2%
sort/mergeSort/480 9.589 ms 9.399 ms -2.0%
sort/mergeSort/490 9.844 ms 9.609 ms -2.4%
sort/mergeSort/500 10.15 ms 9.852 ms -2.9%
sort/quickSort/10 77.32 μs 76.69 μs -0.8%
sort/quickSort/20 288.7 μs 286.7 μs -0.7%
sort/quickSort/30 637.9 μs 633.7 μs -0.7%
sort/quickSort/40 1.142 ms 1.138 ms -0.4%
sort/quickSort/50 1.807 ms 1.795 ms -0.7%
sort/quickSort/60 2.625 ms 2.602 ms -0.9%
sort/quickSort/70 3.571 ms 3.548 ms -0.6%
sort/quickSort/80 4.718 ms 4.663 ms -1.2%
sort/quickSort/90 5.938 ms 5.911 ms -0.5%
sort/quickSort/100 7.301 ms 7.230 ms -1.0%
sort/quickSort/110 8.879 ms 8.735 ms -1.6%
sort/quickSort/120 10.58 ms 10.40 ms -1.7%
sort/quickSort/130 12.42 ms 12.22 ms -1.6%
sort/quickSort/140 14.24 ms 14.05 ms -1.3%
sort/quickSort/150 16.37 ms 16.28 ms -0.5%
sort/quickSort/160 18.61 ms 18.48 ms -0.7%
sort/quickSort/170 21.04 ms 20.85 ms -0.9%
sort/quickSort/180 23.63 ms 23.41 ms -0.9%
sort/quickSort/190 26.24 ms 25.85 ms -1.5%
sort/quickSort/200 29.15 ms 28.66 ms -1.7%
sort/quickSort/210 32.01 ms 31.59 ms -1.3%
sort/quickSort/220 35.23 ms 34.68 ms -1.6%
sort/quickSort/230 38.47 ms 37.89 ms -1.5%
sort/quickSort/240 41.87 ms 41.37 ms -1.2%
sort/quickSort/250 45.37 ms 44.77 ms -1.3%
sort/quickSort/260 48.99 ms 48.43 ms -1.1%
sort/quickSort/270 52.92 ms 52.55 ms -0.7%
sort/quickSort/280 56.92 ms 56.51 ms -0.7%
sort/quickSort/290 60.87 ms 60.64 ms -0.4%
sort/quickSort/300 65.17 ms 64.78 ms -0.6%
sort/quickSort/310 69.92 ms 69.42 ms -0.7%
sort/quickSort/320 74.47 ms 73.98 ms -0.7%
sort/quickSort/330 79.42 ms 78.88 ms -0.7%
sort/quickSort/340 84.35 ms 83.61 ms -0.9%
sort/quickSort/350 89.46 ms 88.65 ms -0.9%
sort/quickSort/360 94.76 ms 94.29 ms -0.5%
sort/quickSort/370 100.4 ms 99.49 ms -0.9%
sort/quickSort/380 105.5 ms 104.7 ms -0.8%
sort/quickSort/390 111.9 ms 110.5 ms -1.3%
sort/quickSort/400 117.3 ms 116.4 ms -0.8%
sort/quickSort/410 123.7 ms 122.4 ms -1.1%
sort/quickSort/420 129.7 ms 128.6 ms -0.8%
sort/quickSort/430 135.6 ms 134.4 ms -0.9%
sort/quickSort/440 141.7 ms 141.6 ms -0.1%
sort/quickSort/450 148.7 ms 148.1 ms -0.4%
sort/quickSort/460 155.4 ms 154.8 ms -0.4%
sort/quickSort/470 162.3 ms 160.0 ms -1.4%
sort/quickSort/480 169.4 ms 167.6 ms -1.1%
sort/quickSort/490 176.2 ms 175.1 ms -0.6%
sort/quickSort/500 183.8 ms 182.5 ms -0.7%
sum/compiled-from-Haskell/sum-right-builtin/10 10.04 μs 9.662 μs -3.8%
sum/compiled-from-Haskell/sum-right-builtin/50 47.59 μs 45.69 μs -4.0%
sum/compiled-from-Haskell/sum-right-builtin/100 96.37 μs 92.74 μs -3.8%
sum/compiled-from-Haskell/sum-right-builtin/500 519.1 μs 497.7 μs -4.1%
sum/compiled-from-Haskell/sum-right-builtin/1000 1.142 ms 1.098 ms -3.9%
sum/compiled-from-Haskell/sum-right-builtin/5000 7.732 ms 7.542 ms -2.5%
sum/compiled-from-Haskell/sum-right-builtin/10000 17.03 ms 16.67 ms -2.1%
sum/compiled-from-Haskell/sum-right-Scott/10 6.831 μs 6.779 μs -0.8%
sum/compiled-from-Haskell/sum-right-Scott/50 31.62 μs 31.56 μs -0.2%
sum/compiled-from-Haskell/sum-right-Scott/100 62.95 μs 62.66 μs -0.5%
sum/compiled-from-Haskell/sum-right-Scott/500 338.1 μs 337.6 μs -0.1%
sum/compiled-from-Haskell/sum-right-Scott/1000 743.8 μs 738.9 μs -0.7%
sum/compiled-from-Haskell/sum-right-Scott/5000 5.960 ms 5.954 ms -0.1%
sum/compiled-from-Haskell/sum-right-Scott/10000 13.87 ms 13.71 ms -1.2%
sum/compiled-from-Haskell/sum-right-data/10 24.61 μs 24.34 μs -1.1%
sum/compiled-from-Haskell/sum-right-data/50 118.0 μs 115.4 μs -2.2%
sum/compiled-from-Haskell/sum-right-data/100 235.4 μs 230.7 μs -2.0%
sum/compiled-from-Haskell/sum-right-data/500 1.323 ms 1.291 ms -2.4%
sum/compiled-from-Haskell/sum-right-data/1000 2.997 ms 2.933 ms -2.1%
sum/compiled-from-Haskell/sum-right-data/5000 17.04 ms 16.80 ms -1.4%
sum/compiled-from-Haskell/sum-right-data/10000 35.73 ms 35.10 ms -1.8%
sum/compiled-from-Haskell/sum-left-builtin/10 9.666 μs 9.673 μs +0.1%
sum/compiled-from-Haskell/sum-left-builtin/50 45.39 μs 44.16 μs -2.7%
sum/compiled-from-Haskell/sum-left-builtin/100 92.41 μs 89.18 μs -3.5%
sum/compiled-from-Haskell/sum-left-builtin/500 493.2 μs 477.6 μs -3.2%
sum/compiled-from-Haskell/sum-left-builtin/1000 1.090 ms 1.059 ms -2.8%
sum/compiled-from-Haskell/sum-left-builtin/5000 7.516 ms 7.367 ms -2.0%
sum/compiled-from-Haskell/sum-left-builtin/10000 16.22 ms 15.86 ms -2.2%
sum/compiled-from-Haskell/sum-left-Scott/10 6.797 μs 6.553 μs -3.6%
sum/compiled-from-Haskell/sum-left-Scott/50 31.05 μs 30.28 μs -2.5%
sum/compiled-from-Haskell/sum-left-Scott/100 61.48 μs 60.64 μs -1.4%
sum/compiled-from-Haskell/sum-left-Scott/500 326.5 μs 320.1 μs -2.0%
sum/compiled-from-Haskell/sum-left-Scott/1000 710.0 μs 695.8 μs -2.0%
sum/compiled-from-Haskell/sum-left-Scott/5000 5.768 ms 5.740 ms -0.5%
sum/compiled-from-Haskell/sum-left-Scott/10000 13.45 ms 13.32 ms -1.0%
sum/compiled-from-Haskell/sum-left-data/10 24.56 μs 23.40 μs -4.7%
sum/compiled-from-Haskell/sum-left-data/50 120.2 μs 115.9 μs -3.6%
sum/compiled-from-Haskell/sum-left-data/100 242.9 μs 234.7 μs -3.4%
sum/compiled-from-Haskell/sum-left-data/500 1.345 ms 1.305 ms -3.0%
sum/compiled-from-Haskell/sum-left-data/1000 3.087 ms 2.995 ms -3.0%
sum/compiled-from-Haskell/sum-left-data/5000 17.41 ms 16.98 ms -2.5%
sum/compiled-from-Haskell/sum-left-data/10000 35.83 ms 35.04 ms -2.2%
sum/hand-written-PLC/sum-right-builtin/10 9.904 μs 9.453 μs -4.6%
sum/hand-written-PLC/sum-right-builtin/50 45.68 μs 43.32 μs -5.2%
sum/hand-written-PLC/sum-right-builtin/100 90.65 μs 85.63 μs -5.5%
sum/hand-written-PLC/sum-right-builtin/500 461.3 μs 441.1 μs -4.4%
sum/hand-written-PLC/sum-right-builtin/1000 956.8 μs 907.3 μs -5.2%
sum/hand-written-PLC/sum-right-builtin/5000 5.529 ms 5.316 ms -3.9%
sum/hand-written-PLC/sum-right-builtin/10000 11.30 ms 10.89 ms -3.6%
sum/hand-written-PLC/sum-right-Scott/10 5.729 μs 5.736 μs +0.1%
sum/hand-written-PLC/sum-right-Scott/50 25.46 μs 25.03 μs -1.7%
sum/hand-written-PLC/sum-right-Scott/100 49.57 μs 48.73 μs -1.7%
sum/hand-written-PLC/sum-right-Scott/500 252.6 μs 245.0 μs -3.0%
sum/hand-written-PLC/sum-right-Scott/1000 527.6 μs 524.4 μs -0.6%
sum/hand-written-PLC/sum-right-Scott/5000 3.767 ms 3.747 ms -0.5%
sum/hand-written-PLC/sum-right-Scott/10000 8.418 ms 8.356 ms -0.7%
sum/hand-written-PLC/sum-left-builtin/10 10.24 μs 9.908 μs -3.2%
sum/hand-written-PLC/sum-left-builtin/50 47.34 μs 45.38 μs -4.1%
sum/hand-written-PLC/sum-left-builtin/100 93.13 μs 89.17 μs -4.3%
sum/hand-written-PLC/sum-left-builtin/500 458.7 μs 439.8 μs -4.1%
sum/hand-written-PLC/sum-left-builtin/1000 913.5 μs 881.2 μs -3.5%
sum/hand-written-PLC/sum-left-builtin/5000 4.522 ms 4.354 ms -3.7%
sum/hand-written-PLC/sum-left-builtin/10000 9.067 ms 8.734 ms -3.7%
sum/hand-written-PLC/sum-left-Scott/10 6.082 μs 5.881 μs -3.3%
sum/hand-written-PLC/sum-left-Scott/50 28.35 μs 26.46 μs -6.7%
sum/hand-written-PLC/sum-left-Scott/100 56.02 μs 53.01 μs -5.4%
sum/hand-written-PLC/sum-left-Scott/500 277.4 μs 264.2 μs -4.8%
sum/hand-written-PLC/sum-left-Scott/1000 567.5 μs 541.6 μs -4.6%
sum/hand-written-PLC/sum-left-Scott/5000 3.446 ms 3.318 ms -3.7%
sum/hand-written-PLC/sum-left-Scott/10000 7.149 ms 6.873 ms -3.9%
TOTAL 5.016 s 4.958 s -1.1%

Copy link
Contributor

@michaelpj michaelpj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@@ -85,6 +86,9 @@ defaultCekMachineCosts =
defaultCekCostModel :: CostModel CekMachineCosts BuiltinCostModel
defaultCekCostModel = CostModel defaultCekMachineCosts defaultBuiltinCostModel

toCekCostModel :: BuiltinSemanticsVariant DefaultFun -> CostModel CekMachineCosts BuiltinCostModel
toCekCostModel _ = defaultCekCostModel
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it therefore have some comment explaining that?

plutus-ledger-api/test/Spec/Eval.hs Outdated Show resolved Hide resolved
Copy link
Contributor

@kwxm kwxm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. Inevitably once I started thinking about this stuff again I became confused by the versions and how they depend on each other. I'm pretty convinced that this does what we want though.

| DefaultFunSemanticsVariant2
deriving stock (Enum, Bounded, Show)
data BuiltinSemanticsVariant DefaultFun
= DefaultFunSemanticsVariant0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I vote that we change these to DefaultSemanticsVariantA/B/C or something similar. It's quite confusing having all these variants and versions in the code, so it might help if they're not all identified by numbers that might lead people to believe that the same number means the same thing in different contexts. That would mean changing the specification too though, so if we do it let's do it later in its own PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine either way. I'm hopeful we'll be able to get rid of the semantics variants entirely and simply rely on the language and protocol versions directly (maybe in the "condensed" form).

>=> mkDynEvaluationContext
PlutusV3
[DefaultFunSemanticsVariant2]
(const Plutus.DefaultFunSemanticsVariant2)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again we could probably do without the Plutus here and in the import.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me check: V1 and V2 have if pv < conwayPV then ... else ... here, but it's safe to have const for V3 because we know that PlutusV3 is going to be introduced at the same time as conwayPV, so a PlutusV3 script can never be executed with a lower PV, so there's no possiblity that pv < conwayPV here (but conceivably we might need if pv < einsteinPV or something in the future). Is that right? Maybe we should try to explain that somewhere, since there's a LL/PV dependency that I don't think is captured in the code anywhere. Maybe we should have note containing a table like we do in the PLC specification (or a big table that contains all of the relevant information, since there are multiple tables in the spec).

[Later] Actually, maybe the dependencies are specified in PlutusLedgerApi.Common.Versions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me check: V1 and V2 have if pv < conwayPV then ... else ... here, but it's safe to have const for V3 because we know that PlutusV3 is going to be introduced at the same time as conwayPV, so a PlutusV3 script can never be executed with a lower PV, so there's no possiblity that pv < conwayPV here (but conceivably we might need if pv < einsteinPV or something in the future). Is that right?

I believe so, yes.

Maybe we should try to explain that somewhere, since there's a LL/PV dependency that I don't think is captured in the code anywhere. Maybe we should have note containing a table like we do in the PLC specification (or a big table that contains all of the relevant information, since there are multiple tables in the spec).

Good point, I'll add a small table somewhere, but I'm not totally sure where given that the logic is spread across multiple files. I guess PlutusLedgerApi.Common.Versions indeed.

plutus-conformance/haskell/Spec.hs Show resolved Hide resolved
evaluation.

Different protocol versions may require different bundles of machine parameters, which allows us for
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe make this into a Note? It seems quite important. It would seem to have some bearing on this, for example.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe make this into a Note?

I kinda want this to be the Haddock for extra transparency. I do reference the Haddock of a function instead of a Note sometimes.

plutus-ledger-api/src/PlutusLedgerApi/Common/Eval.hs Outdated Show resolved Hide resolved
@kwxm
Copy link
Contributor

kwxm commented Apr 22, 2024

/benchmark validation

Copy link
Contributor

Click here to check the status of your benchmark.

@effectfully effectfully force-pushed the effectfully/builtins/both-MajorProtocolVersion-and-multiple-CostModels branch from e6afc0b to 039dce1 Compare April 25, 2024 12:56
@effectfully effectfully merged commit 3944e09 into master Apr 25, 2024
5 checks passed
@effectfully effectfully deleted the effectfully/builtins/both-MajorProtocolVersion-and-multiple-CostModels branch April 25, 2024 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builtins Costing Anything relating to costs, fees, gas, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants