Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generalising bitwise simplifications #2251

Merged
merged 8 commits into from Jan 11, 2024
Merged

Generalising bitwise simplifications #2251

merged 8 commits into from Jan 11, 2024

Conversation

PetarMax
Copy link
Contributor

This PR generalises several lemmas on bitwise operators, streamlining the development.

The corresponding tests are already present.

@PetarMax PetarMax self-assigned this Jan 10, 2024
@PetarMax PetarMax added the enhancement New feature or request label Jan 10, 2024
@ehildenb
Copy link
Member

Can you post performance numbers for the entire KEVM test-suite and the Kontrol test-suite please (comparing this branch to master). I suspect this may have a slowdown in some cases.

@PetarMax
Copy link
Contributor Author

PetarMax commented Jan 10, 2024 •
edited

Running the entire test suite locally for me is complicated. I will wait to see the CI numbers first.

@ehildenb
Copy link
Member

ehildenb commented Jan 10, 2024 •
edited

There are instructions here for KEVM: runtimeverification/hs-backend-booster#355. Note that these instructions are for comparing booster to legacy backend, but you should compare booster on master vs booster on this branch.

Similar instructions work for Kontrol.

@PetarMax
Copy link
Contributor Author

PetarMax commented Jan 11, 2024 •
edited

These is the performance comparison for KEVM on all tests, it does not appear to be statistically significant overall, with the PR times being slightly faster than the master times:

 Test                                        | master-stats time | PR-stats time | (Master-stats/PR-stats) time 
---------------------------------------------+-------------------+---------------+------------------------------
 [mcd/flipper-tau-pass-spec.k]               | 91.76             | 103.7         | 0.8848601735776278           
 [benchmarks/bytes00-spec.k]                 | 58.69             | 62.14         | 0.9444802059864821           
 [mcd/vat-mului-pass-spec.k]                 | 67.28             | 71.21         | 0.9448111220334223           
 [mcd/vat-deny-diff-fail-rough-spec.k]       | 80.21             | 84.76         | 0.9463190184049078           
 [benchmarks/storagevar01-spec.k]            | 51.75             | 54.43         | 0.950762447179864            
 [benchmarks/keccak00-spec.k]                | 44.57             | 46.86         | 0.9511310285958173           
 [erc20/ds/transfer-failure-1-c-spec.k]      | 53.87             | 56.5          | 0.9534513274336283           
 [benchmarks/staticloop00-a0lt10-spec.k]     | 49.16             | 51.45         | 0.9554907677356655           
 [erc20/hkg/transferFrom-success-2-spec.k]   | 65.0              | 67.78         | 0.9589849513130717           
 [erc20/ds/transferFrom-failure-1-c-spec.k]  | 98.5              | 102.64        | 0.9596648480124708           
 [bihu/forwardToHotWallet-failure-3-spec.k]  | 73.86             | 76.91         | 0.9603432583539202           
 [benchmarks/encode-keccak00-spec.k]         | 41.16             | 42.48         | 0.9689265536723164           
 [mcd/dsvalue-peek-pass-rough-spec.k]        | 46.13             | 47.56         | 0.969932716568545            
 [mcd/vat-addui-pass-spec.k]                 | 94.87             | 97.63         | 0.9717300010242754           
 [erc20/ds/transfer-failure-1-a-spec.k]      | 77.48             | 79.38         | 0.9760644998740238           
 [erc20/hkg/approve-spec.k]                  | 37.98             | 38.8          | 0.9788659793814433           
 [bihu/forwardToHotWallet-failure-4-spec.k]  | 98.56             | 100.37        | 0.9819667231244396           
 [mcd/flipper-addu48u48-fail-rough-spec.k]   | 42.24             | 42.93         | 0.9839273235499651           
 [benchmarks/dynamicarray00-spec.k]          | 35.4              | 35.95         | 0.9847009735744088           
 [bihu/functional-spec.k]                    | 74.24             | 75.3          | 0.9859229747675963           
 [mcd/functional-spec.k]                     | 573.16            | 581.06        | 0.9864041579182873           
 [erc20/ds/transferFrom-failure-1-a-spec.k]  | 92.09             | 93.34         | 0.9866080994214699           
 [erc20/ds/transferFrom-failure-2-b-spec.k]  | 87.59             | 88.65         | 0.9880428652002256           
 [benchmarks/address00-spec.k]               | 35.2              | 35.61         | 0.9884863802302725           
 [examples/solidity-code-spec.md]            | 152.47            | 154.24        | 0.988524377593361            
 [benchmarks/overflow00-overflow-spec.k]     | 33.04             | 33.37         | 0.9901108780341624           
 [bihu/forwardToHotWallet-success-1-spec.k]  | 53.91             | 54.43         | 0.9904464449751974           
 [mcd/vat-subui-pass-spec.k]                 | 92.22             | 93.08         | 0.9907606360120327           
 [erc20/ds/approve-failure-spec.k]           | 47.32             | 47.76         | 0.9907872696817421           
 [mcd/dstoken-approve-fail-rough-spec.k]     | 82.5              | 83.21         | 0.9914673717101311           
 [erc20/hkg/transfer-success-1-spec.k]       | 46.95             | 47.35         | 0.9915522703273496           
 [benchmarks/overflow00-nooverflow-spec.k]   | 35.6              | 35.89         | 0.9919197548063527           
 [erc20/ds/approve-success-spec.k]           | 50.46             | 50.87         | 0.9919402398270101           
 [erc20/ds/transfer-failure-2-b-spec.k]      | 43.38             | 43.73         | 0.9919963411845416           
 [benchmarks/requires01-a0gt0-spec.k]        | 36.98             | 37.24         | 0.993018259935553            
 [mcd/dsvalue-read-pass-spec.k]              | 50.43             | 50.78         | 0.9931075226467113           
 [erc20/ds/transferFrom-failure-2-c-spec.k]  | 46.03             | 46.29         | 0.9943832361201124           
 [erc20/ds/allowance-spec.k]                 | 41.78             | 42.01         | 0.9945251130683171           
 [benchmarks/ecrecover00-sigvalid-spec.k]    | 93.01             | 93.5          | 0.9947593582887702           
 [benchmarks/storagevar02-nooverflow-spec.k] | 46.6              | 46.84         | 0.9948761742100768           
 [erc20/ds/transfer-failure-1-b-spec.k]      | 104.37            | 104.79        | 0.9959919839679359           
 [erc20/hkg/transferFrom-failure-1-spec.k]   | 74.54             | 74.82         | 0.996257685110933            
 [mcd/flipper-ttl-pass-spec.k]               | 49.64             | 49.81         | 0.9965870307167235           
 [mcd/vat-muluu-pass-spec.k]                 | 89.74             | 90.0          | 0.9971111111111111           
 [erc20/hkg/transferFrom-success-1-spec.k]   | 63.86             | 64.04         | 0.9971892567145533           
 [benchmarks/structarg00-spec.k]             | 37.87             | 37.95         | 0.9978919631093542           
 [benchmarks/storagevar00-spec.k]            | 44.58             | 44.67         | 0.9979852249832102           
 [benchmarks/staticarray00-spec.k]           | 36.48             | 36.49         | 0.9997259523157028           
 [mcd/end-subuu-pass-spec.k]                 | 45.47             | 45.43         | 1.0008804754567466           
 [erc20/hkg/transferFrom-failure-2-spec.k]   | 70.23             | 70.12         | 1.0015687393040502           
 [erc20/ds/transfer-success-2-spec.k]        | 61.97             | 61.82         | 1.0024263992235523           
 [erc20/ds/balanceOf-spec.k]                 | 40.29             | 40.18         | 1.0027376804380288           
 [erc20/ds/transfer-success-1-spec.k]        | 76.98             | 76.71         | 1.0035197497066877           
 [bihu/forwardToHotWallet-success-2-spec.k]  | 71.98             | 71.58         | 1.0055881531153954           
 [mcd/cat-file-addr-pass-rough-spec.k]       | 48.83             | 48.46         | 1.0076351630210483           
 [erc20/ds/transfer-failure-2-a-spec.k]      | 104.89            | 104.07        | 1.0078793120015375           
 [erc20/hkg/totalSupply-spec.k]              | 30.28             | 30.04         | 1.007989347536618            
 [erc20/hkg/balanceOf-spec.k]                | 34.88             | 34.6          | 1.0080924855491329           
 [mcd/vat-subui-pass-rough-spec.k]           | 105.45            | 104.48        | 1.0092840735068913           
 [benchmarks/storagevar03-spec.k]            | 48.52             | 48.05         | 1.0097814776274716           
 [mcd/dsvalue-read-pass-summarize-spec.k]    | 46.4              | 45.9          | 1.0108932461873639           
 [erc20/ds/transferFrom-failure-1-d-spec.k]  | 57.41             | 56.75         | 1.0116299559471364           
 [erc20/hkg/transfer-success-2-spec.k]       | 45.4              | 44.85         | 1.0122630992196209           
 [erc20/hkg/transfer-failure-2-spec.k]       | 48.6              | 47.93         | 1.0139787189651575           
 [bihu/forwardToHotWallet-failure-1-spec.k]  | 81.6              | 80.39         | 1.0150516233362357           
 [erc20/hkg/allowance-spec.k]                | 38.25             | 37.68         | 1.0151273885350318           
 [mcd/vat-slip-pass-rough-spec.k]            | 140.04            | 137.76        | 1.0165505226480835           
 [mcd/vat-move-diff-rough-spec.k]            | 125.3             | 123.24        | 1.0167153521583903           
 [erc20/ds/totalSupply-spec.k]               | 37.09             | 36.4          | 1.018956043956044            
 [erc20/ds/transferFrom-success-2-spec.k]    | 76.34             | 74.77         | 1.02099772636084             
 [mcd/cat-exhaustiveness-spec.k]             | 228.98            | 224.26        | 1.0210469990189959           
 [mcd/flipper-bids-pass-rough-spec.k]        | 122.45            | 119.69        | 1.023059570557273            
 [examples/erc20-spec.md]                    | 206.74            | 201.9         | 1.0239722634967805           
 [mcd/vat-subui-fail-rough-spec.k]           | 64.79             | 63.27         | 1.0240240240240241           
 [mcd/dai-symbol-pass-spec.k]                | 42.76             | 41.66         | 1.0264042246759482           
 [mcd/flopper-file-pass-rough-spec.k]        | 367.81            | 358.18        | 1.0268859232787984           
 [mcd/dai-adduu-fail-rough-spec.k]           | 38.71             | 37.54         | 1.031166755460842            
 [benchmarks/requires01-a0le0-spec.k]        | 150.25            | 145.55        | 1.0322913088285812           
 [erc20/ds/transferFrom-failure-2-a-spec.k]  | 67.29             | 65.11         | 1.0334818000307173           
 [benchmarks/encodepacked-keccak01-spec.k]   | 40.43             | 39.11         | 1.0337509588340579           
 [functional/evm-int-simplifications-spec.k] | 157.68            | 152.12        | 1.0365500920326058           
 [bihu/forwardToHotWallet-failure-2-spec.k]  | 62.36             | 60.02         | 1.0389870043318894           
 [erc20/ds/transferFrom-failure-1-b-spec.k]  | 149.86            | 143.89        | 1.041490027104038            
 [mcd/flopper-cage-pass-spec.k]              | 52.84             | 50.67         | 1.0428261298598778           
 [erc20/ds/transferFrom-success-1-spec.k]    | 92.22             | 88.11         | 1.0466462376574737           
 [benchmarks/storagevar02-overflow-spec.k]   | 47.54             | 45.42         | 1.0466754733597534           
 [erc20/hkg/transfer-failure-1-spec.k]       | 169.53            | 161.82        | 1.0476455320726734           
 [mcd/vat-addui-fail-rough-spec.k]           | 75.78             | 71.89         | 1.0541104465155098           
 [examples/storage-spec.md]                  | 159.07            | 149.39        | 1.0647968404846375           
 [benchmarks/ecrecover00-siginvalid-spec.k]  | 90.5              | 84.61         | 1.0696135208604185           
 TOTAL                                       | 7348.299999999998 | 7326.02       | 1.0030412147386982

This is likely because the booster can easily discharge the concrete requires conditions when a rule doesn't apply. For example, in this rule

    rule X &Int (Y <<Int Z) => Y <<Int Z
      requires 0 <=Int X andBool X <Int pow256
       andBool 2 ^Int ( log2Int ( pow256 -Int X ) ) ==Int pow256 -Int X
       andBool Z ==Int log2Int ( pow256 -Int X )
       andBool 0 <=Int Y andBool Y <Int 2 ^Int ( 256 -Int Z )
      [simplification, concrete(X, Z), comm]

the first, second, and third condition are all fully concrete always.

@PetarMax PetarMax mentioned this pull request Jan 11, 2024
Merged
@PetarMax @ehildenb
Update kevm-pyk/src/kevm_pyk/kproj/evm-semantics/lemmas/bitwise-simpl…
808e551 
…ification.k

Co-authored-by: Everett Hildenbrandt <everett.hildenbrandt@gmail.com>
@rv-jenkins rv-jenkins merged commit 1c48ff2 into master Jan 11, 2024
11 checks passed
@rv-jenkins rv-jenkins deleted the bitwise-simplifications branch January 11, 2024 23:02
@PetarMax
Copy link
Contributor Author

Just to note, I've noticed a substantial difference in the times when running the test suite with eight parallel processes and with one. Below are the results with PYTEST_PARALLEL=1. The time difference is not statistically significant overall, and the individual tests also show negligible differences.

 Test                                       | master-stats time | PR-stats time     | (master-stats/PR-stats) time 
--------------------------------------------+-------------------+-------------------+------------------------------
[bihu/forwardToHotWallet-failure-1-spec.k]  | 71.02             | 72.65             | 0.977563661390227            
[bihu/functional-spec.k]                    | 62.48             | 63.47             | 0.9844020797227037           
[erc20/hkg/totalSupply-spec.k]              | 28.44             | 28.89             | 0.9844236760124611           
[mcd/flipper-tau-pass-spec.k]               | 81.44             | 82.63             | 0.9855984509258139           
[mcd/vat-mului-pass-spec.k]                 | 66.49             | 67.36             | 0.98708432304038             
[mcd/vat-subui-fail-rough-spec.k]           | 56.86             | 57.57             | 0.9876671877714087           
[functional/evm-int-simplifications-spec.k] | 72.84             | 73.59             | 0.9898083978801467           
[mcd/flipper-bids-pass-rough-spec.k]        | 116.3             | 117.49            | 0.9898714784236956           
[mcd/flopper-cage-pass-spec.k]              | 46.78             | 47.23             | 0.9904721575269957           
[mcd/cat-exhaustiveness-spec.k]             | 90.08             | 90.92             | 0.9907611086669599           
[erc20/ds/transferFrom-failure-2-a-spec.k]  | 60.9              | 61.46             | 0.9908883826879271           
[bihu/forwardToHotWallet-success-1-spec.k]  | 47.76             | 48.18             | 0.9912826899128269           
[mcd/vat-addui-pass-spec.k]                 | 85.14             | 85.85             | 0.9917297612114153           
[benchmarks/storagevar00-spec.k]            | 32.18             | 32.42             | 0.9925971622455274           
[mcd/dsvalue-peek-pass-rough-spec.k]        | 45.6              | 45.92             | 0.9930313588850174           
[benchmarks/bytes00-spec.k]                 | 40.32             | 40.59             | 0.9933481152993348           
[mcd/vat-addui-fail-rough-spec.k]           | 57.52             | 57.9              | 0.9934369602763385           
[erc20/hkg/approve-spec.k]                  | 36.14             | 36.36             | 0.993949394939494            
[erc20/hkg/transfer-success-2-spec.k]       | 41.37             | 41.6              | 0.9944711538461537           
[bihu/forwardToHotWallet-failure-2-spec.k]  | 55.41             | 55.69             | 0.9949721673550008           
[benchmarks/storagevar01-spec.k]            | 36.72             | 36.9              | 0.9951219512195122           
[erc20/hkg/transferFrom-failure-2-spec.k]   | 61.96             | 62.26             | 0.9951814969482814           
[mcd/functional-spec.k]                     | 476.93            | 479.14            | 0.9953875693951664           
[erc20/ds/allowance-spec.k]                 | 37.65             | 37.81             | 0.995768315260513            
[erc20/ds/transferFrom-failure-1-a-spec.k]  | 70.49             | 70.78             | 0.9959027974003956           
[erc20/ds/approve-failure-spec.k]           | 36.79             | 36.92             | 0.9964788732394365           
[erc20/ds/transferFrom-failure-1-b-spec.k]  | 121.75            | 122.16            | 0.9966437459070072           
[bihu/forwardToHotWallet-success-2-spec.k]  | 49.23             | 49.39             | 0.99676047782952             
[mcd/vat-move-diff-rough-spec.k]            | 115.84            | 116.2             | 0.9969018932874355           
[mcd/vat-slip-pass-rough-spec.k]            | 129.51            | 129.91            | 0.9969209452698021           
[benchmarks/keccak00-spec.k]                | 33.35             | 33.44             | 0.9973086124401915           
[mcd/flipper-ttl-pass-spec.k]               | 44.55             | 44.66             | 0.9975369458128079           
[mcd/flipper-addu48u48-fail-rough-spec.k]   | 38.09             | 38.18             | 0.9976427448926141           
[erc20/ds/transferFrom-failure-2-c-spec.k]  | 42.43             | 42.53             | 0.9976487185516106           
[mcd/cat-file-addr-pass-rough-spec.k]       | 43.73             | 43.83             | 0.9977184576773899           
[benchmarks/ecrecover00-sigvalid-spec.k]    | 81.3              | 81.47             | 0.9979133423346017           
[erc20/ds/transferFrom-success-2-spec.k]    | 67.88             | 68.02             | 0.9979417818288738           
[erc20/ds/transfer-failure-2-b-spec.k]      | 38.02             | 38.09             | 0.9981622473090049           
[erc20/ds/transferFrom-failure-1-c-spec.k]  | 89.39             | 89.54             | 0.9983247710520438           
[benchmarks/overflow00-nooverflow-spec.k]   | 31.58             | 31.63             | 0.9984192222573506           
[bihu/forwardToHotWallet-failure-4-spec.k]  | 88.84             | 88.96             | 0.9986510791366907           
[erc20/hkg/transferFrom-failure-1-spec.k]   | 68.51             | 68.6              | 0.9986880466472304           
[mcd/vat-subui-pass-spec.k]                 | 86.27             | 86.38             | 0.9987265570733966           
[erc20/ds/totalSupply-spec.k]               | 33.13             | 33.17             | 0.998794091046126            
[erc20/ds/transferFrom-failure-2-b-spec.k]  | 79.12             | 79.18             | 0.9992422328870927           
[erc20/ds/approve-success-spec.k]           | 47.27             | 47.28             | 0.9997884940778342           
[erc20/ds/transfer-success-2-spec.k]        | 54.74             | 54.75             | 0.9998173515981735           
[bihu/forwardToHotWallet-failure-3-spec.k]  | 67.46             | 67.43             | 1.000444905828266            
[mcd/dai-symbol-pass-spec.k]                | 38.33             | 38.31             | 1.0005220569042024           
[erc20/ds/balanceOf-spec.k]                 | 35.74             | 35.72             | 1.000559910414334            
[erc20/ds/transferFrom-success-1-spec.k]    | 77.92             | 77.87             | 1.0006420958006934           
[benchmarks/structarg00-spec.k]             | 33.96             | 33.93             | 1.0008841732979665           
[erc20/hkg/transfer-success-1-spec.k]       | 43.16             | 43.11             | 1.0011598237067965           
[mcd/vat-muluu-pass-spec.k]                 | 68.7              | 68.62             | 1.0011658408627222           
[erc20/hkg/allowance-spec.k]                | 34.03             | 33.99             | 1.0011768167107973           
[benchmarks/ecrecover00-siginvalid-spec.k]  | 74.73             | 74.64             | 1.0012057877813505           
[erc20/ds/transfer-success-1-spec.k]        | 58.07             | 57.99             | 1.001379548197965            
[erc20/ds/transfer-failure-1-a-spec.k]      | 58.84             | 58.74             | 1.0017024174327545           
[examples/erc20-spec.md]                    | 185.61            | 185.2             | 1.0022138228941686           
[benchmarks/storagevar02-nooverflow-spec.k] | 40.64             | 40.55             | 1.0022194821208386           
[mcd/vat-subui-pass-rough-spec.k]           | 85.7              | 85.51             | 1.0022219623435855           
[erc20/hkg/transferFrom-success-2-spec.k]   | 52.42             | 52.28             | 1.0026778882938026           
[mcd/vat-deny-diff-fail-rough-spec.k]       | 78.27             | 78.06             | 1.0026902382782474           
[erc20/ds/transferFrom-failure-1-d-spec.k]  | 51.44             | 51.28             | 1.0031201248049921           
[mcd/dsvalue-read-pass-summarize-spec.k]    | 41.62             | 41.49             | 1.0031332851289467           
[benchmarks/storagevar03-spec.k]            | 33.7              | 33.59             | 1.003274784161953            
[mcd/end-subuu-pass-spec.k]                 | 41.75             | 41.61             | 1.0033645758231196           
[erc20/hkg/transfer-failure-1-spec.k]       | 46.96             | 46.79             | 1.0036332549690106           
[benchmarks/requires01-a0gt0-spec.k]        | 32.91             | 32.79             | 1.0036596523330283           
[erc20/hkg/transferFrom-success-1-spec.k]   | 57.45             | 57.24             | 1.0036687631027255           
[erc20/ds/transfer-failure-1-c-spec.k]      | 40.37             | 40.22             | 1.0037294878170064           
[benchmarks/staticloop00-a0lt10-spec.k]     | 34.58             | 34.45             | 1.0037735849056602           
[examples/solidity-code-spec.md]            | 133.51            | 132.98            | 1.0039855617386073           
[mcd/flopper-file-pass-rough-spec.k]        | 323.97            | 322.66            | 1.004060001239695            
[examples/storage-spec.md]                  | 135.09            | 134.52            | 1.0042372881355932           
[erc20/hkg/transfer-failure-2-spec.k]       | 45.32             | 45.12             | 1.0044326241134753           
[erc20/ds/transfer-failure-2-a-spec.k]      | 90.41             | 89.99             | 1.0046671852428048           
[benchmarks/encode-keccak00-spec.k]         | 37.32             | 37.09             | 1.0062011323806954           
[benchmarks/staticarray00-spec.k]           | 33.01             | 32.8              | 1.0064024390243902           
[benchmarks/encodepacked-keccak01-spec.k]   | 32.32             | 32.06             | 1.0081097941359949           
[erc20/hkg/balanceOf-spec.k]                | 32.86             | 32.59             | 1.0082847499232892           
[benchmarks/dynamicarray00-spec.k]          | 31.5              | 31.22             | 1.0089686098654709           
[mcd/dstoken-approve-fail-rough-spec.k]     | 75.01             | 74.24             | 1.0103717672413794           
[benchmarks/overflow00-overflow-spec.k]     | 29.73             | 29.41             | 1.0108806528391703           
[benchmarks/storagevar02-overflow-spec.k]   | 40.16             | 39.68             | 1.0120967741935483           
[benchmarks/address00-spec.k]               | 31.58             | 31.2              | 1.0121794871794871           
[benchmarks/requires01-a0le0-spec.k]        | 71.15             | 68.66             | 1.0362656568598894           
[mcd/dsvalue-read-pass-spec.k]              | 48.43             | 45.57             | 1.06276058810621             
 TOTAL                                      | 5905.869999999999 | 5910.149999999997 | 0.9992758221026543

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ehildenb ehildenb ehildenb approved these changes

@anvacaru anvacaru Awaiting requested review from anvacaru

Assignees

@PetarMax PetarMax

Labels
automerge enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@PetarMax @ehildenb @rv-jenkins