Skip to content

v0.7.1: the same code gives the same answer

Latest

Choose a tag to compare

@Yonkoo11 Yonkoo11 released this 20 Sep 22:47
· 101 commits to main since this release
d5bf8f7

This is a defect fix, and the defect was reproducibility.

Three runs of the same benchmark corpus, on the same commit, reported 523, 518 and 517 sightings. The findings the two-witness rule publishes did not move, but the count underneath them did, and on one file an accounting finding appeared in one run and not in the next.

The cause is in _reads, which reports the variables a line mentions without writing. It reported only the first one, chosen by iterating a set, and Python randomises set iteration order per process. So require(balances[msg.sender] >= MinDeposit) reported balances in one run and MinDeposit in the next. payout-lens intersects what a function reads with what it writes, so which name it happened to see decided whether there was a finding at all.

Every match on a line is reported now, in a fixed order. Three runs under three different hash seeds produce byte-identical benchmark files on all three corpora. Two tests pin it, one on the reading itself and one on a payout that lowers the balance it checks.

A note in bench/README.md had already recorded one instance of this and put it down to a lens change. That was wrong, and the history table now says so.

Nothing this project publishes moved. SmartBugs-curated stays at 48% precision and 64% recall, DeFiVulnLabs at 62% and 50%, the recent audit contests at 29% recall with the same two findings. The any-lens count settles at 513, because the duplicates it had been counting were never distinct findings.

Also here, from reviewing the lens released in v0.7.0:

  • the one-shot test read the whole function header when a function carried no modifiers, so a parameter named initializer would have silenced the lens on that function. It reads the modifier list now.
  • the caller test read only a function's own body while the guard test read the helpers it calls. Both read the helpers now.
  • the lens built two write-maps for every function when each is needed for one role. The second is now built only for the function being judged, and only once it is known to carry no guard.

The ecosystem scan also runs projects side by side. Serial it managed four projects in three and a half minutes, which is about twenty-two hours for the listing it reads. Every repository already had its own clone and its own memory file, so nothing about the reading changed.

83 tests.