-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[move-prover][prover-lab] Create a z3 vs cvc5 basic experiment, nuke Rust-Jupyter-Book dependency #8732
Conversation
During review, the correct link to the README for the cvc lab, which includes the rendered SVG, is this: https://github.com/wrwg/diem/blob/bench/language/move-prover/lab/data/cvc/README.md |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice!
.map(|_| ()) | ||
.map_err(|_| "expected number".to_string()) | ||
}; | ||
let cmd_line_parser = App::new("plot") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here, structopt
would make it easy for plot_svg
to take a clean configuration structure (instead of a vector of unparsed arguments). Thanks to the "flatten" option of structopt, the configuration struct can even be equipped with its own reusable command-line parsing logic:
https://github.com/facebookincubator/smt2utils/blob/master/z3tracer/src/main.rs#L18
https://github.com/facebookincubator/smt2utils/blob/master/z3tracer/src/model.rs#L23
(^^ there is only one caveat due to a minor bug in cargo doc)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I'm still using clap, as structopt wasn't ready when I started doing this. Should switch this and other code to it in the future.
/land |
…Rust-Jupyter-Book dependency This PR enables to compare cvc5 with z3 in benchmarks over the whole Diem framework. This is achieved in 3 steps: 1. We remove all dependency from Jupyter Rust notebooks. These turned out to not reliably work. (It was in fact broken at head, but also before fragile and extremely slow.) Instead we provide a new prover-lab command `plot` which generates a .svg file (based on the same refactored plotting logic) which can be included in MD. 2. A new lab is created under `lab/data/cvc`. This lab is intended to benchmark cvc5 vs z3 (later also include vector theories). The output of this lab is found in this [README markdown](language/move-prover/lab/data/cvc/README.md). 3. It turned out that some of the benchmarks lead cvc to not terminate and respect soft timeouts. To innoculate the in-process benchmark infra against such situations, we support now a 'hard timeout' option in the boogie backend. Only with this addition we can now run cvc against the full Diem framework benchmark suite. If cvc does not terminate, we mark the benchmark as error instead of aborting the benchmark run. The older labs aren't yet updated to this new approach, which should be done in subsequent PRs. Closes: diem#8732
Cluster Test Result
Repro cmd:
🎉 Land-blocking cluster test passed! 👌 |
This PR enables to compare cvc5 with z3 in benchmarks over the whole Diem framework. This is achieved in 3 steps:
We remove all dependency from Jupyter Rust notebooks. These turned out to not reliably work. (It was in fact broken at head, but also before fragile and extremely slow.) Instead we provide a new prover-lab command
plot
which generates a .svg file (based on the same refactored plotting logic) which can be included in MD.A new lab is created under
lab/data/cvc
. This lab is intended to benchmark cvc5 vs z3 (later also include vector theories). The output of this lab is found in this README markdown.It turned out that some of the benchmarks lead cvc to not terminate and respect soft timeouts. To innoculate the in-process benchmark infra against such situations, we support now a 'hard timeout' option in the boogie backend. Only with this addition we can now run cvc against the full Diem framework benchmark suite. If cvc does not terminate, we mark the benchmark as error instead of
aborting the benchmark run.
The older labs aren't yet updated to this new approach, which should be done in subsequent PRs.
Motivation
Fix bench-marking infrastructure.
Have you read the Contributing Guidelines on pull requests?
Yes
Test Plan
NA
Related PRs
NA