Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support generating test running code when using dafny translate #3818

Closed
keyboardDrummer opened this issue Mar 29, 2023 · 4 comments · Fixed by #4890
Closed

Support generating test running code when using dafny translate #3818

keyboardDrummer opened this issue Mar 29, 2023 · 4 comments · Fixed by #4890
Labels
kind: enhancement Enhancements, feature requests, etc. These are NOT bugs, but ways we can improve Dafny

Comments

@keyboardDrummer
Copy link
Member

keyboardDrummer commented Mar 29, 2023

Summary

Add an option --with-test-runner to dafny translate so that it generates a main method that runs all the tests, similar to what dafny test does.

Background and Motivation

Currently Dafny programs can run tests using dafny test. However, for Dafny programs that use external code with external dependencies, it might not be possible to compile them using dafny test.

Proposed Feature

See summary

Alternatives

Alternative 0: by default, let dafny translate generate a runTests file that contains a main method that will run the tests. Generating annotations for a particular target language testing framework, such as is done for C# and XUnit now, will only be done when using a back-end specific option.

Alternative 1: Increase the functionality of dafny test so that is support builds with external code and external dependencies.

Alternative 2: enable dafny translate to always generate code for tests that can be executed using a target language testing framework, like is currently done for the C# back-end which generates XUnit tests.

@keyboardDrummer keyboardDrummer added the kind: enhancement Enhancements, feature requests, etc. These are NOT bugs, but ways we can improve Dafny label Mar 29, 2023
@keyboardDrummer
Copy link
Member Author

Related PR that adds support for generating translates tests for Java: #2395

@keyboardDrummer
Copy link
Member Author

keyboardDrummer commented Mar 29, 2023

I'm in favor of alternative 0 because it scales well. Alternative 2 relies on the subjective choice of which testing framework to choose, and adds extra work for each back-end.

@davidcok
Copy link
Collaborator

Note that #3769 has already implemented Alternative #1 in response to a different customer request (namely to add the functionality to dafny test).

@robin-aws
Copy link
Member

I'm more in favor with the --with-test-runner option than alternative 0: emitting a main method by default would mean you can't easily use dafny translate to generate code in a project that needed its own main method. Especially since dafny translate doesn't currently know whether you're translating tests or not, and I think it would be a bit surprising/brittle to dynamically guess at this based on whether the Dafny source has any {:test} methods.

In addition, the C# compiler currently adds [Xunit.Fact] attributes and some code to check IsFailure on results, unless /runAllTests is provided. That means dafny translate cs currently always does that, and we should have an explicit option to stop it so that we're at least not introducing the dependency on xUnit.

keyboardDrummer added a commit that referenced this issue Dec 20, 2023
Fixes #3818
Fixes #4888

### Description
Add the option `--include-test-runner` to `dafny translate`, to enable
getting the results of `dafny test` when doing manual compilation.

### How has this been tested?
Updated `RunAllTestsOption.dfy` so it uses this new feature.

<small>By submitting this pull request, I confirm that my contribution
is made under the terms of the [MIT
license](https://github.com/dafny-lang/dafny/blob/master/LICENSE.txt).</small>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind: enhancement Enhancements, feature requests, etc. These are NOT bugs, but ways we can improve Dafny
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants