Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a type for complex input generation that performs exploratory testing #25

Open
SeanROlszewski opened this issue Sep 25, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@SeanROlszewski
Copy link
Contributor

SeanROlszewski commented Sep 25, 2023

Description

In order to support testing patterns where randomized input generation is used to explore a program's state space (such as fuzz, differential, property-based, or mutation testing) we need to add some affordance to developers to specify the following:

  1. The next value to pass into a @Test
  2. A way to receive the prior value and test result as a result of executing @Test
  3. A way to receive additional information, such as changes in code coverage, from the last @Test
  4. A way to stop input generation based on some heuristic or predicate.

Having these 4 items would be the essential set of primitives to really streamline the creation of tools like QuickCheck, libFuzzer, etc. with Swift Testing.

Perhaps this could take the form of a protocol, potentially named TestCaseInputGenerator, that has the following shape:

protocol TestCaseInputGenerator {
    associatedtype AdditionalContext
    func shouldStop<InputType>(given priorRun: (InputType, TestResult)?, additionalContext: AdditionalContext?) -> Bool
    func nextValue<InputType>(given priorRun: (InputType, TestResult)?, additionalContext: AdditionalContext?) -> InputType
}

AdditionalContext can encapsulate data of interest to the test; previously tried values, their results, and other data like code coverage, crashes, etc.

I'm filing this issue rather quickly, in between tasks, so apologizes in advance for oversights on the type signatures and compilability of the proposed API. :)

@SeanROlszewski SeanROlszewski added the enhancement New feature or request label Sep 25, 2023
@SeanROlszewski SeanROlszewski changed the title Add a type for complex input generator that's perform exploratory actions Add a type for complex input generator that can perform exploratory testing Sep 25, 2023
@SeanROlszewski SeanROlszewski changed the title Add a type for complex input generator that can perform exploratory testing Add a type for complex input generation that can perform exploratory testing Sep 25, 2023
@SeanROlszewski SeanROlszewski changed the title Add a type for complex input generation that can perform exploratory testing Add a type for complex input generation that performs exploratory testing Sep 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant