Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a flexible template-based test structure #3

Closed
slundberg opened this issue Feb 24, 2022 · 1 comment
Closed

Create a flexible template-based test structure #3

slundberg opened this issue Feb 24, 2022 · 1 comment
Labels
enhancement New feature or request

Comments

@slundberg
Copy link
Contributor

Right now all tests are constrained to have the form Prefix "value1" comparator "value2". While this supports the current set of tests, it is a bit obtuse (odd that the changing comparator changes the meaning of the values) and does not support any test types with more than two values. We don't want to change this after release since it would cause compatibility issues. Here is a proposal for a new system:

Allow for a free form test format with placeholders for values and outputs. Such as

  1. {} should not output {}
  2. {} should output {}
  3. {} should have the same output as {}
  4. {} should be more {} than {}
  5. {} should match {} more than {}
  6. etc.

These can all be called "test_format" instead of "comparator". Scorers will need to support each test format specifically, but need not support all of them if they are not used. The columns value1, value2, etc. match the {} placeholders in order.

@slundberg slundberg added the enhancement New feature or request label Feb 24, 2022
@slundberg
Copy link
Contributor Author

Closing this issue since now AdaTest uses local topic models that implicitly define an expectation (instead of having a set of explicit hard coded expectation types).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant