Need a way to edit or delete test cases that don't make sense, on the fly

[luca-martial]

The idea is that the nature of the tests we carry out are such that they are imperfect and the user should be able to take a look at each test case to make sure it is testing what it claims to be testing. If the user spots something a problematic test case (transformed sentence), they should be able to delete that row. This would happen before running the tests basically, and if they only notice it after running the tests, they should be able to come back to the test generation phase and make that edit.

The approach to do this is up for discussion - it could be either integrated in the save/load workflow or elsewhere

[luca-martial]

@JulesBelveze see here an extract from the blogpost explaining the editing functionality:

The meaning of “test case” and “expected result” depends on the test type, but should be human-readable in each case. This is so that after you call h.generate() you can manually review the list of generated test cases, and decide on which ones to keep or edit.

Since the table of tests is a pandas data frame, you can also edit it right within your notebook (with Qgrid) or export it as a CSV and have a business analyst edit it in Excel.

While automation does 80% of the work, you usually will need to manually check the tests. For example, if you are testing a fake news detector, then a replace_to_lower_income_country test editing “Paris is the Capital of France” to “Paris is the Capital of Sudan” will understandably yield a mismatch between the expected prediction and the actual prediction.

You will also have to validate that your tests capture the business requirements of your solution. For example, the FairnessTestFactory example above does not test non-binary or other gender identities, and does not require that accuracy is near-equal across genders. It does, however, make those decisions explicit, human readable, and easy to change.