Authoring guidelines: submission tests

[Blind4Basics]

Here we can discuss more in detail what should be added there. I'll store some ideas at the same time.

First, a lot of things when it'll come to explain how to write "good tests" will be pretty generic. So I believe we might actually writte that generic version first, listing there some of the usual traps to avoid.

Then a more precise guide, python specific.

Generic guide:

Here are some ideas to use as a stump:

structure of the kata/tests suites

• always fixed and random tests (explaining why: not usual TDD practice, needed to enforce the implementation/avoid cheats).
• split the test suite using meaningful names for the different blocks
• write the assertions inside the dedicated structures (groups/blocks/methods) rather than at the base level (when it exists in the language)
• one day you'll maybe stop coming on CW, so other people will have to maintain your kata. So think about them and try to write your best code for your tests. DRY, no functions with tens of line, ...
• explain how the test framework is working (generally speaking), printing the results/data needed for the feeback right to the stdout, meaning that the user can/will print in the middle of all of this. Hence the need to:
  • never do several calls to the user's function then do all the tests later. That makes the debugging a pain because the user cannot track hat assertion is related to waht call to his function
  • when possible and adapted to the test framework you use to have one single call to the user's function per test method.

tests feedback

• be very carful about the assertion messages: they have to be precise and useful. "False should be True" almost always trips up newbies. General guidline: it's good to show the input, either in the assertion message, or in the name of he test (if the test framework allows to name the tests). Like: "28 is not prime. Answer should be False, but solution returned True." would be much better (hobo ; error256).

fixed/sample tests specificities:

• having only 2 fixed tests before the random tests is generally a very bad idea
• do not hide edge cases in the test cases part, provide them int the sample tests too (hobo).
• You can use all the fixed tests of the test cases part as sample tests. But if it's very long, it's not always necessary. But when you remove some of the fixed tests, try to keep at least one meaningful test for every kind of situation (hobo)

random tests specificities:

• for the random tests: prefer, when possible, to avoid to have a reference solution => design the tests so that either you know the answer right at the beginning when the input is generated if possible (hobo)
• performances oriented katas: when the random tests use inputs/outputs that are very hard to debug (strings of hundreds of characters or more, huge arrays, multidimensionnal arrays, ... anything that akes the assertion message unreadable!), split the random tests in 2 sections or more, with a first batch of small random inputs. (error256)
• the "mutation of the input" problem takes all its meaning in the random tests (give examples):
  • why it's bad in general
  • why it's bad on the user side (crazy feedback)
  • why it's bad for the tests (cheatable)

miscellaneous

• don't rely on the random tests to enforce the requirements: all possible edge cases should be exhaustively tested in the fixed tests. As far as possible. When not possible, they should be added later if a user encounter one of them et raise an issue about it. (error256)
• This do not apply to all framework, but keeping it as a habit/default behavior might avoid a lot of troubles to anyone:
  • do never compute directly the expected result with a formula in the same statement than the one calling the user's function (raising an exception reveals the solution) -> is that python specific? (I guess not...)
• all rquirements that are announced have to be tested (hobo). Like, if the user isn't supposed to use BigInt, the author has to enforce it, etc. Adding a note: "Keep in mind that when you end up with this kind of idea, you're generally about to open the pandora's box. Especially in languages that are too much flexible, like JS, pyhton or ruby. So you actually should try to avoid restrictions that may be very hard to ensure. Number of lines or length of the code, on the other hand, is not problematic (code golfing)".

general infos about the guide itself:

• "this guide is still generic, so it might not cover all cases or even not be appropriate for some languages whose framework works in a different way (QuickCheck for Haskell, for example). Please refer to the documentation of your language and/or ask for help to other experienced users." (error256)

Additional sources

• interesting ideas to gather from here (hobovsky's "kata authoring guidelines" draft)

---

python specific guide:

• always put the assertion methods inside a it block rather than at the top level or in a describe block, especially for the random tests. This way, when the tests are passed, the user doesn't end up with a wall of Test Passed! in the output panel (in addition, this allowes to see the timings for the block(s) without the need to scroll all the way down.
• insist one the mutation of the input trouble, because of the order of actual and expected in the assertion methods => always compute the expected value first is the best idea to get

per language:

(or not!? B4B)
add snippet explaining how to access the solution of the user (length/chars constraint)

---

Feel free to dump some ideas.

[Blind4Basics]

first corollary question: shoud the generic guide actually be autonom or a part of the guide about creating a kata?

[hobovsky]

first corollary question: shoud the generic guide actually be autonom or a part of the guide about creating a kata?

I think it should be a part of "making a kata". I am not sure if one is going to create tests (in a way described there) outside of this scope.

What I think would be good to have there:

• some brief explanation why random tests are important in context of CW. Random tests are not common TDD (anti-)practice and I've seen people being surprised by the fact that such thing is used.
• some guide to prefer known answers to reference solutions. Practice of running reference solutions in tests is widespread and in some cases totally unnecessary. Quite often it's possible to generate an input with the answer known upfront.
• a guideline to make sample tests representative - I think that currently sample tests meet with real disregard and are used just as a kind of teaser at best. In my opinion, more attention should be brought to having sample tests which will be smaller in amount than full test suite, but will cover just as many cases (if feasible, that is). [... 10 minutes later...] Okay I know about TDD theme around kata, but 1) I am not sure if it still holds, and 2) people who are not familiar with unit tests do not know how it works
• advice on assertion messages - these are often not clear for people not experienced with unit tests. "False should be True" trips up (almost) every newbie. I think something like "28 is not prime. Answer should be False, but solution returned True." would be much better.

do never compute directly the expected result with a formula in the same statement than the one calling the user's function (raising an exception reveals the solution) -> is that python specific? (I guess not...)

It's not exactly python specific, but for some languages this problem manifests differently or under slightly different circumstances. C, C++ and NASM use macros as assertions, and assertion message shows direct expressions being passed as arguments:

Value of expression __secret_referenceSolution_32587654(arg1, arg2) is not equal to value of expression Kata::solveProblem(arg1, arg2) - or something similar.

[error256]

• It should be clear that the guide covers most common cases and there can be exceptions whenever it's reasonable.

It's not only for non-standard kata ideas, but also for simpler things like... For example, with some libraries like Haskell QuickCheck it should be OK not to have fixed tests if the only edge cases are like the minimum and maximum numbers of the input range. As long as an author can justify what they're doing, it should be OK.

when the random tests use inputs/outputs that are very hard to debug (strings of hundreds of characters or more, huge arrays, multidimensionnal arrays, ... anything that akes the assertion message unreadable!), split the random tests in 2 sections or more, with a first batch of small random inputs.

Splitting the tests may be a good thing when there are performance requirements, but otherwise with proper fixed tests one shouldn't need to look at the randomly generated input because at least one fixed test should fail before that.

do never compute directly the expected result with a formula in the same statement than the one calling the user's function (raising an exception reveals the solution) -> is that python specific? (I guess not...)

Not Python-specific, but there are quite a few languages where the line isn't shown too.

• Custom assertion failure messages? Sometimes input can be a header of it, but it's not always possible. Ideally, it should be the tests that print input, not the user. Only the testing code can print the input only for failed test cases.

[Blind4Basics]

It's not exactly python specific, but for some languages this problem manifests differently or under slightly different circumstances. C, C++ and NASM use macros as assertions, and assertion message shows direct expressions being passed as arguments:

-> but is that a problem, to see those names? (I bet you can then call the ref solution?)

edit: 'added info and reformatted the original message

[hobovsky]

I would add a part about test cases checking some additional constraints, if present. For example, if kata says "do not use X", then this restriction should be either enforced by kata setup, or by assertion that it was not broken.

Edit: maybe a note about how difficult or pointless such restrictions are would be also helpful :)

[Blind4Basics]

added.

I added the info about getting the solution of the user (must be done "per language", there). About that: might be better to write a single generic article, explaining the traps (especially, when/where to put the check), and then just provide all the snippets at the bottom of the page. This sounds far more reasonnable to me.

opinions?

[Blind4Basics]

I added the info about getting the solution of the user (must be done "per language", there). About that: might be better to write a single generic article, explaining the traps (especially, when/where to put the check), and then just provide all the snippets at the bottom of the page. This sounds far more reasonnable to me.

opinions?

up!