Request: no 500-completions-lock for katas with unresolved issues

[MMMAAANNN]

Justification:

Currently, if kata reaches 500 completions, its test cases become read-only. However, there are lots of katas that have long-standing issues due to inactivity of author/contributors, and in this case once they reach 500 completions, those issues may become permanent.

[bkaestner]

Here's my feature request I've submitted three months ago, just in case will be gone (see #130):

Make tests editable even after 500 warriors have solved them, if and only if the new tests don't invalidate the old solutions.

Reason

Better error messages

Sometimes, a user uses Test.expect instead of Test.assertEquals or similar. This leads to bad error messages, see reverse words. The proper test would use assertEquals and would not change the semantics of the tests at all.

Similarly, the arguments are sometimes swapped, e.g. C# uses expected, actual, whereas JavaScript/Ruby/Python use actual, expected. Translators or kata authors don't notice this all the time, which leads to confusing error messages.

Wrong tests

Hands down, I've messed up the translation of is a number prime, where I accidentally added abs to the reference solution. I've added a "bug" note so that people know that they need to use abs, but it would be much better if one was able to simply change the tests so that they only take positive numbers into account and/or check for the current behaviour and the correct one, e.g.

it "should work for negative numbers" $
  property $ \x ->
    oldProperty x .|. newProperty x

it "should work for random examples" $ 
  property $ \x ->
    oldSolutionProperty x .|. newSolutionProperty x

Even though this tests still let negative primes pass, they also let correct solutions pass.

There are other examples where it would be helpful to change the tests afterwards, for example the tree traversal kata, where the built tree is simply wrong in Haskell (the comment mentioning that has more than 20 upvotes).

Implementation

After a change to a 500+ kata, the test cases become locked and get checked asynchronously against the solutions (we're using the solutions as tests for the new tests). If this fails for a solution A, check whether A passes the old tests. If it does, refuse the new tests. If A fails also on the old tests, mark it as non-passing and continue. If all tests pass, notify the kata author, commit the changes, but unlock the tests only after the necessary time for the check has passed twice (e.g. 10 minutes to pass all tests => 30 minutes locked total (10 mins tests + 20 additional)). This test checking should have low-priority compared to other jobs, so it could take much more than the mentioned 10 minutes.

---

EDIT: A copy of the "pull request" idea for katas for completeness:

At the moment, an issue is more or less a statement together with a flag. While some of us try to provide a solution for the problem, most issues are only single-line statements, followed by a long discussion.

This is similar to the GitHub issues. And translations are somewhat similar to pull-requests. But why stop there? Introduce kata-fixing pull-requests, which can be accepted by a moderator, a contributor or the kata author himself. They would probably have another name, like "kata-fix", but the essential part is that they provide working, testable code.

Issues like #123 could be handled that way easily. A +500 kata is broken, the original author is inactive and you're not a contributor? File a kata-fix. You don't have enough honour to edit the kata yourself yet? File a kata-fix. You have a great idea for the description and want to share it with the author? File a kata-fix.

[GiacomoSorbi]

+1 or, alternatively, allow users with 5000+ honour to edit it directly or pull an edit for an admin to approve/reject: currently there are like 15 users with 5000+ points and (except the underwritten of course) they are surely more than capable to improve a kata in need.

[javatlacati]

migrated! https://codewars.ideas.aha.io/ideas/CW-I-93

[jhoffner]

You can now edit any language which itself does not have 500 completions (its been possible for a few days now). Does that satisfy this issue enough to close it?

[javatlacati]

Sure as long you please don't close #163 🎅

[ghost]

+1