Failed community solutions false positives

[552020]

After the tests of the Robot Simulator challenge has been fixed and I was able to pass it, I took a look at the community solutions. Since the exercise has been updated most of them were marked as 'outdated' (which is comprehensible) but since a lot were also marked as 'failed' I copied and tested one, this one by @ErikSchierboom in my editor and actually it passed all tests.

[jiegillet]

I can confirm, Erik's solution does seem to pass the tests. It's great solution actually :)

Marking community solutions as outdated and failed is a site-wide process (and I don't know how it works), so there is not much the Elm track can do about that.

@ErikSchierboom maybe this issue should be transferred to exercism/exercism?

[ErikSchierboom]

Well, they are failing because each exercise is linked against a specific exercise commit. To get them passing, students need to update their exercise to the latest version.

[mpizenberg]

@ErikSchierboom so Outdated also implies Failed right? Is there a way to distinguish outdated solutions that were passing and outdated solutions that were failing?

[ErikSchierboom]

@mpizenberg Nope, they're not the same. Outdated means that there is a newer version of the exercise available, but it doesn't mean that is must be failing. The newer version could just have updated instructions.

[ErikSchierboom]

Is there a way to distinguish outdated solutions that were passing and outdated solutions that were failing?

The community solutions page has three checkboxes with which you can filter the solutions, including outdated and passing the tests (not the reverse).

[mpizenberg]

@mpizenberg Nope, they're not the same. Outdated means that there is a newer version of the exercise available, but it doesn't mean that is must be failing. The newer version could just have updated instructions.

So it's not only the exercise commit that is verified but also the things that changed? If the tests file changed it's also marked Failed but if it's only docs, it's only marked Outdated is that it?

Or we might not be talking about the same thing when saying "exercise commit". Are you referring to the "last commit in which the exercise tests were changed", which is different than the "latest track commit when the exercise was started by the student"?

[mpizenberg]

In any case, it seems that's a question better discussed on the main exercism repo or in slack so I'm closing this issue. Thanks again for the participants.

[552020]

@mpizenberg are you going to open the issue in the main exercism repo or should I? The slack you talk about is a public slack or an internal one?

[mpizenberg]

@552020 I mean the official public slack. But you might be right, actually I don't remember and it's possible it's only for mentors and maintainers. Let me ask there, and answer you when that's clear to me

[mpizenberg]

@552020 So I figured out what is happening by talking to my rubber duck. Erik's solution was submitted at a time where the exercise structure was different. Basically he had a Exercise.elm file and now we have src/Exercise.elm files. When website testing was introduced with v3, the tests ran with the template src/Exercise.elm file and so the tests failed with the todo still being in the template file.

This error will happen for all solutions submitted previous to november 2019 when I updated the track structure for elm 0.19.1