check tests against the config.json

[behrtam]

As discovered by @oalbe while working on the test runner (check-exercises.py #286) it would be good if the actual executed tests are checked against the listed exercises in the config.json. Maybe we could also stop running tests for deprecated exercises.

[kytrinyx]

Yeah, that would be great. Configlet checks that the configured problems all exist, but this sounds like it would solve a different problem.

[oalbe]

I think I did this. I mean, I modified the check-exercises.py script so that it now only runs tests if they are in the 'problems' key of the config.json file and prints something in other cases.

What I was wondering is: is it okay to submit a PR even when one could use some peer-review first? (like I do in this case)

P.S.: I've seen that in the config.json file there is an 'ignore' key to tell it to ignore non-exercise folders. I think it's not needed anymore since now the exercises are in their own folder.

[kytrinyx]

is it okay to submit a PR even when one could use some peer-review first?

Yepp, that's the perfect use case for a PR. Sometimes we open them when we just want to have a discussion about something and it's easier to show what we mean than to describe it.

P.S.: I've seen that in the config.json file there is an 'ignore' key to tell it to ignore non-exercise folders. I think it's not needed anymore since now the exercises are in their own folder.

Yeah, once all of the tracks have moved everything to exercises/ then I'll remove that and update configlet to stop checking.

[oalbe]

Alright, so I will do the PR. Let's see how it goes.