Test-before-submit feature

[canweriotnow]

While working on https://github.com/canweriotnow/exercism-emacs I got to thinking about how easy it is to run ERT on elisp exercises, and how trivial it would be to add a run-test hook before the submit action on exercism.el

Then I got to thinking, since most tracks (sadly excluding my own, i.e. xelisp and xscheme, for the moment) have test runner scripts in their repos, perhaps this could be a useful addition to the CLI as well; I grok some tracks like to do one-at-a-time testing (xruby leaps to mind), requiring hte user to uncomment tests after each feature is implemented, but I don't see this as a downside; I'm thinking something like this (I'll just mock the CLI here, obvi Emacs interaction would be a little different):

# exercism submit some-ex/some-ex.el

=> tests passed, submitting...

(or)

# exercism submit some-ex/some-ex.el

=> 1 of 3 tests failed, not submitting

(or)

# exercism submit -f some-ex/some-ex.el

=> 1 of 3 tests failed, submitting anyway

The underlying premise being that the exercism client (Emacs plugin or CLI) could run the test properly, so we would also get:

# exercism check some-ex/some-ex.el

=> 3 of 3 tests passed

And so on...

@kytrinyx this is most of what I wanted to chat about last week, as I'm thinking through tooling...

Anyhow, thoughts on the utility/practicality of this approach would be appreciated... I think even declaring a testing procedure specification in the config provided by the API might do it, rather than hard-coding it into the client... still in hammocking mode on this.

[kytrinyx]

@canweriotnow Sorry it's taken me so long to get back to you. I'm still traveling, and expect to catch up with a bunch of open issues and discussions on Exercism next week.

[Insti]

I do think that tracks should make running tests as easy as possible for the students.

I don't think test running belongs as part of the Exercism cli.

Disclaimer: I'm an xruby maintainer and a testing extremist.

[petertseng]

I do think that tracks should make running tests as easy as possible for the students.

Yeah I like that. Part of what I realized I was looking for in #117 and looking for tracks' various discussions on the subject.

perhaps this could be a useful addition to the CLI as well

There are some interesting problems to solve when incorporating tests into the CLI, the most prominent being: given a language track and an exercise directory, how does the exercism binary know how to run the test?

• A hard-coded rule for each language that the CLI is made aware of at compile-time
  • Disdavantage: Students will have to update the entire CLI when a new track is added or an existing track changes its way of testing, else they will not be able to run the tests (maybe the CLI could fall back to "run no tests, just submit")
  • Disdavantage: The CLI maintainers will have to release a new CLI whenever a new track is added, for similar reasons
• A rules file containing rules for each language that the CLI is made aware of at run-time
  • Open question: But where will this rules file be placed? How will exercism CLI know where to look for it?
  • Open question: What will the specification language for this rules file look like?
  • Disdavantage: Students will have to update this list when a new track is added or an existing track changes its way of testing, else they will not be able to run the tests.
  • Open question: And what exactly is the update mechanism for this list?
• Other idea that I am missing

[kytrinyx]

I've added a note to the CLI redesign issue, and I'm also tracking this as a question for the nextercism prototype. I'm going to go ahead and close the issue.