Build Test Runner

[ErikSchierboom]

This issue is part of the migration to v3. You can read full details about the various changes here.

In Exercism v3, one of the biggest changes is that we'll automatically check if a submitted solution passes all the tests.

We'll check this via a new, track-specific tool: the Test Runner. Each test runner is track-specific. When a new solution is submitted, we run the track's test runner, which outputs a JSON file that describes the test results.

The test runner must be able to run the tests suites of both Concept Exercises and Practice Exercises. Depending on the test runner implementation, this could mean having to update the Practice Exercises to the format expected by the test runner.

Goal

Build a test runner for your track according to the spec.

If you are building a test runner from scratch, we have a starting guide and a generic test runner that can be used as the base for the new test runner.

If a test runner has already been built for this track, please check if it works on both Concept Exercises and Practice Exercises.

It can be very useful to check how other tracks have implemented their test runner.

Tracking

exercism/v3-launch#4

[davidbruce]

@mfoda Are you willing to work on this?

I've opened an issue for getting the repo created here, but you can get started by forking https://github.com/exercism/generic-test-runner.

[mfoda]

@davidbruce Definitely! I'll get started on that.

Also might be a good idea to discuss over at the exercism slack workspace to plan the rest of v3 tasks and see who else might be interested in helping out. Currently there's a #track-haxe channel there but we should create a #maintaining-haxe one as well (you can ping Erik for an invite).

[mfoda]

Just pushed a fairly minimal implementation to get that started. It is however missing the extraction of command/test_code from the source code. I'm not sure what's the better approach to tackle this and I suspect it might overlap/benefit from work done on the analyzer/representer for code parsing.

[mfoda]

@davidbruce At this point the test runner passes all the generic tests. A few things I could use some help with:

• The Docker/CI scripts and setup as I'm not very familiar with that part
• Which target to pick for fast compilation and returning test results to the user. I was told either the Lua or Hashlink targets but those would probably complicate the Docker image setup a bit
• Any general ideas/suggestions for improving either the test suite or the runner

[BNAndras]

The test runner is built so I think once we remedy exercism/haxe-test-runner#19, this can be closed.