Include journey test in CI build to ensure clients can fetch and build exercises

[jtigger]

As a track maintainer,
I want the CI build to also stand-up a local instance of exercism and fetch exercises through the client
So that I'm confident that end-to-end, my changes don't break the exercism experience.

---

Notes:

• look at README.md for full setup instructions. This should be fun as it requires a Ruby install as part of the build (possible on travis?)

[kytrinyx]

+1000

But... it requires a ruby install? How so?

It seems like it should be enough to download and unpack the linux version of the exercism binary, and then loop through each exercise in the config.json and say exercism fetch java SLUG, no?

The default is to use http://x.exercism.io, and you can fetch individual exercises without having an API key.

[jtigger]

Well, but that setup would be using the deployed set of problems. This would be an attempt to catch defects introduced in the latest commit to the track.

That said, today the client doesn't filter much (just paths with the work "example" in them). Is that something that's likely to change in the near/mid future?

Perhaps it would be better to simply build each exercise individually (disconnected from their parent); that would have caught this issue.

[jtigger]

... looking at the next layer of this onion, "success" would be a little hard to detect: some exercises, when fetched correctly do not compile (because we stop providing a shell of a solution, altogether a few steps down the progression).

I'll be chewing on this.

[kytrinyx]

How about...

• copy the individual exercise
• rename/move example solution
• build

?

[jtigger]

Picking this up as a morning exercise with a couple of folks from work (started with @bronhuston).

The basic plan is to replay the sandbox workstation setup in Travis, fetch, "solve" (copying the example), and test all problems. Not only does this check the validity of the test+example combination, but the plumbing too. When in place, committers have the luxury of focusing on the quality of the submission and not worry whether merging in will break production.

[jtigger]

As a matter of update, this group has nearly finished this work. It's functional on a fork, but needs just a few nips-and-tucks to make it merge-worthy.

[jtigger]

At this point, we've met the spirit of this story.

[jtigger]

Ooops! Forgot to update the README.md accordingly... almost done. ;)

[jtigger]

Okay; docs in place. Done.