Add Cask and ert-runner. Tests refactoring.

[sviridov]

Hi!

This commit adds support for Cask and changes some test related stuff.

Cask and ert-runner

Cask is a project management tool for Emacs Lisp to automate the package development cycle.

Installation

$ curl -fsSL https://raw.githubusercontent.com/cask/cask/master/go | python
$ cat >> ~/.bashrc
export PATH=~/.cask/bin/:$PATH

Examples of usage

$ cd path/to/emacs-elixir
$ cask install # install ert-runner
$ EMACS=`evm bin emacs-24.3-bin` cask install # install ert-runner for Emacs 24.3
$ cask exec ert-runner # run tests
$ cask exec ert-runner -t 24-3 # run tests with 24-3 tag
$ EMACS=`evm bin emacs-24.3-bin` cask exec ert-runner -t 24-3 # run tests with 24-3 tag for Emacs 24.3
$ EMACS=`evm bin emacs-24.3-bin` cask exec ert-runner -t 24-3 --verbose # run it with elixir-smie-verbose-p=t
$ cask exec ert-runner --win # run tests with GUI window

Some other changes

• run_tests script was removed.
• All the tests have been redefined with elixir-deftest.
• elixir-mode-run-tests was removed.

[mattdeboard]

Great contributions! I will probably not be able to get to fully reviewing & merging this until late tomorrow (so about 36 hours maybe). But it looks awesome.

Refs #88

[mattdeboard]

Ok one issue here is that when there's a test failure, I want to see the Actual vs. Expected buffer. If I force a deliberate failure (by deleting a few letters or moving spaces around, etc.) and run cask exec ert-runner --win, it opens up in Emacs but it doesn't actually give me a chance to inspect the errors, which are crucial to sorting through this.

A must for me as I'm working through issues is being able to run failing tests in a fresh copy of emacs, and this setup seems like a step backward in that regard. Am I doing something wrong or missing something?

That aside I like this setup MUCH better. The only thing I'd change is making a rake test wrapper around cask exec ert-runner --win to run the tests (it's easier to remember). But that seems like a separate pull request. Really appreciate your work on this.

[mattdeboard]

Due to the nature of these changes, the excellent documentation you've included with this pull request should be added to CONTRIBUTING.md.

Also, another thing that needs to be added to that document is the steps for installing evm.

[mattdeboard]

After consulting with people in #emacs, I think the answer is that falling back to running tests interactively isn't a thing you can do with ert-runner, which is unfortunate.

I really want a way to fall back automatically to ert-run-tests-interactively automatically.

[mattdeboard]

So I don't think &define is a valid keyword here. I know you didn't add it, but I'm hacking on ert-runner.el to see about falling back to an interactive test session.

When I run ert-runner (after adding (setq ert-runner-exit t) in test-runner.el), I get the following error:

matt@matt-x230:/opt/emacs-elixir$ cask exec ert-runner --verbose
Running tests on Emacs 24.4.50.2
Warning: Unknown macro property &define in elixir-deftest
Wrong type argument: stringp, nil

I've looked around for this and from what I can tell this param is only accepted in Edebug specifications. cl-defmacro only takes &rest, &optional, and &key. I'm not sure the genesis of this macro definition but I don't think it's right.

Again, I know this isn't part of your code necessarily (except the refactor) but I think a bug is being masked here. No action expected on your part, this is a note to my future self (or other contribs).

[mattdeboard]

Or maybe I just don't know what I'm talking about.

[mattdeboard]

Ok I have spent a few hours with this stuff here and fetting familiar with commander, ert-runner, evm, etc., as well as the refactored code you've included.

The test refactoring should be a separate pull request from the evm-related items. There are a couple bugs in the refactored code, which makes it difficult to tell how to start sorting out how to extend the implementation you have included here.

[sviridov]

I really want a way to fall back automatically to ert-run-tests-interactively automatically.

I will try to find solution for this.

Due to the nature of these changes, the excellent documentation you've included with this pull request should be added to CONTRIBUTING.md.

Yep!

The test refactoring should be a separate pull request from the evm-related items.

Ok, I will close this PR and make two new. But it will take some time (about 1-2 days maybe).

[mattdeboard]

Thanks. I know it's kind of a pain but I think (hope) it will pay off in terms of maintainability down the road.

[sviridov]

@mattdeboard Good news! I was able to run test interactively with --win key.

[mattdeboard]

Oh? How? On my machine, emacs closes down immediately once the tests complete.