Consider using a test framework

[joshbax189]

Adding new tests involves touching a lot of files, and test output is quite noisy-- it's hard to see the results within the traces.
It would be preferable to just add the test script and have it included in a test run, then have the results reported in a simple table with traces only shown for failures.

We could use a test framework for either shell, js or elisp since all of those environments are available in the CI images.

We should consider: ease of migrating to the framework, ease of local usage (running and developing), what tests are supported, and format of test reports.

Bash

Using bash test tools, tests look like

# A simple test for the "find" executable
source bash_test_tools

WORK="/tmp/work"

function setup
{
  mkdir -p "$WORK"
  cd "$WORK"
  touch some_file.txt
}

function teardown
{
  cd
  rm -rf "$WORK"
}

function test_find_local_directory
{
  # Run
  run "find ./"
  # Assert
  assert_success
  assert_output_contains "some_file.txt"
}

testrunner

Using bats, tests look like

#!/usr/bin/env bats

@test "addition using bc" {
  result="$(echo 2+2 | bc)"
  [ "$result" -eq 4 ]
}

@test "addition using dc" {
  result="$(echo 2 2+p | dc)"
  [ "$result" -eq 4 ]
}

The major advantage of using bash is that we are testing shell commands, shell output and exit status. We could also reuse the existing test scripts. For now all the test use cases, like checking status and output, are covered.

Disadvantages would be no automatic updates for the test framework, possibly worse dev experience than elisp or js, and maybe problems writing more complex tests using bash.

JS

Could use Jest with something like jest-shell-matcher (it's deprecated), but the nodejs exec command works well:

const process = require("node:process");
const util = require("node:util");
const exec = util.promisify(require("node:child_process").exec);

describe("analyze", () => {
  describe("dsl", () => {
    const cwd = "./test-js/dsl";
    it("handles plain text", async () => {
      const res = await exec("eask analyze", { cwd });
      expect(res).toBeTruthy();

      const res1 = await exec("eask analyze Eask", { cwd });
      expect(res1).toBeTruthy();
    });

    it("handles json", async () => {
      const res = await exec("eask analyze --json", { cwd });
      expect(res).toBeTruthy();

      const res1 = await exec("eask analyze Eask --json", { cwd });
      expect(res1).toBeTruthy();
    });
  });
});

The result includes all stdout and stderr, so can be matched against with snapshots, or checked with string operations.
If it exits with an error then the Promise resolves with an error.

There are probably other choices too, and other frameworks, e.g. Mocha.

Advantages: loads of packages to choose from, features like snapshot matching, portable. Test framework is managed by npm.

Disadvantages: could be some difficulties in working with shell commands or changing files from in node.js.

Elisp

I've added it as a possibility, but I'm skeptical it would work very well since the test scripts would look similar to the files used as part of the fixtures.
E.g. if you use a test runner that uses a naming convention for tests but want to test the Eask version of that same command, you might have a folder structure like

test-the-runner/
- test-runner.test.el  # this is the actual test script
- test-fixture/
  - Eask
  - some-test.test.el  # this fixture may be picked up by the outer runner!

leading to some confusion!

It would have many of the drawbacks as the JS test frameworks in terms of managing shell output and translating exit codes.

[jcs090218]

I choose JS over other choices. Having Batch + Bash together would increase the amount of maintenance needed. That's why I didn't go for what Eldev has gone for. 😓 (Remark: #140)

I've added it as a possibility, but I'm skeptical it would work very well since the test scripts would look similar to the files used as part of the fixtures.

Agreed. 👍 I think this is what all experienced Elisp dev would agree on. Emacs isn't designed for these kinds of tasks.

---

Note: I’m comfortable with Jest since I have some experience with it, but I’d prefer a minimal learning curve, as most Elisp developers don’t favor JavaScript's complexity. 😅

[joshbax189]

I choose JS over other choices

Agree, and I think all the reasons you gave in #140 apply here too!

as most Elisp developers don’t favor JavaScript's complexity.

Multiply that by the number of Elisp devs that are contributing test cases to find out how much you should care 😆 (js-hate-factor * 0 = ???).

That said, creating tests should mostly follow a simple pattern:

• create the test directory with the correct files
• check the command you want to test using shell
• copy the command and path into one of the test patterns

The basic test patterns would be like:

Output should match a snapshot

const res = await exec("eask cmd", { cwd: "./relative_path" });
expect(res).toMatchSnapshot();

Output should match a regexp

const res = await exec("eask cmd", { cwd: "./relative_path" });
expect(res.stdErr).toMatch(".*something.*");

Command should error

expect(exec("eask cmd", { cwd: "./relative_path" })).rejects();

Now compare that to the bash code in the testing.sh script I added to do the same 😱

I'm pretty excited to try it! I'll make a branch and send you a PR so we can discuss it with concrete examples.

[jcs090218]

Multiply that by the number of Elisp devs that are contributing test cases to find out how much you should care 😆 (js-hate-factor * 0 = ???).

😂

I'm pretty excited to try it! I'll make a branch and send you a PR so we can discuss it with concrete examples.

That's awesome! 🚀 😄

I have limited experience with the JS testing framework, but I'll start looking up some tutorials, and hopefully, I'll get all the basic info. ;)