Add a helper for fixture-based tests

[ianstormtaylor]

Hey folks! I'd like to propose a new test helper that makes fixture-based testing convenient and really powerful.

The frustrating part about fixtures is writing the code to read/write them for each test. I think there's room for a simple glob-based helper that matches fixtures and creates tests for each one.

For example:

test.fixtures('./fixtures/**/*.txt', (t, path) => {
  const txt = fs.readFileSync(path, 'utf8')
  // your logic here...
})

That would create a test for each of the files matched by the glob there. And it gives you the t and the path so that you can write your own logic.

The use case for text-file-based fixtures is fairly obvious. But this also unlocks a more powerful type of fixture, where the fixture itself is a JavaScript file with exports. For example, the fixtures that Slate uses for rich text commands:

/** @jsx h */

import h from '../../../helpers/h'

export default function(editor) {
  editor.delete()
}

export const input = (
  <value>
    <document>
      <paragraph>
        word<anchor />
      </paragraph>
      <paragraph>
        <focus />another
      </paragraph>
    </document>
  </value>
)

export const output = (
  <value>
    <document>
      <paragraph>
        word<cursor />another
      </paragraph>
    </document>
  </value>
)

Which can be modeled with the fixtures helper super easily as:

test.fixtures('./fixtures/**/*.js', (t, path) => {
  const module = require(path)
  const { default, input, output } = module
  const actual = default(input)
  t.equal(actual, output)
})

Here are a few real-world examples since I use this pattern a decent amount to make tests much easier to understand and write:

• slate uses it with both input and output being JSX-based representations of a document. Even with JSX these objects are often fairly LOC-heavy to describe the test case properly.
• slate-hyperscript uses it also with input and output, and it makes it easy to define a custom @jsx pragma in the file itself. The resulting objects can be fairly large, so having them in individual files is nice to keep things readable.
• superstruct uses it too, but with more domain-specific names for the exports. This way every test can be defined in terms of the Struct being tested, the data being input, and either a result or an error.

I think this type of testing is under-appreciated partially because writing the logic to recurse/match/read fixtures is pretty annoying to redo for each new repo. (We extracted it somewhat for Slate, but I've rewritten this kind of thing so many times.) To me this pattern is much more powerful than "snapshots", and would be more popular if they were made easier.

I realize that this is asking for a new feature, and that's not a small ask. But I really do think this type of testing is under-appreciated. And I think AVA's simple API could make it really easy to champion. It's really powerful once it's setup. It's just the setup that's annoying.

The name could be fixtures, or potentially since that isn't specific to this type of fixtures, it could be cases:

test.cases('./fixtures/**/*.txt', (t, path) => {
  const txt = fs.readFileSync(path, 'utf8')
  // your logic here...
})

Open to ideas! Thanks for reading.

[novemberborn]

I like it!

A good place to start though would be to build this outside of AVA, e.g.:

import test from 'ava'
import cases from 'testcases'

cases('./fixtures/**/*.txt', test, (t, path) => {})

I'd be happy to promote that, include a recipe etc, and then we can see about adding it to AVA itself. What do you think?