introduce MUST, SHOULD, MAY to allow better reporting

[bourgeoa]

@michielbdejong @ylebre @edwardsph
Each test SHOULD be specified as a MUST, SHOULD, MAY so as to know the specification level required. An html report could be generated with the results (report: work to be done)

I tested a quite easy solution around wrapping jest functions with an itIs function :

• test name will always begin with SKIP, MUST, SHOULD or MAY example itIs('MAY') (this allow for further reporting)
• env variables allow to skip MUST, SHOULD and/or MAY
• default is MUST : itIs()

export function itIs(arg = 'MUST') {
  if (arg === 'SKIP') return (name, runner) => { it.skip(`${arg} ${name}`, runner); }
  if (arg === 'MUST' && process.env.SKIP_MUST) return (name, runner) => { it.skip(`${arg} ${name}`, runner); }
  if (arg === 'SHOULD' && process.env.SKIP_SHOULD) return (name, runner) => { it.skip(`${arg} ${name}`, runner); }
  if (arg === 'MAY' && process.env.SKIP_MAY) return (name, runner) => { it.skip(`${arg} ${name}`, runner); }
  return (name, runner) => { it(`${arg} ${name}`, runner); }
}

The test would look like this :

  describe("Get RDFa", () => {
    describe("As JSON-LD", () => {
....
      itIs('MAY')("Triples", async () => {
        const triples = await asTriples(
          jsonText,
          `${testFolderUrl}example.html`,
          "application/ld+json"
        );
        expect(triples).toIncludeAllMembers(triplesFromHtml);
      });
    });
    describe("As Turtle", () => {
....
      itIs()("Triples", async () => {
        const triples = await asTriples(
          text,
          `${testFolderUrl}example.html`,
          "text/turtle"
        );
        expect(triples).toIncludeAllMembers(triplesFromHtml);
      });
    });
  });
  describe("GET Turtle", () => {
    describe("As JSON-LD", () => {
...
      itIs('MUST')("Triples", async () => {
        const triples = await asTriples(
          jsonText,
          `${testFolderUrl}example.ttl`,
          "application/ld+json"
        );
        expect(triples).toEqual(triplesFromTurtle);
      });
    });

The result for SKIP_MAY=1 npm run jest conneg.test.ts -- --verbose
the use of verbose is not needed here but allow to display each test result when there is a default folder of tests.

 PASS  test/surface/conneg.test.ts
  Alice's pod
    Get RDFa
      As JSON-LD
        ○ skipped MAY Triples
      As Turtle
        ✓ MUST Triples (13 ms)
    GET Turtle
      As JSON-LD
        ✓ MUST Triples (7 ms)
      As Turtle
        ✓ MUST Triples (2 ms)
    GET JSON-LD
      As Turtle
        ✓ MUST Triples (3 ms)

Test Suites: 1 passed, 1 total
Tests:       1 skipped, 4 passed, 5 total
Snapshots:   0 total
Time:        2.291 s
Ran all test suites matching /conneg.test.ts/i.

[ylebre]

I really like this idea!
A few considerations I have (maybe for future versions, because I think just this change would be a great improvement already)

• Should we keep a list of the rules we have, together with a link to where in which spec the rule came from?
• Should we have some kind of versioning in the rules, so we can run the test suite against a specific spec version?
• I think we need to be able to single out MAY rules, so a server can choose to run the test for the one they implement but not the rest, to that the applicable MAY rules can be run in CI as well.

[bourgeoa]

I think we need to be able to single out MAY rules, so a server can choose to run the test for the one they implement but not the rest, to that the applicable MAY rules can be run in CI as well.

May be this could be done with an options parameter in function itIs (arg = 'MUST', options) or multiple MAY depending on the complexity

[ylebre]

I think it would be best to require the level for itIs - it will require a bit of work behorehand, but will be more explicit from there on.
Also, it might be a good idea to have an ID for every test so that you can explicitly toggle each one.

And just to put it out there: maybe we should make 'MUST' and 'SHOULD' required by default, so you'd have to opt-in to skip them.
For 'MAY', the tests are skipped by default and you'd have to opt-in to include them;

It would end up something like this:

export function itIs(level, id) {
  switch (level) {
    case 'SKIP':
      return (name, runner) => { it.skip(`${level} ${name}`, runner); }
    break;
    case 'MUST':
      if (
        (process.env.SKIP_MUST) ||
        (process.env_SKIP_MUST_' + id])
      ) {{
          return (name, runner) => { it.skip(`${level} ${name}`, runner); }
      } else {
          return (name, runner) => { it(`${level} ${name}`, runner); }
      }
    break;
    case 'SHOULD':
      if (
        (process.env.SKIP_SHOULD) ||
        (process.env['SKIP_SHOULD_' + id])
      ) {
        return (name, runner) => { it.skip(`${level} ${name}`, runner); }
      } else {
        return (name, runner) => { it(`${level} ${name}`, runner); }
      }
    break;
    case 'MAY':
      if (
        (process.env.INCLUDE_MAY) ||
        (process.env['SKIP_SHOULD_' + id])
      ) {
        return (name, runner) => { it(`${level} ${name}`, runner); }
      } else {
        return (name, runner) => { it.skip(`${level} ${name}`, runner); }
      }
    break;
  }
  return (name, runner) => { it(`${level} ${name}`, runner); }
}

[bourgeoa]

If the ID option is retained, if defined the ID shall be displayed explicitly in the test name
it(${level} ${Id} ${name}, runner)

I agree to require a level.

For practical reason it may be better to keep the ID optional until the tests and spec are stabilised. Changes imply a review of the CI.

The ID could be the link with the specification.