What is the "correct" notation for nested tests?

[xmojmr]

According to http://www.nntp.perl.org/group/perl.qa/2009/03/msg12165.html it seems that nested test notation similar to this:

1..3
ok 1
   1..3
   ok 1
   ok 2
      ok 1
      ok 2
      ok 3
      ok 4
      1..4
   ok 3 - some name
ok 2 - some name
ok 3
ok

would not be too difficult to support. But the official site linked from Wikipedia: Test Anything Protocol does not mention this as the way to go.

Is there any other notation suggested for nested tests? If not then can you add this one to the specification?

[kinow]

IIRC, the latest release of TAP (13?) doesn't include an official description of subtests. In the old testanything.org Wiki, there were archives of conversations about it, and some proposals. You can use the Internet Archive Wayback Machine to take a look. TAP producers in some programming languages, in special Perl, implemented different notations, but as far as I know, TAP 13 didn't provide an official notation.

[xmojmr]

@kinow according to the Internet Archive Wayback Machine (e.g. http://web.archive.org/web/20070501024548/http://testanything.org/wiki/index.php/Main_Page) the TAP version 13 was specified already 7 years ago and the nested tests were not there.

That is why I'm asking about the "current" state of affairs. What can the TAP specification site say about it today? Is the attempt to standardize the TAP protocol already dead?

[kinow]

Is the attempt to standardize the TAP protocol already dead?

There has been some conversation in the TAP mailing list, including a thread on subtests. You can join the conversation and help us working a new version.

http://www.ietf.org/mail-archive/web/tap/current/maillist.html

Hope that helps :)

[xmojmr]

@kinow from the archive it seems that current status as of 25 Oct 2014 is "we're stuck"

Thank you for the link, I'll check if there is anything useful I can do for the project 😉

[kinow]

Thanks @xmojmr ! Ping me if you need anything :-)

[jonathanKingston]

Sample code and documentation would be a start to progress this onwards.
We would need to start a draft version 14 spec to integrate it into.

[jonathanKingston]

Also @kinow can help as he is also an owner of the group too now - he deserved it 😄

[kinow]

Thanks @jonathanKingston ! Will triage issues and continue to comment here. Let me know if you need help with anything in special. @xmojmr if you have time to work on this issue sometime this month drop a message here. We can hop on some IRC channel and try a few iterations until we get some draft to post here and in the mailing list :-D

[xmojmr]

@jonathanKingston, @kinow IMO formal IETF-approved specification would be nice to have but a specification signed by a less authoritative authority would be good as well.

My current competence when it comes to the TAP protocol is not anything impressive. I have implemented closed-source TAP producer in TypeScript which supports nested tests (subtests) hopefully compliant with the specifications I have seen. That's all

IMO the most notable deliverables composing the TAP version 14 specification would be

• MUST include visual syntax diagrams like in Introducing JSON
• MUST include formal machine-consumable grammar to be used by tools like Gunther Rademacher's Railroad Diagram Generator and by parser generators (so that a validating parser might be generated)
• MUST include visual UML diagrams of a reference storage model like in YAML Ain’t Markup Language (YAML™) Version 1.2 → 3. Processing YAML Information → 3.2. Information Models
• SHOULD include representative suite of tests for validating TAP consumers against TAP producers (similar to what "W3C Markup Validation Service" does for HTML on the producer part and what "The Can I use... test suite" does for the consumer part)

@kinow I'll drop you my contact link through the e-mail

[kinow]

MUST include visual syntax diagrams like in Introducing JSON

+1

MUST include formal machine-consumable grammar to be used by tools like Gunther Rademacher's Railroad Diagram Generator and by parser generators (so that a validating parser might be generated)

A BIG +1 for this one. Apache Jena contains parsers for SPARQL with a javacc grammar following what was specified by the W3C group. This specification includes a grammar in an EBNF notation that is easy to be adapted to different tools like yacc, javacc, antlr and even Railroad Diagram Generator (which I didn't know, thanks @xmojmr!)

If you are curious about the SPARQL grammar by W3C, get it, remove the leading number (or copy this gist), paste in Railroad Diagram Generator and export the graph (example below).

MUST include visual UML diagrams of a reference storage model like in YAML Ain’t Markup Language (YAML™) Version 1.2 → 3. Processing YAML Information → 3.2. Information Models

+1

SHOULD include representative suite of tests for validating TAP consumers against TAP producers (similar to what "W3C Markup Validation Service" does for HTML on the producer part and what "The Can I use... test suite" does for the consumer part)

Another big +1 for this, but as a MUST. I like the JSON-LD Test Suite.

@xmojmr we probably must move this conversation to the mailing list, or link it in some thread there, as several other members could chime in and cast their votes/opinions too. WDYT?

[xmojmr]

@kinow 👍 ... 👍 for your comments and links.

I've found (randomly) another test suite example recommended in the mailing list

Andrew de Andrade, 11 Aug 2014:

...Having worked a bit with JSON schema, I found that one of the most useful tools was a standardized set of tests to check if a particular implementation conforms to the protocol standard. (see: https://github.com/json-schema/JSON-Schema-Test-Suite)

As for the where should the conversation move I don't have a clear view. Involving the TAP mailing list members surely looks like way forward.

As for the where should the conversation happen and where should be some beta-version-artifacts placed (https://github.com/TestAnything/Specification ❓) and how should be the project management handled, that's an unknown for me.

I think that it should be clarified upfront in order to avoid another "we're stuck" and @Leont ("RFC Status?: ..I'm fine with doing the administrative parts of this") should express his opinion.

So, what's the next step?

[kinow]

As for the where should the conversation move I don't have a clear view. Involving the TAP mailing list members surely looks like way forward.

Ok, if you have no objections then I'll drop a message in the mailing list linking to this issue. I'll check the previous threads to see if we must continue an existing one or start a new one.

As for the where should the conversation happen and where should be some beta-version-artifacts placed (https://github.com/TestAnything/Specification ❓)(...)

I think it's a good place. We can create a document detailing the TAP 14 specification, with a grammar and basic examples. What do you think? Not sure if @jonathanKingston was thinking of putting something else there.

and how should be the project management handled, that's an unknown for me.

I think that it should be clarified upfront in order to avoid another "we're stuck" and @Leont ("RFC Status?: ..I'm fine with doing the administrative parts of this") should express his opinion.

That sounds like a very good idea. This topic must probably be decided in the mailing list, so others will be aware of that too, and may even join and help. Once we've decided anything we can document in the web site too.

So, what's the next step?

• @kinow: ping the mailing list about this discussion
• @kinow / @xmojmr / others: start a draft and maybe send a pull request to https://github.com/TestAnything/Specification
• @xmojmr: ping the mailing list about the administrative parts

How does that sound? Can you take care of the last bullet item @xmojmr ?

[Leont]

IMO the most notable deliverables composing the TAP version 14 specification would be

Quite frankly, I'd consider all of that nothing more than formatting. It's useful to explain our behavior to outsiders, but it would seem rather premature to get tied up to that at this stage.

We don't even have a list of features that should be in TAP14 yet (other than the ones that existed in the TAP 13 draft, which isn't much more than TAP 12).

[Leont]

MUST include formal machine-consumable grammar to be used by tools like Gunther Rademacher's Railroad Diagram Generator and by parser generators (so that a validating parser might be generated)

Also, I'm not sure that is possible. The whitespace-sensitive nature of subtests as they currently exist means the grammar isn't easily describable in common grammar toolkits because it's not context free.

[xmojmr]

MUST include formal machine-consumable grammar to be used by tools...so that a validating parser might be generated

I'm not sure that is possible. The whitespace-sensitive nature of subtests...grammar...common grammar toolkits...it's not context free

I'm not sure if whitespaces would make the grammar context-sensitive neither what are the useful common grammar tools. Yaml format also uses nested white spaces and nesting in general, yet they were able to produce formal machine-consumable(?) grammar through what they call 4.1. Production Parameters. IMO the requirement should not be dropped just because it may turn to be difficult to fulfil. Personally I see the "MUST include visual syntax diagrams" more important as hand-coding some picture-compliant consumer/producer should not be that hard in any programming language

---

@kinow in the light of what @Leont said I don't see the proposed next steps as clear, I agree fully only with the "ping the mailing list about this discussion" action point.

Collecting requirements for the TAP 14 draft and keeping it's design goals aligned with the declared TAP Philosophy seems to me like the 1st thing that the TAP mailing list members should do.

But I'm newbie in the TAP land with not enough reputation points to cast serious votes.

BTW: drawing a reasonable incompatibility line between TAP 13 and TAP 14 (e.g. TAP 14 might set the spacing rule for nested tests to be fixed to \t) might be liberating decision.

This issue#20 aimed to resolve:

• TAP notation for nested tests is not standardized
• approved TAP specification is 7 years old
• the specification update process is currently stalled
• what should TAP consumers/producers interested in the nested tests do?

@kinow So, what's the next step?

[Leont]

I'm not sure if whitespaces would make the grammar context-sensitive neither what are the useful common grammar tools. Yaml format also uses nested white spaces and nesting in general, yet they were able to produce formal machine-consumable(?) grammar through what they call 4.1. Production Parameters.

I've only written one parser for it, and in its grammar I had to cheat in context to make it work. I know it's possible to write such a parser, I'm not familiar with anything that describes them usefully. I would honestly love to be proven wrong.

Collecting requirements for the TAP 14 draft and keeping it's design goals aligned with the declared TAP Philosophy seems to me like the 1st thing that the TAP mailing list members should do.

Indeed.

[jonathanKingston]

@kinow the https://github.com/TestAnything/Specification repo was created from the conversation based in the mailing list. It seemed a better place to work upon a draft specification. We did however have @gaurav contribute his work to the group on a draft IETF submission.

I suggest to reduce the risk of this spinning into another 'we are stuck':

• Raising feature requests in: https://github.com/TestAnything/Specification
  • For now this repo should have zero content.
  • Can hold all changes and discussions
  • At some point we can tag the release and turn it into a real document.
• Keeping this repo only for the site
  • Which for now will not mention the work on TAP 14
  • If people are interested they will come to github, the site should speak only of the current specification and software using it
• Work on evolving the TAP 13 IETF submission in that repo separately to help @gaurav
  • When the changes mature enough in the specification repo we can create a TAP 14 branch here also.

[kinow]

@Leont

I've only written one parser for it, and in it's grammar I had to cheat in context to make it work. I know it's possible to write such a parser, I'm not familiar with anything that describes them usefully. I would honestly love to be proven wrong.

I remember trying to write a parser with Antlr some time ago... can't recall exactly why I ended up writing a regex rule parser instead for tap4j, but IIRC it was something related to recursion of some rules, plus my noobness with grammars and compilers :-)

However I was trying by myself, perhaps with more heads we'll achieve something closer to a formal and machine consumable grammar - or just something useful for using as basis when writing producer/consumers.

[kinow]

@jonathanKingston Agreed! After reading your comment I realized we are having this conversation in an issue on the testanything.github.io repository. I apologize, I hadn't notice we were using the website repository :-)

@Leont , @xmojmr ,I'm closing this issue and have filed an issue under the Specification repository: TestAnything/Specification#2

@xmojmr :

@kinow in the light of what @Leont said I don't see the proposed next steps as clear, I agree fully only with the "ping the mailing list about this discussion" action point.

I'm writing an e-mail right now to the mailing list ;-) Feel free to raise new issues in that repository for other things (maybe the specification process?).

[jonathanKingston]

👍 thanks @kinow