doc/test-catalog.tags.xml

<?xml-stylesheet type="text/xsl" href="../lib/tsd-to-html.xsl"?>
<TEI xmlns="http://www.tei-c.org/ns/1.0"
     xmlns:rng="http://relaxng.org/ns/structure/1.0">
  <teiHeader>
    <fileDesc>
      <titleStmt>
	<title>XML elements for ixml test catalogs</title>
	<author>C. M. Sperberg-McQueen</author>
      </titleStmt>
      <publicationStmt>
	<authority>Black Mesa Technologies LLC</authority>
	<date>2021</date>
      </publicationStmt>
      <sourceDesc>
	<p>No source; created in electronic form.</p>
      </sourceDesc>
    </fileDesc>
    <revisionDesc>
      <list>
	<item>
	  <p>2022-02-22 : CMSMcQ : add skeletal entries for test-report elements.</p>
	</item>
	<item>
	  <p>2021-12-22 : CMSMcQ : copy edits.</p>
	</item>
	<item>
	  <p>2021-11-11 : CMSMcQ : substantial copy edits
	  and revision.</p>
	</item>
	<item>
	  <p>2021-11-07 : CMSMcQ : first version of this
	  tag set description.</p>
	</item>
      </list>
    </revisionDesc>
  </teiHeader>
  <text>
    <front>
      <titlePage>
	<docTitle>
	  <titlePart>XML elements for ixml test catalogs</titlePart>
	</docTitle>
	<docAuthor>C. M. Sperberg-McQueen</docAuthor>
	<docDate>7 November 2021, last rev. 22 February 2022</docDate>
      </titlePage>
    </front>
    <body>
      <div xml:id="intro">
	<head>Introduction</head>
	<p>This document lists the elements in an XML vocabulary for
	ixml test catalogs and test-result reports, giving a short
	description of each.  Fuller description and discussion
	may be added later if felt to be needed; for now this is
	mostly an alphabetical reference list to explain what the
	elements are intended to mean and how they are to be used.</p>
	<p>The vocabulary described was developed by C. M. Sperberg-McQueen
	as a proposal for the invisible-XML Community Group at the
	World Wide Web Consortium, but it has not yet been considered
	by the group, still less adopted, and it has no official status
	of any kind.</p>
	<p>The vocabulary describes two different types of document:
	<list>
	  <item>
	    <p>Test catalogs describe a potentially complex set of
	    test cases for an invisible XML processor; they can
	    be organized internally into multiple sets of test cases,
	    and test sets can nest.</p>
	    <p>Each test set may specify an ixml grammar in either
	    ixml or XML form, together with a set of test cases and
	    optionally nested test sets.  An optional
	    <gi>grammar-test</gi> element signals that the grammar
	    itself should be regarded as a test case and identifies
	    the expected result of processing the grammar.</p>
	    <p>Each test case in a test set supplies input to be
	    parsed against the grammar for the test set, and
	    specifies a result.</p>
	    <p>Grammars and test-case input can be given inline
	    or in separate files.  The expected results can also
	    be given inline or externally.</p>
	  </item>
	  <item>
	    <p>Test reports describe the results of running a
	    given processor against the tests in a test catalog.</p>
	    <p>A metadata section provides information about the
	    test catalog used and the processor tested.  Then,
	    for each test set and test case, the result produced
	    by the processor is reported and whether it is or is not
	    the result expected in the test catalog.</p>
	    <p><hi>Note:  the current version of this vocabulary
	    does not yet define elements for test reports.  Real
	    soon now.</hi></p>
	  </item>
	</list></p>
	<p>A <ref target="catalog-sketch.xml">sample test
	catalog</ref> is given in the <code>doc</code> directory of
	the ixml-tests Git repository (where this document is also to
	be found) in both <ref target="catalog-sketch.xml">XML</ref>
	and <ref target="catalog-sketch.html">HTML</ref> (for
	situations where dynamic XSLT processing is not possible).
	</p>
      </div>
      
      <div xml:id="issues">
	<head>Known issues</head>
	<p>Among the shortcomings of the current state of the
	test catalog schema are these.</p>
	<p xml:id="reports"><label>No reporting of test
	results.</label> XML vocabularies are needed to allow a test
	harness to report the detailed results of running a test suite
	and comparing the output to the expected results, and to allow
	the developers of an implementation to report (perhaps in
	somewhat less detail) which tests they have run and not run,
	and of those they ran which tests they passed and which they
	failed.</p>
	<p>An initial sketch of such a vocabulary is expected
	soon, but it's not here yet.</p>
	<p xml:id="ambiguity"><label>Awkward treatment of ambiguous
	sentences.</label> An ambiguous sentence has more than one
	parse tree, and the choice of which tree to return is
	implementation-dependent (or possibly implementation-defined),
	so that a correct implementation may easily produce a parse
	tree which is correct but does not match the parse tree
	specified in the test case.</p>
	
	<p>The current approach is to allow multiple parse trees, with
	the meaning being that if the processor produces any of them,
	it passes that test.  If the processor returns a parse forest
	instead of a single tree, every tree in the returned forest
	should be among those specified in the result.  This has the
	drawback that in some cases the set of parse trees specified
	will not be and cannot be complete: in some grammars, some
	sentences have an infinite number of parse trees.</p>
	<p>For practical purposes, however, we can expect there to be
	a finite number of processors using any given test suite, each
	of which will return a finite number of trees.  If a processor
	fails a test on an ambiguous grammar, then the tree returned
	by the processor should be checked manually and added to the
	specified result.  Unless a processor returns a different
	result each time it parses the test input, the second and
	later runs against the test suite should be problem-free.</p>
	<p>There are of course alternatives:
	<list>
	  <item>
	    <p>Allow an assertion about the result, expressed in
	    XPath.  This is the approach taken in the QT test suite
	    and in XSpec. Drawback: unless the assertion is very
	    thorough, the test case will not fully specify the
	    required test result, allowing incorrect results to pass
	    the test.</p>
	  </item>
	  <item>
	    <p>Allow a set of assertions about the result, expressed
	    in XPath.  To pass the test, a processor's result must
	    satisfy all assertions.  Drawback: as before, though it's
	    easier to be thorough.</p> 
	  </item>	  
	  <item>
	    <p>Allow an and/or tree of assertions about the result,
	    expressed in XPath.  To pass the test, a processor's
	    result must satisfy all children of an <gi>and</gi> and at
	    least one child of an <gi>or</gi>.  Drawback: logical
	    conjunction and disjunction are already built in to XPath
	    so it's not really necessary to provide a new set of
	    machinery for the purpose.</p>
	  </item>
	</list>
	As the examples of the QT3 test suite and XSpec show, all of
	these involve building additional machinery both into the
	vocabulary for the test catalog and also into the test
	harness.  An XPath assertion must be evaluated in a dynamic
	context and must say something about the result returned, so
	we now end up having to build a dynamic context and bind the
	result to some well known variable like <code>$result</code>,
	so that the assertion can say something useful about the
	result.</p>
	<p>The current approach has the comparative advantage
	that it does not require an XPath evaluator: one could in
	principle handle the comparison of test results to the
	expected output by writing out temporary files and calling a
	diff program.  The occasional false negative in the test
	results seems a small price to pay for that advantage.
	</p>
      </div>
      
      <div xml:id="elements">
	<head>Element reference list</head>
	
	<elementSpec ident="app-info">
	  <gloss>application information</gloss>
	  <desc>includes information needed by any
	  application which is to process the
	  test catalog; this is an extension mechanism
	  to allow the inclusion of arbitrary XML
	  elements in the catalog.  May occur
	  wherever metadata is allowed:
	  as a child of <gi>test-catalog</gi>,
	  <gi>test-set</gi>,
	  <gi>test-case</gi>, etc.</desc>
	  <exemplum>
	    <p>The process by which a test case is generated may
	    produce, as a side effect, a <soCalled>raw parse
	    tree</soCalled> in which all nonterminals are
	    present.  This can be helpful in diagnosing
	    problems in an ixml parser, so it may be thought
	    useful to record the raw parse tree, even though
	    there is no standard place for it in the schema.
	    The following example shows a raw parse tree
	    embedded in a test case.</p>
	    <eg><![CDATA[
<test-case xml:id="S-1" name="S1">
   <created by="Gingersnap (CMSMcQ)" 
            on="2021-01-26"/>
   <test-string/>
   <result>
      <assert-xml>
         <S/>
      </assert-xml>
   </result>
   <app-info>
      <raw-parse-tree xmlns="">
         <S/>
      </raw-parse-tree>
   </app-info>
</test-case>]]></eg>
	  </exemplum>
	</elementSpec>
	
	<elementSpec ident="assert-not-a-grammar">
	  <desc>signals that the grammar specified is not a conforming
	  ixml grammar (and thus cannot be used to parse the test
	  input).  Correct behavior in this case is to report that
	  the input cannot be parsed using the grammar and optionally
	  describe the nature of the problem.</desc>
	  <remarks>
	    <p>Appears as child of <gi>result</gi>.</p>
	    <p>The spec does not require a particular way of signaling
	    errors in the grammar, so each processor may require a
	    different setup.</p>
	    <p>Within the <gi>result</gi> element for a grammar test,
	    there is no useful distinction between this element and
	    <gi>assert-not-a-sentence</gi>.  Casuists can argue the
	    case each way.  (<hi>The purported grammar is not a
	    grammar, so this element applies; it is also not a
	    sentence in the language defined by the grammar for ixml
	    grammars, so the <gi>assert-not-a-sentence</gi> also
	    applies. If the problem with the grammar is not failure to
	    follow the grammar for grammars but violation of some
	    extra-grammatical constraint like having only one rule per
	    nonterminal, the input <emph>is</emph> a sentence but
	    <emph>is not</emph> a grammar.  Etc., etc., etc.</hi>) In
	    practice both elements will and should be treated as
	    interchangeable in that particular case.</p>
	    <p>This element can contain any elements of the
	    <code>metadata</code> class (so: <gi>description</gi>,
	    <gi>app-info</gi>, <gi>doc</gi>), which leads to a lack of
	    parallelism with <gi>assert-xml</gi> and
	    <gi>assert-xml-ref</gi>.  Remove?</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="assert-not-a-sentence">
	  <desc>signals that the test input is not
	  a sentence in the language described by the
	  grammar indicated; correct behavior in this
	  case is to report that the input cannot be
	  parsed using the grammar and optionally
	  describe the nature of the problem.</desc>
	  <remarks>
	    <p>Appears as child of <gi>result</gi>.</p>
	    <p>The spec does not require a particular
	    way of signaling failure to parse the input,
	    so each processor may require a different
	    setup.</p>
	    <p>This element can contain any elements of the
	    <code>metadata</code> class (so: <gi>description</gi>,
	    <gi>app-info</gi>, <gi>doc</gi>), which leads to a lack of
	    parallelism with <gi>assert-xml</gi> and
	    <gi>assert-xml-ref</gi>.  Remove?</p>
	    <p>Within the <gi>result</gi> element for a grammar test,
	    there is no useful distinction between this element and
	    <gi>assert-not-a-grammar</gi>; in that context, the two
	    should in practice be treated as interchangeable.</p>
	  </remarks>
	</elementSpec>

	<elementSpec ident="assert-xml">
	  <desc>signals that the test input is a sentence in the
	  language described by the grammar indicated and contains an
	  acceptable XML representation of one parse of the
	  sentence.</desc>
	  <remarks>
	    <p>Appears as child of <gi>result</gi>.</p>
	    <p>For ambiguous sentences, more than one parse is
	    correct, and this element may occur more than once.</p>
	  </remarks>
	</elementSpec>

	<elementSpec ident="assert-xml-ref">
	  <desc>signals that the test input is a sentence in the
	  language described by the grammar indicated and give a URI
	  for the prescribed XML representation of one parse of the
	  sentence.</desc>
	  <remarks>
	    <p>Appears as child of <gi>result</gi>.</p>
	    <p>For ambiguous sentences, more than one parse is
	    correct, and this element may occur more than once.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="catalog-date">
	  <desc>gives the release date of the test catalog describing the tests
	  whose results are included in a test report.</desc>
	  <remarks>
	    <p>Occurs as a child of <gi>metadata</gi> at the top of
	    the test report.</p>
	    <p>The <gi>catalog-date</gi> will normally just repeat the
	    information given in the <att>release-date</att> attribute
	    on the <gi>test-catalog</gi> element.  The purpose of
	    including it in the test report is to help document which
	    version of the test catalog was used, if the test catalog
	    is revised.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="catalog-uri">
	  <desc>gives the URI of the test catalog describing the tests
	  whose results are included in a test report.</desc>
	  <remarks>
	    <p>Occurs as a child of <gi>metadata</gi> at the top of
	    the test report.</p>
	  </remarks>
	</elementSpec>

	<elementSpec ident="code">
	  <desc>marks inline code fragments in prose.</desc>
	  <remarks>
	    <p>Typical typographic treatment is to display in a
	    monospaced font.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="created">
	  <desc>a metadata element containing information about
	  the creation of a test catalog, test set, or test case.</desc>
	  <attList>
	    <attDef ident="who">
	      <desc>specifies who created the item.</desc>
	    </attDef>
	    <attDef ident="when">
	      <desc>specifies the date the item was created.</desc>
	    </attDef>
	  </attList>
	  <remarks>
	    <p>The element is allowed but optional on individual test
	    cases and grammar tests.  If no creation date is given for
	    an individual test case, it is assumed to have been
	    created at the same time as the test set as a whole; if it
	    was added later, it can (and ideally should) have its date
	    of creation and revision history noted.  When a test suite
	    is changed over time, it can be helpful if every test case
	    has a complete revision history.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="description">
	  <desc>contains a sequence of paragraphs containing a
	  natural-language description of a test catalog, test set, or
	  test case.</desc>
	  <remarks>
	    <p>Ideally, the <gi>description</gi> element should be
	    able to contain prose tagged in XHTML, or in the XML
	    vocabulary of the author's choice.  Currently this is not
	    the case.  <!-- <hi>Experience with XSD makes me lean
	    toward prescribing HTML, to make the opportunity of using
	    full markup easier to grasp.</hi> --></p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="doc">
	  <desc>refers to external documentation for the test catalog,
	  test set, or test case, if there is any.</desc>
	  <attList>
	    <attDef ident="href">
	      <desc>contains a URI at which the external documentation
	      can be found.</desc>
	    </attDef>
	  </attList>
	  <remarks>
	    <p>If a test case exhibits a particularly tricky situation
	    which has been extensively discussed or analysed by
	    the community group (or anyone else), for example, it's
	    helpful to be able to provide pointers instead of having
	    to reproduce the discussion inside the test catalog,
	    or omitting it entirely.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="emph">
	  <desc>marks emphasized text in prose.</desc>
	  <remarks>
	    <p>This element is intended to mark rhetorical emphasis or
	    stress, not arbitrary special typographic treatment.  (If
	    we turn out to need markup for arbitrary typographic
	    treatment, we'll want to introduce <gi>hi</gi> for
	    typographic highlighting, or <gi>seg</gi> or <gi>span</gi>
	    with a <att>style</att> or <att>rendering</att>
	    element.)</p>
	    <p><hi>Note:  if we turn out to need markup for arbitrary
	    typographic treatment, we would do better to specify that
	    the prose should be marked up in XHTML.</hi></p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="grammar-result">
	  <desc>provides the result, in a test report, of running a
	  given <gi>grammar-test</gi> in the test catalog.</desc>
	  <attList>
	    <attDef ident="result">
	      <desc>Summarizes the result of running the test; the
	      value is one of:
	      <list>
		<item><code>pass</code>, indicating that the reported
		result agrees with the expected result</item>
		<item><code>fail</code>, indicating that the test was
		run and the reported result did not agree with the
		expected result</item>
		<item><code>not-run</code>, indicating that the
		grammar test was not run, for whatever reason
		(the reason can be given in a <gi>description</gi>
		element)</item>
		<item><code>other</code>, indicating that none of the
		other values apply (an explanation can be given in a
		<gi>description</gi> or <gi>app-info</gi>
		element)</item>
	      </list>
	      </desc>
	    </attDef>
	  </attList>
	  <remarks>
	    <p>A child of <gi>test-set-results</gi>.</p>
	    <p>The results of the grammar test can be given in greater
	    or lesser detail.  The minimum <gi>grammar-result</gi>
	    element gives the result of the test.  If desired,
	    however, the test result can also contain or point to the
	    grammar tested, the expected result, and the reported
	    result.</p>
	    <p>For convenience, the schema allows the
	    <gi>grammar-result</gi> element to be the root element of
	    a document.</p>
	  </remarks>
	</elementSpec>

	
	<elementSpec ident="grammar-test">
	  <desc>indicates that the grammar associated with a given
	  test set should be treated as a test case and checked for
	  conformance; contains a <gi>result</gi> element indicating
	  expected result of the test.</desc>
	  <remarks>
	    <p>A child of <gi>test-set</gi>.</p>
	    <p>If the grammar is given in an <gi>ixml-grammar</gi>
	    element or pointed to with an <gi>ixml-grammar-ref</gi>
	    element, the grammar should be parsed with the grammar for
	    ixml and the result compared with the expected result.
	    The extra-grammatical constraints imposed by the spec
	    should also be checked.  If the grammar given is in XML
	    (using the <gi>vxml-grammar</gi> or
	    <gi>vxml-grammar-ref</gi> elements), the processor should
	    check it for structural correctness (the spec requires
	    that the XML grammar <q>can be derived from an ixml
	    grammar in ixml form by parsing as described in this
	    specification</q>) and for conformance to the
	    extra-grammatical or non-structural constraints in the
	    ixml specification.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="ixml-grammar">
	  <desc>contains a grammar in ixml form.</desc>
	  <exemplum>
	    <p>The following example shows an ixml grammar for
	    arithmetic expressions.
	    </p>
	    <eg><![CDATA[     <ixml-grammar>
{ Grammar for simple arithmetic expressions, 
  for testing of test-case generation and
  grammar-manipulation tools.

  Revisions:
  2020-12-30 : CMSMcQ : made grammar
}
expr: s*, term+addop.
term:  factor+mulop.
factor:  num; var;
         '(', expr, ')'; 
         '[', expr, ']'; 
         '{', expr, '}'.
num:  digit+, ('.', digit*)?.
var:  ['a'-'z'].
addop: s*, ["+-"], s*.
mulop: s*, ["*/×÷"], s*.
digit: ['0'-'9'].
s: #20; #9; #a.       
     </ixml-grammar>
]]></eg>
	  </exemplum>
	  <remarks>
	    <p>Appears as a child of <gi>test-set</gi> and gives
	    the grammar for the test cases of the set.</p>
	    <p>This is one of several elements that can be used to
	    specify a grammar; grammars may be given in ixml or XML
	    form, inline or externally.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="ixml-grammar-ref">
	  <gloss>reference to an external grammar in ixml form</gloss>
	  <desc>points to an external resource containing
	  an ixml grammar.</desc>
	  <attList>
	    <attDef ident="href">
	      <desc>contains a URI at which the external grammar
	      can be found.</desc>
	    </attDef>
	  </attList>
	  <remarks>
	    <p>Appears as a child of <gi>test-set</gi> and gives
	    the grammar for the test cases of the set.</p>
	    <p>This is one of several elements that can be used to
	    specify a grammar; grammars may be given in ixml or XML
	    form, inline or externally.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="metadata">
	  <desc>contains a short sequence of metadata elements describing
	  the contents of a test report.</desc>
	  <remarks>
	    <p>Children include <gi>name</gi> (of the report),
	    <gi>report-date</gi>,
	    <gi>processor</gi>,
	    <gi>processor-version</gi>,
	    <gi>catalog-uri</gi>, and
	    <gi>catalog-date</gi>.
	    </p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="modified">
	  <desc>contains an entry in the change log for a test case
	  or test set describing a modification.</desc>
	  <attList>
	    <attDef ident="change">
	      <desc>describes the change.</desc>
	    </attDef>
	    <attDef ident="who">
	      <desc>specifies who made the change.</desc>
	    </attDef>
	    <attDef ident="when">
	      <desc>specifies the date the change was made.</desc>
	    </attDef>
	  </attList>
	  <remarks>
	    <p></p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="name">
	  <desc>gives the name assigned to a test report.</desc>
	  <remarks>
	    <p>Occurs as a child of <gi>metadata</gi> at the top of
	    the test report.</p>
	    <p>The analogous name of a test catalog is given as
	    an attribute on <gi>test-catalog</gi>, not as an
	    element.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="p">
	  <desc>a paragraph of natural-language prose.</desc>
	  <remarks>
	    <p>A very small number of phrase-level elements
	    are provided for the markup of the prose; if the
	    set proves inconveniently small, it can easily be
	    grown.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="processor">
	  <desc>indicates the processor used in the testing described
	  in a a test report.</desc>
	  <remarks>
	    <p>Occurs as a child of <gi>metadata</gi> at the top of
	    the test report.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="processor-version">
	  <desc>gives the version number or identifier, if any, of
	  the processor for which a test report was prepared.</desc>
	  <remarks>
	    <p>Occurs as a child of <gi>metadata</gi> at the top of
	    the test report.  Optional.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="report-date">
	  <desc>gives the date on which a test report was prepared.</desc>
	  <remarks>
	    <p>Occurs as a child of <gi>metadata</gi> at the top of
	    the test report.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="reported-not-a-grammar">
	  <desc>in a test report, signals that the processor reported
	  that the grammar supplied for a test is not a conforming
	  ixml grammar (and thus cannot be used to parse the test
	  input).</desc>
	  <remarks>
	    <p>Appears as child of <gi>result</gi>.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="reported-not-a-sentence">
	  <desc>in a test report, signals that the test input is not a
	  sentence in the language described by the grammar supplied for the test.</desc>
	  <remarks>
	    <p>Appears as child of <gi>result</gi>.</p>
	  </remarks>
	</elementSpec>

	<elementSpec ident="reported-xml">
	  <desc>in a test report, signals that the processor parsed
	  the test input as a sentence in the language described by
	  the grammar supplied; contains the XML element produced as
	  output.</desc>
	  <remarks>
	    <p>Appears as child of <gi>result</gi>.</p>
	    <p>For ambiguous sentences, more than one parse is
	    correct, and this element may occur more than once.</p>
	  </remarks>
	</elementSpec>

	<elementSpec ident="reported-xml-ref">
	  <desc>in a test report, signals that the processor parsed
	  the test input as a sentence in the language described by
	  the grammar supplied; gives a URI for the XML output
	  produced by the processor.</desc>
	  <remarks>
	    <p>Appears as child of <gi>result</gi>.</p>
	    <p>For ambiguous sentences, more than one parse is
	    correct, and this element may occur more than once.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="result">
	  <desc>in a test catalog specifies the expected result of a
	  test, using the <gi>assert-xml</gi>, <gi>assert-xml-ref</gi>
	  or the <gi>assert-not-a-sentence</gi> elements depending on
	  what is to be asserted and how; in a test report may
	  optionally recapitulate the expected result and may give the
	  output reported by the processor.</desc>
	  <remarks>
	    <p>Unlike a test catalog, which is required to specify the
	    expected result exactly once, a test report may give the
	    expected result in multiple forms (i.e. both inline and by
	    reference to an external document).  The same holds true
	    for the reported result.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="test-case">
	  <desc>contains input and expected result for one test
	  case.</desc>
	  <attList>
	    <attDef ident="name" usage="req">
	      <desc>a name by which the test case can be referred
	      to; should be unique within the test set</desc>
	    </attDef>
	  </attList>
	  <exemplum>
	    <p>The following example shows a test case for the grammar
	    <code>S: {nil}.</code> (not shown, because it's specified
	    earlier in the catalog).  The input is the empty string,
	    as specified here by the <gi>test-string</gi> element.
	    The expected output is an empty <gi>S</gi> element, as
	    specified here by the <gi>assert-xml</gi> element within
	    <gi>result</gi>.</p>
	    <eg><![CDATA[<test-case xml:id="S-1" name="S1">
   <created by="Gingersnap (CMSMcQ)" 
            on="2021-01-26"/>
   <test-string/>
   <result>
      <assert-xml>
         <S/>
      </assert-xml>
   </result>
</test-case>]]></eg>
	  </exemplum>
	  <remarks>
	    <p>A processor being tested should parse the specified
	    input using the grammar supplied by a containing
	    <gi>test-set</gi> element.  The processor passes the test
	    if the processor's output matches the expected result
	    (specified in the <gi>result</gi> element).</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="test-catalog">
	  <desc>contains a collection of test cases organized into one
	  or more test sets and released at the same time.</desc>
	  <content>
	    <rng:interleave>
	      <rng:zeroOrMore>
		<rng:ref name="metadata"/>
              </rng:zeroOrMore>
	      <rng:zeroOrMore>
		<rng:choice>
		  <rng:ref name="test-set"/>
		  <rng:ref name="test-set-ref"/>
		</rng:choice>
	      </rng:zeroOrMore>
	    </rng:interleave>
	  </content>
	  <attList>
	    <attDef ident="name" usage="req">
	      <desc>a name by which the collection of tests can be
	      referred to</desc>
	    </attDef>
	    <attDef ident="release-date" usage="req">
	      <desc>date of the test suite described by this
	      catalog; more specifically, date of its release
	      or official publication</desc>
	    </attDef>
	  </attList>
	  <remarks>
	    <p>When a suite of related tests is published, it may
	    happen that there are different test-catalog files (and
	    <gi>test-catalog</gi> elements) for different groups of
	    tests within the larger collection.  It is a good idea for
	    there to be one test catalog that includes all of the
	    others, so as to provide a single point of entry for
	    processing the entire collection of tests.  The
	    <gi>test-set-ref</gi> element is used to include other
	    test-catalog files.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="test-report">
	  <desc>contains a collection of test-case results derived by
	  running the tests from a given test catalog through a
	  particular processor.</desc>
	  <remarks>
	    <p>The overall structure of the test report mirrors the
	    structure of the test catalog: for each <gi>test-set</gi>
	    in the catalog, there will normally be a
	    <gi>test-set-results</gi> element in the report, for each
	    <gi>grammar-test</gi> a <gi>grammar-result</gi>, and for
	    each <gi>test-case</gi> and <gi>test-result</gi>.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="test-result">
	  <desc>provides the result, in a test report, of running a
	  given <gi>test-case</gi> in the test catalog.</desc>
	  <attList>
	    <attDef ident="name">
	      <desc>The name of the test case.</desc>
	    </attDef>
	    <attDef ident="result">
	      <desc>Summarizes the result of running the test; the
	      value is one of:
	      <list>
		<item><code>pass</code>, indicating that the reported
		result agrees with the expected result</item>
		<item><code>fail</code>, indicating that the test was
		run and the reported result did not agree with the
		expected result</item>
		<item><code>not-run</code>, indicating that the
		grammar test was not run, for whatever reason
		(the reason can be given in a <gi>description</gi>
		element)</item>
		<item><code>other</code>, indicating that none of the
		other values apply (an explanation can be given in a
		<gi>description</gi> or <gi>app-info</gi>
		element)</item>
	      </list>
	      </desc>
	    </attDef>
	  </attList>
	  <remarks>
	    <p>A child of <gi>test-set-results</gi>.</p>
	    <p>The results of the test case can be given in greater or
	    lesser detail.  The minimum <gi>test-result</gi> element
	    gives the name of the test case and the result.  If
	    desired, however, the test result can also contain or
	    point to the grammar used for the test, the input string,
	    the expected result, and the reported result.</p>
	    <p>For convenience, the schema allows the
	    <gi>test-result</gi> element to be the root element of a
	    document.  It may be desired, for example, that a test
	    harness produce a full test report for a given test
	    catalog with only summary information on each test, and
	    that a detailed report be written out for each failing
	    test case as a separate document.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="test-set">
	  <desc>a set of test cases, nested test sets, or both.</desc>
	  <attList>
	    <attDef ident="name" usage="req">
	      <desc>a name by which the test set can be referred
	      to; should be unique within the test catalog</desc>
	    </attDef>
	  </attList>
	  <remarks>
	    <p>The grammar for a test set may be given in any of
	    several ways:
	    <list>
	      <item>
		<p>in ixml form inline, using an <gi>ixml-grammar</gi>
		element, or</p>
	      </item>
	      <item>		
		<p>in ixml form in an external resource, using an
		<gi>ixml-grammar-ref</gi> element, or</p>
	      </item>
	      <item>
		<p>in XML form inline, using the <gi>vxml-grammar</gi>
		element, or</p>
	      </item>
	      <item>
		<p>in XML form in an external resource, using the
		<gi>vxml-grammar-ref</gi> element.</p>
	      </item>
	    </list>
	    Here, <mentioned>vxml</mentioned> is short for
	    <gloss>visible XML</gloss>, as distinct from
	    <gloss>invisible XML</gloss>.</p>
	    
	    <p>The <gi>grammar-test</gi> element can be used to
	    specify that the grammar itself is to be treated as a test
	    case and parsed or checked against the grammar for ixml
	    grammars.  In this case, the test set need not contain any
	    test cases.</p>
	    
	    <p>Test sets can be nested to reflect any desired
	    structure for the test suite.  Nested test sets can take
	    the form of child <gi>test-set</gi> elements or of child
	    <gi>test-set-ref</gi> elements, which give the URI of a
	    subordinate test set.  (In practice the test set reference
	    is usually to an XML document rooted in
	    <gi>test-catalog</gi>, rather than <gi>test-set</gi>, but
	    the effect is the same: all test sets in the external
	    resource are logically subordinate to the parent of the
	    <gi>test-set-ref</gi> element.)</p>
	    <p>If a grammar is specified by a <gi>test-set</gi>
	    element or by one of its ancestor <gi>test-set</gi>
	    elements, then the test set can contain any number of test
	    cases.  A grammar specified on a descendant overrides any
	    grammar specified on an ancestor; the first grammar found,
	    when ascending from the test case, is the one used.</p>
	    <p>Note that grammars are not inherited across references
	    to external test sets.</p>
	  </remarks>
	  
	</elementSpec>
	
	<elementSpec ident="test-set-ref">
	  <gloss>reference to an external test set</gloss>
	  <desc>points to an external resource with a set of test
	  cases.</desc>
	  <attList>
	    <attDef ident="href">
	      <desc>contains a URI at which the external test set
	      can be found.</desc>
	    </attDef>
	  </attList>
	  <remarks>
	    <p>Despite the name of the element, the usual target of a
	    <gi>test-set-ref</gi> element will be a document rooted in
	    a <gi>test-catalog</gi> element.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="test-set-results">
	  <desc>a collection of test results for a <gi>test-set</gi>
	  in a test catalog.</desc>
	  <attList>
	    <attDef ident="name" usage="req">
	      <desc>the name of the source test set</desc>
	    </attDef>
	  </attList>	  
	</elementSpec>
	
	<elementSpec ident="test-string">
	  <desc>contains a string to be used as input in a test
	  case.</desc>
	  <remarks>
	    <p>If the string is long or otherwise inconvenient
	    to include literally in the test catalog,
	    the <gi>test-string-ref</gi> element may be
	    used instead.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="test-string-ref">
	  <desc>points to a string to be used as input in a test
	  case.</desc>
	
	  <attList>
	    <attDef ident="href">
	      <desc>contains a URI at which the external input
	      can be found.</desc>
	    </attDef>
	  </attList>
	  <remarks>
	    <p>If the string is short and simple, it may be more
	    convenient to include it directly in the test catalog;
	    when this is so, the <gi>test-string</gi> element may be
	    used instead of the <gi>test-string-ref</gi> element.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="vxml-grammar">
	  <gloss>reference to grammar in visible-XML form</gloss>
	  <desc>contains a grammar in XML form.</desc>
	  <remarks>
	    <p>Note that the schema for test catalogs does not
	    constrain the XML used in the content of this element; in
	    cases where the purpose of the test is to check the
	    behavior of a processor when given XML input which is not
	    a valid XML representation of an ixml grammar, the XML
	    content of the <gi>vxml-grammar</gi> element will not in
	    fact conform to the schema for the XML form of ixml
	    grammars.</p>
	    <p>This is one of several elements that can be used to
	    specify a grammar; grammars may be given in ixml or XML
	    form, inline or externally.</p>
	  </remarks>
	</elementSpec>
	
	<elementSpec ident="vxml-grammar-ref">
	  <gloss>reference to grammar in visible-XML form</gloss>
	  <desc>points to a grammar in XML form.</desc>
	  <attList>
	    <attDef ident="href">
	      <desc>contains a URI at which the external grammar
	      can be found.</desc>
	    </attDef>
	  </attList>
	  <remarks>
	    <p>This is one of several elements that can be used to
	    specify a grammar; grammars may be given in ixml or XML
	    form, inline or externally.</p>
	  </remarks>
	</elementSpec>

	<!--
	<elementSpec ident="description">
	  <gloss>test catalog</gloss>
	  <desc>description here</desc>
	  <content>
	    <rng:interleave>
	      <rng:zeroOrMore>
		<rng:ref name="metadata"/>
              </rng:zeroOrMore>
	      <rng:zeroOrMore>
		<rng:choice>
		  <rng:ref name="test-set"/>
		  <rng:ref name="test-set-ref"/>
		</rng:choice>
	      </rng:zeroOrMore>
	    </rng:interleave>
	  </content>
	  <attList>
	    <attDef ident="type" usage="opt">
	      <gloss>ggg</gloss>
	      <desc>the language of the code</desc>
	      <datatype>
		<dataRef key="teidata.enumerated"/>
	      </datatype>
	    </attDef>
	  </attList>
	  <exemplum>
	    <p>
	    </p>
	    <eg><![CDATA[]]></eg>
	    <p>
	    </p>
	  </exemplum>
	  <remarks>
	    <p>
	    </p>
	  </remarks>
	</elementSpec>
	-->
      </div>
    </body>
  </text>
</TEI>