-
Notifications
You must be signed in to change notification settings - Fork 2
/
test-catalog.tags.xml
1003 lines (954 loc) · 36.1 KB
/
test-catalog.tags.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml-stylesheet type="text/xsl" href="../lib/tsd-to-html.xsl"?>
<TEI xmlns="http://www.tei-c.org/ns/1.0"
xmlns:rng="http://relaxng.org/ns/structure/1.0">
<teiHeader>
<fileDesc>
<titleStmt>
<title>XML elements for ixml test catalogs</title>
<author>C. M. Sperberg-McQueen</author>
</titleStmt>
<publicationStmt>
<authority>Black Mesa Technologies LLC</authority>
<date>2021</date>
</publicationStmt>
<sourceDesc>
<p>No source; created in electronic form.</p>
</sourceDesc>
</fileDesc>
<revisionDesc>
<list>
<item>
<p>2022-02-22 : CMSMcQ : add skeletal entries for test-report elements.</p>
</item>
<item>
<p>2021-12-22 : CMSMcQ : copy edits.</p>
</item>
<item>
<p>2021-11-11 : CMSMcQ : substantial copy edits
and revision.</p>
</item>
<item>
<p>2021-11-07 : CMSMcQ : first version of this
tag set description.</p>
</item>
</list>
</revisionDesc>
</teiHeader>
<text>
<front>
<titlePage>
<docTitle>
<titlePart>XML elements for ixml test catalogs</titlePart>
</docTitle>
<docAuthor>C. M. Sperberg-McQueen</docAuthor>
<docDate>7 November 2021, last rev. 22 February 2022</docDate>
</titlePage>
</front>
<body>
<div xml:id="intro">
<head>Introduction</head>
<p>This document lists the elements in an XML vocabulary for
ixml test catalogs and test-result reports, giving a short
description of each. Fuller description and discussion
may be added later if felt to be needed; for now this is
mostly an alphabetical reference list to explain what the
elements are intended to mean and how they are to be used.</p>
<p>The vocabulary described was developed by C. M. Sperberg-McQueen
as a proposal for the invisible-XML Community Group at the
World Wide Web Consortium, but it has not yet been considered
by the group, still less adopted, and it has no official status
of any kind.</p>
<p>The vocabulary describes two different types of document:
<list>
<item>
<p>Test catalogs describe a potentially complex set of
test cases for an invisible XML processor; they can
be organized internally into multiple sets of test cases,
and test sets can nest.</p>
<p>Each test set may specify an ixml grammar in either
ixml or XML form, together with a set of test cases and
optionally nested test sets. An optional
<gi>grammar-test</gi> element signals that the grammar
itself should be regarded as a test case and identifies
the expected result of processing the grammar.</p>
<p>Each test case in a test set supplies input to be
parsed against the grammar for the test set, and
specifies a result.</p>
<p>Grammars and test-case input can be given inline
or in separate files. The expected results can also
be given inline or externally.</p>
</item>
<item>
<p>Test reports describe the results of running a
given processor against the tests in a test catalog.</p>
<p>A metadata section provides information about the
test catalog used and the processor tested. Then,
for each test set and test case, the result produced
by the processor is reported and whether it is or is not
the result expected in the test catalog.</p>
<p><hi>Note: the current version of this vocabulary
does not yet define elements for test reports. Real
soon now.</hi></p>
</item>
</list></p>
<p>A <ref target="catalog-sketch.xml">sample test
catalog</ref> is given in the <code>doc</code> directory of
the ixml-tests Git repository (where this document is also to
be found) in both <ref target="catalog-sketch.xml">XML</ref>
and <ref target="catalog-sketch.html">HTML</ref> (for
situations where dynamic XSLT processing is not possible).
</p>
</div>
<div xml:id="issues">
<head>Known issues</head>
<p>Among the shortcomings of the current state of the
test catalog schema are these.</p>
<p xml:id="reports"><label>No reporting of test
results.</label> XML vocabularies are needed to allow a test
harness to report the detailed results of running a test suite
and comparing the output to the expected results, and to allow
the developers of an implementation to report (perhaps in
somewhat less detail) which tests they have run and not run,
and of those they ran which tests they passed and which they
failed.</p>
<p>An initial sketch of such a vocabulary is expected
soon, but it's not here yet.</p>
<p xml:id="ambiguity"><label>Awkward treatment of ambiguous
sentences.</label> An ambiguous sentence has more than one
parse tree, and the choice of which tree to return is
implementation-dependent (or possibly implementation-defined),
so that a correct implementation may easily produce a parse
tree which is correct but does not match the parse tree
specified in the test case.</p>
<p>The current approach is to allow multiple parse trees, with
the meaning being that if the processor produces any of them,
it passes that test. If the processor returns a parse forest
instead of a single tree, every tree in the returned forest
should be among those specified in the result. This has the
drawback that in some cases the set of parse trees specified
will not be and cannot be complete: in some grammars, some
sentences have an infinite number of parse trees.</p>
<p>For practical purposes, however, we can expect there to be
a finite number of processors using any given test suite, each
of which will return a finite number of trees. If a processor
fails a test on an ambiguous grammar, then the tree returned
by the processor should be checked manually and added to the
specified result. Unless a processor returns a different
result each time it parses the test input, the second and
later runs against the test suite should be problem-free.</p>
<p>There are of course alternatives:
<list>
<item>
<p>Allow an assertion about the result, expressed in
XPath. This is the approach taken in the QT test suite
and in XSpec. Drawback: unless the assertion is very
thorough, the test case will not fully specify the
required test result, allowing incorrect results to pass
the test.</p>
</item>
<item>
<p>Allow a set of assertions about the result, expressed
in XPath. To pass the test, a processor's result must
satisfy all assertions. Drawback: as before, though it's
easier to be thorough.</p>
</item>
<item>
<p>Allow an and/or tree of assertions about the result,
expressed in XPath. To pass the test, a processor's
result must satisfy all children of an <gi>and</gi> and at
least one child of an <gi>or</gi>. Drawback: logical
conjunction and disjunction are already built in to XPath
so it's not really necessary to provide a new set of
machinery for the purpose.</p>
</item>
</list>
As the examples of the QT3 test suite and XSpec show, all of
these involve building additional machinery both into the
vocabulary for the test catalog and also into the test
harness. An XPath assertion must be evaluated in a dynamic
context and must say something about the result returned, so
we now end up having to build a dynamic context and bind the
result to some well known variable like <code>$result</code>,
so that the assertion can say something useful about the
result.</p>
<p>The current approach has the comparative advantage
that it does not require an XPath evaluator: one could in
principle handle the comparison of test results to the
expected output by writing out temporary files and calling a
diff program. The occasional false negative in the test
results seems a small price to pay for that advantage.
</p>
</div>
<div xml:id="elements">
<head>Element reference list</head>
<elementSpec ident="app-info">
<gloss>application information</gloss>
<desc>includes information needed by any
application which is to process the
test catalog; this is an extension mechanism
to allow the inclusion of arbitrary XML
elements in the catalog. May occur
wherever metadata is allowed:
as a child of <gi>test-catalog</gi>,
<gi>test-set</gi>,
<gi>test-case</gi>, etc.</desc>
<exemplum>
<p>The process by which a test case is generated may
produce, as a side effect, a <soCalled>raw parse
tree</soCalled> in which all nonterminals are
present. This can be helpful in diagnosing
problems in an ixml parser, so it may be thought
useful to record the raw parse tree, even though
there is no standard place for it in the schema.
The following example shows a raw parse tree
embedded in a test case.</p>
<eg><![CDATA[
<test-case xml:id="S-1" name="S1">
<created by="Gingersnap (CMSMcQ)"
on="2021-01-26"/>
<test-string/>
<result>
<assert-xml>
<S/>
</assert-xml>
</result>
<app-info>
<raw-parse-tree xmlns="">
<S/>
</raw-parse-tree>
</app-info>
</test-case>]]></eg>
</exemplum>
</elementSpec>
<elementSpec ident="assert-not-a-grammar">
<desc>signals that the grammar specified is not a conforming
ixml grammar (and thus cannot be used to parse the test
input). Correct behavior in this case is to report that
the input cannot be parsed using the grammar and optionally
describe the nature of the problem.</desc>
<remarks>
<p>Appears as child of <gi>result</gi>.</p>
<p>The spec does not require a particular way of signaling
errors in the grammar, so each processor may require a
different setup.</p>
<p>Within the <gi>result</gi> element for a grammar test,
there is no useful distinction between this element and
<gi>assert-not-a-sentence</gi>. Casuists can argue the
case each way. (<hi>The purported grammar is not a
grammar, so this element applies; it is also not a
sentence in the language defined by the grammar for ixml
grammars, so the <gi>assert-not-a-sentence</gi> also
applies. If the problem with the grammar is not failure to
follow the grammar for grammars but violation of some
extra-grammatical constraint like having only one rule per
nonterminal, the input <emph>is</emph> a sentence but
<emph>is not</emph> a grammar. Etc., etc., etc.</hi>) In
practice both elements will and should be treated as
interchangeable in that particular case.</p>
<p>This element can contain any elements of the
<code>metadata</code> class (so: <gi>description</gi>,
<gi>app-info</gi>, <gi>doc</gi>), which leads to a lack of
parallelism with <gi>assert-xml</gi> and
<gi>assert-xml-ref</gi>. Remove?</p>
</remarks>
</elementSpec>
<elementSpec ident="assert-not-a-sentence">
<desc>signals that the test input is not
a sentence in the language described by the
grammar indicated; correct behavior in this
case is to report that the input cannot be
parsed using the grammar and optionally
describe the nature of the problem.</desc>
<remarks>
<p>Appears as child of <gi>result</gi>.</p>
<p>The spec does not require a particular
way of signaling failure to parse the input,
so each processor may require a different
setup.</p>
<p>This element can contain any elements of the
<code>metadata</code> class (so: <gi>description</gi>,
<gi>app-info</gi>, <gi>doc</gi>), which leads to a lack of
parallelism with <gi>assert-xml</gi> and
<gi>assert-xml-ref</gi>. Remove?</p>
<p>Within the <gi>result</gi> element for a grammar test,
there is no useful distinction between this element and
<gi>assert-not-a-grammar</gi>; in that context, the two
should in practice be treated as interchangeable.</p>
</remarks>
</elementSpec>
<elementSpec ident="assert-xml">
<desc>signals that the test input is a sentence in the
language described by the grammar indicated and contains an
acceptable XML representation of one parse of the
sentence.</desc>
<remarks>
<p>Appears as child of <gi>result</gi>.</p>
<p>For ambiguous sentences, more than one parse is
correct, and this element may occur more than once.</p>
</remarks>
</elementSpec>
<elementSpec ident="assert-xml-ref">
<desc>signals that the test input is a sentence in the
language described by the grammar indicated and give a URI
for the prescribed XML representation of one parse of the
sentence.</desc>
<remarks>
<p>Appears as child of <gi>result</gi>.</p>
<p>For ambiguous sentences, more than one parse is
correct, and this element may occur more than once.</p>
</remarks>
</elementSpec>
<elementSpec ident="catalog-date">
<desc>gives the release date of the test catalog describing the tests
whose results are included in a test report.</desc>
<remarks>
<p>Occurs as a child of <gi>metadata</gi> at the top of
the test report.</p>
<p>The <gi>catalog-date</gi> will normally just repeat the
information given in the <att>release-date</att> attribute
on the <gi>test-catalog</gi> element. The purpose of
including it in the test report is to help document which
version of the test catalog was used, if the test catalog
is revised.</p>
</remarks>
</elementSpec>
<elementSpec ident="catalog-uri">
<desc>gives the URI of the test catalog describing the tests
whose results are included in a test report.</desc>
<remarks>
<p>Occurs as a child of <gi>metadata</gi> at the top of
the test report.</p>
</remarks>
</elementSpec>
<elementSpec ident="code">
<desc>marks inline code fragments in prose.</desc>
<remarks>
<p>Typical typographic treatment is to display in a
monospaced font.</p>
</remarks>
</elementSpec>
<elementSpec ident="created">
<desc>a metadata element containing information about
the creation of a test catalog, test set, or test case.</desc>
<attList>
<attDef ident="who">
<desc>specifies who created the item.</desc>
</attDef>
<attDef ident="when">
<desc>specifies the date the item was created.</desc>
</attDef>
</attList>
<remarks>
<p>The element is allowed but optional on individual test
cases and grammar tests. If no creation date is given for
an individual test case, it is assumed to have been
created at the same time as the test set as a whole; if it
was added later, it can (and ideally should) have its date
of creation and revision history noted. When a test suite
is changed over time, it can be helpful if every test case
has a complete revision history.</p>
</remarks>
</elementSpec>
<elementSpec ident="description">
<desc>contains a sequence of paragraphs containing a
natural-language description of a test catalog, test set, or
test case.</desc>
<remarks>
<p>Ideally, the <gi>description</gi> element should be
able to contain prose tagged in XHTML, or in the XML
vocabulary of the author's choice. Currently this is not
the case. <!-- <hi>Experience with XSD makes me lean
toward prescribing HTML, to make the opportunity of using
full markup easier to grasp.</hi> --></p>
</remarks>
</elementSpec>
<elementSpec ident="doc">
<desc>refers to external documentation for the test catalog,
test set, or test case, if there is any.</desc>
<attList>
<attDef ident="href">
<desc>contains a URI at which the external documentation
can be found.</desc>
</attDef>
</attList>
<remarks>
<p>If a test case exhibits a particularly tricky situation
which has been extensively discussed or analysed by
the community group (or anyone else), for example, it's
helpful to be able to provide pointers instead of having
to reproduce the discussion inside the test catalog,
or omitting it entirely.</p>
</remarks>
</elementSpec>
<elementSpec ident="emph">
<desc>marks emphasized text in prose.</desc>
<remarks>
<p>This element is intended to mark rhetorical emphasis or
stress, not arbitrary special typographic treatment. (If
we turn out to need markup for arbitrary typographic
treatment, we'll want to introduce <gi>hi</gi> for
typographic highlighting, or <gi>seg</gi> or <gi>span</gi>
with a <att>style</att> or <att>rendering</att>
element.)</p>
<p><hi>Note: if we turn out to need markup for arbitrary
typographic treatment, we would do better to specify that
the prose should be marked up in XHTML.</hi></p>
</remarks>
</elementSpec>
<elementSpec ident="grammar-result">
<desc>provides the result, in a test report, of running a
given <gi>grammar-test</gi> in the test catalog.</desc>
<attList>
<attDef ident="result">
<desc>Summarizes the result of running the test; the
value is one of:
<list>
<item><code>pass</code>, indicating that the reported
result agrees with the expected result</item>
<item><code>fail</code>, indicating that the test was
run and the reported result did not agree with the
expected result</item>
<item><code>not-run</code>, indicating that the
grammar test was not run, for whatever reason
(the reason can be given in a <gi>description</gi>
element)</item>
<item><code>other</code>, indicating that none of the
other values apply (an explanation can be given in a
<gi>description</gi> or <gi>app-info</gi>
element)</item>
</list>
</desc>
</attDef>
</attList>
<remarks>
<p>A child of <gi>test-set-results</gi>.</p>
<p>The results of the grammar test can be given in greater
or lesser detail. The minimum <gi>grammar-result</gi>
element gives the result of the test. If desired,
however, the test result can also contain or point to the
grammar tested, the expected result, and the reported
result.</p>
<p>For convenience, the schema allows the
<gi>grammar-result</gi> element to be the root element of
a document.</p>
</remarks>
</elementSpec>
<elementSpec ident="grammar-test">
<desc>indicates that the grammar associated with a given
test set should be treated as a test case and checked for
conformance; contains a <gi>result</gi> element indicating
expected result of the test.</desc>
<remarks>
<p>A child of <gi>test-set</gi>.</p>
<p>If the grammar is given in an <gi>ixml-grammar</gi>
element or pointed to with an <gi>ixml-grammar-ref</gi>
element, the grammar should be parsed with the grammar for
ixml and the result compared with the expected result.
The extra-grammatical constraints imposed by the spec
should also be checked. If the grammar given is in XML
(using the <gi>vxml-grammar</gi> or
<gi>vxml-grammar-ref</gi> elements), the processor should
check it for structural correctness (the spec requires
that the XML grammar <q>can be derived from an ixml
grammar in ixml form by parsing as described in this
specification</q>) and for conformance to the
extra-grammatical or non-structural constraints in the
ixml specification.</p>
</remarks>
</elementSpec>
<elementSpec ident="ixml-grammar">
<desc>contains a grammar in ixml form.</desc>
<exemplum>
<p>The following example shows an ixml grammar for
arithmetic expressions.
</p>
<eg><![CDATA[ <ixml-grammar>
{ Grammar for simple arithmetic expressions,
for testing of test-case generation and
grammar-manipulation tools.
Revisions:
2020-12-30 : CMSMcQ : made grammar
}
expr: s*, term+addop.
term: factor+mulop.
factor: num; var;
'(', expr, ')';
'[', expr, ']';
'{', expr, '}'.
num: digit+, ('.', digit*)?.
var: ['a'-'z'].
addop: s*, ["+-"], s*.
mulop: s*, ["*/×÷"], s*.
digit: ['0'-'9'].
s: #20; #9; #a.
</ixml-grammar>
]]></eg>
</exemplum>
<remarks>
<p>Appears as a child of <gi>test-set</gi> and gives
the grammar for the test cases of the set.</p>
<p>This is one of several elements that can be used to
specify a grammar; grammars may be given in ixml or XML
form, inline or externally.</p>
</remarks>
</elementSpec>
<elementSpec ident="ixml-grammar-ref">
<gloss>reference to an external grammar in ixml form</gloss>
<desc>points to an external resource containing
an ixml grammar.</desc>
<attList>
<attDef ident="href">
<desc>contains a URI at which the external grammar
can be found.</desc>
</attDef>
</attList>
<remarks>
<p>Appears as a child of <gi>test-set</gi> and gives
the grammar for the test cases of the set.</p>
<p>This is one of several elements that can be used to
specify a grammar; grammars may be given in ixml or XML
form, inline or externally.</p>
</remarks>
</elementSpec>
<elementSpec ident="metadata">
<desc>contains a short sequence of metadata elements describing
the contents of a test report.</desc>
<remarks>
<p>Children include <gi>name</gi> (of the report),
<gi>report-date</gi>,
<gi>processor</gi>,
<gi>processor-version</gi>,
<gi>catalog-uri</gi>, and
<gi>catalog-date</gi>.
</p>
</remarks>
</elementSpec>
<elementSpec ident="modified">
<desc>contains an entry in the change log for a test case
or test set describing a modification.</desc>
<attList>
<attDef ident="change">
<desc>describes the change.</desc>
</attDef>
<attDef ident="who">
<desc>specifies who made the change.</desc>
</attDef>
<attDef ident="when">
<desc>specifies the date the change was made.</desc>
</attDef>
</attList>
<remarks>
<p></p>
</remarks>
</elementSpec>
<elementSpec ident="name">
<desc>gives the name assigned to a test report.</desc>
<remarks>
<p>Occurs as a child of <gi>metadata</gi> at the top of
the test report.</p>
<p>The analogous name of a test catalog is given as
an attribute on <gi>test-catalog</gi>, not as an
element.</p>
</remarks>
</elementSpec>
<elementSpec ident="p">
<desc>a paragraph of natural-language prose.</desc>
<remarks>
<p>A very small number of phrase-level elements
are provided for the markup of the prose; if the
set proves inconveniently small, it can easily be
grown.</p>
</remarks>
</elementSpec>
<elementSpec ident="processor">
<desc>indicates the processor used in the testing described
in a a test report.</desc>
<remarks>
<p>Occurs as a child of <gi>metadata</gi> at the top of
the test report.</p>
</remarks>
</elementSpec>
<elementSpec ident="processor-version">
<desc>gives the version number or identifier, if any, of
the processor for which a test report was prepared.</desc>
<remarks>
<p>Occurs as a child of <gi>metadata</gi> at the top of
the test report. Optional.</p>
</remarks>
</elementSpec>
<elementSpec ident="report-date">
<desc>gives the date on which a test report was prepared.</desc>
<remarks>
<p>Occurs as a child of <gi>metadata</gi> at the top of
the test report.</p>
</remarks>
</elementSpec>
<elementSpec ident="reported-not-a-grammar">
<desc>in a test report, signals that the processor reported
that the grammar supplied for a test is not a conforming
ixml grammar (and thus cannot be used to parse the test
input).</desc>
<remarks>
<p>Appears as child of <gi>result</gi>.</p>
</remarks>
</elementSpec>
<elementSpec ident="reported-not-a-sentence">
<desc>in a test report, signals that the test input is not a
sentence in the language described by the grammar supplied for the test.</desc>
<remarks>
<p>Appears as child of <gi>result</gi>.</p>
</remarks>
</elementSpec>
<elementSpec ident="reported-xml">
<desc>in a test report, signals that the processor parsed
the test input as a sentence in the language described by
the grammar supplied; contains the XML element produced as
output.</desc>
<remarks>
<p>Appears as child of <gi>result</gi>.</p>
<p>For ambiguous sentences, more than one parse is
correct, and this element may occur more than once.</p>
</remarks>
</elementSpec>
<elementSpec ident="reported-xml-ref">
<desc>in a test report, signals that the processor parsed
the test input as a sentence in the language described by
the grammar supplied; gives a URI for the XML output
produced by the processor.</desc>
<remarks>
<p>Appears as child of <gi>result</gi>.</p>
<p>For ambiguous sentences, more than one parse is
correct, and this element may occur more than once.</p>
</remarks>
</elementSpec>
<elementSpec ident="result">
<desc>in a test catalog specifies the expected result of a
test, using the <gi>assert-xml</gi>, <gi>assert-xml-ref</gi>
or the <gi>assert-not-a-sentence</gi> elements depending on
what is to be asserted and how; in a test report may
optionally recapitulate the expected result and may give the
output reported by the processor.</desc>
<remarks>
<p>Unlike a test catalog, which is required to specify the
expected result exactly once, a test report may give the
expected result in multiple forms (i.e. both inline and by
reference to an external document). The same holds true
for the reported result.</p>
</remarks>
</elementSpec>
<elementSpec ident="test-case">
<desc>contains input and expected result for one test
case.</desc>
<attList>
<attDef ident="name" usage="req">
<desc>a name by which the test case can be referred
to; should be unique within the test set</desc>
</attDef>
</attList>
<exemplum>
<p>The following example shows a test case for the grammar
<code>S: {nil}.</code> (not shown, because it's specified
earlier in the catalog). The input is the empty string,
as specified here by the <gi>test-string</gi> element.
The expected output is an empty <gi>S</gi> element, as
specified here by the <gi>assert-xml</gi> element within
<gi>result</gi>.</p>
<eg><![CDATA[<test-case xml:id="S-1" name="S1">
<created by="Gingersnap (CMSMcQ)"
on="2021-01-26"/>
<test-string/>
<result>
<assert-xml>
<S/>
</assert-xml>
</result>
</test-case>]]></eg>
</exemplum>
<remarks>
<p>A processor being tested should parse the specified
input using the grammar supplied by a containing
<gi>test-set</gi> element. The processor passes the test
if the processor's output matches the expected result
(specified in the <gi>result</gi> element).</p>
</remarks>
</elementSpec>
<elementSpec ident="test-catalog">
<desc>contains a collection of test cases organized into one
or more test sets and released at the same time.</desc>
<content>
<rng:interleave>
<rng:zeroOrMore>
<rng:ref name="metadata"/>
</rng:zeroOrMore>
<rng:zeroOrMore>
<rng:choice>
<rng:ref name="test-set"/>
<rng:ref name="test-set-ref"/>
</rng:choice>
</rng:zeroOrMore>
</rng:interleave>
</content>
<attList>
<attDef ident="name" usage="req">
<desc>a name by which the collection of tests can be
referred to</desc>
</attDef>
<attDef ident="release-date" usage="req">
<desc>date of the test suite described by this
catalog; more specifically, date of its release
or official publication</desc>
</attDef>
</attList>
<remarks>
<p>When a suite of related tests is published, it may
happen that there are different test-catalog files (and
<gi>test-catalog</gi> elements) for different groups of
tests within the larger collection. It is a good idea for
there to be one test catalog that includes all of the
others, so as to provide a single point of entry for
processing the entire collection of tests. The
<gi>test-set-ref</gi> element is used to include other
test-catalog files.</p>
</remarks>
</elementSpec>
<elementSpec ident="test-report">
<desc>contains a collection of test-case results derived by
running the tests from a given test catalog through a
particular processor.</desc>
<remarks>
<p>The overall structure of the test report mirrors the
structure of the test catalog: for each <gi>test-set</gi>
in the catalog, there will normally be a
<gi>test-set-results</gi> element in the report, for each
<gi>grammar-test</gi> a <gi>grammar-result</gi>, and for
each <gi>test-case</gi> and <gi>test-result</gi>.</p>
</remarks>
</elementSpec>
<elementSpec ident="test-result">
<desc>provides the result, in a test report, of running a
given <gi>test-case</gi> in the test catalog.</desc>
<attList>
<attDef ident="name">
<desc>The name of the test case.</desc>
</attDef>
<attDef ident="result">
<desc>Summarizes the result of running the test; the
value is one of:
<list>
<item><code>pass</code>, indicating that the reported
result agrees with the expected result</item>
<item><code>fail</code>, indicating that the test was
run and the reported result did not agree with the
expected result</item>
<item><code>not-run</code>, indicating that the
grammar test was not run, for whatever reason
(the reason can be given in a <gi>description</gi>
element)</item>
<item><code>other</code>, indicating that none of the
other values apply (an explanation can be given in a
<gi>description</gi> or <gi>app-info</gi>
element)</item>
</list>
</desc>
</attDef>
</attList>
<remarks>
<p>A child of <gi>test-set-results</gi>.</p>
<p>The results of the test case can be given in greater or
lesser detail. The minimum <gi>test-result</gi> element
gives the name of the test case and the result. If
desired, however, the test result can also contain or
point to the grammar used for the test, the input string,
the expected result, and the reported result.</p>
<p>For convenience, the schema allows the
<gi>test-result</gi> element to be the root element of a
document. It may be desired, for example, that a test
harness produce a full test report for a given test
catalog with only summary information on each test, and
that a detailed report be written out for each failing
test case as a separate document.</p>
</remarks>
</elementSpec>
<elementSpec ident="test-set">
<desc>a set of test cases, nested test sets, or both.</desc>
<attList>
<attDef ident="name" usage="req">
<desc>a name by which the test set can be referred
to; should be unique within the test catalog</desc>
</attDef>
</attList>
<remarks>
<p>The grammar for a test set may be given in any of
several ways:
<list>
<item>
<p>in ixml form inline, using an <gi>ixml-grammar</gi>
element, or</p>
</item>
<item>
<p>in ixml form in an external resource, using an
<gi>ixml-grammar-ref</gi> element, or</p>
</item>
<item>
<p>in XML form inline, using the <gi>vxml-grammar</gi>
element, or</p>
</item>
<item>
<p>in XML form in an external resource, using the
<gi>vxml-grammar-ref</gi> element.</p>
</item>
</list>
Here, <mentioned>vxml</mentioned> is short for
<gloss>visible XML</gloss>, as distinct from
<gloss>invisible XML</gloss>.</p>
<p>The <gi>grammar-test</gi> element can be used to
specify that the grammar itself is to be treated as a test
case and parsed or checked against the grammar for ixml
grammars. In this case, the test set need not contain any
test cases.</p>
<p>Test sets can be nested to reflect any desired
structure for the test suite. Nested test sets can take
the form of child <gi>test-set</gi> elements or of child
<gi>test-set-ref</gi> elements, which give the URI of a
subordinate test set. (In practice the test set reference
is usually to an XML document rooted in
<gi>test-catalog</gi>, rather than <gi>test-set</gi>, but
the effect is the same: all test sets in the external
resource are logically subordinate to the parent of the
<gi>test-set-ref</gi> element.)</p>
<p>If a grammar is specified by a <gi>test-set</gi>
element or by one of its ancestor <gi>test-set</gi>
elements, then the test set can contain any number of test
cases. A grammar specified on a descendant overrides any
grammar specified on an ancestor; the first grammar found,
when ascending from the test case, is the one used.</p>
<p>Note that grammars are not inherited across references
to external test sets.</p>
</remarks>
</elementSpec>
<elementSpec ident="test-set-ref">
<gloss>reference to an external test set</gloss>
<desc>points to an external resource with a set of test
cases.</desc>
<attList>
<attDef ident="href">
<desc>contains a URI at which the external test set
can be found.</desc>
</attDef>
</attList>
<remarks>
<p>Despite the name of the element, the usual target of a
<gi>test-set-ref</gi> element will be a document rooted in
a <gi>test-catalog</gi> element.</p>
</remarks>
</elementSpec>
<elementSpec ident="test-set-results">
<desc>a collection of test results for a <gi>test-set</gi>
in a test catalog.</desc>
<attList>
<attDef ident="name" usage="req">
<desc>the name of the source test set</desc>
</attDef>
</attList>
</elementSpec>
<elementSpec ident="test-string">
<desc>contains a string to be used as input in a test
case.</desc>
<remarks>
<p>If the string is long or otherwise inconvenient
to include literally in the test catalog,
the <gi>test-string-ref</gi> element may be
used instead.</p>
</remarks>
</elementSpec>
<elementSpec ident="test-string-ref">
<desc>points to a string to be used as input in a test
case.</desc>
<attList>
<attDef ident="href">
<desc>contains a URI at which the external input
can be found.</desc>
</attDef>
</attList>
<remarks>
<p>If the string is short and simple, it may be more
convenient to include it directly in the test catalog;
when this is so, the <gi>test-string</gi> element may be
used instead of the <gi>test-string-ref</gi> element.</p>
</remarks>
</elementSpec>
<elementSpec ident="vxml-grammar">
<gloss>reference to grammar in visible-XML form</gloss>
<desc>contains a grammar in XML form.</desc>
<remarks>
<p>Note that the schema for test catalogs does not
constrain the XML used in the content of this element; in
cases where the purpose of the test is to check the
behavior of a processor when given XML input which is not
a valid XML representation of an ixml grammar, the XML
content of the <gi>vxml-grammar</gi> element will not in
fact conform to the schema for the XML form of ixml
grammars.</p>
<p>This is one of several elements that can be used to
specify a grammar; grammars may be given in ixml or XML
form, inline or externally.</p>
</remarks>
</elementSpec>
<elementSpec ident="vxml-grammar-ref">
<gloss>reference to grammar in visible-XML form</gloss>
<desc>points to a grammar in XML form.</desc>
<attList>
<attDef ident="href">
<desc>contains a URI at which the external grammar
can be found.</desc>
</attDef>
</attList>
<remarks>
<p>This is one of several elements that can be used to
specify a grammar; grammars may be given in ixml or XML
form, inline or externally.</p>
</remarks>
</elementSpec>
<!--
<elementSpec ident="description">
<gloss>test catalog</gloss>
<desc>description here</desc>
<content>
<rng:interleave>
<rng:zeroOrMore>
<rng:ref name="metadata"/>
</rng:zeroOrMore>
<rng:zeroOrMore>
<rng:choice>
<rng:ref name="test-set"/>
<rng:ref name="test-set-ref"/>
</rng:choice>
</rng:zeroOrMore>
</rng:interleave>
</content>
<attList>
<attDef ident="type" usage="opt">
<gloss>ggg</gloss>
<desc>the language of the code</desc>
<datatype>
<dataRef key="teidata.enumerated"/>
</datatype>
</attDef>
</attList>
<exemplum>
<p>
</p>
<eg><![CDATA[]]></eg>
<p>
</p>
</exemplum>
<remarks>
<p>
</p>
</remarks>
</elementSpec>
-->
</div>