/
XmlSchema.shtml
604 lines (504 loc) · 25.5 KB
/
XmlSchema.shtml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html lang="en">
<head>
<meta name="generator" content=
"HTML Tidy for Mac OS X (vers 31 October 2006 - Apple Inc. build 15.17), see www.w3.org">
<title>JMRI: Xml Schema Usage</title>
<meta name="author" content="Bob Jacobsen">
<meta name="keywords" content=
"JMRI technical code xml schema usage">
<!-- The combination of "Define" and {Header,Style, Logo and Footer} comments -->
<!-- are an arbitrary design pattern used by the update.pl script to -->
<!-- easily replace the common header/footer code for all the web pages -->
<!-- delete the following 2 Defines if you want to use the default JMRI logo -->
<!-- or change them to reflect your alternative logo -->
<!--#include virtual="/Style.shtml" -->
</head><!--#include virtual="/Header.shtml" -->
<body>
<div id="mBody">
<!--#include virtual="Sidebar.shtml" -->
<div id="mainContent">
<h1>JMRI: XML Schema Usage</h1>
<p>JMRI uses XML for a number of
purposes: to hold decoder definitions, for its <a href=
"XmlPersistance.shtml">persistance system</a> for
configuration and panel information, and to create parts of
the web site from other files. This page describes how we
specify the content of those files using <a href=
"http://www.w3schools.com/schema/schema_intro.asp">XML
schema</a>.</p>
<p>For examples (not a tutorial!) on the structure of our
schema, see the <a href="XmlSchemaExamples.shtml">examples
page</a>.</p>
<p>The current schema can be seen online in the <a href=
"http://jmri.org/xml/schema">schema directory</a>. The most
commonly used one is the <a href=
"http://jmri.org/xml/schema/layout.xsd">layout.xsd schema for
panel files</a>. See below on how it's organized.</p>
<h2>Access to Schema Definitions</h2>JMRI uses XML Schema to
define the format of its files.
<p>Those XML Schema may need to be available to the program
when it reads the files, as they define the default values of
missing attributes and other needed information.</p>
<p>In the JMRI distributions, these are stored in the
<a href="/xml/schema">xml/schema</a> directory. Note that
they are not stored in each directory alongside the XML
files. There are just too many locations to keep such a set
of schema definition files up to date. JMRI itself, via the
jmri.jmrit.XmlFile class, provides support for locating those
files when the XML parser needs them.</p>
<h2 id="modschema">Modifying JMRI Schema</h2>This section
talks about how to handle small changes to existing schema,
for example added or removing an attribute or element. For
more extensive changes, including creating entirely new types
or new file formats, see the next section on "<a href=
"#devschema">Developing JMRI Schema</a>".
<p>Every time you change what JMRI writes (and therefore
reads) in an XML file, you must also do the following:</p>
<ol>
<li>You need to change the code that does the reading and
writing.</li>
<li>You need to change the schema file(s) so that the XML
format can be properly checked.</li>
<li>You have to provide new test XML files to make sure
nothing has been broken, and in some cases have to adjust
old ones.</li>
</ol>
<p>Please don't skip the latter steps. They matter a lot to
the long-term stability of JMRI code.</p>
<p>If possible, it's best to make changes by addition, so
that existing files can continue to be read unchanged. JMRI
strongly values backward compatibility, where any newer
version of JMRI can still load and use a file written by an
older version.</p>
<p>If you can make a change that's just an addition, then the
process is:</p>
<ol>
<li>Change your code</li>
<li>Add the new elements and attributes to the most recent
version of the schema file(s).</li>
<li>Run "ant headlesstest" to make sure that the older
files already present in the tests can still be processed.
Fix anything that breaks. (You may discover at this point
that you didn't actually make a backward compatible change,
in which case either improve that or see the section on
"<a href="#versioning">Schema Versioning</a>" below).</li>
<li>Create a test file with the new content. Ideally, this
won't require using the screen, so it can actually be
loaded and stored as part of headlesstest. In that case,
put your file in a "load" test subdirectory (see below). At
a minimum, though, put your file in a "verify" test
subdirectory so that your schema changes will be tested.
For more on this, see <a href=
"#junitschematest">below</a>.</li>
</ol>
<h3 id="versioning">Schema Versioning</h3>"Versioning" schema
allows us to have different schema for older and newer files.
This lets newer versions of JMRI continue to check and
read files that were written by older versions of JMRI. This
backward compatibility is an important JMRI feature which we
don't want to lose.
<p>In practical terms, versioning consists of having
multiple-but-related versions of the schema definition files
which are labeled by the first JMRI version that can read
them.</p>
<p>When do you need to create a new version?</p>
<ul>
<li>You <em>do not</em> need to create a new version of the
schema if you add or change things such that existing files
will continue to validate.
<p>In that case, just make your schema changes in the
current schema document, and commit them back to the JMRI
code repository.</p>
</li>
<li>You <em>do</em> need to version a schema when you make a
change to the schema such that previous files will no
longer validate with the current schema.
<p>In this case, the steps to version the schema are:</p>
<ol>
<li>Copy the current schema file to a new one with the
number of the <em>next</em> JMRI release. E.g. copy
types/turnouts-2-9-6.xsd to types/turnouts-3.7.3.xsd if
you're doing this before JMRI 3.7.3 is released. Make
your changes and commit that new version.</li>
<li>If this is a subfile, such as the
types/turnouts-2-9-6.xsd, that is included in a main
schema such as layout-2-9-6.xsd, that main file also
needs to be copied, have the include changed, and
commited. Now you've got a new layout-3-7-3.xsd schema
for output files to reference.</li>
<li>Then, change the Java code that writes the schema
reference to the top of output files to use the new
filename. For example, layout (panel) files are written
by
<code>src/jmri/configurexml/ConfigXmlManager.java</code>.
Look for the<br>
<code>static final public String schemaVersion =
"-3-7-3"</code><br>
line and change it to your new version number
suffix.</li>
<li>Run "ant headlesstest" and when tests fail, fix them.
If a LoadAndStoreTest flags a compare inLine/outLine
error, update the schema version in the header of the
reference file. For example in
<code>java/test/jmri/configurexml/loadref/BlockAndSignalMastTest.xml</code>
point to the most recent layout.xsd version in line 3:<br>
<code><layout-config
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://jmri.org/xml/schema/layout-4-7-2.xsd">
</code>
</li>
<li>If there's an associated XML stylesheet(s), its
name has to be changed in a coordinated way. (You
should also update the stylesheet to show your new XML
content, of course).</li>
</ol>
</li>
</ul>In either case, it's important to include sufficient
test files that the unit tests catch any problems with the
new and old schema. See the <a href="#test">test section
below</a>.
<p>Note that the unlabeled schema is the primordial, oldest,
now-obsolete schema. For example layout.xsd is <em>older</em>
than layout-2-9-6.xsd, and therefore not to be used for new
files anymore. Don't assume that layout.xsd is the default
for new files!</p>
<h2>Checking JMRI Schema</h2>It's important that the JMRI
schema definitions be kept semantically correct. If we let
too many problems build up, we'll eventually have a lot of
back-fitting to do. The W3C online <a href=
"http://www.w3.org/2001/03/webdata/xsv">schema validation
tool</a> is a very good tool for checking that changes to the
JMRI schema are still technically correct. You should check
your changes with it before commiting them to the repository.
Unfortunately, it doesn't seem to check compliance with
nested schema elements, e.g. from DocBook (see below) or JMRI
schema, but it's still a very useful check.
<p>Using the JMRI "Validate XML File" tool in the "Debug"
menu to validate a .xml file ("instance file") that uses your
new or updated schema is an important check of both. Use it
often during development. You can also use it from the
command line via e.g.:</p>
<pre>
<code>
./runtest.csh jmri/jmrit/XmlFileValidateAction xml/decoders/0NMRA.xml
</code>
</pre>
<p>For a quick file check, Linux and Mac OS X users can
validate from the command line with e.g.</p>
<pre>
<code>
cd xml
xmllint -schema schema/aspecttable.xsd -noout signals/sample-aspects.xml
</code>
</pre><code>xmllint</code> can't check schema files themselves,
unfortunately, because their schema isn't something it can handle.
<p>Your schema docs should point to our standard stylesheet
in their head matter:</p>
<pre>
<code>
<?xml-stylesheet href="schema2xhtml.xsl" type="text/xsl"?>
</code>
</pre>Stylesheets turn XML code, like this schema, into a
human-readable form when the XML is parsed and displayed by a
browser. For an example of that, click on this link to the
<a href="http://jmri.org/xml/schema/aspecttable.xsd">aspecttable.xsd</a>
schema file. Our standard stylesheet is pretty basic. It just
shows basic structure. If anybody knows of a better
stylesheet, we can certainly switch to it.
<h3><a id="test" name="test">JUnit testing</a></h3>
You should also add a
<a href="JUnit.shtml">JUnit test</a> that checks a typical file.
There are three kinds of checks that can be done:
<ol>
<li>You should always have a class that validates a typical
file against the schema.
<p>That's done by having a SchemaTest class in your
test-tree package (see e.g.
<a href="https://github.com/JMRI/JMRI/blob/master/java/test/jmri/configurexml/SchemaTest.java">
test/jmri/configurexml/SchemaTest.java</a> for
an example) that checks all the XML files kept there. If
that's in place, just put a copy of a (new) typical XML
file in the existing "verify" subdirectory.</p>
<p>To more extensively check your schema, you can check
that it fails XML files that you think are not valid.
There are lots of ways to be not valid, and you don't
need to check them all, but if there something specific
you want to be sure of, put an example of that in the
"invalid" subdirectory. Those files are expected to fail
for some specific reason. You should document that reason
via comments in the file itself so your colleagues can
figure it out later.</p>
<p>If there's no "verify" subdirectory, create one and
add it to the end of the SchemaTest class in that
package. If there's no SchemaTest class, create one by
replicating an existing one, see link above. Don't forget
to add it to PackageTest so it gets invoked!</p>
</li>
<li>If you're working on configurexml files (panel files),
and your new code isn't involved in the actual display of
panels (e.g. can run as part of headlesstest), you should
add a test that a sample file can be loaded and stored back
successfully. Beyond just checking the schema, this also
checks that all the load and store plumbing is working. (We
run these tests headless as part of Jenkins, so please
don't add tests that pop screen windows; they'll just cause
errors).
<p>That's done by having a LoadAndStoreTest class in your
test-tree package (see e.g.
<a href="https://github.com/JMRI/JMRI/blob/master/java/test/jmri/configurexml/LoadAndStoreTest.java">
test/jmri/configurexml/LoadAndStoreTest.java</a>
for an example) that checks all the XML files kept there.
If that's in place, just put a copy of a (new) typical
XML file in the existing "load" subdirectory.</p>
<p>If there's no "load" subdirectory, create one and add
it to the end of the LoadAndStoreTest class in that
package. If there's no LoadAndStoreTest class, create one
by replicating an existing one, see link above. Don't
forget to add it to PackageTest so it gets invoked!</p>
<p>When LoadAndStoreTest runs, it loads the files in the
load directory one-by-one, storing each back out to the
"temp" directory within the local preferences directory,
and then compares the input and output files. Sometimes
this load-and-store process results in something that's
in a different order, or contains more info (e.g.
attributes missing from the input file are written with
default values in the output file). If the comparison
fails, but the output file is still OK when you manually
inspect it, copy that output file to the "loadref"
directory (create it if needed) within your test package.
See <a href=
"http://sourceforge.net/p/jmri/code/HEAD/tree/trunk/jmri/java/test/jmri/configurexml/loadref"
target="_blank">test/jmri/configurexml/loadref</a> for an
example. LoadAndStoreTest will compare to files it finds
here, instead of to the original file in the "load"
subdirectory.</p>
<p>If your changes to the code cause this testing to fail
with older versions of the file, <em>do not change the
older file version.</em> Instead, either put an updated
output reference in the "loadref" directory, version the
schema to allow the older file to continue to load, or
improve your code. Backward compatibility is
important!</p>
</li>
<li>You can also add a custom JUnit test that reads your
sample file and makes sure that the proper objects have
been created, that they have the correct data and state,
etc. This could be anything from a "load and check that new
beans exist in the new manager" to something quite more
extensive.</li>
</ol>At a minimum, please do the schema checks. They're easy,
and will save lots of trouble in the future. If your new
features don't involve displaying on the screen, adding
load-and-store checks is also valuable, and isn't so hard.
<p><em>Note: Do not remove or modify any existing XML file
checks. Those keep old versions of the files working!</em> If
your new code and/or schema breaks the processing of the
existing files, you need to either fix your code or <a href=
"#versioning">version the schema</a> to allow multiple
formats to coexist.</p>
<h2 id="devschema">Developing JMRI Schema</h2>For some
examples of the XML schema structures described here, see the
separate <a href="XmlSchemaExamples.shtml">XML Schema
Examples page</a>/
<p>Our preferred organization for XML schema is based on the
structure of the underlying code: A particular *Xml class is
the unit of reuse.</p>
<p>Lots of classes descend from jmri.configurexml.XmAdapter:
(<a href=
"http://jmri.org/JavaDoc/doc/jmri/configurexml/XmlAdapter.html">see
Javadoc</a>)</p>
<p>By convention, provide
<xsd:annotation><xsd:appinfo> element containing
the class name that reads/writes the element:</p>
<pre>
<code>
<xs:annotation>
<xs:documentation>
Some human readable docs go here
</xs:documentation>
<xs:appinfo>
<jmri:usingclass configurexml="false">jmri.managers.DefaultSignalSystemManager</jmri:usingclass>
</xs:appinfo>
</xs:annotation>
</code>
</pre>
<h3>The Venetian Blinds Pattern</h3>We are moving toward
structuring our XML using the "<a href=
"http://www.xfront.com/GlobalVersusLocal.html#ThirdDesignCharacteristics">Venetian
Blinds pattern</a>". In this style, the top level elements
that are written by classes have types defined for them. Any
elements that are within those are defined anonymously,
within those elements. For an example of this, see the
<a href=
"http://jmri.org/xml/schema/types/sensors.xsd">types/sensors.xsd</a>
file, which defines a type for the "sensors" element written
for SensorManagers. Within that, there is included a
definition of a "Sensor" element, and a "comment" element
within that.
<p>This limits the number of types, and keeps the schema
files roughly aligned with the classes that do the reading
and writing.</p>
<p>There are a few items (elements and attribute groups) that
extend across multiple types. They are defined in the
<a href="http://jmri.org/xml/schema/types/general.xsd">types/general.xsd</a>
file.</p>
<p>More information on XML schema design patterns is
available at <a href=
"http://www.ibm.com/developerworks/xml/library/ws-soa-xmlwsdl.html#N1012B">
DeveloperWorks</a> and the <a href=
"http://www.oracle.com/technetwork/java/design-patterns-142138.html">
Oracle Java site</a>.</p>
<h3>On Elements vs Attributes</h3>When defining how to store
new classes or updates to classes, the basic questions are:
<ul>
<li>Are we storing data? In that case, it should be stored
in an element of its own. Comments, speed values, user and
system names are all examples of data that should be
separately stored.</li>
<li>Is this a modifier that provides information about the
data in the element? In that case, it's OK to store the
modifier information in an attribute. Color of a label,
whether a turnout is inverted, which icon of the following
list to load are examples of modifiers.</li>
</ul>JMRI XML originally leaned heavily to attributes due to
limitations in the JDOM library. These limitations are long
gone, and we're now moving toward using elements in the
proper way.
<h3>Available Defined Types</h3>The JMRI schema provides a
large number of pre-defined data types. These (generally)
check their content, and will be maintained in the future as
valid content changes, so it's best to use these where
possible instead of defining your own.
<p>A partial list of the pre-defined types:</p>
<dl>
<dt>systemNameType</dt>
<dd>System names, to eventually be tightened up to a real
test of validity</dd>
<dt>userNameType</dt>
<dd>User names, not including the empty name</dd>
<dt>nullUserNameType</dt>
<dd>User names, with an empty value allowed</dd>
<dt>beanNameType</dt>
<dd>Either user or system name</dd>
<dt>turnoutStateType</dt>
<dd>closed, thrown</dd>
<dt>signalColorType</dt>
<dd>red, yellow, etc</dd>
<dt>trueFalseType</dt>
<dd>true, false</dd>
<dt>yesNoType</dt>
<dd>yes, no</dd>
<dt>yesNoMaybeType</dt>
<dd>yes, no, maybe</dd>
</dl>For others, browse the <a href=
"http://jmri.org/xml/schema/types/general.xsd">general types
schema</a>.
<h2>External Standards and Future Work</h2>The <a href=
"http://www.oasis-open.org/">OASIS collaboration</a> defines
a number of schema and schema elements that have become
well-known standards. Were possible, we should use those
<a href="http://www.oasis-open.org/specs/index.php">standard
elements</a> to improve inter-operability. The first ones of
interest are:
<ul>
<li>
<a href="http://docbook.org/">DockBook</a> defines
elements for several concepts we use:
<ul>
<li>author
(http://www.docbook.org/tdg/en/html/author.html)</li>
<li>address
(http://www.docbook.org/tdg/en/html/address.html)</li>
<li>revision history
(http://www.docbook.org/tdg/en/html/revhistory.html)</li>
</ul>See
<ul>
<li><a href=
"http://www.docbook.org/specs/docbook-5.0-spec-cs-01.html">
http://www.docbook.org/specs/docbook-5.0-spec-cs-01.html</a></li>
<li><a href=
"http://www.docbook.org/xml/5.0/xsd/">http://www.docbook.org/xml/5.0/xsd/</a></li>
<li><a href=
"http://www.docbook.org/xml/5.0/xsd/docbook.xsd">http://www.docbook.org/xml/5.0/xsd/docbook.xsd</a></li>
</ul>
<p>We have our own DocBook subset that we use, because
the full DocBook 5.0 schema takes a very long time to
parse, and isn't fully consistent with versions of other
software that we use. We use the normal DocBook 5.0
namespace, so we can later easily convert to a more
complete schema transparently. Our smaller schema is
located at <a href=
"http://jmri.org/xml/schema/docbook/docbook.xsd">http://jmri.org/xml/schema/docbook/docbook.xsd</a>
(our usual schema location). It is <em>only</em>
referenced from JMRI schema files, not instance files, so
that we can later convert with finite work.</p>
</li>
<li><a href=
"http://www.oasis-open.org/committees/ubl/faq.php">UBL</a>,
though aimed at business transactions, defines elements to
represent parties (companies, people), devices, model
numbers, etc.</li>
<li><a href=
"http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=office">
OpenDocument</a> (OODF) defines a set of elements and
structures for computations as part of its spreadsheet
module. (But they provide Relax-NG schema, not W3C XML
Schema, so that doesn't help so much)</li>
</ul>Learning to use these will require some work, as we
can't assume that computers using JMRI have internet access,
so can't just reference the entire schema as remote entities.
<h2>Copyright, Author and Revision Information</h2>For
various reasons, we need to move to DocBook format for
Copyright, Author and Revision information in our XML files
(instance files).
<p>Sample XML:</p>
<pre>
<code>
<db:copyright>
<db:year>2009</db:year>
<db:year>2010</db:year><
db:holder>JMRI</db:holder></db:copyright>
<db:authorgroup>
<db:author>
<db:personname><db:firstname>Sample</db:firstname><db:surname>Name</db:surname></db:personname>
<db:email>name@com.domain</db:email>
</db:author>
</db:authorgroup>
<db:revhistory>
<db:revision>
<db:revnumber>1</db:revnumber>
<db:date>2009-12-28</db:date>
<db:authorinitials>initials</db:authorinitials>
</db:revision>
</db:revhistory>
</code>
</pre>
<p>Sample schema description: (But see the real one, which is
provided in schema/docbook)</p>
<pre>
<code>
<xs:element ref="docbook:copyright" minOccurs="1" maxOccurs="1" >
<xs:annotation><xs:documentation>
DocBook element(s) providing copyright information in standard form.
Must be present.
</xs:documentation></xs:annotation>
</xs:element>
<xs:element ref="docbook:authorgroup" minOccurs="1" maxOccurs="unbounded" >
<xs:annotation><xs:documentation>
DocBook element(s) describing the authors in standard form
</xs:documentation></xs:annotation>
</xs:element>
<xs:element ref="docbook:revhistory" minOccurs="1" maxOccurs="unbounded" >
<xs:annotation><xs:documentation>
DocBook element(s) describing the revision history in standard form
</xs:documentation></xs:annotation>
</xs:element>
</code>
</pre>
<!--#include virtual="/Footer.shtml" -->
</div><!-- closes #mainContent-->
</div><!-- closes #mBody-->
</body>
</html>