help/en/html/doc/Technical/XmlSchema.shtml

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">

<html lang="en">
<head>
  <meta name="generator" content=
  "HTML Tidy for Mac OS X (vers 31 October 2006 - Apple Inc. build 15.17), see www.w3.org">

  <title>JMRI: Xml Schema Usage</title>
    <meta name="author" content="Bob Jacobsen">
  <meta name="keywords" content=
  "JMRI technical code xml schema usage">
  <!-- The combination of "Define" and {Header,Style, Logo and Footer} comments -->
  <!-- are an arbitrary design pattern used by the update.pl script to -->
  <!-- easily replace the common header/footer code for all the web pages -->
  <!-- delete the following 2 Defines if you want to use the default JMRI logo -->
  <!-- or change them to reflect your alternative logo -->
  <!--#include virtual="/Style" -->
</head><!--#include virtual="/Header" -->

<body>
  <div id="mBody">
    <!--#include virtual="Sidebar" -->

    <div id="mainContent">
      <h1>JMRI: XML Schema Usage</h1>
      <p>JMRI uses XML for a number of
      purposes: to hold decoder definitions, for its <a href=
      "XmlPersistance.shtml">persistance system</a> for
      configuration and panel information, and to create parts of
      the web site from other files. This page describes how we
      specify the content of those files using <a href=
      "http://www.w3schools.com/schema/schema_intro.asp">XML
          schema</a>.</p>

      <p>For examples (not a tutorial!) on the structure of our
      schema, see the <a href="XmlSchemaExamples.shtml">examples
      page</a>.</p>

      <p>The current schema can be seen online in the <a href=
      "http://jmri.org/xml/schema">schema directory</a>. The most
      commonly used one is the <a href=
      "http://jmri.org/xml/schema/layout.xsd">layout.xsd schema for
      panel files</a>. See below on how it's organized.</p>

      <h2>Access to Schema Definitions</h2>JMRI uses XML Schema to
      define the format of its files.

      <p>Those XML Schema may need to be available to the program
      when it reads the files, as they define the default values of
      missing attributes and other needed information.</p>

      <p>In the JMRI distributions, these are stored in the
      <a href="/xml/schema">xml/schema</a> directory. Note that
      they are not stored in each directory alongside the XML
      files. There are just too many locations to keep such a set
      of schema definition files up to date. JMRI itself, via the
      jmri.jmrit.XmlFile class, provides support for locating those
      files when the XML parser needs them.</p>

      <h2 id="modschema">Modifying JMRI Schema</h2>This section
      talks about how to handle small changes to existing schema,
      for example added or removing an attribute or element. For
      more extensive changes, including creating entirely new types
      or new file formats, see the next section on "<a href=
      "#devschema">Developing JMRI Schema</a>".

      <p>Every time you change what JMRI writes (and therefore
      reads) in an XML file, you must also do the following:</p>

      <ol>
        <li>You need to change the code that does the reading and
        writing.</li>

        <li>You need to change the schema file(s) so that the XML
        format can be properly checked.</li>

        <li>You have to provide new test XML files to make sure
        nothing has been broken, and in some cases have to adjust
        old ones.</li>
      </ol>

      <p>Please don't skip the latter steps. They matter a lot to
      the long-term stability of JMRI code.</p>

      <p>If possible, it's best to make changes by addition, so
      that existing files can continue to be read unchanged. JMRI
      strongly values backward compatibility, where any newer
      version of JMRI can still load and use a file written by an
      older version.</p>

      <p>If you can make a change that's just an addition, then the
      process is:</p>

      <ol>
        <li>Change your code</li>

        <li>Add the new elements and attributes to the most recent
        version of the schema file(s).</li>

        <li>Run "ant headlesstest" to make sure that the older
        files already present in the tests can still be processed.
        Fix anything that breaks. (You may discover at this point
        that you didn't actually make a backward compatible change,
        in which case either improve that or see the section on
        "<a href="#versioning">Schema Versioning</a>" below).</li>

        <li>Create a test file with the new content. Ideally, this
        won't require using the screen, so it can actually be
        loaded and stored as part of headlesstest. In that case,
        put your file in a "load" test subdirectory (see below). At
        a minimum, though, put your file in a "verify" test
        subdirectory so that your schema changes will be tested.
        For more on this, see <a href=
        "#junitschematest">below</a>.</li>
      </ol>

      <h3 id="versioning">Schema Versioning</h3>"Versioning" schema
      allows us to have different schema for older and newer files.
      This lets newer versions of JMRI continue to check and
      read files that were written by older versions of JMRI. This
      backward compatibility is an important JMRI feature which we
      don't want to lose.

      <p>In practical terms, versioning consists of having
      multiple-but-related versions of the schema definition files
      which are labeled by the first JMRI version that can read
      them.</p>

      <p>When do you need to create a new version?</p>

      <ul>
        <li>You <em>do not</em> need to create a new version of the
        schema if you add or change things such that existing files
        will continue to validate.

          <p>In that case, just make your schema changes in the
          current schema document, and commit them back to the JMRI
          code repository.</p>
        </li>

        <li>You <em>do</em> need to version a schema when you make a
        change to the schema such that previous files will no
        longer validate with the current schema.

          <p>In this case, the steps to version the schema are:</p>

          <ol>
            <li>Copy the current schema file to a new one with the
            number of the <em>next</em> JMRI release. E.g. copy
            types/turnouts-2-9-6.xsd to types/turnouts-3.7.3.xsd if
            you're doing this before JMRI 3.7.3 is released. Make
            your changes and commit that new version.</li>

            <li>If this is a subfile, such as the
            types/turnouts-2-9-6.xsd, that is included in a main
            schema such as layout-2-9-6.xsd, that main file also
            needs to be copied, have the include changed, and
            commited. Now you've got a new layout-3-7-3.xsd schema
            for output files to reference.</li>

            <li>Then, change the Java code that writes the schema
            reference to the top of output files to use the new
            filename. For example, layout (panel) files are written
            by
            <code>src/jmri/configurexml/ConfigXmlManager.java</code>.
            Look for the<br>
            <code>static final public String schemaVersion =
            "-3-7-3"</code><br>
            line and change it to your new version number
            suffix.</li>

            <li>Run "ant headlesstest" and when tests fail, fix them.
                If a LoadAndStoreTest flags a compare inLine/outLine
                error, update the schema version in the header of the
                reference file. For example in
                <code>java/test/jmri/configurexml/loadref/BlockAndSignalMastTest.xml</code>
                point to the most recent layout.xsd version in line 3:<br>
                <code>&lt;layout-config
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xsi:noNamespaceSchemaLocation="http://jmri.org/xml/schema/layout-4-7-2.xsd"&gt;
                </code>
            </li>

            <li>If there's an associated XML stylesheet(s), its
            name has to be changed in a coordinated way. (You
            should also update the stylesheet to show your new XML
            content, of course).</li>
          </ol>
        </li>
      </ul>In either case, it's important to include sufficient
      test files that the unit tests catch any problems with the
      new and old schema. See the <a href="#test">test section
      below</a>.

      <p>Note that the unlabeled schema is the primordial, oldest,
      now-obsolete schema. For example layout.xsd is <em>older</em>
      than layout-2-9-6.xsd, and therefore not to be used for new
      files anymore. Don't assume that layout.xsd is the default
      for new files!</p>

      <h2>Checking JMRI Schema</h2>It's important that the JMRI
      schema definitions be kept semantically correct. If we let
      too many problems build up, we'll eventually have a lot of
      back-fitting to do. The W3C online <a href=
      "http://www.w3.org/2001/03/webdata/xsv">schema validation
      tool</a> is a very good tool for checking that changes to the
      JMRI schema are still technically correct. You should check
      your changes with it before commiting them to the repository.
      Unfortunately, it doesn't seem to check compliance with
      nested schema elements, e.g. from DocBook (see below) or JMRI
      schema, but it's still a very useful check.

      <p>Using the JMRI "Validate XML File" tool in the "Debug"
      menu to validate a .xml file ("instance file") that uses your
      new or updated schema is an important check of both. Use it
      often during development. You can also use it from the
      command line via e.g.:</p>
      <pre>
<code>
./runtest.csh jmri/jmrit/XmlFileValidateAction xml/decoders/0NMRA.xml 
</code>
</pre>

      <p>For a quick file check, Linux and Mac OS X users can
      validate from the command line with e.g.</p>
      <pre>
<code>
 cd xml
 xmllint -schema schema/aspecttable.xsd -noout signals/sample-aspects.xml
</code>
</pre><code>xmllint</code> can't check schema files themselves,
      unfortunately, because their schema isn't something it can handle.

      <p>Your schema docs should point to our standard stylesheet
      in their head matter:</p>
      <pre>
<code>
&lt;?xml-stylesheet href="schema2xhtml.xsl" type="text/xsl"?&gt;
</code>
</pre>Stylesheets turn XML code, like this schema, into a
      human-readable form when the XML is parsed and displayed by a
      browser. For an example of that, click on this link to the
      <a href="http://jmri.org/xml/schema/aspecttable.xsd">aspecttable.xsd</a>
      schema file. Our standard stylesheet is pretty basic. It just
      shows basic structure. If anybody knows of a better
      stylesheet, we can certainly switch to it. 

      <h3><a id="test" name="test">JUnit testing</a></h3>
      You should also add a 
      <a href="JUnit.shtml">JUnit test</a> that checks a typical file.
      There are three kinds of checks that can be done:

      <ol>
        <li>You should always have a class that validates a typical
        file against the schema.

          <p>That's done by having a SchemaTest class in your
          test-tree package (see e.g. 
          <a href="https://github.com/JMRI/JMRI/blob/master/java/test/jmri/configurexml/SchemaTest.java">
            test/jmri/configurexml/SchemaTest.java</a> for
          an example) that checks all the XML files kept there. If
          that's in place, just put a copy of a (new) typical XML
          file in the existing "verify" subdirectory.</p>

          <p>To more extensively check your schema, you can check
          that it fails XML files that you think are not valid.
          There are lots of ways to be not valid, and you don't
          need to check them all, but if there something specific
          you want to be sure of, put an example of that in the
          "invalid" subdirectory. Those files are expected to fail
          for some specific reason. You should document that reason
          via comments in the file itself so your colleagues can
          figure it out later.</p>

          <p>If there's no "verify" subdirectory, create one and
          add it to the end of the SchemaTest class in that
          package. If there's no SchemaTest class, create one by
          replicating an existing one, see link above. Don't forget
          to add it to PackageTest so it gets invoked!</p>
        </li>

        <li>If you're working on configurexml files (panel files),
        and your new code isn't involved in the actual display of
        panels (e.g. can run as part of headlesstest), you should
        add a test that a sample file can be loaded and stored back
        successfully. Beyond just checking the schema, this also
        checks that all the load and store plumbing is working. (We
        run these tests headless as part of Jenkins, so please
        don't add tests that pop screen windows; they'll just cause
        errors).

          <p>That's done by having a LoadAndStoreTest class in your
          test-tree package (see e.g. 
          <a href="https://github.com/JMRI/JMRI/blob/master/java/test/jmri/configurexml/LoadAndStoreTest.java">
            test/jmri/configurexml/LoadAndStoreTest.java</a>
          for an example) that checks all the XML files kept there.
          If that's in place, just put a copy of a (new) typical
          XML file in the existing "load" subdirectory.</p>

          <p>If there's no "load" subdirectory, create one and add
          it to the end of the LoadAndStoreTest class in that
          package. If there's no LoadAndStoreTest class, create one
          by replicating an existing one, see link above. Don't
          forget to add it to PackageTest so it gets invoked!</p>

          <p>When LoadAndStoreTest runs, it loads the files in the
          load directory one-by-one, storing each back out to the
          "temp" directory within the local preferences directory,
          and then compares the input and output files. Sometimes
          this load-and-store process results in something that's
          in a different order, or contains more info (e.g.
          attributes missing from the input file are written with
          default values in the output file). If the comparison
          fails, but the output file is still OK when you manually
          inspect it, copy that output file to the "loadref"
          directory (create it if needed) within your test package.
          See <a href=
          "http://sourceforge.net/p/jmri/code/HEAD/tree/trunk/jmri/java/test/jmri/configurexml/loadref"
          target="_blank">test/jmri/configurexml/loadref</a> for an
          example. LoadAndStoreTest will compare to files it finds
          here, instead of to the original file in the "load"
          subdirectory.</p>

          <p>If your changes to the code cause this testing to fail
          with older versions of the file, <em>do not change the
          older file version.</em> Instead, either put an updated
          output reference in the "loadref" directory, version the
          schema to allow the older file to continue to load, or
          improve your code. Backward compatibility is
          important!</p>
        </li>

        <li>You can also add a custom JUnit test that reads your
        sample file and makes sure that the proper objects have
        been created, that they have the correct data and state,
        etc. This could be anything from a "load and check that new
        beans exist in the new manager" to something quite more
        extensive.</li>
      </ol>At a minimum, please do the schema checks. They're easy,
      and will save lots of trouble in the future. If your new
      features don't involve displaying on the screen, adding
      load-and-store checks is also valuable, and isn't so hard.

      <p><em>Note: Do not remove or modify any existing XML file
      checks. Those keep old versions of the files working!</em> If
      your new code and/or schema breaks the processing of the
      existing files, you need to either fix your code or <a href=
      "#versioning">version the schema</a> to allow multiple
      formats to coexist.</p>

      <h2 id="devschema">Developing JMRI Schema</h2>For some
      examples of the XML schema structures described here, see the
      separate <a href="XmlSchemaExamples.shtml">XML Schema
      Examples page</a>/

      <p>Our preferred organization for XML schema is based on the
      structure of the underlying code: A particular *Xml class is
      the unit of reuse.</p>

      <p>Lots of classes descend from jmri.configurexml.XmAdapter:
      (<a href=
      "http://jmri.org/JavaDoc/doc/jmri/configurexml/XmlAdapter.html">see
      Javadoc</a>)</p>

      <p>By convention, provide
      &lt;xsd:annotation&gt;&lt;xsd:appinfo&gt; element containing
      the class name that reads/writes the element:</p>
      <pre>
<code>
    &lt;xs:annotation&gt;
        &lt;xs:documentation&gt;
        Some human readable docs go here
        &lt;/xs:documentation&gt;
        &lt;xs:appinfo&gt;
            &lt;jmri:usingclass configurexml="false"&gt;jmri.managers.DefaultSignalSystemManager&lt;/jmri:usingclass&gt;
        &lt;/xs:appinfo&gt;
    &lt;/xs:annotation&gt;
</code>
</pre>

      <h3>The Venetian Blinds Pattern</h3>We are moving toward
      structuring our XML using the "<a href=
      "http://www.xfront.com/GlobalVersusLocal.html#ThirdDesignCharacteristics">Venetian
      Blinds pattern</a>". In this style, the top level elements
      that are written by classes have types defined for them. Any
      elements that are within those are defined anonymously,
      within those elements. For an example of this, see the
      <a href=
      "http://jmri.org/xml/schema/types/sensors.xsd">types/sensors.xsd</a>
      file, which defines a type for the "sensors" element written
      for SensorManagers. Within that, there is included a
      definition of a "Sensor" element, and a "comment" element
      within that.

      <p>This limits the number of types, and keeps the schema
      files roughly aligned with the classes that do the reading
      and writing.</p>

      <p>There are a few items (elements and attribute groups) that
      extend across multiple types. They are defined in the
      <a href="http://jmri.org/xml/schema/types/general.xsd">types/general.xsd</a>
      file.</p>

      <p>More information on XML schema design patterns is
      available at <a href=
      "http://www.ibm.com/developerworks/xml/library/ws-soa-xmlwsdl.html#N1012B">
      DeveloperWorks</a> and the <a href=
      "http://www.oracle.com/technetwork/java/design-patterns-142138.html">
      Oracle Java site</a>.</p>

      <h3>On Elements vs Attributes</h3>When defining how to store
      new classes or updates to classes, the basic questions are:

      <ul>
        <li>Are we storing data? In that case, it should be stored
        in an element of its own. Comments, speed values, user and
        system names are all examples of data that should be
        separately stored.</li>

        <li>Is this a modifier that provides information about the
        data in the element? In that case, it's OK to store the
        modifier information in an attribute. Color of a label,
        whether a turnout is inverted, which icon of the following
        list to load are examples of modifiers.</li>
      </ul>JMRI XML originally leaned heavily to attributes due to
      limitations in the JDOM library. These limitations are long
      gone, and we're now moving toward using elements in the
      proper way.

      <h3>Available Defined Types</h3>The JMRI schema provides a
      large number of pre-defined data types. These (generally)
      check their content, and will be maintained in the future as
      valid content changes, so it's best to use these where
      possible instead of defining your own.

      <p>A partial list of the pre-defined types:</p>

      <dl>
        <dt>systemNameType</dt>

        <dd>System names, to eventually be tightened up to a real
        test of validity</dd>

        <dt>userNameType</dt>

        <dd>User names, not including the empty name</dd>

        <dt>nullUserNameType</dt>

        <dd>User names, with an empty value allowed</dd>

        <dt>beanNameType</dt>

        <dd>Either user or system name</dd>

        <dt>turnoutStateType</dt>

        <dd>closed, thrown</dd>

        <dt>signalColorType</dt>

        <dd>red, yellow, etc</dd>

        <dt>trueFalseType</dt>

        <dd>true, false</dd>

        <dt>yesNoType</dt>

        <dd>yes, no</dd>

        <dt>yesNoMaybeType</dt>

        <dd>yes, no, maybe</dd>
      </dl>For others, browse the <a href=
      "http://jmri.org/xml/schema/types/general.xsd">general types
      schema</a>.

      <h2>External Standards and Future Work</h2>The <a href=
      "http://www.oasis-open.org/">OASIS collaboration</a> defines
      a number of schema and schema elements that have become
      well-known standards. Were possible, we should use those
      <a href="http://www.oasis-open.org/specs/index.php">standard
      elements</a> to improve inter-operability. The first ones of
      interest are:

      <ul>
        <li>
          <a href="http://docbook.org/">DockBook</a> defines
          elements for several concepts we use:

          <ul>
            <li>author
            (http://www.docbook.org/tdg/en/html/author.html)</li>

            <li>address
            (http://www.docbook.org/tdg/en/html/address.html)</li>

            <li>revision history
            (http://www.docbook.org/tdg/en/html/revhistory.html)</li>
          </ul>See

          <ul>
            <li><a href=
            "http://www.docbook.org/specs/docbook-5.0-spec-cs-01.html">
            http://www.docbook.org/specs/docbook-5.0-spec-cs-01.html</a></li>

            <li><a href=
            "http://www.docbook.org/xml/5.0/xsd/">http://www.docbook.org/xml/5.0/xsd/</a></li>

            <li><a href=
            "http://www.docbook.org/xml/5.0/xsd/docbook.xsd">http://www.docbook.org/xml/5.0/xsd/docbook.xsd</a></li>
          </ul>

          <p>We have our own DocBook subset that we use, because
          the full DocBook 5.0 schema takes a very long time to
          parse, and isn't fully consistent with versions of other
          software that we use. We use the normal DocBook 5.0
          namespace, so we can later easily convert to a more
          complete schema transparently. Our smaller schema is
          located at <a href=
          "http://jmri.org/xml/schema/docbook/docbook.xsd">http://jmri.org/xml/schema/docbook/docbook.xsd</a>
          (our usual schema location). It is <em>only</em>
          referenced from JMRI schema files, not instance files, so
          that we can later convert with finite work.</p>
        </li>

        <li><a href=
        "http://www.oasis-open.org/committees/ubl/faq.php">UBL</a>,
        though aimed at business transactions, defines elements to
        represent parties (companies, people), devices, model
        numbers, etc.</li>

        <li><a href=
        "http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=office">
        OpenDocument</a> (OODF) defines a set of elements and
        structures for computations as part of its spreadsheet
        module. (But they provide Relax-NG schema, not W3C XML
        Schema, so that doesn't help so much)</li>
      </ul>Learning to use these will require some work, as we
      can't assume that computers using JMRI have internet access,
      so can't just reference the entire schema as remote entities.

      <h2>Copyright, Author and Revision Information</h2>For
      various reasons, we need to move to DocBook format for
      Copyright, Author and Revision information in our XML files
      (instance files).

      <p>Sample XML:</p>
      <pre>
<code>
  &lt;db:copyright&gt;
        &lt;db:year&gt;2009&lt;/db:year&gt;
        &lt;db:year&gt;2010&lt;/db:year&gt;&lt;
        db:holder&gt;JMRI&lt;/db:holder&gt;&lt;/db:copyright&gt;

  &lt;db:authorgroup&gt;
    &lt;db:author&gt;
        &lt;db:personname&gt;&lt;db:firstname&gt;Sample&lt;/db:firstname&gt;&lt;db:surname&gt;Name&lt;/db:surname&gt;&lt;/db:personname&gt;
        &lt;db:email&gt;name@com.domain&lt;/db:email&gt;
    &lt;/db:author&gt;    
  &lt;/db:authorgroup&gt;

  &lt;db:revhistory&gt;
    &lt;db:revision&gt;
        &lt;db:revnumber&gt;1&lt;/db:revnumber&gt;
        &lt;db:date&gt;2009-12-28&lt;/db:date&gt;
        &lt;db:authorinitials&gt;initials&lt;/db:authorinitials&gt;
    &lt;/db:revision&gt;    
  &lt;/db:revhistory&gt;
</code>
</pre>

      <p>Sample schema description: (But see the real one, which is
      provided in schema/docbook)</p>
      <pre>
<code>
    &lt;xs:element ref="docbook:copyright" minOccurs="1" maxOccurs="1" &gt;
      &lt;xs:annotation&gt;&lt;xs:documentation&gt;
      DocBook element(s) providing copyright information in standard form.
      Must be present.
      &lt;/xs:documentation&gt;&lt;/xs:annotation&gt;
    &lt;/xs:element&gt;

    &lt;xs:element ref="docbook:authorgroup" minOccurs="1" maxOccurs="unbounded" &gt;
      &lt;xs:annotation&gt;&lt;xs:documentation&gt;
      DocBook element(s) describing the authors in standard form
      &lt;/xs:documentation&gt;&lt;/xs:annotation&gt;
    &lt;/xs:element&gt;

    &lt;xs:element ref="docbook:revhistory" minOccurs="1" maxOccurs="unbounded" &gt;
      &lt;xs:annotation&gt;&lt;xs:documentation&gt;
      DocBook element(s) describing the revision history in standard form
      &lt;/xs:documentation&gt;&lt;/xs:annotation&gt;
    &lt;/xs:element&gt;
</code>
</pre>

        <!--#include virtual="/Footer" -->
    </div><!-- closes #mainContent-->
  </div><!-- closes #mBody-->
</body>
</html>