Testing SCL

Opportunities for testing

• Run fedex_plus on IS schemas and on specially-crafted test schemas (done)
  • We have BNF and WSN representations of EXPRESS, so it should be possible to automate the generation of schemas that are definitely good and definitely bad. Generate thousands, and run fedex on each and ensure that it passes or fails as it should.
  • Verify usable schemas (i.e. APs, IFC, etc) using bnfparser2
• Compile schema code (done)
• Test p21read: known good file -> output file (done)
  • Will p21read create output file from known good file? If so,
    • Use Craig Lanning's Express Engine to verify that the input and output comply with the schema: $ ./expresso --validate -source-schema path/to/schema.exp -input-encoding p21 -input path/to/p21.stp
    • Compare input with both outputs, using Craig Lanning's STEP Instance File Diff Tool and the schema? (see below for info on building and using p21diff)
    • Use bnfparser2 on the part 21 files as well?
• Other apps described in src/test/README
• Other?

Methodology

• Automate for ease of use and to reduce or eliminate human error
• Automation via:
  • cmake (cross-platform, requires no additional software) (done)
  • python (easy to use, cross-platform) (to replace bash script?) (done)
  • bash script (obsolete)
• Output
  • CDash
    • command: ctest -S run_ctest.cmake
    • will not upload unless a file named .SCL_CTEST_PREFS.cmake (note the dot) exists in the dir above scl/
  • GitHub wiki
    • command: ctest -S ctest_matrix.cmake
    • requires that the github wiki repo be located at scl/../wiki-scl, and that your github ssh passphrase (if any) has been given to ssh-add
    • could be modified to allow manual upload of the wiki page (i.e. copy file to clipboard, paste into browser) - currently, the python script exits immediately if it doesn't find the wiki repo
  • include execution time for fedex_plus, gcc, p21read (done)
  • lump similar errors/warnings together, listing the affected schemas, to aid prioritization
    • this would be nice, but requires that the test script be able to recognize all of SCL's error messages
    • number the error messages, like MSVC does, to make parsing them easier?

p21diff

$ git clone git://step-toybox.git.sourceforge.net/gitroot/step-toybox/step-toybox
$ cd step-toybox/apps/
$ git clone git://step-toybox.git.sourceforge.net/gitroot/step-toybox/p21diff
$ cd p21diff/
$ sbcl --load sbdeliver.lisp
$ ./p21diff in.stp out.stp schema.exp