Skip to content
Permalink
Browse files
Run TDML tests with user-chosen TDML implementation
Define TDMLImplementation using simple enumerated type in dafext.xsd.
Add TDML implementation option to Daffodil's CLI test subcommand and
pass Option[TDMLImplementation] argument to Runner to allow CLI to run
TDML tests with different TDMLImplementations (formerly we had to edit
the TDML file, but that become undesirable).  Remove tunable
tdmlImplementation from dafext.xsd and runtime2 TDML files.  Modify
runtime2 Scala tests to pass TDMLImplementation.DaffodilC to Runner
instead.  Look up TDML_IMPLEMENTATION environment variable in
DFDLTestSuite to allow "sbt test" to run tests with different TDML
implementations too.  Add CLI integration tests to increase test
coverage of new and changed lines.

Interaction between environment variable, CLI option, and JUnit tests
is simple to understand.  CLI option or JUnit argument passed to
Runner always specifies TDML implementation.  In their absence,
environment variable specifies TDML implementation.  Otherwise, tests
run with TDMLImplementation.Daffodil (runtime1).

Also fix some inconsistencies with fill bits overlooked by a previous
pull request which extended runtime2 to read and write N-bit numbers.
See more detailed changelog below.

DAFFODIL-2697

testNonCompatibleImplementation.tdml: Add another test schema and test
case to increase test coverage of "daffodil test -I" option.

Util.scala: Use standard Scala idioms for getting DAFFODIL_HOME
environment variable and composing environment variables.

TestCLIexecuting.scala: Add new error and normal test cases to
increase test coverage of "daffodil test -I" option.  Update "not
compatible" error test case due to change in error message.

Main.scala: Import TDMLImplementation (defined in dafext.xsd) to tell
CLI and user which TDML implementations Daffodil supports.  Add TDML
implementation option to CLI test subcommand.  Pass specified TDML
implementation or None to Runner for TDML file's tests.  Include TDML
implementation's name in "not compatible with implementation" message.

tdml.xsd: Update names of TDML implementations to match names of TDML
implementations in dafext.xsd.  Change runtime2 implementation name
from "daffodil-runtime2" to "daffodilC" because a TDML implementation
name must be a valid identifier in both XML and Scala and you can't
embed a dash within a Scala identifier.

dafext.xsd: Remove tdmlImplementation tunable to avoid overly
complicated interaction between tunable, environment variable, and CLI
-I option.  Add TunableTDMLImplementation simple type (can't remove
"Tunable" from its name because TunableGenerator filters out types not
beginning with "Tunable") with enumerations listing every TDML
implementation Daffodil supports, which also creates a
TDMLImplementation enumerated type we can import into Scala files.
Add comment to clarify you still need different classpaths to call ibm
as a cross tester.  Change all enumerated simple types to use standard
xs:token idiom instead of xs:string.

parsers.c: Rename parse_fill_bytes to parse_fill_bits and make it read
exactly N fill bits (no more, no less).

parsers.h: Rename parse_fill_bytes to parse_fill_bits and rename its
first parameter to correct name end_bitPos0b (not end_bytePos0b).

unparsers.c: Rename unparse_fill_bytes to unparse_fill_bits and make
it write exactly N fill bits (no more, no less).

unparsers.h: Rename unparse_fill_bytes to unparse_fill_bits and rename
its first parameter to correct name end_bitPos0b (not end_bytePos0b).

CodeGenerator.scala: Use standard Scala idiom for getting CC and PATH
environment variables.

CodeGeneratorState.scala: Rename calls of (un)parse_fill_bytes to
(un)parse_fill_bits in generated C code.

ex_nums.tdml: Make comments show how to run CLI test subcommand with
both daffodil and daffodilC implementations.  Remove config-runtime1
and config-runtime2 since tunable tdmlImplementation doesn't exist
anymore.  Add implementation-specific error tests with
implementations="daffodil" and implementations="daffodilC" to
demonstrate how runtime1 and runtime2 handle non-fixed values in
elements with fixed attributes slightly differently depending on
validation levels.  Combine runtime2 error_limited and error_on tests
into error_parse since runtime2 doesn't have a validation option.

nested.tdml: Replace tdmlImplementation configs with comments showing
how to run CLI test subcommand with different TDML implementations.

RunnerFactory.scala: Extend Runner API with TDMLImplementation
parameter in first apply method, Option[TDMLImplementation] parameter
in new two-argument apply method, and Option[TDMLImplementation]
parameter in base constructor to allow CLI and JUnit code to pass TDML
implementation to Runner(...) if they choose to.  Put all known TDML
implementations in defaultImplementationsDefaultDefault to make it
easier to run TDML tests with any TDML implementation.

TDMLRunner.scala: Extend DFDLTestSuite constructor with
Option[TDMLImplementation] parameter to allow Runner to pass TDML
implementation, also remove __nl parameter which was only difference
between first and second constructors since second constructor isn't
called anymore and can be removed.  Make DFDLTestSuite responsible for
defining defaultTDMLImplementation (either
sys.env.get("TDML_IMPLEMENTATION") or daffodil) and
tdmlDFDLProcessorFactory (moved to DFDLTestSuite from TestCase) to try
to minimize number of times code looks up TDML_IMPLEMENTATION and
matching TDMLDFDLProcessorFactory class and constructs new factory.
Unfortunately, both CLI Main class and JUnit tests run individual TDML
test cases one by one, not en masse, and each individual test calls
Runner.getTS which always creates a new DFDLTestSuite, so every single
test will fail and throw TDMLException if TDML_IMPLEMENTATION names an
invalid TDML implementation.  Didn't try it, but maybe could move
defaultTDMLImplementation and tdmlDFDLProcessorFactory to Runner and
pass tdmlDFDLProcessorFactory instead of optTDMLImplementation to
DFDLTestSuite.  Then only first test might fail and rest of tests
might run with daffodil implementation.

Runtime2TDMLDFDLProcessor.scala: Don't set implementationName to
"daffodil-runtime2" literal string, set it to
TDMLImplementation.DaffodilC.toString to get the correct name instead.

TestRunnerFactory.java: Pass Option.apply(null) [closest Java
equivalent to passing None] to Runner in first test case.  Noticed
that IDEA complains TestRunnerFactory is calling private Runner
constructor, but sbt still compiles TestRunnerFactory with no problem.

ISRM_green_to_orange_60000.tdml: Replace tunable configs with comments
showing how to run CLI test subcommand with different TDML
implementations.

ISRM_orange_to_green_60002.tdml: Replace tunable configs with comments
showing how to run CLI test subcommand with different TDML
implementations.

MPU_green_to_orange_60004.tdml: Replace tunable configs with comments
showing how to run CLI test subcommand with different TDML
implementations.

MPU_orange_to_green_60006.tdml: Replace tunable configs with comments
showing how to run CLI test subcommand with different TDML
implementations.

collisions.tdml: Replace tunable configs with comments showing how to
run CLI test subcommand with different TDML implementations.

egress_xdcc_bw.tdml: Replace tunable configs with comments showing how
to run CLI test subcommand with different TDML implementations.

ingress_xdcc_bw.tdml: Replace tunable configs with comments showing
how to run CLI test subcommand with different TDML implementations.

orion.tdml: Replace tunable configs with comments showing how to run
CLI test subcommand with different TDML implementations.

TestCollisions.scala: Pass TDMLImplementation.DaffodilC to Runner.

TestEgressXdccBw.scala: Pass TDMLImplementation.DaffodilC to Runner.

TestExNums.scala: Pass TDMLImplementation.Daffodil and
TDMLImplementation.DaffodilC to two Runners to run both runtime1 and
runtime2 tests side by side.  Combine runtime2 error_limited and
error_on tests into error_parse since runtime2 doesn't have a
validation option.

TestIngressXdccBw.scala: Pass TDMLImplementation.DaffodilC to Runner.

TestIsrmGreenToOrange60000.scala: Pass TDMLImplementation.DaffodilC to
Runner.

TestIsrmOrangeToGreen60002.scala: Pass TDMLImplementation.DaffodilC to
Runner.

TestMpuGreenToOrange60004.scala: Pass TDMLImplementation.DaffodilC to
Runner.

TestMpuOrangeToGreen60006.scala: Pass TDMLImplementation.DaffodilC to
Runner.

TestNested.scala: Pass TDMLImplementation.DaffodilC to Runner.

TestOrion.scala: Pass TDMLImplementation.DaffodilC to Runner.
  • Loading branch information
tuxji committed Jun 8, 2022
1 parent c81b213 commit 382d870d640995bbb05280f3f5c1c1eb854aff71
Showing 36 changed files with 390 additions and 298 deletions.
@@ -27,10 +27,20 @@
<tdml:defineSchema name="s1">
<xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
<dfdl:format ref="ex:GeneralFormat" />

<xs:element name="e1" dfdl:lengthKind="explicit"
dfdl:length="2" type="xs:int" />
</tdml:defineSchema>

<tdml:defineSchema name="s2">
<xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
<dfdl:format ref="ex:GeneralFormat" representation="binary" />
<xs:element name="e1">
<xs:complexType>
<xs:sequence>
<xs:element name="e2" type="xs:int"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</tdml:defineSchema>

<tdml:parserTestCase name="testNotCompatibleImplementation1" root="e1"
@@ -44,4 +54,19 @@
</tdml:infoset>
</tdml:parserTestCase>

</tdml:testSuite>
<tdml:parserTestCase name="testDaffodilCImplementation1" root="e1"
model="s2" description="this test specifies daffodilC as implementation."
implementations="daffodilC">
<tdml:document>
<tdml:documentPart type="byte">0000002a</tdml:documentPart>
</tdml:document>
<tdml:infoset>
<tdml:dfdlInfoset>
<e1>
<e2>42</e2>
</e1>
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>

</tdml:testSuite>
@@ -26,7 +26,6 @@ import org.apache.daffodil.Main.ExitCode

import java.nio.file.Paths
import java.io.{File, PrintWriter}
import scala.collection.JavaConverters._
import java.util.concurrent.TimeUnit
import org.apache.daffodil.xml.XMLUtils
import org.junit.Assert.fail
@@ -39,7 +38,7 @@ object Util {

val isWindows = System.getProperty("os.name").toLowerCase().startsWith("windows")

val dafRoot = sys.env.get("DAFFODIL_HOME").getOrElse(".")
val dafRoot = sys.env.getOrElse("DAFFODIL_HOME", ".")

def daffodilPath(dafRelativePath: String): String = {
XMLUtils.slashify(dafRoot) + dafRelativePath
@@ -82,7 +81,7 @@ object Util {
// The inputStream will be at index 0
// The errorStream will be at index 1
def getShell(cmd: String, spawnCmd: String, envp: Map[String, String] = Map.empty[String, String], timeout: Long): Expect = {
val newEnv = System.getenv().asScala ++ envp
val newEnv = sys.env ++ envp

val envAsArray = newEnv.toArray.map { case (k, v) => k + "=" + v }
val process = Runtime.getRuntime().exec(spawnCmd, envAsArray)
@@ -194,7 +194,47 @@ class TestCLIexecuting {
val cmd = String.format("%s test %s testNotCompatibleImplementation1", Util.binPath, testTdmlFile)
println(cmd)
shell.sendLine(cmd)
shell.expect(contains("[Skipped] testNotCompatibleImplementation1 (Not compatible implementation.)"))
shell.expect(contains("[Skipped] testNotCompatibleImplementation1 (not compatible"))

Util.expectExitCode(ExitCode.Success, shell)
shell.sendLine("exit")
} finally {
shell.close()
}
}

@Test def test_CLI_catch_TestBadArguments(): Unit = {
val tdmlFile = Util.daffodilPath("daffodil-cli/src/it/resources/org/apache/daffodil/CLI/testNonCompatibleImplementation.tdml")

val testTdmlFile = if (Util.isWindows) Util.cmdConvert(tdmlFile) else tdmlFile

val shell = Util.start("")

try {
val cmd = String.format("%s test -I notDaffodilC %s", Util.binPath, testTdmlFile)
println(cmd)
shell.sendLine(cmd)
shell.expectIn(1, contains("[error] Bad arguments for option 'implementation'"))

Util.expectExitCode(ExitCode.Usage, shell)
shell.sendLine("exit")
} finally {
shell.close()
}
}

@Test def test_CLI_Executing_implementation(): Unit = {
val tdmlFile = Util.daffodilPath("daffodil-cli/src/it/resources/org/apache/daffodil/CLI/testNonCompatibleImplementation.tdml")

val testTdmlFile = if (Util.isWindows) Util.cmdConvert(tdmlFile) else tdmlFile

val shell = Util.start("")

try {
val cmd = String.format("%s test -I daffodilC %s testDaffodilCImplementation1", Util.binPath, testTdmlFile)
println(cmd)
shell.sendLine(cmd)
shell.expect(contains("[Pass] testDaffodilCImplementation1"))

Util.expectExitCode(ExitCode.Success, shell)
shell.sendLine("exit")
@@ -57,6 +57,7 @@ import org.apache.daffodil.api.DFDL.ParseResult
import org.apache.daffodil.api.DFDL.UnparseResult
import org.apache.daffodil.api.DaffodilConfig
import org.apache.daffodil.api.DaffodilTunables
import org.apache.daffodil.api.TDMLImplementation
import org.apache.daffodil.api.URISchemaSource
import org.apache.daffodil.api.ValidationMode
import org.apache.daffodil.api.WithDiagnostics
@@ -199,6 +200,15 @@ class CLIConf(arguments: Array[String]) extends scallop.ScallopConf(arguments) {
}
})

implicit def implementationConverter = singleArgConverter[TDMLImplementation]((s: String) => {
val optImplementation = TDMLImplementation.optionStringToEnum("implementation", s)
if (!optImplementation.isDefined) {
throw new Exception("Unrecognized TDML implementation '%s'. Must be one of %s"
.format(s, TDMLImplementation.allValues.mkString(", ")))
}
optImplementation.get
})

def qnameConvert(s: String): RefQName = {
val eQN = QName.refQNameFromExtendedSyntax(s)
eQN.get
@@ -388,7 +398,7 @@ class CLIConf(arguments: Array[String]) extends scallop.ScallopConf(arguments) {

// Test Subcommand Options
object test extends scallop.Subcommand("test") {
banner("""|Usage: daffodil test [-l] [-r] [-i] <tdmlfile> [testnames...]
banner("""|Usage: daffodil test [-I <implementation>] [-l] [-r] [-i] <tdmlfile> [testnames...]
|
|List or execute tests in a TDML file
|
@@ -397,6 +407,10 @@ class CLIConf(arguments: Array[String]) extends scallop.ScallopConf(arguments) {
descr("List or execute TDML tests")
helpWidth(width)

val implementation = opt[TDMLImplementation](short = 'I', argName = "implementation",
descr = "Implementation to run TDML tests. Choose one of %s. Defaults to %s."
.format(TDMLImplementation.allValues.mkString(", "), TDMLImplementation.Daffodil.toString),
default = None)
val info = tally(descr = "Increment test result information output level, one level for each -i")
val list = opt[Boolean](descr = "Show names and descriptions instead of running test cases")
val regex = opt[Boolean](descr = "Treat <testnames...> as regular expressions")
@@ -1257,7 +1271,8 @@ object Main {
val testOpts = conf.test

val tdmlFile = testOpts.tdmlfile()
val tdmlRunner = new Runner(new java.io.File(tdmlFile))
val optTDMLImplementation = testOpts.implementation.toOption
val tdmlRunner = Runner(tdmlFile, optTDMLImplementation)

val tests = {
if (testOpts.testnames.isDefined) {
@@ -1330,7 +1345,8 @@ object Main {
case s: scala.util.control.ControlThrowable => throw s
case u: UnsuppressableException => throw u
case e: TDMLTestNotCompatibleException => {
println("[Skipped] %s (Not compatible implementation.)".format(name))
println("[Skipped] %s (not compatible with implementation %s)"
.format(name, e.implementation.getOrElse("<none>")))
}
case e: Throwable => {
println("[Fail] %s".format(name))
@@ -101,7 +101,7 @@
<simpleType name="implementationItem">
<restriction base="xs:token">
<enumeration value="daffodil"/>
<enumeration value="daffodil-runtime2"/>
<enumeration value="daffodilC"/>
<enumeration value="ibm"/>
</restriction>
</simpleType>
@@ -45,7 +45,7 @@
</xs:complexType>

<xs:simpleType name="PropertyNameType">
<xs:restriction base="xs:string">
<xs:restriction base="xs:token">
<xs:enumeration value="parseUnparsePolicy"/>
</xs:restriction>
</xs:simpleType>
@@ -502,14 +502,6 @@
</xs:documentation>
</xs:annotation>
</xs:element>
<xs:element name="tdmlImplementation" type="xs:string" default="daffodil" minOccurs="0">
<xs:annotation>
<xs:documentation>
TDMLDFDLProcessorFactory implementation to use when running TDML tests.
Allowed values are "daffodil" (default), "daffodil-runtime2", and "ibm".
</xs:documentation>
</xs:annotation>
</xs:element>
<xs:element name="tempFilePath" type="xs:string" default="This string is ignored. Default value is taken from java.io.tmpdir property" minOccurs="0">
<xs:annotation>
<xs:documentation>
@@ -587,15 +579,27 @@
<xs:restriction base="dfdlx:ParseUnparsePolicyEnum" />
</xs:simpleType>
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:restriction base="xs:token">
<xs:enumeration value="fromRoot" />
</xs:restriction>
</xs:simpleType>
</xs:union>
</xs:simpleType>

<!--
TDML implementation to use when running TDML tests. Currently daffodil and ibm use the
same classname, so you still need different classpaths to call ibm as a cross tester.
-->
<xs:simpleType name="TunableTDMLImplementation">
<xs:restriction base="xs:token">
<xs:enumeration value="daffodil" />
<xs:enumeration value="daffodilC" />
<xs:enumeration value="ibm" />
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="TunableUnqualifiedPathStepPolicy">
<xs:restriction base="xs:string">
<xs:restriction base="xs:token">
<xs:enumeration value="defaultNamespace" />
<xs:enumeration value="noNamespace" />
<xs:enumeration value="preferDefaultNamespace" />
@@ -507,26 +507,25 @@ parse_le_uint8(uint8_t *number, size_t num_bits, PState *pstate)
*number = (uint8_t)integer;
}

// Skip fill bytes until end bitPos0b is reached
// Parse fill bits until end bitPos0b is reached

void
parse_fill_bytes(size_t end_bitPos0b, PState *pstate)
parse_fill_bits(size_t end_bitPos0b, PState *pstate)
{
union
{
uint8_t bytes[1];
} buffer;
assert(pstate->bitPos0b <= end_bitPos0b);

// Update our last successful parse position only if this loop
// finishes without any errors
size_t current_bitPos0b = pstate->bitPos0b;
while (current_bitPos0b < end_bitPos0b)
size_t fill_bits = end_bitPos0b - pstate->bitPos0b;
uint8_t bytes[1];
while (fill_bits)
{
read_bits(buffer.bytes, BYTE_WIDTH, pstate);
size_t num_bits = (fill_bits >= BYTE_WIDTH) ? BYTE_WIDTH : fill_bits;
read_bits(bytes, num_bits, pstate);
if (pstate->error) return;
current_bitPos0b += BYTE_WIDTH;
fill_bits -= num_bits;
}
pstate->bitPos0b = current_bitPos0b;

// If we got all the way here, update our last successful parse position
pstate->bitPos0b = end_bitPos0b;
}

// Allocate memory for hexBinary array
@@ -57,9 +57,9 @@ extern void parse_le_uint32(uint32_t *number, size_t num_bits, PState *pstate);
extern void parse_le_uint64(uint64_t *number, size_t num_bits, PState *pstate);
extern void parse_le_uint8(uint8_t *number, size_t num_bits, PState *pstate);

// Parse fill bytes until end bytePos0b is reached
// Parse fill bits until end bitPos0b is reached

extern void parse_fill_bytes(size_t end_bytePos0b, PState *pstate);
extern void parse_fill_bits(size_t end_bitPos0b, PState *pstate);

// Allocate memory for hexBinary array

@@ -401,27 +401,24 @@ unparse_le_uint8(uint8_t number, size_t num_bits, UState *ustate)
unparse_endian_uint64(LITTLE_ENDIAN_DATA, number, num_bits, ustate);
}

// Add fill bytes until end bitPos0b is reached
// Unparse fill bits until end bitPos0b is reached

void
unparse_fill_bytes(size_t end_bitPos0b, const uint8_t fill_byte, UState *ustate)
unparse_fill_bits(size_t end_bitPos0b, const uint8_t fill_byte, UState *ustate)
{
union
{
uint8_t bytes[1];
} buffer;
buffer.bytes[0] = fill_byte;
assert(ustate->bitPos0b <= end_bitPos0b);

// Update our last successful write position only if this loop
// finishes without any errors
size_t current_bitPos0b = ustate->bitPos0b;
while (current_bitPos0b < end_bitPos0b)
size_t fill_bits = end_bitPos0b - ustate->bitPos0b;
while (fill_bits)
{
write_bits(buffer.bytes, BYTE_WIDTH, ustate);
size_t num_bits = (fill_bits >= BYTE_WIDTH) ? BYTE_WIDTH : fill_bits;
write_bits(&fill_byte, num_bits, ustate);
if (ustate->error) return;
current_bitPos0b += BYTE_WIDTH;
fill_bits -= num_bits;
}
ustate->bitPos0b = current_bitPos0b;

// If we got all the way here, update our last successful write position
ustate->bitPos0b = end_bitPos0b;
}

// Unparse opaque bytes from hexBinary field
@@ -57,9 +57,9 @@ extern void unparse_le_uint32(uint32_t number, size_t num_bits, UState *ustate);
extern void unparse_le_uint64(uint64_t number, size_t num_bits, UState *ustate);
extern void unparse_le_uint8(uint8_t number, size_t num_bits, UState *ustate);

// Unparse fill bytes until end bytePos0b is reached
// Unparse fill bits until end bitPos0b is reached

extern void unparse_fill_bytes(size_t end_bytePos0b, const uint8_t fill_byte, UState *ustate);
extern void unparse_fill_bits(size_t end_bitPos0b, const uint8_t fill_byte, UState *ustate);

// Unparse opaque bytes from hexBinary field

@@ -161,9 +161,9 @@ class CodeGenerator(root: Root) extends DFDL.CodeGenerator {
* sequence if no compiler could be found in the user's PATH.
*/
lazy val pickCompiler: Seq[String] = {
val ccEnv = System.getenv("CC")
val ccEnv = sys.env.getOrElse("CC", "zig cc")
val compilers = Seq(ccEnv, "zig cc", "cc", "clang", "gcc")
val path = System.getenv("PATH").split(File.pathSeparatorChar)
val path = sys.env.getOrElse("PATH", ".").split(File.pathSeparatorChar)
def inPath(compiler: String): Boolean = {
(compiler != null) && {
val exec = compiler.takeWhile(_ != ' ')
@@ -331,10 +331,10 @@ class CodeGeneratorState(private val root: ElementBase) {
if (context.maybeFixedLengthInBits.isDefined && context.maybeFixedLengthInBits.get > 0) {
val octalFillByte = context.fillByteEv.constValue.toByte.toOctalString
val parseStatement =
s""" parse_fill_bytes(end_bitPos0b, pstate);
s""" parse_fill_bits(end_bitPos0b, pstate);
| if (pstate->error) return;""".stripMargin
val unparseStatement =
s""" unparse_fill_bytes(end_bitPos0b, '\\$octalFillByte', ustate);
s""" unparse_fill_bits(end_bitPos0b, '\\$octalFillByte', ustate);
| if (ustate->error) return;""".stripMargin

structs.top.parserStatements += parseStatement

0 comments on commit 382d870

Please sign in to comment.