Skip to content
Permalink
Browse files

sanitycheck: order results.csv and discards.csv deterministically

One of the first things needed when comparing builds of tests across
different environments/systems is to make sure the same (sub)tests were
selected and run in the first place. For that purpose sort the output of
--testcase-report and --discard-report as they were in random order.

Actually make the entire class TestInstance sortable by adding a
standard __lt__() method comparing unique instance names; it could be
useful again.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
  • Loading branch information...
marc-hb authored and nashif committed Apr 5, 2019
1 parent f098b44 commit 0f7255c3b8f4a00d4be7d239c53e0b4c058829bd
Showing with 5 additions and 2 deletions.
  1. +5 −2 scripts/sanitycheck
@@ -1804,6 +1804,9 @@ class TestInstance:
or self.check_dependency()
self.results = {}

def __lt__(self, other):
return self.name < other.name

def check_dependency(self):
build_only = False
if self.test.harness == 'console':
@@ -2402,7 +2405,7 @@ class TestSuite:
fieldnames = ["test", "arch", "platform", "reason"]
cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
cw.writeheader()
for instance, reason in self.discards.items():
for instance, reason in sorted(self.discards.items()):
rowdict = {"test": instance.test.name,
"arch": instance.platform.arch,
"platform": instance.platform.name,
@@ -2625,7 +2628,7 @@ class TestSuite:
"rom_size"]
cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
cw.writeheader()
for name, goal in self.goals.items():
for name, goal in sorted(self.goals.items()):
i = self.instances[name]
rowdict = {"test": i.test.name,
"arch": i.platform.arch,

0 comments on commit 0f7255c

Please sign in to comment.
You can’t perform that action at this time.