Skip to content

Commit

Permalink
Added reasons to incomplete scores
Browse files Browse the repository at this point in the history
  • Loading branch information
@rgerkin
rgerkin committed Jan 27, 2020
1 parent 601204c commit 4ac6465
Showing 1 changed file with 10 additions and 8 deletions.
18 changes: 10 additions & 8 deletions sciunit/scores/incomplete.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,21 +19,24 @@ def __init__(self, score, related_data=None):
super(NoneScore, self).__init__(score, related_data=related_data)
else:
raise InvalidScoreError("Score must be a string or None")

@property
def norm_score(self):
return None

def __str__(self):
return 'Unknown'
if self.score:
s = '%s (%s)' % (self.description, self.score)
else:
s = self.description
return s


class TBDScore(NoneScore):
"""A TBD (to be determined) score. Indicates that the model has
capabilities required by the test but has not yet taken it."""

def __str__(self):
return 'TBD'
description = 'None'

This comment has been minimized.

Sign in to view

Copy link
@zsinnema

zsinnema Mar 17, 2020

Contributor

@rgerkin : in the scidash in the compatibility matrix we test on the return value to be TBD which then is interpreted as "compatible" now with this change the return value is None. Should this be interpreted also as being "compatible"?

This comment has been minimized.

Sign in to view

Copy link
@rgerkin

rgerkin Mar 21, 2020

Author Contributor

@zsinnema Basically, yes. If it was incompatible it should return an NAScore, which would return 'N/A' rather than None. A brief summary of behavior can be obtained by:

import sciunit
from sciunit.scores import TBDScore, NAScore
s1 = TBDScore('Compatible')
s2 = NAScore('Incompatible')
s3 = TBDScore(None)
s4 = NAScore(None)
for s in [s1, s2, s3, s4]:
    print(s)

where the argument to the score is only for annotation to produce a more rich return value. From the example above, the return value of __str__ should always begin with 'N/A' for actual incompatibilities (which return NAScore), while the __str__ of TBDScore should always return something that begins with None. In any case, the score type (using isinstance) will always be more reliable than the return value of __str__, if you want to use that instead.


class NAScore(NoneScore):
Expand All @@ -42,13 +45,12 @@ class NAScore(NoneScore):
Indicates that the model doesn't have the
capabilities that the test requires."""

def __str__(self):
return 'N/A'
description = 'N/A'


class InsufficientDataScore(NoneScore):
"""A score returned when the model or test data
is insufficient to score the test."""

def __str__(self):
return 'Insufficient Data'
description = 'Insufficient Data'

0 comments on commit 4ac6465

Please sign in to comment.