PartialCredit appears to not be working correctly for lists of values.

[Jmdwow]

Describe the bug
When using PartialCredit on a list of pipelines and a list of values, it raises an error upon calling the 'score' function. This appears to be due to this code from partialcredit.py:

line 31: self.max_score = value
and
line 41: return min(self.max_score, round(self._score, 2))

Since value can be an int or a list inside init, max_score can be set to a list, and an error pops up when trying to compare that list to the proper score.

[thoward27]

Hey @Jmdwow can you provide a test case to work from? Even just a small snippet demonstrating how you are using partialcredit?

[Jmdwow]

Sorry about that, I was rushing. Here's an example from my code:

        thresholds: List[float] = [threshold, threshold*1.1, threshold*2, threshold*4, 1.0]
        points: List[int] = [1, 1, 2, 2, 1]
        test_orders: List[Tuple[str, str]] = [
            (self.student_program, self.student_program),
            (self.student_program, self.reference_program),
            (self.reference_program, self.student_program),
        ]

        results = PartialCredit([self.run_test(image, th, test_order) for th in thresholds for test_order in test_orders] , points)()

        return int(results.score)

Where run_test returns a Pipeline. The error doesn't show up until results.score is called, and the error message points to line 41 of partialcredit.py

[Jmdwow]

For a more simple example that still produces an error:

results = PartialCredit(
    [Pipeline(Run(['ls'])) for _ in range(2)],
    [1, 2]
)()
self.score = results.score()

[Jmdwow]

I made a PR that should fix the error:
https://github.com/thoward27/grade/pull/27

[thoward27]

@Jmdwow that second example is much easier to wrap my head around. Going forward, could you also include the erroneous output? Whether it's a traceback or just an incorrect value?

[Jmdwow]

Will do! Here it was complaining about the '<' operator not being defined for a list and an int, but in future, I'll give the exact output.

[Jmdwow]

An example for the other small issue my code handled would be:

results = PartialCredit(
    [Pipeline(Run(['ls'])) for _ in range(5)],
    [1, 2]
)()
self.score = results.score()

Here, the way the documentation describes using PartialCredit would imply that this would work fine, assigning a score of 1, 2, 1, 2, and 1 to the 5 pipelines.

[thoward27]

Makes sense! As I mentioned in the PR, I'm not sure we actually want this behavior, do you see any reason to allow mismatched lengths between Pipelines and Scores?