Leaderboard not well ordered

[Didayolo]

The leaderboard the validation phase of the following competition is not well ordered:

https://www.codabench.org/competitions/1219/

(the competition is private, if you need the secret URL please ask me)

---

UPDATE:

This problem is linked to another problem: the score in the tab "Submissions" (either participant tab or admin tab) does not reflect perfectly the score in the leaderboard.

This is likely due to the javascript rounding found in submission_manager.tag:

        self.get_score = function (submission) {
            try{
                return parseFloat(submission.scores[0].score).toFixed(2)
           [...]

It seems that javascript standard for floats is not precise. The order is based on the scores that you can find in the "Submissions" tab, not the actual scores. The leaderboard reflect the true score.

---

So, it is ordered by primary column, but using a bad rounding.

[Didayolo]

Even clicking on "Score" to re-order the rows does not give the right order

[ihsaan-ullah]

I think this is not competition specific problem.

In this competition: https://www.codabench.org/competitions/984/#/results-tab
you can see there is a problem

DESC

ASC

[Didayolo]

I don't understand where the order comes from. It could be based on another score, or on the best score of the participant, maybe on the date... Maybe we should investigate the code directly.

[ihsaan-ullah]

I don't understand where the order comes from. It could be based on another score, or on the best score of the participant, maybe on the date... Maybe we should investigate the code directly.

When you load the leaderboard, the order seems correct. I think the order is changed and is buggy because of the reordering when you click the column

[Didayolo]

No, in my case, the leaderboard is wrong when you load the page. You can see that the medals (1st, 2nd, etc.) are not well ordered as well.

[Didayolo]

UPDATE:

This problem is linked to another problem: the score in the tab "Submissions" (either participant tab or admin tab) does not reflect perfectly the score in the leaderboard.

This is likely due to the javascript rounding found in submission_manager.tag:

        self.get_score = function (submission) {
            try{
                return parseFloat(submission.scores[0].score).toFixed(2)
           [...]

It seems that javascript standard for floats is not precise. The order is based on the scores that you can find in the "Submissions" tab, not the actual scores. The leaderboard reflect the true score.

[Didayolo]

In https://github.com/codalab/codabench/blob/develop/src/apps/api/serializers/leaderboards.py :

    def get_submissions(self, instance):
        # desc == -colname
        # asc == colname
        primary_col = instance.leaderboard.columns.get(index=instance.leaderboard.primary_index)
        ordering = [f'{"-" if primary_col.sorting == "desc" else ""}primary_col']
        submissions = Submission.objects.filter(
            phase=instance,
            has_children=False,
            is_specific_task_re_run=False,
            leaderboard__isnull=False, ) \
            .select_related('owner').prefetch_related('scores') \
            .annotate(primary_col=Sum('scores__score', filter=Q(scores__column=primary_col)))

        for column in instance.leaderboard.columns.exclude(id=primary_col.id, hidden=False).order_by('index'):
            col_name = f'col{column.index}'
            ordering.append(f'{"-" if column.sorting == "desc" else ""}{col_name}')
            kwargs = {
                col_name: Sum('scores__score', filter=Q(scores__column__index=column.index))
            }
            submissions = submissions.annotate(**kwargs)

        submissions = submissions.order_by(*ordering, 'created_when')
        return SubmissionLeaderBoardSerializer(submissions, many=True).data

[bbearce]

Not seeing this right now and I'm not sure what the difference is:

fresh page load:

click Accuracy:

click Accuracy again:

local:
commit 1c3d366

bbearce@pop-os:~/Documents/codabench$ git branch

• develop

[bbearce]

That being said I wonder if we should make a leaderboard issue to encapsulate them all:

• Leaderboard not well ordered #1171
• n/a scores on leaderboard if Submission Rule is Force Last #1055
• Multiple leaderboards #873
• [Feature] Ranking program: organizer provides the code for computation between leaderboard columns #406

I have one that's new and I can make it its own issue:
While order above isn't my issue I see that computations aren't working right and options I specifically unset and saved persist when I go back into the editor:

Before:

After:

Cols in DB (wrong changes are being propagated to db):

Update:
Reuploaded the competition with leaderboard:

leaderboards:
  - title: Results
    key: main
    columns:
      - title: Correct1
        key: correct1
        index: 0
        sorting: desc
      - title: Correct2
        key: correct2
        index: 1
        sorting: desc
      - title: Computed
        key: computed
        index: 2
        sorting: desc
        computation: avg
        computation_indexes:
          - 0
          - 1

It works on fresh upload. This leads me to believe as we edit the leaderboard and add\remove columns, old data (whether deleted or not) is contributing to incorrect results.

Lastly I change Computed to (max) and results don't update. I recommend backend code to recalculate computed values.

Diving a little deeper I notice when a submission is deleted, its leaderboard score is not. I think this should be changed. I believe we've been discussing whether or not we should invalidate submissions when a task is updated. I think we need to think about scores on the leaderboard as well. It seems to me that we would want to trigger a re-calculation of leaderboard scores if the leaderboard has been edited. This may be too big a feature. We could instead simplify by marking old submissions before leaderboard change as invalid or not synced with leaderboard. his is because we may not be able to generate scores for new columns (because of stale scoring program) or may have too many to fit on new leaderboard (again because maybe the scoring program generated too many). In these cases I recommend we mark all scores n/a if the leaderboard has been updated and leave a note on leaderboard or in participant's submission view.

Quick summary of above:
If leaderboard change is minor, re-running submission should suffice to generate new scores and old submissions can be deleted. This gets complicated if submissions take a long time to run.

If leaderboard change is major (new columns\removed columns), the old scoring program will not match leaderboard. It is my belief that editing the task to make a new scoring program that matches the new leaderboard is necessary.

On the topic of submissions tied to tasks:
My two cents on "keeping" old submissions valid or associating submissions to tasks is that is all fine and good as the task has the scoring program that generated the scores associated with submission (this is good for reproduceability). I don't know that I think the task\submission relationship is a good idea. It adds an overwhelming amount of complexity to track. In my opinion, challenges shouldn't change once they are started. We may need to make a couple tweaks here and there, but if the scoring program changes, the task changes, and we should just re-run old submissions to generate new scores. If the old submission doesn't match the new valid format required by a new scoring program\task, I believe it should be considered invalid for the new task and or challenge. These should be marked for deletion or as stale and shouldn't be allowed on the leaderboard at all. They can maybe stay around just for the benefit of the participant's history.

I know that participants are worried about their old "working" submission is not a "failed" one, but I don't see why challenge organizers would change the task last minute in a challenge anyways. Even if they did, they have a duty to re-run the submissions that already happened and if there are too many to reasonably run, I don't think it's a good idea. If the change is absolutely necessary last minute, then why can't every participant remake their submission? If the change is that drastically different and necessary, then it's everyone's responsibility to do their part to make new submissions. If this is annoying to participants, organizers should think wisely whether this change is absolutely necessary.

In summary: Anyways, I think edits to the leaderboard should flag all submissions as stale and require a re-submission to be allowed on leaderboard and only allow successful non-stale submissions to leaderboard.

Sorry for the massive wall of text ;)

[bbearce]

Hi all,

Picking up where we left off the new test was to make a basic scoring program that copies scores from submission directly to results.txt in the right leaderboard format. This is to make is to I can submit direct scores and have them show up on the leaderboard exactly how I want them. With that said I can recreate the leaderboard from the first image posted about this issue.

These were the input text files:

When I click the score column to sort I do run into the sort issue still: