Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

algorithm: some consistency corrections required (consider add more ranks and unit tests for each rank) #98

Closed
Dentrax opened this issue Jul 18, 2020 · 4 comments
Labels
help wanted Extra attention is needed. ranks Feature, Bug fix, improvement related to ranking system. stale Issue is marked as stale.

Comments

@Dentrax
Copy link

Dentrax commented Jul 18, 2020

Describe the bug
Base algorithm has been changed to Cumulative Distribution Function, in order to get more accurate results, in this commit. But I'm not too sure either CDF really improves our rank algorithm. We have still have some consistency issues. (See the 'Additional context' below)

Brainstorming

  1. We can handle this problem if we add more ranks like: A, A+, A++, B, B+, B++, ... or AA, BA, BB, CB, CC, ...
  2. We should add more unit tests here for each rank, to make sure that there is no consistency problems between ranks there.
  3. We can add a rank-up table in README file, in order to know what values required to rank up and demonstrate how rank system works.
  4. Would Relative Grading solve this problem? (S+, S, A++, A+, B are not enough. )

Additional context

Some A+ Examples:

Should these be really A+ profiles? What about F ?

Total Stars: 4
Total Commits: 3
Total PRs: 1
Total Issues: 0
Contributed to: 0
Total Stars: 1
Total Commits: 1
Total PRs: 5
Total Issues: 6
Contributed to: 0

These are A+ profiles too, but shouldn't be. These demonstrates us why we need more intermediate values between ranks.

Total Stars: 3
Total Commits: 86
Total PRs: 133
Total Issues: 26
Contributed to: 4
Total Stars: 14
Total Commits: 91
Total PRs: 33
Total Issues: 38
Contributed to: 45
Total Stars: 497
Total Commits: 83
Total PRs: 11
Total Issues: 118
Contributed to: 38
@anuraghazra anuraghazra added help wanted Extra attention is needed. ranks Feature, Bug fix, improvement related to ranking system. labels Jul 18, 2020
@anuraghazra
Copy link
Owner

Yes @Dentrax the ranking algorithm isn't really consistent throughout,
@b1f6c1c4 did a pr which includes h-index #76 which might also work maybe.

I'm not good at maths so i don't have any idea which algorithm will result in good balance. so any help would be appreciated.
if anyone wanted to implement this PR feel free to do so.

@b1f6c1c4
Copy link

  1. There is no ultimate and universal way to merge multiple metrics into one metric. Weighted arithmetic average is the closest one. However metrics needs to be scaled before they are averaged.
  2. The best way to determine how to scale the metric, as well as how to divide the continuous combined metrics into discrete ranks, is through statistics. It's definitely necessary to sample a moderate amount of GitHub users (not just famous users), collect their metrics, and do statistics.

@NeoTse
Copy link

NeoTse commented Jul 20, 2020

if we let the score is zero, then

// TOTAL_VALUES = RANK_S_VALUE + RANK_A2_VALUE + RANK_A3_VALUE + RANK_B_VALUE = 206
// ALL_OFFSETS = CONTRIBS_OFFSET + ISSUES_OFFSET + STARS_OFFSET + PRS_OFFSET + FOLLOWERS_OFFSET + REPO_OFFSET = 5.35
normalcdf(0, 206, 5.35) * 100 = 51.03597635931825

but A+ rank requires the result between [0.45, 0.65), and the score will never be a negative. So any rank below it is useless.

@stale
Copy link

stale bot commented Dec 6, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale Issue is marked as stale. label Dec 6, 2020
@stale stale bot closed this as completed Dec 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed. ranks Feature, Bug fix, improvement related to ranking system. stale Issue is marked as stale.
Projects
None yet
Development

No branches or pull requests

4 participants