Skip to content

fix: use total solved_count for issue credibility instead of valid_solved_count#643

Merged
anderdc merged 6 commits intoentrius:testfrom
edwin-rivera-dev:fix/issue-credibility-uses-total-solved-count
Apr 23, 2026
Merged

fix: use total solved_count for issue credibility instead of valid_solved_count#643
anderdc merged 6 commits intoentrius:testfrom
edwin-rivera-dev:fix/issue-credibility-uses-total-solved-count

Conversation

@edwin-rivera-dev
Copy link
Copy Markdown
Contributor

@edwin-rivera-dev edwin-rivera-dev commented Apr 21, 2026

Summary

  • Pass both solved_count and valid_solved_count to check_issue_eligibility
  • Use solved_count for credibility formula (matches docs and OSS pattern)
  • Use valid_solved_count for the minimum count gate (unchanged behavior)
  • Also update evaluation.issue_credibility to reflect the corrected value

Closes #642

Problem

check_issue_eligibility receives valid_solved_count and uses it for both the eligibility
gate and the credibility formula. The credibility formula should use ALL solved issues, not
just valid ones. This matches the OSS credibility pattern which passes all merged PRs to
calculate_credibility and separately counts valid ones.

Test plan

  • Miner with 10 solved (7 valid) + 2 closed gets credibility 0.91 not 0.875
  • Eligibility gate still requires 7 valid solved issues

@anderdc anderdc added the bug Something isn't working label Apr 21, 2026
Copy link
Copy Markdown
Collaborator

@anderdc anderdc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix is correct — the bug is real, the math in the issue is verified, and the approach matches the existing OSS credibility pattern cited in #642. Just requesting a test before merge.

Why a test matters for this one:

This is a scoring-formula change with direct impact on every miner's issue credibility. The bug was introduced by a silent parameter collapse during a previous refactor (solved_count and valid_solved_count got merged into one slot), and without a test locking down the corrected behavior, the same class of regression could happen again on the next refactor.

Suggested test shape (~15 lines in tests/validator/test_issue_discovery_scoring.py or similar):

@pytest.mark.parametrize(
    'solved_count,valid_solved_count,closed_count,expected_credibility,expected_eligible',
    [
        # Issue #642 example: 10 total solved, 7 valid, 2 closed
        # Credibility uses total (10), eligibility gate uses valid (7)
        (10, 7, 2, pytest.approx(10 / 11, abs=1e-3), True),
        # Valid count below threshold → not eligible regardless of credibility
        (10, 6, 2, pytest.approx(10 / 11, abs=1e-3), False),
        # Zero solved → credibility 0, not eligible
        (0, 0, 2, 0.0, False),
        # All solved counts equal (no low-token solvers) → fix is no-op for this case
        (7, 7, 2, pytest.approx(7 / 8, abs=1e-3), True),
    ],
)
def test_check_issue_eligibility_uses_total_for_credibility_valid_for_gate(
    solved_count, valid_solved_count, closed_count, expected_credibility, expected_eligible
):
    is_eligible, credibility, _ = check_issue_eligibility(solved_count, valid_solved_count, closed_count)
    assert credibility == expected_credibility
    assert is_eligible is expected_eligible

That locks down:

  • The 10/7/2 example from the issue body
  • The valid-below-threshold case (eligibility fails even with good credibility)
  • The zero-input edge case
  • The solved_count == valid_solved_count case (no-op for miners without low-token solvers)

Parametrizes cleanly, covers the branches, and documents the intent for future refactorers.

Everything else looks good — scope is minimal (4 code lines), approach matches the OSS pattern, signature change is local, and the caller is updated correctly. Once the test lands, ready to merge.

statxc added a commit to statxc/gittensor that referenced this pull request Apr 21, 2026
…lved_count

`check_issue_eligibility` received `data.valid_solved_count` as its first
parameter and used it for BOTH the eligibility minimum check and the
credibility formula. Per the docs, credibility should use ALL solved
issues (not just those whose solving PR has token_score >= 5). The OSS
credibility implementation (`oss_contributions/credibility.py:49-52`)
separates these correctly.

Example from entrius#642: a miner with 10 solved (7 valid, 3 low-token) and
2 closed issues gets credibility 7/8 = 0.875 today, but the correct
value is 10/11 = 0.91 — a 4% scoring penalty for having solved issues
with small PRs.

Splits `check_issue_eligibility` to accept `(solved_count,
valid_solved_count, closed_count)`: credibility uses `solved_count`,
the minimum-issues gate uses `valid_solved_count`. Caller in
`score_discovered_issues` is updated to pass both.

Adds a parametrized regression test locking the corrected behavior
per the maintainer's review of entrius#643: 4 cases covering the entrius#642
example, valid-below-threshold, zero input, and the no-op case where
`solved_count == valid_solved_count`.

Fixes entrius#642.
@edwin-rivera-dev edwin-rivera-dev force-pushed the fix/issue-credibility-uses-total-solved-count branch 2 times, most recently from 5b27127 to a4c615b Compare April 21, 2026 18:35
@edwin-rivera-dev edwin-rivera-dev force-pushed the fix/issue-credibility-uses-total-solved-count branch from a4c615b to 6aee1ce Compare April 21, 2026 18:41
@edwin-rivera-dev edwin-rivera-dev force-pushed the fix/issue-credibility-uses-total-solved-count branch from 631f182 to c312247 Compare April 21, 2026 22:01
@edwin-rivera-dev
Copy link
Copy Markdown
Contributor Author

Hi, @anderdc
Thanks for your feedback. Could you review again please?

@anderdc
Copy link
Copy Markdown
Collaborator

anderdc commented Apr 22, 2026

ci failed

@edwin-rivera-dev edwin-rivera-dev force-pushed the fix/issue-credibility-uses-total-solved-count branch from 30b9bc9 to ca60069 Compare April 23, 2026 04:55
@edwin-rivera-dev
Copy link
Copy Markdown
Contributor Author

Hi, @anderdc
CI Error has been resolved. Please let me know your recommendation.
Thanks.

@edwin-rivera-dev
Copy link
Copy Markdown
Contributor Author

🐛 CI Bug was fixed

Hi, @anderdc
How are you? Please review again and share your opinion.
Thanks.

@anderdc anderdc merged commit 4be8b22 into entrius:test Apr 23, 2026
3 checks passed
anderdc added a commit that referenced this pull request Apr 23, 2026
Origin/test's #643 changed check_issue_eligibility to take (solved_count,
valid_solved_count, closed_count) — previously (valid_solved_count,
closed_count). Mirror issue discovery call site needed the same update to
pass solved_count through.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Issue discovery credibility uses valid_solved_count instead of total solved_count - lowers credibility unfairly

3 participants