Skip to content

Add challenge prequalification fairness guard#423

Open
KoiosSG wants to merge 4 commits into
SCIBASE-AI:mainfrom
KoiosSG:challenge-prequalification-fairness-18
Open

Add challenge prequalification fairness guard#423
KoiosSG wants to merge 4 commits into
SCIBASE-AI:mainfrom
KoiosSG:challenge-prequalification-fairness-18

Conversation

@KoiosSG
Copy link
Copy Markdown

@KoiosSG KoiosSG commented May 28, 2026

/claim #18

Summary

Adds a distinct challenge-prequalification-fairness-guard/ slice for Scientific Bounty System issue #18.

The guard evaluates sponsor-side prequalification rounds before solver teams are accepted or rejected. It checks published criteria, weighted score threshold consistency, anonymous-screening requirements, reviewer conflicts, reviewer quorum, rejection reason completeness, appeal windows, and deterministic audit evidence. Conflicted reviewer scores are excluded from threshold scoring while the conflict remains auditable. Unfair or incomplete screening decisions are held for remediation before challenge access changes.

Hardening Update

  • Excludes conflicted reviewer scores from weighted threshold evidence, preventing sponsor-adjacent scores from inflating acceptance/rejection decisions while still surfacing the conflict finding.

Non-overlap

This is scoped to prequalification fairness before solver acceptance/rejection. It does not duplicate the broad bounty-system modules, intake compliance, workspace privacy, clarification freeze, arbitration/scoring, payout eligibility, sponsor data-room access, benchmark leakage, evaluator calibration, reviewer workload SLA, award transparency, appeals, escrow settlement, or sponsor reliability slices.

Validation

  • Wrote failing test first: node challenge-prequalification-fairness-guard\test.js failed on missing ./index
  • Added hardening regression first: npm test failed with 75 == 50 before excluding conflicted reviewer scores
  • npm run check from challenge-prequalification-fairness-guard passed: 6 tests, demo, video
  • node --check challenge-prequalification-fairness-guard\index.js
  • node --check challenge-prequalification-fairness-guard\demo.js
  • node --check challenge-prequalification-fairness-guard\test.js
  • ffprobe -v error -select_streams v:0 -show_entries stream=codec_name,width,height,duration,avg_frame_rate -show_entries format=size,duration -of default=noprint_wrappers=1 challenge-prequalification-fairness-guard\reports\demo.mp4 -> H.264, 1280x720, 4s, 30fps, 45,643 bytes
  • git diff --check
  • git diff --cached --check
  • rg -n "(password|secret|wallet|paypal|bank|passport|private key|api key)" challenge-prequalification-fairness-guard -> no matches

Demo Artifacts

  • challenge-prequalification-fairness-guard/reports/prequalification-fairness-packet.json
  • challenge-prequalification-fairness-guard/reports/prequalification-fairness-report.md
  • challenge-prequalification-fairness-guard/reports/summary.svg
  • challenge-prequalification-fairness-guard/reports/demo.mp4

Synthetic data only. No credentials, payment processors, identity providers, private workspaces, sponsor systems, solver accounts, payout systems, or external APIs are used.

AI-assisted with OpenAI Codex; I reviewed and locally verified the diff before submitting.

@KoiosSG
Copy link
Copy Markdown
Author

KoiosSG commented May 28, 2026

Hardening update pushed in 25af825: conflicted reviewer scores are now excluded from weighted threshold evidence, while the conflict finding remains auditable. I added a regression that failed before the fix with 75 == 50 and now passes. Validation refreshed locally: npm run check, npm test (6 tests), node --check on index/demo/test, ffprobe on demo.mp4, git diff --check, and sensitive-term scan returned no matches.

@KoiosSG
Copy link
Copy Markdown
Author

KoiosSG commented May 28, 2026

Follow-up hardening pass for the challenge prequalification fairness guard.

What changed:

  • Added an explicit regression for rejected applicants whose appeal deadline has already expired before the prequalification packet is generated.
  • Treat expired appeal windows as expired-appeal-window fairness holds rather than allowing them through as reject-with-audit.
  • Route the remediation to publish-rejection-reasons-and-appeal-window, matching the applicant-facing fairness action for missing appeal windows.

Validation:

  • Confirmed the new regression failed before the implementation with reject-with-audit instead of hold-for-fairness-review.
  • npm test -> 7 challenge prequalification fairness tests passed.
  • npm run check -> test, demo, and demo video generation passed.
  • ffprobe verified reports/demo.mp4 as H.264, 1280x720, 30 fps, 4.0s.
  • git diff --check and git diff --cached --check passed, with only Git line-ending normalization warnings on Windows.
  • Sensitive-term scan found only expected local hashing/test assertions, no payout or credential strings.

@KoiosSG
Copy link
Copy Markdown
Author

KoiosSG commented May 28, 2026

Follow-up competitive hardening pass for the challenge prequalification fairness guard.

What changed:

  • Added a regression for malformed published rubric weights, where criteria totaled 145 instead of 100.
  • Hold affected applicants for fairness review when the published criterion weights do not sum to 100, before sponsor accept/reject decisions can take effect.
  • Added a high-priority remediation action: publish-valid-weighted-scoring-rubric.
  • Updated the README so the documented guard explicitly covers valid criterion weight totals.

Why this matters:

Validation:

  • Confirmed the new regression failed before the implementation: applicant was incorrectly accept-prequalified with invalid 145-total criterion weights.
  • npm test -> 8 challenge prequalification fairness tests passed.
  • npm run check -> test, demo, and video generation passed.
  • npm run demo -> generated prequalification packet/report/SVG with expected accepted/held counts.
  • ffprobe verified reports/demo.mp4 as H.264, 1280x720, 30 fps, 4.0s, 45,643 bytes.
  • git diff --check and git diff --cached --check passed; the only messages were Git line-ending normalization warnings on Windows.
  • Sensitive-term scan of the code/test/docs patch found no payout or credential strings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant