docs: add scoring system, rules, and contribution track#45
Merged
msaroufim merged 23 commits intogpu-mode:mainfrom Mar 14, 2026
Merged
docs: add scoring system, rules, and contribution track#45msaroufim merged 23 commits intogpu-mode:mainfrom
msaroufim merged 23 commits intogpu-mode:mainfrom
Conversation
Documents ENABLE_TILE=0 vs ENABLE_TILE=1 and the TileIR compilation pipeline available via nvtriton on B200 instances. Covers how to enable TileIR with Helion (ENABLE_TILE=1 + HELION_BACKEND=tileir), the different tunables (num_ctas/occupancy vs num_warps/maxnreg), and how to hardcode TileIR configs in submissions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Group both sections under a single "Optional: Extra Performance Knobs" heading to emphasize neither is required. Streamline both into step 1 (autotune) / step 2 (hardcode) format. Add a "Which combination" section showing all 4 options to try. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove duplicate bash export block — the Python os.environ in the code example is sufficient for both local autotuning and submissions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add point allocation table, scoring formula (correctness + performance ranking), rules & requirements, and the separate open-ended contribution track for non-kernel Helion contributions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each submission uses one static helion.Config for all shapes, not per-shape configs. Simplified rules to reflect this. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This reverts commit 4fac3a8.
Use a factory function (_make_kernel) to create kernel variants with different helion.Config objects, and dispatch in custom_kernel() based on input tensor shapes. This lets participants optimize each benchmark shape independently. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…st vs benchmark shapes Test shapes: TODO to replace with default config or any config that passes correctness. Benchmark shapes: TODO to replace with autotuned config. Also add instructions on getting default config via autotune_effort="none". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per-shape configs are the recommended approach. Remove mentions of using a single config across all shapes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Configs are always participant-provided. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous description incorrectly stated geometric mean of 100 runs. The actual helion eval uses CUDA graphs with L2 cache clearing, 10 measurements, and arithmetic mean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The LOC-based rule was gameable (denominator inflation with padding code), so switch to a qualitative rule: inline triton/asm is allowed as escape hatches, but predominantly inline submissions may be disqualified. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Spawn mode isolates each autotuner trial in a subprocess with timeout protection, preventing hangs or crashes from killing the entire run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
msaroufim
approved these changes
Mar 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
🤖 Generated with Claude Code