Fix benchmarks.yml: use ec2-gha, remove FOMO benchmark#399
Merged
Conversation
The addition of FOMO model testing in PR #384 caused benchmark failures due to excessive memory usage. FOMO with Perceiver layers uses ~170GB RAM for CPU benchmarks and exceeds 22GB GPU memory. This commit removes FOMO from the benchmark suite, restoring it to the last known working state. A separate commit will follow to add a lightweight FOMO configuration suitable for CI benchmarks. Fixes benchmark OOM failures that started with #384. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
jder
approved these changes
Sep 24, 2025
jder
left a comment
jder
left a comment
Member
There was a problem hiding this comment.
Thanks! Looking into why these were broken was next on my list so hope it wasn't too bad :)
Co-authored-by: Jesse Rusak <jesse@openathena.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
benchmarks.ymlto use ec2-gha, which resulted in an invalid workflow file.benchmarks.ymlhas been broken onmainsince #384, which added a FOMO model benchmark that uses more memory (and GPU memory)This PR fixes the former by updating
benchmarks.ymlto use ec2-gha, and works around the latter by restoring the benchmarks configs to the pre-#384 state.benchmarks#172 is a passing run from
f313865