Move tests to GPU by maxjeblick · Pull Request #132 · NVIDIA/kvpress

maxjeblick · 2025-08-27T09:15:49Z

PR description

This PR

moves existing tests to gpu (if run on a GPU)
make test will only pass if all tests pass (i.e. we do not allow to skip tests)

Fixes #131

Checklist

Tests are working (make test)
Code is formatted correctly (make style, on errors try fix with make format)
Copyright header is included
All commits are signed-off using git commit -s
(new press) mypress_press.py is in the presses directory
(new press) MyPress is in __init__.py
(new press) README.md is updated with a 1 liner about the new press in the Available presses section
(new press) New press is in the default_presses list in tests/default_presses.py
(new press) A docstring is provided that follows the same structure as the existing ones

Signed-off-by: Maximilian Jeblick <maximilianjeblick@gmail.com>

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

copy-pr-bot · 2025-08-27T09:15:54Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

alessiodevoto · 2025-08-27T09:18:28Z

/ok to test c6383f2

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

maxjeblick · 2025-08-27T09:31:21Z

/ok to test f83914e

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

maxjeblick · 2025-08-27T09:37:24Z

/ok to test fa3e9c0

maxjeblick · 2025-08-27T09:44:35Z

/ok to test f83914e

copy-pr-bot · 2025-08-27T09:44:37Z

/ok to test f83914e

@maxjeblick, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

maxjeblick · 2025-08-27T09:46:41Z

/ok to test 59f3156

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

maxjeblick · 2025-08-27T10:02:48Z

/ok to test 9eaf5a7

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

maxjeblick · 2025-08-27T10:07:48Z

/ok to test 92d6e0e

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

maxjeblick · 2025-08-27T11:55:08Z

/ok to test 4987ef9

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

maxjeblick · 2025-08-27T12:07:11Z

/ok to test 7ad2049

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

maxjeblick · 2025-08-27T12:57:07Z

/ok to test a58c28b

maxjeblick · 2025-08-27T13:17:27Z

Works now except for qfilter press

Jack-Yu-815 · 2025-10-02T08:25:09Z

/ok to test df1b283

Signed-off-by: Jack Yu <jacyu@nvidia.com>

Jack-Yu-815 · 2025-10-02T08:46:11Z

/ok to test 5bd4b6c

Signed-off-by: Jack Yu <jacyu@nvidia.com>

Jack-Yu-815 · 2025-10-02T09:16:19Z

/ok to test a62ed14

…usly and OOM Signed-off-by: Jack Yu <jacyu@nvidia.com>

Jack-Yu-815 · 2025-10-02T10:21:32Z

/ok to test 5bd69ed

Signed-off-by: Jack Yu <jacyu@nvidia.com>

Jack-Yu-815 · 2025-10-02T11:30:05Z

/ok to test 9fbadec

Signed-off-by: Jack Yu <jacyu@nvidia.com>

Jack-Yu-815 · 2025-10-02T11:55:24Z

/ok to test 2876b69

Jack-Yu-815 · 2025-10-02T12:59:53Z

@maxjeblick, I think this PR is ready to merge. Changes from main is already merged to this branch after conflict resolution.

Changes since my takeover:

a separate RULER test specifically for QFilterPress
allow RULER test's prep_answer to be wrong. Give a warning instead of AssertionError.
changing LLM fixture scope to "class", allowing GPU memory to be freed first before loading another LLM. This avoids OOM on the runner.
refactor some tests (that require large GPU memory for LLM) into test class, which enables the fixture scope "class"

Signed-off-by: Jack Yu <jacyu@nvidia.com>

Jack-Yu-815 · 2025-10-06T15:05:49Z

/ok to test 6abef90

maxjeblick · 2025-10-07T08:05:03Z

tests/integration/test_ruler.py::TestRuler::test_ruler_is_correct[0-dynamic-press_dict0] FAILED [  3%]
FAILED tests/integration/test_ruler.py::TestRuler::test_ruler_is_correct[0-dynamic-press_dict0] - assert '9375710' in 'The special magic number for **fair-sprout** mentioned in the provided text is **937510**.\n\nWait — let\'s double-check.\n\nThe text says:\n\n> "One of the special magic numbers for fair-sprout'

The test is failing for one sample (as you mentioned). I propose finding a sample index that works for all presses.

Signed-off-by: Jack Yu <jacyu@nvidia.com>

Jack-Yu-815 · 2025-10-08T06:51:46Z

/ok to test 2aedcc5

Signed-off-by: Jack Yu <jacyu@nvidia.com>

Jack-Yu-815 · 2025-10-08T22:33:50Z

/ok to test defd68f

Signed-off-by: Jack Yu <jacyu@nvidia.com>

Jack-Yu-815 · 2025-10-09T00:26:26Z

/ok to test e4ae240

Signed-off-by: Jack Yu <jacyu@nvidia.com>

Jack-Yu-815 · 2025-10-09T02:09:27Z

/ok to test 9073cf3

Jack-Yu-815

LGTM

maxjeblick added 4 commits August 27, 2025 10:20

update to gpu tests

3bf1d63

Signed-off-by: Maximilian Jeblick <maximilianjeblick@gmail.com>

enfore all tests to run

08649f0

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

remove add in favor of pip install

c4fae0f

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

update pr template

c6383f2

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

maxjeblick requested a review from alessiodevoto August 27, 2025 09:15

alessiodevoto reviewed Aug 27, 2025

View reviewed changes

Comment thread pyproject.toml

set cuda env in test workflow

f83914e

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

maxjeblick added 2 commits August 27, 2025 11:36

set cuda env in test workflow

ecac1ba

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

set cuda env in test workflow

fa3e9c0

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

set cuda env in test workflow

59f3156

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

test Jimver/cuda-toolkit@v0.2.16 workflow

9eaf5a7

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

update cuda version

92d6e0e

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

alessiodevoto reviewed Aug 27, 2025

View reviewed changes

Comment thread Makefile

maxjeblick added 2 commits August 27, 2025 13:48

switch to Qwen/Qwen3-4B-Instruct-2507

423baa8

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

try flash attn

4987ef9

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

add back failing

7ad2049

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

add back cuda setup

a58c28b

Signed-off-by: Max Jeblick <maximilianjeblick@gmail.com>

skip QFilter RULER test

5bd4b6c

Signed-off-by: Jack Yu <jacyu@nvidia.com>

allow some RULER answer to be incorrect. Log a warning instead.

a62ed14

Signed-off-by: Jack Yu <jacyu@nvidia.com>

set LLM fixture scope to "class" to avoid loaded in memory simultaneo…

5bd69ed

…usly and OOM Signed-off-by: Jack Yu <jacyu@nvidia.com>

corrected test method parameter order

9fbadec

Signed-off-by: Jack Yu <jacyu@nvidia.com>

bug fix for test class

2876b69

Signed-off-by: Jack Yu <jacyu@nvidia.com>

maxjeblick commented Oct 2, 2025

View reviewed changes

Comment thread tests/integration/test_ruler.py Outdated

Jack-Yu-815 added 2 commits October 6, 2025 06:56

revert to using assertion for RULER correctness test

af5a08c

Signed-off-by: Jack Yu <jacyu@nvidia.com>

revert to using assertion for RULER correctness test

6abef90

Signed-off-by: Jack Yu <jacyu@nvidia.com>

maxjeblick commented Oct 7, 2025

View reviewed changes

Comment thread tests/presses/test_flash_attention.py Outdated

Jack-Yu-815 added 2 commits October 7, 2025 23:48

change RULER question index

04ec456

Signed-off-by: Jack Yu <jacyu@nvidia.com>

align test_fa_works model and tokenizer

2aedcc5

Signed-off-by: Jack Yu <jacyu@nvidia.com>

test RULER idx 0 to 19

defd68f

Signed-off-by: Jack Yu <jacyu@nvidia.com>

test RULER idx 0 to 19

e4ae240

Signed-off-by: Jack Yu <jacyu@nvidia.com>

change RULER test idx to 6

9073cf3

Signed-off-by: Jack Yu <jacyu@nvidia.com>

Jack-Yu-815 approved these changes Oct 9, 2025

View reviewed changes

maxjeblick merged commit d31a190 into main Oct 9, 2025
3 checks passed

maxjeblick deleted the max/gpu_tests branch October 9, 2025 19:52

Conversation

maxjeblick commented Aug 27, 2025

PR description

Checklist

Uh oh!

copy-pr-bot Bot commented Aug 27, 2025

Uh oh!

alessiodevoto commented Aug 27, 2025

Uh oh!

Uh oh!

maxjeblick commented Aug 27, 2025

Uh oh!

maxjeblick commented Aug 27, 2025

Uh oh!

maxjeblick commented Aug 27, 2025

Uh oh!

copy-pr-bot Bot commented Aug 27, 2025

Uh oh!

maxjeblick commented Aug 27, 2025

Uh oh!

maxjeblick commented Aug 27, 2025

Uh oh!

maxjeblick commented Aug 27, 2025

Uh oh!

Uh oh!

maxjeblick commented Aug 27, 2025

Uh oh!

maxjeblick commented Aug 27, 2025

Uh oh!

maxjeblick commented Aug 27, 2025

Uh oh!

maxjeblick commented Aug 27, 2025

Uh oh!

Jack-Yu-815 commented Oct 2, 2025

Uh oh!

Jack-Yu-815 commented Oct 2, 2025

Uh oh!

Jack-Yu-815 commented Oct 2, 2025

Uh oh!

Jack-Yu-815 commented Oct 2, 2025

Uh oh!

Jack-Yu-815 commented Oct 2, 2025

Uh oh!

Jack-Yu-815 commented Oct 2, 2025

Uh oh!

Jack-Yu-815 commented Oct 2, 2025

Uh oh!

Uh oh!

Jack-Yu-815 commented Oct 6, 2025

Uh oh!

Uh oh!

maxjeblick commented Oct 7, 2025

Uh oh!

Jack-Yu-815 commented Oct 8, 2025

Uh oh!

Jack-Yu-815 commented Oct 8, 2025

Uh oh!

Jack-Yu-815 commented Oct 9, 2025

Uh oh!

Jack-Yu-815 commented Oct 9, 2025

Uh oh!

Jack-Yu-815 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants