Skip to content

Conversation

@kshitij12345
Copy link
Collaborator

Ref: #2363

@kshitij12345 kshitij12345 marked this pull request as ready for review July 28, 2025 12:43
@kshitij12345
Copy link
Collaborator Author

Need to relax more -

=========================== short test summary info ============================
FAILED thunder/tests/test_grad.py::test_vjp_correctness_softsign_nvfuser_cuda_thunder.dtypes.float64 - AssertionError: Scalars are not close!

Expected 0.018125936040412825 but got 0.018194965055091755.
Absolute difference: 6.902901467892991e-05 (up to 1e-05 allowed)
Relative difference: 0.0038083006872045517 (up to 1e-05 allowed)
FAILED thunder/tests/test_grad.py::test_vjp_correctness_normalize_nvfuser_cuda_thunder.dtypes.float64 - AssertionError: Scalars are not close!

Expected 1.6720222046537816 but got 1.6720607368208813.
Absolute difference: 3.8532167099702974e-05 (up to 1e-05 allowed)
Relative difference: 2.304524843776322e-05 (up to 1e-05 allowed)
FAILED thunder/tests/test_grad.py::test_vjp_correctness_abs_nvfuser_cuda_thunder.dtypes.float64 - AssertionError: Scalars are not close!

Expected 1.6447549289287395 but got 1.644292826080398.
Absolute difference: 0.00046210284834158344 (up to 1e-05 allowed)
Relative difference: 0.0002809554421840583 (up to 1e-05 allowed)
= 3 failed, 733 passed, 55 skipped, 23 xfailed, 15 xpassed, 92480 warnings in 830.74s (0:13:50) =

CI - https://dev.azure.com/Lightning-AI/lightning/_build/results?buildId=238962&view=logs&j=83d7ca60-c2d2-50c8-1ed9-9f99750fc4f0&t=d88aa08c-a7b7-5504-bdb7-50942829cc54&l=2233

@kshitij12345
Copy link
Collaborator Author

Still more tol required

=========================== short test summary info ============================
FAILED thunder/tests/test_grad.py::test_vjp_correctness_softsign_nvfuser_cuda_thunder.dtypes.float64 - AssertionError: Scalars are not close!

Expected 0.28762750612550503 but got 0.2877591463516204.
Absolute difference: 0.0001316402261153926 (up to 0.0001 allowed)
Relative difference: 0.0004576760682198175 (up to 0.0001 allowed)
FAILED thunder/tests/test_grad.py::test_vjp_correctness_abs_nvfuser_cuda_thunder.dtypes.float64 - AssertionError: Scalars are not close!

Expected -0.39983096480067737 but got -0.40020568936399437.
Absolute difference: 0.0003747245633169971 (up to 0.0001 allowed)
Relative difference: 0.0009372074609173999 (up to 0.0001 allowed)
= 2 failed, 734 passed, 55 skipped, 23 xfailed, 15 xpassed, 92918 warnings in 830.85s (0:13:50) =

https://dev.azure.com/Lightning-AI/lightning/_build/results?buildId=238966&view=logs&j=83d7ca60-c2d2-50c8-1ed9-9f99750fc4f0&t=d88aa08c-a7b7-5504-bdb7-50942829cc54&l=2137

@t-vi
Copy link
Collaborator

t-vi commented Jul 28, 2025

The funny part is that it's only on PT nightly. Maybe an upstream thing, even?

@kshitij12345
Copy link
Collaborator Author

Ping @t-vi for stamping

@kshitij12345 kshitij12345 enabled auto-merge (squash) July 29, 2025 12:08
Copy link
Collaborator

@t-vi t-vi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @kshitij12345

@kshitij12345 kshitij12345 merged commit b5f00a1 into main Jul 29, 2025
53 of 54 checks passed
@kshitij12345 kshitij12345 deleted the relax-double-grad-test-tol branch July 29, 2025 13:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants