Skip to content

test: adjust is_timing_close #17178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Mar 24, 2023
Merged

test: adjust is_timing_close #17178

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
7488586
is_timing_close
Borda Mar 23, 2023
9f28b8a
tol
Borda Mar 23, 2023
ceb33f8
revert min
Borda Mar 23, 2023
eafc67d
tols
Borda Mar 23, 2023
6b23bcd
Merge branch 'master' into test/is_timing_close
Borda Mar 24, 2023
d70d00d
Merge branch 'master' into test/is_timing_close
Borda Mar 24, 2023
d6e101b
Devices
Borda Mar 24, 2023
d699104
Merge branch 'master' into test/is_timing_close
Borda Mar 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions tests/tests_fabric/parity/test_parity_ddp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,8 +130,8 @@ def train_fabric_ddp(fabric):
@pytest.mark.parametrize(
"accelerator, devices, tolerance",
[
("cpu", 2, 0.01),
pytest.param("cuda", 2, 0.005, marks=RunIf(min_cuda_gpus=2)),
("cpu", 2, 0.02),
pytest.param("cuda", 2, 0.01, marks=RunIf(min_cuda_gpus=2)),
],
)
def test_parity_ddp(accelerator, devices, tolerance):
Expand Down
2 changes: 1 addition & 1 deletion tests/tests_fabric/parity/test_parity_simple.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ def test_parity_single_device(precision, accelerator):
assert is_state_dict_equal(state_dict_torch, state_dict_fabric)

# Compare the time per iteration
assert is_timing_close(timings_torch, timings_fabric, rtol=1e-3, atol=1e-3)
assert is_timing_close(timings_torch, timings_fabric, rtol=1e-2, atol=0.1)

# Compare memory usage
if accelerator == "cuda":
Expand Down
2 changes: 1 addition & 1 deletion tests/tests_fabric/parity/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def is_state_dict_equal(state0, state1):
return all(eq_fn(w0.cpu(), w1.cpu()) for w0, w1 in zip(state0.values(), state1.values()))


def is_timing_close(timings_torch, timings_fabric, rtol=1e-3, atol=1e-3):
def is_timing_close(timings_torch, timings_fabric, rtol=1e-2, atol=0.1):
# Drop measurements of the first iterations, as they may be slower than others
# The median is more robust to outliers than the mean
# Given relative and absolute tolerances, we want to satisfy: |torch – fabric| < RTOL * torch + ATOL
Expand Down