add patches to fix test issues for PyTorch 2.1.2 with `foss/2023a` + CUDA 12.1.1 #20156

Flamefire · 2024-03-19T15:55:10Z

(created using eb --new-pr)

Fixes #19946

easybuild/easyconfigs/p/PyTorch/PyTorch-2.1.2-foss-2023a-CUDA-12.1.1.eb

casparvl · 2024-03-21T08:44:14Z

Test report by @casparvl
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
gcn6.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/casparvl/127d18ffbe2dfa016b8e2b31db21583b for a full test report.

Flamefire · 2024-03-22T00:25:57Z

Test report by @Flamefire
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
i8001 - Linux Rocky Linux 8.7 (Green Obsidian), x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.8.13
See https://gist.github.com/Flamefire/03f217eae6950077bb56d51d164aecf4 for a full test report.

jfgrimm · 2024-03-23T06:18:34Z

Test report by @jfgrimm
SUCCESS
Build succeeded (with --ignore-test-failure) for 1 out of 1 (1 easyconfigs in total)
gpu22.viking2.yor.alces.network - Linux Rocky Linux 8.8, x86_64, AMD EPYC 7413 24-Core Processor, 1 x NVIDIA NVIDIA H100 PCIe, 535.86.10, Python 3.6.8
See https://gist.github.com/jfgrimm/d7245f91e629eca8cd8c2cc129b142de for a full test report.

casparvl · 2024-03-23T09:35:19Z

Test report by @casparvl
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
gcn6.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/casparvl/d1a5eb6abbfedbecdc909e6d2db8076a for a full test report.

Flamefire · 2024-03-23T09:57:30Z

Test report by @casparvl FAILED Build succeeded for 0 out of 1 (1 easyconfigs in total) gcn6.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8 See https://gist.github.com/casparvl/d1a5eb6abbfedbecdc909e6d2db8076a for a full test report.

Looks like we need to increase the allowed failures to ~10. Yours report 8. 6 of them are caught in detail: test_Conv1d_pad_same_cuda_tf32, test_constant_specialization, test_delayed_optim_step_offload_true_no_shard, test_file_reader_no_memory_leak, test_file_reader_no_memory_leak, test_file_reader_no_memory_leak

The first is from test_nn which is kinda know, the last 3 are known on your machine from other runs. the other 2 (4) I don't know.

The full test log might be useful to enhance the RegEx to capture the other 2 tests too. I think it really helps having the individual tests listed conveniently at a single place to judge the failure (see e.g. the last 3 where you can see that it is the same cause, not only: "test_jit_foo", "test_jit_bar", "test_jit" files failed)

casparvl · 2024-03-24T12:03:32Z

@boegelbot please test @ generoso
CORE_CNT=16

boegelbot · 2024-03-24T12:05:09Z

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=20156 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_20156 --ntasks="16" ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 13194

Test results coming soon (I hope)...

- notification for comment with ID 2016786483 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

casparvl · 2024-03-24T12:18:02Z

test_jit 1/1 (1 failed, 2388 passed, 106 skipped, 12 xfailed, 2 rerun)
test_proxy_tensor 1/1 (1 failed, 2078 passed, 613 skipped, 80 xfailed, 2 rerun)
distributed/fsdp/test_fsdp_core 1/1 (2 failed, 58 passed, 10 rerun)
test_jit_legacy 1/1 (1 failed, 2390 passed, 104 skipped, 12 xfailed, 2 rerun)
test_jit_profiling 1/1 (1 failed, 2388 passed, 106 skipped, 12 xfailed, 2 rerun)
test_nn 1/1 (1 failed, 2804 passed, 122 skipped, 3 xfailed, 2 rerun)
distributed/test_c10d_nccl 1/1 (1 unit test(s) failed)

Detailed log

test_jit
Ok, we know this from #20171 , I'm not too worried here as it might be that kernel memory leak issue I mentioned.

================================================================ FAILURES ================================================================
_______________________________________________ TestScript.test_file_reader_no_memory_leak _______________________________________________
Traceback (most recent call last):
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 57, in testPartExecutor
    yield
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 623, in run
    self._callTestMethod(testMethod)
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 2407, in wrapper
    method(*args, **kwargs)
  File "/gpfs/nvme1/1/casparl/ebbuildpath/PyTorch/2.1.2/foss-2023a-CUDA-12.1.1/pytorch-v2.1.2/test/test_jit.py", line 12862, in test_file_reader_no_memory_leak
    assert peak_from_file < peak_from_string * 500
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
======================================================== short test summary info =========================================================
FAILED [2.4948s] test_jit.py::TestScript::test_file_reader_no_memory_leak - AssertionError
============================== 1 failed, 2388 passed, 106 skipped, 12 xfailed, 2 rerun in 166.02s (0:02:46) ==============================
FINISHED PRINTING LOG FILE of test_jit 1/1 (/gpfs/nvme1/1/casparl/ebbuildpath/PyTorch/2.1.2/foss-2023a-CUDA-12.1.1/pytorch-v2.1.2/test/test-reports/test_jit_g6he3ln_.log)

test_proxy_tensor
I have the feeling this should not be there, since you included Z3 as dependency (right?)...:

================================================================= RERUNS =================================================================
____________________________________________ TestSymbolicTracing.test_constant_specialization ____________________________________________
Traceback (most recent call last):
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 57, in testPartExecutor
    yield
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 623, in run
    self._callTestMethod(testMethod)
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 2407, in wrapper
    method(*args, **kwargs)
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/contextlib.py", line 81, in inner
##[endgroup]
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/gpfs/nvme1/1/casparl/ebbuildpath/PyTorch/2.1.2/foss-2023a-CUDA-12.1.1/pytorch-v2.1.2/test/test_proxy_tensor.py", line 1491, in test_constant_specialization
    tensor = make_fx(f, tracing_mode="symbolic")(torch.randn(10))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 739, in wrapped
    shape_env = ShapeEnv()
                ^^^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/fx/experimental/symbolic_shapes.py", line 2116, in __init__
    if _translation_validation_enabled():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/fx/experimental/symbolic_shapes.py", line 1483, in _translation_validation_enabled
    return translation_validation_enabled()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/fx/experimental/validator.py", line 537, in translation_validation_enabled
    assert_z3_installed_if_tv_set()
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/fx/experimental/validator.py", line 546, in assert_z3_installed_if_tv_set
    assert _HAS_Z3 or not config.translation_validation, (
AssertionError: translation validation requires Z3 package. Please, either install z3-solver or disable translation validation.

To execute this test, run the following from the base repo dir:
     python test/test_proxy_tensor.py -k test_constant_specialization

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

distributed/fsdp/test_fsdp_core
These are ... potentially worrying:

================================================================= RERUNS =================================================================
____________________________________ TestParityWithDDP.test_delayed_optim_step_offload_true_no_shard _____________________________________
Traceback (most recent call last):
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 57, in testPartExecutor
    yield
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 623, in run
    self._callTestMethod(testMethod)
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 506, in wrapper
    self._join_processes(fn)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 725, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 775, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 2 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
    getattr(self, test_name)()
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 508, in wrapper
    fn()
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 2407, in wrapper
    method(*args, **kwargs)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 356, in instantiated_test
    test(self, **param_kwargs)
  File "/gpfs/nvme1/1/casparl/ebbuildpath/PyTorch/2.1.2/foss-2023a-CUDA-12.1.1/pytorch-v2.1.2/test/distributed/fsdp/test_fsdp_core.py", line 188, in test_delayed_optim_step
    self.run_subtests(
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_fsdp.py", line 896, in run_subtests
    return run_subtests(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_fsdp.py", line 848, in run_subtests
    test_fn(*test_args, **test_kwargs, **subtest_kwargs)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_fsdp.py", line 1170, in _test_fsdp_parity
    torch.testing.assert_close(ref_loss, fsdp_loss, check_dtype=False)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_comparison.py", line 1520, in assert_close
    raise error_metas[0].to_error(msg)
AssertionError: Scalars are not close!

Expected -1.2194809913635254 but got -2.304675579071045.
Absolute difference: 1.0851945877075195 (up to 1e-05 allowed)
Relative difference: 0.8898823314122694 (up to 1.3e-06 allowed)

To execute this test, run the following from the base repo dir:
     python test/distributed/fsdp/test_fsdp_core.py -k test_delayed_optim_step_offload_true_no_shard

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

______________________________________ TestParityWithDDP.test_mixture_of_experts_offload_true_none _______________________________________
Traceback (most recent call last):
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 57, in testPartExecutor
    yield
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 623, in run
    self._callTestMethod(testMethod)
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 506, in wrapper
    self._join_processes(fn)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 725, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 775, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 1 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
    getattr(self, test_name)()
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 508, in wrapper
    fn()
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 2407, in wrapper
    method(*args, **kwargs)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 356, in instantiated_test
    test(self, **param_kwargs)
  File "/gpfs/nvme1/1/casparl/ebbuildpath/PyTorch/2.1.2/foss-2023a-CUDA-12.1.1/pytorch-v2.1.2/test/distributed/fsdp/test_fsdp_core.py", line 231, in test_mixture_of_experts
    self.run_subtests(
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_fsdp.py", line 896, in run_subtests
    return run_subtests(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_fsdp.py", line 848, in run_subtests
    test_fn(*test_args, **test_kwargs, **subtest_kwargs)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_fsdp.py", line 1170, in _test_fsdp_parity
    torch.testing.assert_close(ref_loss, fsdp_loss, check_dtype=False)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_comparison.py", line 1520, in assert_close
    raise error_metas[0].to_error(msg)
AssertionError: Scalars are not close!

Expected -3.869016647338867 but got -2.7315802574157715.
Absolute difference: 1.1374363899230957 (up to 1e-05 allowed)
Relative difference: 0.29398591259756696 (up to 1.3e-06 allowed)

_________________________ TestParityWithDDP.test_mixture_of_experts_with_delay_before_free_offload_true_no_shard _________________________
Traceback (most recent call last):
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 57, in testPartExecutor
    yield
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 623, in run
    self._callTestMethod(testMethod)
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 506, in wrapper
    self._join_processes(fn)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 725, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 775, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 3 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
    getattr(self, test_name)()
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_distributed.py", line 508, in wrapper
    fn()
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 2407, in wrapper
    method(*args, **kwargs)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 356, in instantiated_test
    test(self, **param_kwargs)
  File "/gpfs/nvme1/1/casparl/ebbuildpath/PyTorch/2.1.2/foss-2023a-CUDA-12.1.1/pytorch-v2.1.2/test/distributed/fsdp/test_fsdp_core.py", line 248, in test_mixture_of_experts_with_delay_before_free
    self.run_subtests(
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_fsdp.py", line 896, in run_subtests
    return run_subtests(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_fsdp.py", line 848, in run_subtests
    test_fn(*test_args, **test_kwargs, **subtest_kwargs)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_fsdp.py", line 1170, in _test_fsdp_parity
    torch.testing.assert_close(ref_loss, fsdp_loss, check_dtype=False)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_comparison.py", line 1520, in assert_close
    raise error_metas[0].to_error(msg)
AssertionError: Scalars are not close!

Expected -1.8912922143936157 but got -2.122039318084717.
Absolute difference: 0.23074710369110107 (up to 1e-05 allowed)
Relative difference: 0.12200499845291383 (up to 1.3e-06 allowed)

To execute this test, run the following from the base repo dir:
     python test/distributed/fsdp/test_fsdp_core.py -k test_mixture_of_experts_with_delay_before_free_offload_true_no_shard

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

test_jit_legacy
Ok, we know this from #20171 , I'm not too worried here as it might be that kernel memory leak issue I mentioned.

================================================================ FAILURES ================================================================
_______________________________________________ TestScript.test_file_reader_no_memory_leak _______________________________________________
Traceback (most recent call last):
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 57, in testPartExecutor
    yield
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 623, in run
    self._callTestMethod(testMethod)
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 2407, in wrapper
    method(*args, **kwargs)
  File "/gpfs/nvme1/1/casparl/ebbuildpath/PyTorch/2.1.2/foss-2023a-CUDA-12.1.1/pytorch-v2.1.2/test/test_jit.py", line 12862, in test_file_reader_no_memory_leak
    assert peak_from_file < peak_from_string * 500
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
======================================================== short test summary info =========================================================
FAILED [2.4620s] test_jit_legacy.py::TestScript::test_file_reader_no_memory_leak - AssertionError
============================== 1 failed, 2390 passed, 104 skipped, 12 xfailed, 2 rerun in 166.18s (0:02:46) ==============================
FINISHED PRINTING LOG FILE of test_jit_legacy 1/1 (/gpfs/nvme1/1/casparl/ebbuildpath/PyTorch/2.1.2/foss-2023a-CUDA-12.1.1/pytorch-v2.1.2/test/test-reports/test_jit_legacy_mhxi2r5m.log)

test_jit_profiling
Ok, we know this from #20171 , I'm not too worried here as it might be that kernel memory leak issue I mentioned.

================================================================ FAILURES ================================================================
_______________________________________________ TestScript.test_file_reader_no_memory_leak _______________________________________________
Traceback (most recent call last):
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 57, in testPartExecutor
    yield
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 623, in run
    self._callTestMethod(testMethod)
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 2407, in wrapper
    method(*args, **kwargs)
  File "/gpfs/nvme1/1/casparl/ebbuildpath/PyTorch/2.1.2/foss-2023a-CUDA-12.1.1/pytorch-v2.1.2/test/test_jit.py", line 12862, in test_file_reader_no_memory_leak
    assert peak_from_file < peak_from_string * 500
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
======================================================== short test summary info =========================================================
FAILED [2.4602s] test_jit_profiling.py::TestScript::test_file_reader_no_memory_leak - AssertionError

test_nn

================================================================= RERUNS =================================================================
_________________________________________________ TestNN.test_Conv1d_pad_same_cuda_tf32 __________________________________________________
Traceback (most recent call last):
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 57, in testPartExecutor
    yield
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 623, in run
    self._callTestMethod(testMethod)
  File "/scratch-nvme/1/casparl/generic/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 2407, in wrapper
    method(*args, **kwargs)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 2407, in wrapper
    method(*args, **kwargs)
  File "/gpfs/nvme1/1/casparl/ebbuildpath/PyTorch/2.1.2/foss-2023a-CUDA-12.1.1/pytorch-v2.1.2/test/test_nn.py", line 7474, in with_tf32_on
    test.test_cuda(self, **kwargs)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_nn.py", line 4486, in test_cuda
    test_case.assertEqual(cpu_d_i, gpu_d_i, atol=self.precision, rtol=0, exact_dtype=False)
  File "/scratch-nvme/1/casparl/ebtmpdir/eb-ofx1wu0a/tmp00_2xzah/lib/python3.11/site-packages/torch/testing/_internal/common_utils.py", line 3304, in assertEqual
    raise error_metas.pop()[0].to_error(
AssertionError: Tensor-likes are not close!

Mismatched elements: 2 / 60 (3.3%)
Greatest absolute difference: 0.007216689253091602 at index (1, 0, 0) (up to 0.005 allowed)
Greatest relative difference: 0.0016201659975987861 at index (3, 0, 0) (up to 0 allowed)

To execute this test, run the following from the base repo dir:
     python test/test_nn.py -k test_Conv1d_pad_same_cuda_tf32

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

casparvl · 2024-03-24T12:25:30Z

@boegelbot please test @ jsc-zen3
CORE_CNT=16

boegelbot · 2024-03-24T12:30:17Z

@casparvl: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=20156 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_20156 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 3853

Test results coming soon (I hope)...

- notification for comment with ID 2016792987 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-03-24T19:16:13Z

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.3, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/9a5df52bc276b0e7cbeed75b48b94327 for a full test report.

boegelbot · 2024-03-24T21:11:28Z

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
cnx1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/dd225639683307147b137315b189be0d for a full test report.

Flamefire · 2024-03-25T11:42:29Z

@casparvl Can you upload the compressed log of the failures (#20156 (comment)) such that the easyblock can be enhanced to also detect the 2 other failing tests by name?

casparvl · 2024-03-25T14:04:54Z

I'll send it to you in a DM. I don't assume there to be much privacy-sensitive info in there, but just to be safe I'll not share it with the world ;-)

Flamefire · 2024-03-25T14:14:18Z

Thanks.

As for the failures:

test_file_reader_no_memory_leak -> OK
test_proxy_tensor - An import error. Certainly strange but of course they messed up the check such that it isn't clear if it really is z3 failing to import or anything else. Try with sourcing the --dump-env and python -c 'import z3' to see if it is z3 or something else
distributed/fsdp/test_fsdp_core - True those are potential issues and seem to be related. But as most others work I'd dismiss them stll
test_nn (especially test_Conv1d_pad_same_cuda_tf32) has known issues, so can be ignored

casparvl · 2024-03-26T10:31:00Z

I think I know: I probably didn't rebuild Z3 after #20050 , and the old one is a non-python one...
Ok, I think we have an understanding of most of these. I agree that test_fsdp_core is supsicious, but I'd also dismiss it since the others work.

I'll merge this PR: there are sufficient succesful test-reports, and on my system, we have a reasonable understanding of the failing tests.

casparvl

Lgtm

casparvl · 2024-03-26T10:33:33Z

Going in, thanks @Flamefire!

jfgrimm · 2024-03-28T02:51:24Z

Test report by @jfgrimm
SUCCESS
Build succeeded (with --ignore-test-failure) for 1 out of 1 (1 easyconfigs in total)
gpu25.viking2.yor.alces.network - Linux Rocky Linux 8.8, x86_64, AMD EPYC 7413 24-Core Processor, 1 x NVIDIA NVIDIA H100 PCIe, 535.86.10, Python 3.6.8
See https://gist.github.com/jfgrimm/b0a1f84ef02126a1229ff263966a163d for a full test report.

Fix test issues in PyTorch-2.1.2-foss-2023a-CUDA-12.1.1

4c3d938

casparvl reviewed Mar 19, 2024

View reviewed changes

easybuild/easyconfigs/p/PyTorch/PyTorch-2.1.2-foss-2023a-CUDA-12.1.1.eb Outdated Show resolved Hide resolved

Flamefire mentioned this pull request Mar 20, 2024

{ai}[foss/2023a,CUDA/12.3.0] PyTorch 2.1.2 #19824

Open

Remove extension from excluded test

a8dab8e

Flamefire mentioned this pull request Mar 20, 2024

{ai}[foss/2023b] PyTorch v2.1.2 #20171

Merged

Flamefire mentioned this pull request Mar 21, 2024

{ai}[foss/2022b] PyTorch v2.1.2 w/ CUDA 12.0.0 #20155

Open

akesandgren mentioned this pull request Mar 25, 2024

Fix some more tests in PyTorch-2.1.2-foss-2023a-CUDA-12.1.1.eb #20204

Closed

casparvl approved these changes Mar 26, 2024

View reviewed changes

casparvl added the bug fix label Mar 26, 2024

casparvl added this to the release after 4.9.0 milestone Mar 26, 2024

casparvl merged commit ffde1f5 into easybuilders:develop Mar 26, 2024
9 checks passed

Flamefire deleted the 20240319165505_new_pr_PyTorch212 branch March 26, 2024 11:23

lexming mentioned this pull request Mar 27, 2024

PyTorch-2.1.2-foss-2023a-CUDA-12.1.1.eb still fails with too many errors after #20156 #20222

Open

boegel changed the title ~~Fix test issues in PyTorch-2.1.2-foss-2023a-CUDA-12.1.1~~ add patches to fix test issues for PyTorch 2.1.2 with foss/2023a + CUDA 12.1.1 Apr 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add patches to fix test issues for PyTorch 2.1.2 with `foss/2023a` + CUDA 12.1.1 #20156

add patches to fix test issues for PyTorch 2.1.2 with `foss/2023a` + CUDA 12.1.1 #20156

Flamefire commented Mar 19, 2024 •

edited

Loading

casparvl commented Mar 21, 2024

Flamefire commented Mar 22, 2024

jfgrimm commented Mar 23, 2024

casparvl commented Mar 23, 2024

Flamefire commented Mar 23, 2024

casparvl commented Mar 24, 2024

boegelbot commented Mar 24, 2024

casparvl commented Mar 24, 2024

casparvl commented Mar 24, 2024

boegelbot commented Mar 24, 2024

boegelbot commented Mar 24, 2024

boegelbot commented Mar 24, 2024

Flamefire commented Mar 25, 2024

casparvl commented Mar 25, 2024

Flamefire commented Mar 25, 2024

casparvl commented Mar 26, 2024

casparvl left a comment

casparvl commented Mar 26, 2024

jfgrimm commented Mar 28, 2024

add patches to fix test issues for PyTorch 2.1.2 with foss/2023a + CUDA 12.1.1 #20156

add patches to fix test issues for PyTorch 2.1.2 with foss/2023a + CUDA 12.1.1 #20156

Conversation

Flamefire commented Mar 19, 2024 • edited Loading

casparvl commented Mar 21, 2024

Flamefire commented Mar 22, 2024

jfgrimm commented Mar 23, 2024

casparvl commented Mar 23, 2024

Flamefire commented Mar 23, 2024

casparvl commented Mar 24, 2024

boegelbot commented Mar 24, 2024

casparvl commented Mar 24, 2024

casparvl commented Mar 24, 2024

boegelbot commented Mar 24, 2024

boegelbot commented Mar 24, 2024

boegelbot commented Mar 24, 2024

Flamefire commented Mar 25, 2024

casparvl commented Mar 25, 2024

Flamefire commented Mar 25, 2024

casparvl commented Mar 26, 2024

casparvl left a comment

Choose a reason for hiding this comment

casparvl commented Mar 26, 2024

jfgrimm commented Mar 28, 2024

add patches to fix test issues for PyTorch 2.1.2 with `foss/2023a` + CUDA 12.1.1 #20156

add patches to fix test issues for PyTorch 2.1.2 with `foss/2023a` + CUDA 12.1.1 #20156

Flamefire commented Mar 19, 2024 •

edited

Loading