GradcheckError with deepwave v0.0.13 on nixpkgs

[AtilaSaraiva]

Hello!

I'm the maintainer for the deepwave package on the Nixpkgs package repository. While updating it to the v0.0.13 version I noticed a particular test is failing and I'd like to report it here. Here is the full test phase log:

    _gradcheck_real_imag(gradcheck_fn, func, func_out, tupled_inputs, outputs, eps,
/nix/store/13hks8798xqfz1xflajl1v29di04r8z1-python3.9-pytorch-1.11.0/lib/python3.9/site-packages/torch/autograd/gradcheck.py:1061: in _gradcheck_real_imag
    gradcheck_fn(func, func_out, tupled_inputs, outputs, eps,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

func = <function scalarbornprop at 0x7ffea053f3a0>
func_out = (tensor([[[ 2.0908e-02,  2.7553e-01,  1.2889e+00,  1.8004e+00,  1.8285e+00,
           1.8062e+00,  1.7811e+00,  1.759...   6.3811e-06, -9.2151e-07, -1.8668e-07]]], dtype=torch.float64,
       grad_fn=<CppNode<ScalarBornCPUFunction>>), ...)
tupled_inputs = (tensor([[1506.1053, 1522.4555, 1523.4253],
        [1517.7099, 1555.6068, 1510.9444],
        [1546.0913, 1570.8365, ...orch.float64,
       requires_grad=True), tensor([[[0, 2],
         [1, 2]],

        [[2, 2],
         [3, 2]]]), ...)
outputs = (tensor([[[ 2.0908e-02,  2.7553e-01,  1.2889e+00,  1.8004e+00,  1.8285e+00,
           1.8062e+00,  1.7811e+00,  1.759...   6.3811e-06, -9.2151e-07, -1.8668e-07]]], dtype=torch.float64,
       grad_fn=<CppNode<ScalarBornCPUFunction>>), ...)
eps = 1e-06, rtol = 1e-05, atol = 1e-08, check_grad_dtypes = True
nondet_tol = 0.001

    def _slow_gradcheck(func, func_out, tupled_inputs, outputs, eps, rtol, atol, check_grad_dtypes,
                        nondet_tol, *, use_forward_ad=False, complex_indices=None, test_imag=False):
        func_out = _as_tuple(func_out)
        if not outputs:
            return _check_no_differentiable_outputs(func, tupled_inputs, func_out, eps)
    
        numerical = _transpose(_get_numerical_jacobian(func, tupled_inputs, outputs, eps=eps, is_forward_ad=use_forward_ad))
    
        if use_forward_ad:
            analytical_forward = _get_analytical_jacobian_forward_ad(func, tupled_inputs, func_out, check_grad_dtypes=check_grad_dtypes)
    
            for i, n_per_out in enumerate(numerical):
                for j, n in enumerate(n_per_out):
                    a = analytical_forward[j][i]
                    if not _allclose_with_type_promotion(a, n.to(a.device), rtol, atol):
                        raise GradcheckError(_get_notallclose_msg(a, n, i, j, complex_indices, test_imag,
                                                                  is_forward_ad=True))
        else:
            for i, o in enumerate(outputs):
                analytical = _check_analytical_jacobian_attributes(tupled_inputs, o, nondet_tol, check_grad_dtypes)
    
                for j, (a, n) in enumerate(zip(analytical, numerical[i])):
                    if not _allclose_with_type_promotion(a, n.to(a.device), rtol, atol):
>                       raise GradcheckError(_get_notallclose_msg(a, n, i, j, complex_indices, test_imag))
E                       torch.autograd.gradcheck.GradcheckError: Jacobian mismatch for output 6 with respect to input 4,
E                       numerical:tensor([[ 3.0910e-06,  3.2390e-04,  1.0494e-02,  ...,  0.0000e+00,
E                                 0.0000e+00,  0.0000e+00],
E                               [-3.1462e-05, -4.6165e-04, -4.2815e-03,  ...,  0.0000e+00,
E                                 0.0000e+00,  0.0000e+00],
E                               [ 1.4742e-05, -3.6820e-03,  4.0949e-03,  ...,  0.0000e+00,
E                                 0.0000e+00,  0.0000e+00],
E                               ...,
E                               [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  2.1815e-09,
E                                -9.9844e-11, -4.1739e-12],
E                               [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -3.1082e-10,
E                                 1.0950e-11,  3.4516e-13],
E                               [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  2.6232e-11,
E                                -8.3089e-13, -2.1348e-14]], dtype=torch.float64)
E                       analytical:tensor([[ 3.0910e-06,  3.2390e-04,  1.0494e-02,  ..., -0.0000e+00,
                    if not _allclose_with_type_promotion(a, n.to(a.device), rtol, atol):
>                       raise GradcheckError(_get_notallclose_msg(a, n, i, j, complex_indices, test_imag))
E                       torch.autograd.gradcheck.GradcheckError: Jacobian mismatch for output 6 with respect to input 4,
E                       numerical:tensor([[ 3.0910e-06,  3.2390e-04,  1.0494e-02,  ...,  0.0000e+00,
E                                 0.0000e+00,  0.0000e+00],
E                               [-3.1462e-05, -4.6165e-04, -4.2815e-03,  ...,  0.0000e+00,
E                                 0.0000e+00,  0.0000e+00],
E                               [ 1.4742e-05, -3.6820e-03,  4.0949e-03,  ...,  0.0000e+00,
E                                 0.0000e+00,  0.0000e+00],
E                               ...,
E                               [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  2.1815e-09,
E                                -9.9844e-11, -4.1739e-12],
E                               [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -3.1082e-10,
E                                 1.0950e-11,  3.4516e-13],
E                               [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  2.6232e-11,
E                                -8.3089e-13, -2.1348e-14]], dtype=torch.float64)
E                       analytical:tensor([[ 3.0910e-06,  3.2390e-04,  1.0494e-02,  ..., -0.0000e+00,
E                                -0.0000e+00, -0.0000e+00],
E                               [-3.1462e-05, -4.6165e-04, -4.2815e-03,  ..., -0.0000e+00,
E                                -0.0000e+00, -0.0000e+00],
E                               [ 1.4742e-05, -3.6820e-03,  4.0949e-03,  ..., -0.0000e+00,
E                                -0.0000e+00, -0.0000e+00],
E                               ...,
E                               [-0.0000e+00, -0.0000e+00, -0.0000e+00,  ...,  2.1815e-09,
E                                -9.9844e-11, -4.1739e-12],
E                               [-0.0000e+00, -0.0000e+00, -0.0000e+00,  ..., -3.1083e-10,
E                                 1.0950e-11,  3.4519e-13],
E                               [-0.0000e+00, -0.0000e+00, -0.0000e+00,  ...,  2.6227e-11,
E                                -8.3129e-13, -2.1350e-14]], dtype=torch.float64)

/nix/store/13hks8798xqfz1xflajl1v29di04r8z1-python3.9-pytorch-1.11.0/lib/python3.9/site-packages/torch/autograd/gradcheck.py:1114: GradcheckError
=========================== short test summary info ============================
FAILED tests/test_scalar_born.py::test_born_gradcheck_2d_different_pml - torc...
================== 1 failed, 74 passed in 2372.49s (0:39:32) ===================

[AtilaSaraiva]

Seems like the values of the tensors are quite similar, but the test fails nevertheless. I prefer waiting for the next release than updating to the latest one and running the risk of breaking the users projects. Either way, thank you for your amazing project, it surprised me when I discovered.

[ar4]

Oh wow, I didn't know that you were doing that. Thank you! Thank you also for your care in making sure that the tests pass and for reporting this problem. All of the tests pass on my computer and on GitHub Actions, but I must have set the tolerances a bit too tight for it to pass everywhere. I plan to have another release within about two weeks, so I will adjust the tolerances on that test before then. Would it be helpful if I committed the new version as a branch on GitHub and notified you so that you could check it before I release it?

[AtilaSaraiva]

Oh no problem! I love to support geophysics related project by packaging then, I've done it to devito, pyrevolve and segyio too.
oh about the branch pre-release, that would help a lot! I don't know much about writing github actions workflows, but it is also possible to create one that does this test with nix. I will talk to my nix friends so I can figure this out.

[ar4]

I have now released v0.0.14. The Nixpkgs tests pass, so hopefully there won't be any problems this time.

[AtilaSaraiva]

Ohh thanks, I will bump it right away!

[AtilaSaraiva]

Merged