Compile and run test_corr::test_sp in stdlib/test/stats

[czgdp1807]

Work in Progress. Will continue on Monday and Tuesday next week.

[Pranavchiku]

Please push https://github.com/czgdp1807/stdlib/tree/lf-stats-test forward, I had made progress here.

[Pranavchiku]

I have made progress on this, will push on branch soon.

[czgdp1807]

@Pranavchiku I am already working on it. The problems with putting all the submodules into the stdlib_stats module makes things unmanageable. I would suggest to parallelise the workflow. This deals with stdlib_stats_corr. You can take stdlib_stats_var and others. I will complete this on Thursday and Friday.

Also the report is already submitted. So I would recommend do things right instead of creating an unmanageable mess. Once we run stdlib_stats after compiling all the submodules into it, the module will not be easily debugged.

[czgdp1807]

subroutine test_sp(x, x2)
        real(sp), intent(in) :: x(:)
        real(sp), intent(in) :: x2(:, :)
        real(sp), allocatable :: y1(:, :), y2(:, :), y3(:, :), y4(:, :)
        allocate(y1(size(x2, 2), size(x2, 2)), &
            y2(size(x2, 1), size(x2, 1)), &
            y3(size(x2, 2), size(x2, 2)), &
            y4(size(x2, 1), size(x2, 1)))
        integer :: yshape(2)

        call check( abs(corr(x, 1) - 1._sp) < sptol&
            , 'sp check 1')
        call check( ieee_is_nan(corr(x, 1, .false.))&
            , 'sp check 2')
        call check( ieee_is_nan(corr(x, 1, x == 1.)), 'sp check 3')
        call check( abs(corr(x, 1, x < 5) - 1._sp) < sptol, 'sp check 4')
        call check( ieee_is_nan(corr(x, 1, x < -999)), 'sp check 5')

        call check( any(ieee_is_nan(corr(x2, 1, mask = .false.)))&
            , 'sp check 6')
        call check( any(ieee_is_nan(corr(x2, 2, mask = .false.)))&
            , 'sp check 7')

        yshape(1) = size(x2, 2)
        yshape(2) = size(x2, 2)
        y1 = reshape([&
            1._sp, 0.9994439103600_sp, -0.870544389237152_sp, 0.99944391036_sp,&
            1._sp, -0.86261576629742_sp, -0.87054438923715_sp,  -0.862615766297428_sp,&
            1._sp ], yshape)
        call check( all(abs(corr(x2, 1) - y1) < sptol)&
                , 'sp check 8')

        yshape(1) = size(x2, 1)
        yshape(2) = size(x2, 1)
        y2 = reshape([&
             1._sp, 0.998742137866914_sp,  0.999846989517886_sp, -0.998337488459582_sp,&
             0.998742137866914_sp, 1._sp, 0.999466429486246_sp, -0.99419162560192020_sp,&
             0.999846989517886_sp, 0.999466429486246_sp, 1._sp, -0.99717646495273815_sp,&
             -0.998337488459582_sp, -0.994191625601920_sp, -0.997176464952738_sp, 1._sp]&
            ,yshape)
        call check( all( abs( corr(x2, 2) - y2) < sptol)&
            , 'sp check 9')

        call check( any(ieee_is_nan(corr(x2, 1, mask = x2 < 10)))&
            , 'sp check 10')
        yshape(1) = size(x2, 2)
        yshape(2) = size(x2, 2)
        y3 = reshape([&
              1._sp, 0.981980506061965_sp, -1._sp&
              ,0.981980506061965_sp, 1._sp, -0.862615766297428_sp&
              ,-1._sp, -0.862615766297428_sp, 1._sp]&
            ,yshape)
        call check( all( abs( corr(x2, 1, mask = x2 < 22) - y3) < sptol)&
            , 'sp check 11')
        
        ! fails below. will make it work tomorrow.
        yshape(1) = size(x2, 1)
        yshape(2) = size(x2, 1)
        y4 = reshape([&
            1._sp, 0.998742137866914_sp, 0.999846989517886_sp, -1._sp&
            ,0.998742137866914_sp, 1._sp, 0.999466429486246_sp, -1._sp&
            ,0.999846989517886_sp, 0.999466429486246_sp, 1._sp, -1._sp&
            ,-1._sp, -1._sp, -1._sp, 1._sp]&
            ,yshape)
        print *, "corr: ", corr(x2, 2, mask = x2 < 22), x2, x2 < 22
        call check( all( abs( corr(x2, 2, mask = x2 < 22) - y4) < sptol)&
            , 'sp check 12')


        ! call check( all(abs(corr(x2, 1, mask = x2 < 1000) - corr(x2, 1))&
        !     < sptol)&
        !     , 'sp check 13')

        ! call check( all(abs(corr(x2, 2, mask = x2 < 1000) - corr(x2, 2))&
        !     < sptol)&
        !     , 'sp check 14')

    end subroutine test_sp

Only the last test is remaining in this. Once it works then all other test types are similarly defined. So everything should be working.

[czgdp1807]

With a small work around, test_rsp fully works. Now I will enable the tests for all the types in test_corr and complete this PR tomorrow.

[czgdp1807]

So I progressed as much as I could. test_rsp, test_dp, test_iint64 (partially) work. While test_csp, test_cdp don't in test_corr. I have commented those out in stdlib.

I will fix the failing integration_tests (commented out in integration_tests/CMakeLists.txt) on Monday and then hopefully merge this PR. Once this is in, and we have time at our hand, I will re-write the array_op pass and also do some re-arrangements in pass_manager.h to reduce the issues related to allocatables.

[czgdp1807]

I have reproduced the segfault on my linux machine. Working on fixing it.

[czgdp1807]

So everything passes here. Now I will merge this and wrap it up. I will investigate the segfault a bit deeper tomorrow, just for knowing the cause. Since we have time at our hands now, so its a good opportunity to fix up array_op pass and also intrinsic_function pass (like reducing the auxiliary variable creation, reducing the number of allocatables, a cleaner management of allocatables inside these passes). Next steps would be as follows,

1. Sync libasr across LPython, LFortran and LC (I will do it on 25 March (Monday next week)).
2. Fix up array_op pass and also other passes - a major problem (common across many bugs is auxiliary variable creation and allocatables, so these passes should handle these two very cleanly).
3. Continue stdlib/stats compilation and anything else left for array programming modules in stdlib
4. Then start reverting workarounds one by one and add integration tests for each of those workarounds in LFortran.

[certik]

I think that's fine to merge, thanks for the fixes.