CI: Merge scipy build of specfun, minpack, fitpack, amos and mach with no patches! #2673
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Make sure everything works on local, and then push. I can see a test segfaults. |
|
Yes, I executed it locally, all the tests passed for me. I'll check once again. |
HarshitaKalani
force-pushed
the
ci_amos_mach
branch
from
ca5f34c
to
cb749fa
Compare
(scipy-dev) harshita@harshita-linux:~/scipy$ python dev.py build
💻 meson setup /home/harshita/scipy/build --prefix
/home/harshita/scipy/build-install
The Meson build system
Version: 1.2.1
Source dir: /home/harshita/scipy
Build dir: /home/harshita/scipy/build
Build type: native build
Project name: SciPy
Project version: 1.12.0.dev0
C compiler for the host machine: /home/harshita/miniconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc (gcc 12.3.0 "x86_64-conda-linux-gnu-cc (conda-forge gcc 12.3.0-0) 12.3.0")
C linker for the host machine: /home/harshita/miniconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc ld.bfd 2.40
C++ compiler for the host machine: /home/harshita/miniconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++ (gcc 12.3.0 "x86_64-conda-linux-gnu-c++ (conda-forge gcc 12.3.0-0) 12.3.0")
C++ linker for the host machine: /home/harshita/miniconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++ ld.bfd 2.40
Cython compiler for the host machine: cython (cython 3.0.2)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python3 found: YES (/home/harshita/miniconda3/envs/scipy-dev/bin/python3.11)
Found pkg-config: /home/harshita/miniconda3/envs/scipy-dev/bin/pkg-config (0.29.2)
Run-time dependency python found: YES 3.11
Program cython found: YES (/home/harshita/miniconda3/envs/scipy-dev/bin/cython)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES
Compiler for C supports arguments -Wno-unused-function: YES
Compiler for C supports arguments -Wno-conversion: YES
Compiler for C supports arguments -Wno-misleading-indentation: YES
Library m found: YES
meson.build:82:0: ERROR: Unknown compiler(s): [['/home/harshita/Desktop/lfortran/src/bin/lfortran', 'cmake']]
A full log can be found at /home/harshita/scipy/build/meson-logs/meson-log.txt
Meson build setup failed!I did |
|
One suggestion at certik/scipy@e8c93e5, please avoid creating new CMakeLists.txt file, this makes finding difference very hard ( had to be done manually ) |
|
I pushed changes at https://github.com/certik/scipy/commits/merge_specfun_minpack_fitpack_amos_mach_03, I feel this has clear diff and continuation of work. The tests segfaults for me, we need to create MRE and see what is causing trouble. |
|
There are a lot more tests that segfaults, here is a small diff of the same: index d990a7a4e..f1ca7dcc5 100644
--- a/scipy/special/tests/test_basic.py
+++ b/scipy/special/tests/test_basic.py
@@ -54,8 +54,8 @@ class TestCephes:
def test_airy(self):
cephes.airy(0)
- def test_airye(self):
- cephes.airye(0)
+ # def test_airye(self):
+ # cephes.airye(0)
def test_binom(self):
n = np.array([0.264, 4, 5.2, 17])
@@ -421,17 +421,17 @@ class TestCephes:
def test_gdtrix(self):
cephes.gdtrix(1,1,.1)
- def test_hankel1(self):
- cephes.hankel1(1,1)
+ # def test_hankel1(self):
+ # cephes.hankel1(1,1)
- def test_hankel1e(self):
- cephes.hankel1e(1,1)
+ # def test_hankel1e(self):
+ # cephes.hankel1e(1,1)
- def test_hankel2(self):
- cephes.hankel2(1,1)
+ # def test_hankel2(self):
+ # cephes.hankel2(1,1)
- def test_hankel2e(self):
- cephes.hankel2e(1,1)
+ # def test_hankel2e(self):
+ # cephes.hankel2e(1,1)
def test_hyp1f1(self):
assert_approx_equal(cephes.hyp1f1(1,1,1), exp(1.0))
@@ -480,8 +480,8 @@ class TestCephes:
def test_iv(self):
assert_equal(cephes.iv(1,0),0.0)
- def test_ive(self):
- assert_equal(cephes.ive(1,0),0.0)
+ # def test_ive(self):
+ # assert_equal(cephes.ive(1,0),0.0)
def test_j0(self):
assert_equal(cephes.j0(0),1.0)
@@ -489,14 +489,14 @@ class TestCephes:
def test_j1(self):
assert_equal(cephes.j1(0),0.0)
- def test_jn(self):
- assert_equal(cephes.jn(0,0),1.0)
+ # def test_jn(self):
+ # assert_equal(cephes.jn(0,0),1.0)
- def test_jv(self):
- assert_equal(cephes.jv(0,0),1.0)
+ # def test_jv(self):
+ # assert_equal(cephes.jv(0,0),1.0)
- def test_jve(self):
- assert_equal(cephes.jve(0,0),1.0)
+ # def test_jve(self):
+ # assert_equal(cephes.jve(0,0),1.0)
def test_k0(self):
cephes.k0(2)
@@ -525,8 +525,8 @@ class TestCephes:
def test_kelvin(self):
cephes.kelvin(2)
- def test_kn(self):
- cephes.kn(1,1)
+ # def test_kn(self):
+ # cephes.kn(1,1)
def test_kolmogi(self):
assert_equal(cephes.kolmogi(1),0.0)
@@ -545,11 +545,11 @@ class TestCephes:
assert_equal(cephes._kolmogci(0), 0.0)
assert_(np.isnan(cephes._kolmogci(np.nan)))
- def test_kv(self):
- cephes.kv(1,1)
+ # def test_kv(self):
+ # cephes.kv(1,1)
- def test_kve(self):
- cephes.kve(1,1)
+ # def test_kve(self):
+ # cephes.kve(1,1)
def test_log1p(self):
log1p = cephes.log1p
@@ -930,11 +930,11 @@ class TestCephes:
def test_yn(self):
cephes.yn(1,1)
- def test_yv(self):
- cephes.yv(1,1)
+ # def test_yv(self):
+ # cephes.yv(1,1)
- def test_yve(self):
- cephes.yve(1,1)
+ # def test_yve(self):
+ # cephes.yve(1,1)
def test_wofz(self):
z = [complex(624.2,-0.26123), complex(-0.4,3.), complex(0.6,2.),
@@ -992,15 +992,15 @@ class TestAiry:
x = special.airy(-.36)@HarshitaKalani go ahead, pick one test, create MRE, and then try to fix it. |
|
I also noticed that mach has some patches in our scipy branch that does not actually use the new equivalence feature. So we just need to figure out MRE for each bug and fix it. |
|
When I'm running the tests for Amos with https://github.com/certik/scipy/commits/merge_specfun_minpack_fitpack_amos_mach_03 And when I remove the use of Edit: these are resolved now. |
|
When I run the tests for scipy.optimize, I see some of the tests fail there too. |
double precision function d1mach(i)
use iso_c_binding, only: c_loc, c_f_pointer
integer i
! integer small(2)
! integer large(2)
! integer right(2)
! integer diver(2)
! integer log10(2)
integer sc, cray1(38), j
common /d9mach/ cray1
! save small, large, right, diver, log10,
save sc
! double precision dmach(5)
! equivalence (dmach(1),small(1))
! equivalence (dmach(2),large(1))
! equivalence (dmach(3),right(1))
! equivalence (dmach(4),diver(1))
! equivalence (dmach(5),log10(1))
! implicit none
DOUBLE PRECISION, target :: DMACH(5)
INTEGER*4, pointer :: SMALL(:)
INTEGER*4, pointer :: LARGE(:)
INTEGER*4, pointer :: RIGHT(:)
INTEGER*4, pointer :: DIVER(:)
INTEGER*4, pointer :: LOG10(:)
call c_f_pointer(c_loc(dmach(1)), small, [2])
call c_f_pointer(c_loc(dmach(2)), large, [2])
call c_f_pointer(c_loc(dmach(3)), right, [2])
call c_f_pointer(c_loc(dmach(4)), diver, [2])
call c_f_pointer(c_loc(dmach(5)), log10, [2])
DATA SC/0/
if (sc .ne. 987) then
dmach(1) = 1.d13
if ( small(1) .eq. 1117925532 .and. small(2) .eq. -448790528) then
small(1) = 1048576
small(2) = 0
large(1) = 2146435071
large(2) = -1
right(1) = 1017118720
right(2) = 0
diver(1) = 1018167296
diver(2) = 0
log10(1) = 1070810131
log10(2) = 1352628735
else if ( small(2) .eq. 1117925532 .and. small(1) .eq. -448790528) then
small(2) = 1048576
small(1) = 0
large(2) = 2146435071
large(1) = -1
right(2) = 1017118720
right(1) = 0
diver(2) = 1018167296
diver(1) = 0
log10(2) = 1070810131
log10(1) = 1352628735
else if ( small(1) .eq. -2065213935 .and. small(2) .eq. 10752) then
small(1) = 128
small(2) = 0
large(1) = -32769
large(2) = -1
right(1) = 9344
right(2) = 0
diver(1) = 9472
diver(2) = 0
log10(1) = 546979738
log10(2) = -805796613
else if ( small(1) .eq. 1267827943 .and. small(2) .eq. 704643072) then
small(1) = 1048576
small(2) = 0
large(1) = 2147483647
large(2) = -1
right(1) = 856686592
right(2) = 0
diver(1) = 873463808
diver(2) = 0
log10(1) = 1091781651
log10(2) = 1352628735
else if ( small(1) .eq. 1120022684 .and. small(2) .eq. -448790528) then
small(1) = 1048576
small(2) = 0
large(1) = 2147483647
large(2) = -1
right(1) = 1019215872
right(2) = 0
diver(1) = 1020264448
diver(2) = 0
log10(1) = 1072907283
log10(2) = 1352628735
else if ( small(1) .eq. 815547074 .and. small(2) .eq. 58688) then
small(1) = 16
small(2) = 0
large(1) = -32769
large(2) = -1
right(1) = 15552
right(2) = 0
diver(1) = 15568
diver(2) = 0
log10(1) = 1142112243
log10(2) = 2046775455
else
dmach(2) = 1.d27 + 1
dmach(3) = 1.d27
large(2) = large(2) - right(2)
if (large(2) .eq. 64 .and. small(2) .eq. 0) then
! Ideally lfortran should come here
cray1(1) = 67291416
do 10 j = 1, 20
cray1(j+1) = cray1(j) + cray1(j)
10 continue
cray1(22) = cray1(21) + 321322
do 20 j = 22, 37
cray1(j+1) = cray1(j) + cray1(j)
20 continue
if (cray1(38) .eq. small(1)) then
call i1mcry(small(1), j, 8285, 8388608, 0)
small(2) = 0
call i1mcry(large(1), j, 24574, 16777215, 16777215)
call i1mcry(large(2), j, 0, 16777215, 16777214)
call i1mcry(right(1), j, 16291, 8388608, 0)
right(2) = 0
call i1mcry(diver(1), j, 16292, 8388608, 0)
diver(2) = 0
call i1mcry(log10(1), j, 16383, 10100890, 8715215)
call i1mcry(log10(2), j, 0, 16226447, 9001388)
else
write(*,9000)
stop 779
end if
else
write(*,9000)
! Problem is here, lfortran comes here and stops
stop 779
end if
end if
sc = 987
end if
if (dmach(4) .ge. 1.0d0) stop 778
if (i .lt. 1 .or. i .gt. 5) then
write(*,*) 'd1mach(i): i =',i,' is out of bounds.'
stop
end if
d1mach = dmach(i)
return
9000 FORMAT(/' Adjust D1MACH by uncommenting data statements appropriate for your machine.')
end
subroutine i1mcry(a, a1, b, c, d)
integer a, a1, b, c, d
a1 = 16777216*b + c
a = 16777216*a1 + d
end
program main
interface
double precision function d1mach(i)
integer i
end function
end interface
print *, d1mach(4)
end program$ gfortran a.f90 && ./a.out
2.2204460492503131E-016
$ lfortran a.f90 --no-warnings
2.22044604925031308e-16There is a slight difference in the output for this. And for the following, we get the following errors double precision function d1mach(i)
integer i
integer small(2)
integer large(2)
integer right(2)
integer diver(2)
integer log10(2)
integer sc, cray1(38), j
common /d9mach/ cray1
save small, large, right, diver, log10,sc
double precision dmach(5)
equivalence (dmach(1),small(1))
equivalence (dmach(2),large(1))
equivalence (dmach(3),right(1))
equivalence (dmach(4),diver(1))
equivalence (dmach(5),log10(1))
DATA SC/0/
if (sc .ne. 987) then
dmach(1) = 1.d13
if ( small(1) .eq. 1117925532 .and. small(2) .eq. -448790528) then
small(1) = 1048576
small(2) = 0
large(1) = 2146435071
large(2) = -1
right(1) = 1017118720
right(2) = 0
diver(1) = 1018167296
diver(2) = 0
log10(1) = 1070810131
log10(2) = 1352628735
else if ( small(2) .eq. 1117925532 .and. small(1) .eq. -448790528) then
small(2) = 1048576
small(1) = 0
large(2) = 2146435071
large(1) = -1
right(2) = 1017118720
right(1) = 0
diver(2) = 1018167296
diver(1) = 0
log10(2) = 1070810131
log10(1) = 1352628735
else if ( small(1) .eq. -2065213935 .and. small(2) .eq. 10752) then
small(1) = 128
small(2) = 0
large(1) = -32769
large(2) = -1
right(1) = 9344
right(2) = 0
diver(1) = 9472
diver(2) = 0
log10(1) = 546979738
log10(2) = -805796613
else if ( small(1) .eq. 1267827943 .and. small(2) .eq. 704643072) then
small(1) = 1048576
small(2) = 0
large(1) = 2147483647
large(2) = -1
right(1) = 856686592
right(2) = 0
diver(1) = 873463808
diver(2) = 0
log10(1) = 1091781651
log10(2) = 1352628735
else if ( small(1) .eq. 1120022684 .and. small(2) .eq. -448790528) then
small(1) = 1048576
small(2) = 0
large(1) = 2147483647
large(2) = -1
right(1) = 1019215872
right(2) = 0
diver(1) = 1020264448
diver(2) = 0
log10(1) = 1072907283
log10(2) = 1352628735
else if ( small(1) .eq. 815547074 .and. small(2) .eq. 58688) then
small(1) = 16
small(2) = 0
large(1) = -32769
large(2) = -1
right(1) = 15552
right(2) = 0
diver(1) = 15568
diver(2) = 0
log10(1) = 1142112243
log10(2) = 2046775455
else
dmach(2) = 1.d27 + 1
dmach(3) = 1.d27
large(2) = large(2) - right(2)
if (large(2) .eq. 64 .and. small(2) .eq. 0) then
! Ideally lfortran should come here
cray1(1) = 67291416
do 10 j = 1, 20
cray1(j+1) = cray1(j) + cray1(j)
10 continue
cray1(22) = cray1(21) + 321322
do 20 j = 22, 37
cray1(j+1) = cray1(j) + cray1(j)
20 continue
if (cray1(38) .eq. small(1)) then
call i1mcry(small(1), j, 8285, 8388608, 0)
small(2) = 0
call i1mcry(large(1), j, 24574, 16777215, 16777215)
call i1mcry(large(2), j, 0, 16777215, 16777214)
call i1mcry(right(1), j, 16291, 8388608, 0)
right(2) = 0
call i1mcry(diver(1), j, 16292, 8388608, 0)
diver(2) = 0
call i1mcry(log10(1), j, 16383, 10100890, 8715215)
call i1mcry(log10(2), j, 0, 16226447, 9001388)
else
write(*,9000)
stop 779
end if
else
write(*,9000)
! Problem is here, lfortran comes here and stops
stop 779
end if
end if
sc = 987
end if
if (dmach(4) .ge. 1.0d0) stop 778
if (i .lt. 1 .or. i .gt. 5) then
write(*,*) 'd1mach(i): i =',i,' is out of bounds.'
stop
end if
d1mach = dmach(i)
return
9000 FORMAT(/' Adjust D1MACH by uncommenting data statements appropriate for your machine.')
end
subroutine i1mcry(a, a1, b, c, d)
integer a, a1, b, c, d
a1 = 16777216*b + c
a = 16777216*a1 + d
end
program main
interface
double precision function d1mach(i)
integer i
end function
end interface
print *, d1mach(4)
end program$ gfortran b.f90 && ./a.out
2.2204460492503131E-016
$ lfortran b.f90 --no-warnings
ASR verify pass error: ASR verify: The symbol table was not found in the scope of `symtab`.
--> b.f90:185:18
|
185 | double precision dmach(5)
| ^^^^^^^^ failed here
ASR verify pass error: ASR verify: Var::m_v `dmach` cannot point outside of its symbol table
--> b.f90:186:14
|
186 | equivalence (dmach(1),small(1))
| ^^^^^^^^ failed here
Note: Please report unclear or confusing messages as bugs at
https://github.com/lfortran/lfortran/issues.Edit: I checked with the workarounds again and realised that the tests seg fault because of this issue. |
|
The following is the MRE double precision function d1mach(i)
integer i
integer small(2)
integer large(2)
integer right(2)
integer diver(2)
integer log10(2)
double precision dmach(5)
equivalence (dmach(1),small(1))
equivalence (dmach(2),large(1))
equivalence (dmach(3),right(1))
equivalence (dmach(4),diver(1))
equivalence (dmach(5),log10(1))
end
program main
interface
double precision function d1mach(i)
integer i
end function
end interface
print *, d1mach(4)
end program
$ gfortran x.f90 && ./a.out
4.6681084772624576E-310
$ lfortran x.f90 --no-warnings
ASR verify pass error: ASR verify: The symbol table was not found in the scope of `symtab`.
--> x.f90:8:18
|
8 | double precision dmach(5)
| ^^^^^^^^ failed here
ASR verify pass error: ASR verify: Var::m_v `dmach` cannot point outside of its symbol table
--> x.f90:9:14
|
9 | equivalence (dmach(1),small(1))
| ^^^^^^^^ failed here
Note: Please report unclear or confusing messages as bugs at
https://github.com/lfortran/lfortran/issues. |
|
Great job! It is getting tricky day by day to create MRE for SciPy. To see what is happening just |
|
This is what I get in |
|
This PR will be good to go once we have #2716 merged. |
|
@Pranavchiku can you check that the patches are good? This is what I can see: |
|
These are the final patches that I see: $ git log
commit 5cc5edb03d0ae5b1a54639cece63d6263a111f5f (HEAD -> merge_specfun_minpack_fitpack_amos_mach_02, ondrej/merge_specfun_minpack_fitpack_amos_mach_02)
Author: HarshitaKalani <kalani.1@iitj.ac.in>
Date: Tue Oct 24 14:27:28 2023 +0530
CMake: Add build for specfun, minpack, fitpack, amos, mach
commit 63b55b62dd71671b0ade7409c7b5b9e2712af4fc
Author: HarshitaKalani <kalani.1@iitj.ac.in>
Date: Tue Oct 24 14:26:59 2023 +0530
Amos and Mach: use our own library
commit 58f4e763c9c59741eac707a2101afc04570e48f7
Author: HarshitaKalani <kalani.1@iitj.ac.in>
Date: Mon Sep 11 18:23:18 2023 +0530
Fitpack: use our own library
commit 6e8022db6a479dceba043d3f8ce276096341ae3f
Author: Pranavchiku <goswami.4@iitj.ac.in>
Date: Sat Sep 2 22:24:03 2023 +0530
Minpack: use our own library
commit 31c2d62641434da9e2a70a9ae54a8f7fcfea0477
Author: Pranavchiku <goswami.4@iitj.ac.in>
Date: Sat Sep 9 20:45:01 2023 +0530
Specfun: use our own library
commit 6e31999696f4f78570b3cf42733fdef87b86e6f9
Author: Pranavchiku <goswami.4@iitj.ac.in>
Date: Fri Sep 1 16:16:37 2023 +0530
use exact versions in environment.yml
commit c66a6462ddfa9e9bdb15330ffbf4181f4229072c
Merge: 9d12e2c55 bdde09c5e
Author: Tyler Reddy <tyler.je.reddy@gmail.com>
Date: Sat Sep 23 16:56:46 2023 -0600
Merge pull request #19285 from rgommers/fix-depr-non-prototype-2 |
|
@HarshitaKalani the commits that I posted above correspond to the change in this PR: - git checkout -t ondrej/merge_specfun_minpack_fitpack_amos_mach
- git checkout ef9dea634b8d1f85299844d60fd975001b4e0f7e
+ git checkout -t ondrej/merge_specfun_minpack_fitpack_amos_mach_02
+ git checkout e8c93e595a97351eb2527971c1bf1b0824ffe03eThe top-most commit is e8c93e595. The top-most commit in your comment is So that's why I was asking if we can clarify what commits we are supposed to test here. |
HarshitaKalani
force-pushed
the
ci_amos_mach
branch
from
26e4cb5
to
69b5595
Compare
HarshitaKalani
force-pushed
the
ci_amos_mach
branch
from
69b5595
to
1c1022e
Compare
|
We are now testing the following commits: I reviewed them by hand, this seems good (no Fortran source code modifications, only build system). So I set this to merge. |
|
This is not working for me (Macbook air m2) [ 97%] Linking C shared library amos.so
ld: warning: -pie being ignored. It is only used when linking a main executable
Undefined symbols for architecture arm64:
"_d1mach_", referenced from:
_dgamln_ in dgamln.f.o
_zacai_ in zacai.f.o
_zacon_ in zacon.f.o
_zairy_ in zairy.f.o
_zasyi_ in zasyi.f.o
_zbesh_ in zbesh.f.o
_zbesi_ in zbesi.f.o
...
"_i1mach_", referenced from:
_dgamln_ in dgamln.f.o
_zairy_ in zairy.f.o
_zbesh_ in zbesh.f.o
_zbesi_ in zbesi.f.o
_zbesj_ in zbesj.f.o
_zbesk_ in zbesk.f.o
_zbesy_ in zbesy.f.o
...
ld: symbol(s) not found for architecture arm64
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [amos.so] Error 1
make[1]: *** [CMakeFiles/amos.dir/all] Error 2
make: *** [all] Error 2 |
|
I am planning to use https://github.com/certik/scipy/commits/merge_specfun_minpack_fitpack_amos_mach_03 in a new PR, this works for me. CI will test if it works on Linux or not. I feel confident that it should. |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference: https://github.com/certik/scipy/commits/merge_specfun_minpack_fitpack_amos_mach_02