f2py error when using assumed shape arrays and parametrized kinds (migrated from Trac #1835)

[thouis]

Original ticket http://projects.scipy.org/numpy/ticket/1835
Reported 2011-05-18 by trac user lorenz, assigned to atmention:pearu.

I have a small problem when using both the new assumed shape support (see ticket #1532) and kind statements via a parameter (see ticket #1750). Sorry it didn't spot this earlier!

Consider the following two files:

{{{precision.f90}}}
module precision
integer, parameter :: rk = 8
end module

{{{test.f90}}}
module test

contains

subroutine testfunc(x, res)
  use precision

  implicit none

  real(kind=rk), intent(in) :: x(:)
  real(kind=rk), intent(out) :: res

  integer :: i

  print *, "x =", x

  res = 0.0
  do i = 1, size(x)
    res = res + x(i)
  enddo
end subroutine

end module

The compilation fails with:
#> gfortran -c precision.f90
#> f2py --build-dir build -c -m test test.f90 precision.o --fcompiler=gnu95
Reading .f2py_f2cmap ...
Mapping "real(kind=rk)" to "double"
Mapping "integer(kind=ik)" to "int"
Succesfully applied user defined changes from .f2py_f2cmap
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "test" sources
f2py options: []
adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources.
adding 'build/src.linux-x86_64-2.7' to include_dirs.
adding 'build/src.linux-x86_64-2.7/test-f2pywrappers2.f90' to sources.
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize Gnu95FCompiler
Found executable /afs/mpa/atmention:sys/system/MPA-6.00/usr/bin/gfortran
customize Gnu95FCompiler using build_ext
building 'test' extension
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

compile options: '-Ibuild/src.linux-x86_64-2.7 -I/usr/common/pdsoft/appl/python-2.7.1-amd64/lib/python2.7/site-packages/numpy/core/include -I/usr/common/pdsoft/appl/python-2.7.1-amd64/include/python2.7 -c'
gcc: build/src.linux-x86_64-2.7/fortranobject.c
gcc: build/src.linux-x86_64-2.7/testmodule.c
compiling Fortran 90 module sources
Fortran f77 compiler: /afs/mpa/atmention:sys/system/MPA-6.00/usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /afs/mpa/atmention:sys/system/MPA-6.00/usr/bin/gfortran -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /afs/mpa/atmention:sys/system/MPA-6.00/usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-Ibuild/src.linux-x86_64-2.7 -I/usr/common/pdsoft/appl/python-2.7.1-amd64/lib/python2.7/site-packages/numpy/core/include -I/usr/common/pdsoft/appl/python-2.7.1-amd64/include/python2.7 -c'
extra options: '-Jbuild/ -Ibuild/'
gfortran:f90: test.f90
gfortran:f90: precision.f90
compiling Fortran sources
Fortran f77 compiler: /afs/mpa/atmention:sys/system/MPA-6.00/usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /afs/mpa/atmention:sys/system/MPA-6.00/usr/bin/gfortran -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /afs/mpa/atmention:sys/system/MPA-6.00/usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-Ibuild/src.linux-x86_64-2.7 -I/usr/common/pdsoft/appl/python-2.7.1-amd64/lib/python2.7/site-packages/numpy/core/include -I/usr/common/pdsoft/appl/python-2.7.1-amd64/include/python2.7 -c'
extra options: '-Jbuild/ -Ibuild/'
gfortran:f90: build/src.linux-x86_64-2.7/test-f2pywrappers2.f90
build/src.linux-x86_64-2.7/test-f2pywrappers2.f90:17.16:

      real(kind=rk) res
                1
Error: Parameter 'rk' at (1) has not been declared or is a variable, which does not reduce to a constant expression
build/src.linux-x86_64-2.7/test-f2pywrappers2.f90:19.16:

      real(kind=rk) x(f2py_x_d0)
                1
Error: Parameter 'rk' at (1) has not been declared or is a variable, which does not reduce to a constant expression
build/src.linux-x86_64-2.7/test-f2pywrappers2.f90:17.16:

      real(kind=rk) res
                1
Error: Parameter 'rk' at (1) has not been declared or is a variable, which does not reduce to a constant expression
build/src.linux-x86_64-2.7/test-f2pywrappers2.f90:19.16:

      real(kind=rk) x(f2py_x_d0)
                1
Error: Parameter 'rk' at (1) has not been declared or is a variable, which does not reduce to a constant expression

The cause is the omission of an "use precision" statement in the interface declaration for {{{f2pywrap_test_testfunc}}} in the generated routine {{{f2pyinittest()}}} in {{{test-f2pywrappers2.f90}}}:
! -- f90 --
! This file is autogenerated with f2py (version:2)
! It contains Fortran 90 wrappers to fortran functions.

      subroutine f2pywrap_test_testfunc (x, res, f2py_x_d0)
      use test, only : testfunc
                use precision
      real(kind=rk) res
      integer f2py_x_d0
      real(kind=rk) x(f2py_x_d0)
      call testfunc(x, res)
      end subroutine f2pywrap_test_testfunc

      subroutine f2pyinittest(f2pysetupfunc)
      interface 
      subroutine f2pywrap_test_testfunc (x, res, f2py_x_d0)
      real(kind=rk) res
      integer f2py_x_d0
      real(kind=rk) x(f2py_x_d0)
      end subroutine f2pywrap_test_testfunc
      end interface
      external f2pysetupfunc
      call f2pysetupfunc(f2pywrap_test_testfunc)
      end subroutine f2pyinittest

I.e. a patch to it would look like:
--- build/src.linux-x86_64-2.7/test-f2pywrappers2.f90 2011-05-18 15:33:47.000000000 +0200
+++ test-f2pywrappers2.f90 2011-05-18 15:33:43.000000000 +0200
atmention:atmention: -14,6 +14,7 atmention:atmention:
subroutine f2pyinittest(f2pysetupfunc)
interface
subroutine f2pywrap_test_testfunc (x, res, f2py_x_d0)
+ use precision
real(kind=rk) res
integer f2py_x_d0
real(kind=rk) x(f2py_x_d0)

I would suggest that all modules {{{use}}d by the original function should - to be on the safe side - also be stated in any interface declarations.

Note that this error is only present when using both assumed shape support and parametrized {{{kind}}} statements. If {{{test.f90}}} is changed to
...
subroutine testfunc(x, n, res)
use precision

  implicit none

  integer, intent(in) :: n
  real(kind=rk), intent(in) :: x(n)
...

everything works as expected.

Regards,
Lorenz

[thouis]

Comment in Trac by atmention:pearu, 2011-05-18

Note that if you would use (the order of .f90 files is important):
f2py --build-dir build -c -m test precision.f90 test.f90 --fcompiler=gnu9
then there is no issue, f2py successfully will build the extension module.

Will this approach be acceptable for your application?

Regarding your suggestion about specifying use statement in generated Fortran wrapper files:
when all Fortran files that define {{{use}}} modules are specified in f2py command line then this is not needed: f2py can automatically evaluate kind expressions and substitute the values in
generated Fortran files. In fact, all Fortran files defining {{{use}}} modules should be specified in f2py command line so that f2py could reliably map the type specifications with kind expressions to the corresponding C types. Otherwise, f2py will look to .f2py_f2cmap file (as in your example) for the mapping. The disadvantages of relying to .f2py_f2cmap file are 1) it must be created by the user and 2) such a creation is error prone. So, with recent f2py I would not recommend using the .f2py_f2cmap hack.

Unfortunately it is not easy for f2py automatically to find all Fortran files that define {{{use}}} modules: the files may not be in the current working directory and the module names
may not match with file names that define the modules.

Regarding the interference between assumed shape support and parametrized kind statements:
IIRC, the use statements had to be removed from the generated Fortran wrapper files
when a wrapper function contains an interface block needed for the assumed shape support.
Otherwise compiler would bark.

[thouis]

Comment in Trac by atmention:rgommers, 2011-05-29

Should commit:026bc920 be backported to 1.6.x, and is that all for this issue for now? There should probably be a 1.6.1 release relatively soon, this could be included.

[thouis]

Comment in Trac by atmention:pearu, 2011-05-30

Yes, the commit should be backported to 1.6.1.

Regarding this issue, I think the task of finding all use-dependencies is too work-some to implement in f2py (I would not be sure that there won't be some corner cases that will be missed) and the explicit specification of all needed modules in command line resolves the problem.

[thouis]

Comment in Trac by atmention:rgommers, 2011-05-30

Backported in commit:08953ea

[thouis]

Comment in Trac by trac user lorenz, 2011-05-30

Hi, and sorry for not commenting earlier!

Regarding the work-around by specifying precision.f90 also on the command-line: This would work, although it would be quite cumbersome, I usually supply only a single fortran frontend file with the routine(s) I want wrapped and all my other pre-compiled object files with dependet code along, for now, precision.f90 was part of that bunch.

I can of course change that, but that would make precision.f90 special in a way it does not really deserve.

Regarding the compiler barks: What was the problem with supplying complete {{{use}}} statements? If those are not supplied, the interface is not really complete and thus possibly even erroneus, I think. Think of parameters specifying the size of some input and output varibles, for example.

There should not need to be any parsing involed, just pass the use statements as they are, right?

Regards and thanks for your great software,
Lorenz