Order of f2py files handling

[zerothi]

Hi

I am trying to convert my project (www.github.com/zerothi/sisl) to scikit-build. And I have some problems when running on the ci system (testing the build-system).

First, the root cause seems to be related to the f2py build order (in some unknown way).

I have a pretty large fortran source base consisting of many different source files.

My procedure can be seen here: siesta
Basically:

1. Use f2py to generate the signature file (my prior build process was also directly using the unediting signature file)
2. Create the signature c-file
3. Compile everything as a library.

Now, locally everything works as expected, I can get correct builds and have been doing ~20 builds without problems.

When running this on the ci system, the order of the cmake processing is different and it results in a crash.

local output

  -- Build files have been written to: /home/nicpa/codes/sisl/_skbuild/linux-x86_64-3.11/cmake-build
  [1/107] Building C object CMakeFiles/_f2py_runtime_library.dir/opt/gnu/12.2.0/python/packages/3.11.2/numpy/1.24.2/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c.o
  [2/107] Linking C static library lib_f2py_runtime_library.a
  [3/107] Generating C source src/sisl/_lattice_cython.c
  [4/107] Generating signature file from all siesta/_src/ files
  Reading fortran codes...
...
  vars2fortran: No typespec for argument "iostat_query".
  [5/107] Building C object src/sisl/CMakeFiles/_lattice.dir/_lattice_cython.c.o
  [6/107] Linking C shared module src/sisl/_lattice.cpython-311-x86_64-linux-gnu.so
  [7/107] Generating C source src/sisl/_math_small_cython.c
  [8/107] Generating C source src/sisl/_indices_cython.c
  [9/107] Generating C source src/sisl/_sparse_cython.c
  [10/107] Generating C source src/sisl/physics/_bloch_cython.c
  [11/107] Generating _siestamodule.c, _siesta-f2pywrappers.f, _siesta-f2pywrappers2.f90
...
      Wrote C/API module "_siesta" to file "./_siestamodule.c"
      Fortran 90 wrappers are saved to "./_siesta-f2pywrappers2.f90"
  [12/107] Generating C source src/sisl/physics/_phase_cython.c
  [13/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_siesta-f2pywrappers.f-pp.f
  [14/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_siesta-f2pywrappers2.f90-pp.f90
  [15/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/io_m.f90-pp.f90
  [16/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/siesta_sc_off.f90-pp.f90
  [17/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/hsx_read.f90-pp.f90
  [18/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/hsx_write.f90-pp.f90
  [19/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/dm_read.f90-pp.f90
  [20/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/dm_write.f90-pp.f90
  [21/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/tshs_read.f90-pp.f90
  [22/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/tshs_write.f90-pp.f90
  [23/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/grid_read.f90-pp.f90
  [24/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/grid_write.f90-pp.f90
  [25/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/gf_read.f90-pp.f90
  [26/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/gf_write.f90-pp.f90
  [27/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/tsde_read.f90-pp.f90
  [28/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/tsde_write.f90-pp.f90
  [29/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/hs_read.f90-pp.f90
  [30/107] Building Fortran preprocessed src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/wfsx_read.f90-pp.f90
  [31/107] Generating Fortran dyndep file src/sisl/io/siesta/CMakeFiles/_siesta.dir/Fortran.dd
  [32/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_siesta-f2pywrappers.f.o
  [33/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/io_m.f90.o
  [34/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_siesta-f2pywrappers2.f90.o
  [35/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/siesta_sc_off.f90.o
  [36/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/hsx_read.f90.o
  [37/107] Building C object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_siestamodule.c.o
  [38/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/hsx_write.f90.o
  [39/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/dm_write.f90.o
  [40/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/dm_read.f90.o
  [41/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/tshs_write.f90.o
  [42/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/grid_read.f90.o
  [43/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/grid_write.f90.o
  [44/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/tshs_read.f90.o
  [45/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/gf_read.f90.o
  [46/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/gf_write.f90.o
  [47/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/tsde_write.f90.o
  [48/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/hs_read.f90.o
  [49/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/tsde_read.f90.o
  [50/107] Building Fortran object src/sisl/io/siesta/CMakeFiles/_siesta.dir/_src/wfsx_read.f90.o
  [51/107] Linking Fortran shared module src/sisl/io/siesta/_siesta.cpython-311-x86_64-linux-gnu.so
  [52/107] Generating C source src/sisl/physics/_matrix_utils_cython.c
...

However, on the ci I get a different order, and then a crash, see here: https://github.com/zerothi/sisl/actions/runs/4666932759/jobs/8262317695

It is unclear to me why this happens, is there some dependency that isn't set in the cmake configuration? Or?

[itcarroll]

Came here searching on "f2pywrappers2.f90" because I have just started seeing a very similar error. The following is on attempting to build the simple sample project at scikit-build/scikit-build-sample-projects#35. I have an older, working build tree and it does not include _hello-f2pywrappers2.f90, so I believe some dependency changed (no idea what!) to make these "f2pywrappers2.f90" an expected output even though they're not generated. Can you compare numpy, scikit-build, etc. versions between your local system and the ci system?

Or maybe my issue is unrelated ... very confused.

[3/13] Generating _hellomodule.c, _hello-f2pywrappers.f, _hello-f2pywrappers2.f90
      Reading fortran codes...
          Reading file '/home/icarroll/tmp/skb/projects/hello-f2py/hello_f2py/_hello.pyf' (format:free)
      Post-processing...
          Block: _hello
                          Block: hello
      Applying post-processing hooks...
        character_backward_compatibility_hook
      Post-processing (stage 2)...
      Building modules...
          Building module "_hello"...
          Generating possibly empty wrappers"
          Maybe empty "_hello-f2pywrappers.f"
              Constructing wrapper function "hello"...
                hello(a)
          Wrote C/API module "_hello" to file "./_hellomodule.c"
      [4/13] Building Fortran preprocessed hello_f2py/CMakeFiles/_hello.dir/_hello-f2pywrappers2.f90-pp.f90
      FAILED: hello_f2py/CMakeFiles/_hello.dir/_hello-f2pywrappers2.f90-pp.f90 hello_f2py/CMakeFiles/_hello.dir/_hello-f2pywrappers2.f90.o.ddi
      /usr/bin/gfortran -cpp -D_hello_EXPORTS -I/tmp/pip-build-env-uk49bule/overlay/lib/python3.10/site-packages/numpy/f2py/src -I/tmp/pip-build-env-uk49bule/overlay/lib/python3.10/site-packages/numpy/core/include -I/home/icarroll/miniconda3/envs/hello-f2py/include/python3.10 -O3 -DNDEBUG -O3 -Jhello_f2py -fPIC -E /home/icarroll/tmp/skb/projects/hello-f2py/_skbuild/linux-x86_64-3.10/cmake-build/hello_f2py/_hello-f2pywrappers2.f90 -o hello_f2py/CMakeFiles/_hello.dir/_hello-f2pywrappers2.f90-pp.f90 && /tmp/pip-build-env-uk49bule/overlay/lib/python3.10/site-packages/cmake/data/bin/cmake -E cmake_ninja_depends --tdi=hello_f2py/CMakeFiles/_hello.dir/FortranDependInfo.json --lang=Fortran --pp=hello_f2py/CMakeFiles/_hello.dir/_hello-f2pywrappers2.f90-pp.f90 --dep=hello_f2py/CMakeFiles/_hello.dir/_hello-f2pywrappers2.f90-pp.f90.d --obj=hello_f2py/CMakeFiles/_hello.dir/_hello-f2pywrappers2.f90.o --ddi=hello_f2py/CMakeFiles/_hello.dir/_hello-f2pywrappers2.f90.o.ddi
      gfortran: error: /home/icarroll/tmp/skb/projects/hello-f2py/_skbuild/linux-x86_64-3.10/cmake-build/hello_f2py/_hello-f2pywrappers2.f90: No such file or directory
      gfortran: fatal error: no input files
      compilation terminated.

[zerothi]

One cannot fix this except by forcefully create the file that one knows shouldn't be there.

The wrappers* files are generated according to the rules listed here:
https://numpy.org/doc/stable/f2py/buildtools/index.html#basic-concepts

Therefore, on some ci-systems one needs to create the files if they are not created since scikit-build has no option to decide this for your self.

I have converted to using scikit-build-core and added my own custom command that forces the user to option which wrapper is created. See scikit-build/scikit-build-core#301 for details.

I think the same procedure should be adopted here, but I can understand that scikit-build-core will replace scikit-build at some time.