BUG: f2py compile error with full path of signature file on Windows

[jmrohwer]

Describe the issue:

I am trying to port the build system for a python package (https://github.com/PySCeS/pysces) from numpy.distutils to use scikit-build because of the deprecation of distutils (the package contains 2 Fortran extension modules wrapped with f2py). To get to know the system I’ve started with a minimal example. I have been getting compile failures on Windows (on Linux and macOS everything works).

I have posted this on SO and on the Fortran discourse group but have not had any replies despite quite a number of reads. Links for cross-reference:

• python - f2py compile error with signature file on Windows - Stack Overflow
• Fortran language discourse group

During compilation of the module with python setup.py build the build fails. In tracking it down I saw that cmake expands the full path of the source files and this is where it fails. To ensure that it is not a problem with the skbuild build system I verified that the failure can be reproduced on the command line using f2py on its own as follows - save the Fortran files in the code block below to the same folder and then run in a command prompt:

(skbuild) z:\Documents\Python\skbuild_test\simple\hello\shout>f2py.exe -m hi -c hi.pyf hi.f

This works as expected and generates the .pyd and .dll files. However, when specifying the full path as follows (and as it is called in skbuild), it fails:

(skbuild) z:\Documents\Python\skbuild_test\simple\hello\shout>f2py.exe -m hi -c z:/Documents/Python/skbuild_test/simple/hello/shout/hi.pyf z:/Documents/Python/skbuild_test/simple/hello/shout/hi.f

The traceback produced is appended below. The problem seems to be with the creating C:Users part leading to the malformed filename in the last line. Specifying only the filename for hi.pyf but the full path for hi.f does work strangely enough, but as soon as the full path for hi.pyf is specified it fails. As mentioned, on Linux and macOS all works fine.

Note: I could of course skip the .pyf file and then it works. However, the Fortran code I’m wrapping is F77 and needs to be redistributed in unmodified form due to licensing conditions, so I do need a custom .pyf file to integrate it into my Python module.

Reproduce the code example:

**hi.f**

      SUBROUTINE HELLO()
      PRINT*,"Hello from fortran"
      END

**hi.pyf**

!    -*- f90 -*-
! Note: the context of this file is case sensitive.

python module hi ! in 
    interface  ! in :hi
        subroutine hello ! in :hi:hi.f
        end subroutine hello
    end interface 
end python module hi

! This file was auto-generated with f2py (version:1.23.2).
! See:
! https://web.archive.org/web/20140822061353/http://cens.ioc.ee/projects/f2py2e

Error message:

running build
running config_cc
INFO: unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
INFO: unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
INFO: build_src
INFO: building extension "hi" sources
creating C:
creating C:Users
creating C:Users\jr
creating C:Users\jr\AppData
creating C:Users\jr\AppData\Local
creating C:Users\jr\AppData\Local\Temp
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents\Python
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents\Python\skbuild_test
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents\Python\skbuild_test\simple
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents\Python\skbuild_test\simple\hello
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents\Python\skbuild_test\simple\hello\shout
INFO: f2py options: []
INFO: f2py: Z:/Documents/Python/skbuild_test/simple/hello/shout/hi.pyf
error: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'C:C:\\Users\\jr\\AppData\\Local\\Temp\\tmpdbudw6jh\\src.win-amd64-3.10\\Documents\\Python\\skbuild_test\\simple\\hello\\shout'

NumPy/Python version information:

• Windows 10 64-bit
• MSVC 2022 build tools
• mingw 11.2.0.07112021
• python 3.10.6
• numpy 1.23.2

Context for the issue:

Any project utilizing f2py to wrap Fortran code, and which includes a custom .pyf file, cannot be built with scikit-build which becomes problematic once distutils and numpy.distutils are no longer available.

[jmrohwer]

I've tracked down the origin of this to the expansion of tempfile.mkdtemp() here:

numpy/numpy/f2py/f2py2e.py

Lines 514 to 525 in 5bc5976

	remove_build_dir = 0
	try:
	i = sys.argv.index('--build-dir')
	except ValueError:
	i = None
	if i is not None:
	build_dir = sys.argv[i + 1]
	del sys.argv[i + 1]
	del sys.argv[i]
	else:
	remove_build_dir = 1
	build_dir = tempfile.mkdtemp()

If I manually specify a --build-dir on the command line, or in the CMakeLists.txt call to f2py in the project build, then it works. This is how I'm currently working around the issue for PySCeS, e.g. here:

https://github.com/PySCeS/pysces/blob/7aedc3e1f0fa060fb414052474457f09d9cc30e9/pysces/nleq2/CMakeLists.txt#L21-L28

[jmrohwer]

Incidentially, I see that the f2py compile process itself makes use of numpy.distutils extensively, e.g.

numpy/numpy/f2py/f2py2e.py

Line 607 in 5bc5976

from numpy.distutils.command.build_src import get_f2py_modulename

numpy/numpy/f2py/f2py2e.py

Line 636 in 5bc5976

from numpy.distutils.core import setup, Extension

What is the numpy strategy going forward with this, with the deprecation of distutils and numpy.distutils?

[rgommers]

Incidentially, I see that the f2py compile process itself makes use of numpy.distutils extensively, e.g.

This is only for f2py -c, and that is being ported to Meson as part of @NamamiShanker's GSoC project: https://blogs.python-gsoc.org/en/namamishankers-blog/

What is the numpy strategy going forward with this, with the deprecation of distutils and numpy.distutils?

Note that the core functionality of f2py (generating bindings) is independent of build system, and there are docs for multiple build systems (including scikit-build) at https://numpy.org/devdocs/f2py/buildtools/index.html.

[rgommers]

@jmrohwer I just saw the comment on gh-22225, and that reminded me of this issue. Is it still relevant?

I'll note that you should not be using f2py -c at all if you're using CMake (scikit-build / scikit-build-core) or Meson/meson-python. You want to use f2py as a transpiler only, and integrate the resulting .c/.f/.h files into your regular CMake or Meson targets.

[jmrohwer]

@jmrohwer I just saw the comment on gh-22225, and that reminded me of this issue. Is it still relevant?

@rgommers I am happy to close this. Apologies for the delayed reply, I was travelling.

I'll note that you should not be using f2py -c at all if you're using CMake (scikit-build / scikit-build-core) or Meson/meson-python. You want to use f2py as a transpiler only, and integrate the resulting .c/.f/.h files into your regular CMake or Meson targets.

Is there any good documentation or examples on this I could use as a reference or starting point? I am not an expert in these build systems and the project (PySCeS) started out using f2py -c with a custom .pyf file using numpy.distutils. I did get this to work with scikit-build, still calling f2py-c but am happy to give it a stab to do it "properly" if I have a pointer where to start. Note that we use a .pyf file to only expose a small subset of the Fortran methods in Python.

[HaoZeke]

@jmrohwer I just saw the comment on gh-22225, and that reminded me of this issue. Is it still relevant?

@rgommers I am happy to close this. Apologies for the delayed reply, I was travelling.

I'll note that you should not be using f2py -c at all if you're using CMake (scikit-build / scikit-build-core) or Meson/meson-python. You want to use f2py as a transpiler only, and integrate the resulting .c/.f/.h files into your regular CMake or Meson targets.

Is there any good documentation or examples on this I could use as a reference or starting point? I am not an expert in these build systems and the project (PySCeS) started out using f2py -c with a custom .pyf file using numpy.distutils. I did get this to work with scikit-build, still calling f2py-c but am happy to give it a stab to do it "properly" if I have a pointer where to start. Note that we use a .pyf file to only expose a small subset of the Fortran methods in Python.

Tracking in #25199, so I'll close this here 😄 .