BUG: AssertError from f2py in crackfortran.py

[gmloose]

Describe the issue:

When processing one of the Fortran files from the International Reference Ionosphere Software (namely iriflip.for) with f2py an AssertError is raised. I stripped most of the source code, and retained only a small snippet that still produces the same error.

Reproduce the code example:

#!/bin/sh
cat > bug.for <<EOF
      SUBROUTINE CNOPV(JPR,I,JPT,Z,RTS,ON,O2N,N2N,NE,P1,NOP,OPLS
     >  ,N2PLS,O2P,N4S,NNO,NPLUS,N2P,PLYNOP,VCON,N2D,OP2D)
      IMPLICIT NONE
      INTEGER JPR,I,JPT,INV,IJ,IV,K
      PARAMETER (INV=20)            !.. NO+(v) array dimensions
      REAL Z,RTS(99),   !.. Altitude, rate coefficients
     >  ON,O2N,N2N,NE,              !.. O, N2, O2, electron densities
     >  P1,                         !.. total source output for finding [e]
     >  NOP,OPLS,N2PLS,O2P,         !.. NO+, )+,N2+,O2+ densities
     >  N4S,NNO,NPLUS,N2P,          !.. N(4S), NO, N+, N(2P) densities
     >  PLYNOP,VCON,                !.. Lyman-a source, N2(v) rate factor
     >  N2D,OP2D,                   !.. N(2D), O+(2D) densities
     >  NOPV(INV),NOPTOT,           !.. NO+(v) densities and total NO+ density
     >  LR(22),PR(22),              !.. storage for NO+ sources and sinks
     >  EINSCO1(INV),EINSCO2(INV),  !.. Einstein coeffs for delv=1,2
     >  LRV(INV),                   !.. NO+(v) + e rate factors
     >  PNOPV,LNOPV,                !.. Sources and sinks of NO+(v)
     >  PCASC,LRAD,                 !.. Temp total cascade source, sink
     >  K_N2_Q,P_N2_Q,L_N2_Q        !.. N2 queching rate :- coeff, source, sink

      RETURN
      END
EOF

f2py -c bug.for

Error message:

Traceback (most recent call last):
  File "/home/marcel/code/RMextract/venv/bin/f2py", line 8, in <module>
    sys.exit(main())
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/f2py/f2py2e.py", line 766, in main
    run_compile()
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/f2py/f2py2e.py", line 738, in run_compile
    builder.compile()
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/f2py/_backends/_distutils.py", line 71, in compile
    setup(ext_modules=[ext])
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/distutils/core.py", line 170, in setup
    return old_setup(**new_attr)
  File "/usr/lib/python3.10/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.10/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/distutils/command/build.py", line 62, in run
    old_build.run(self)
  File "/usr/lib/python3.10/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.10/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/distutils/command/build_src.py", line 144, in run
    self.build_sources()
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/distutils/command/build_src.py", line 161, in build_sources
    self.build_extension_sources(ext)
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/distutils/command/build_src.py", line 321, in build_extension_sources
    sources = self.f2py_sources(sources, ext)
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/distutils/command/build_src.py", line 562, in f2py_sources
    f2py2e.run_main(f2py_options + ['--lower',
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/f2py/f2py2e.py", line 463, in run_main
    postlist = callcrackfortran(files, options)
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/f2py/f2py2e.py", line 347, in callcrackfortran
    postlist = crackfortran.crackfortran(files)
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/f2py/crackfortran.py", line 3531, in crackfortran
    readfortrancode(files, crackline)
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/f2py/crackfortran.py", line 550, in readfortrancode
    dowithline(finalline)
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/f2py/crackfortran.py", line 856, in crackline
    analyzeline(m, pat[1], line)
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/f2py/crackfortran.py", line 1308, in analyzeline
    for e in markoutercomma(ll).split('@,@'):
  File "/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy/f2py/crackfortran.py", line 893, in markoutercomma
    assert not f, repr((f, line, l))
AssertionError: (-1, 'inv=20)            !.. no+(v) array dimension', 'inv=20)            !.. no+(v) array dimension')

Python and NumPy Versions:

1.26.4
3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

Runtime Environment:

[{'numpy_version': '1.26.4',
  'python': '3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]',
  'uname': uname_result(system='Linux', node='dop540', release='5.15.0-102-generic', version='#112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024', machine='x86_64')},
 {'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
                      'found': ['SSSE3',
                                'SSE41',
                                'POPCNT',
                                'SSE42',
                                'AVX',
                                'F16C',
                                'FMA3',
                                'AVX2'],
                      'not_found': ['AVX512F',
                                    'AVX512CD',
                                    'AVX512_KNL',
                                    'AVX512_KNM',
                                    'AVX512_SKX',
                                    'AVX512_CLX',
                                    'AVX512_CNL',
                                    'AVX512_ICL']}},
 {'architecture': 'Haswell',
  'filepath': '/home/marcel/code/RMextract/venv/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so',
  'internal_api': 'openblas',
  'num_threads': 20,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.23.dev'}]
None

Context for the issue:

I cannot build the python bindings to the aforementioned Fortran sources (International Reference Ionosphere Software).
A temporary work-around is that I preprocess the Fortran source files by stripping all comments, e.g.:

for i in *.for; do sed -i 's,[[:blank:]]*!.*$,,' $i; done

[RibomBalt]

Same BUG encountered. With some simple testing I know that this would happen if ! comment is after the same line of dimension or parameter declaration.

After digging into the code, I think the problem is that, f2py didn't actually ignore the comment when processing the code.

There is a regexp that preprocess lines and determine the type

numpy/numpy/f2py/crackfortran.py

Lines 586 to 589 in ea02e84

	# Crack line
	beforethisafter = r'\s*(?P<before>%s(?=\s*(\b(%s)\b)))' + \
	r'\s*(?P<this>(\b(%s)\b))' + \
	r'\s*(?P<after>%s)\s*\Z'

numpy/numpy/f2py/crackfortran.py

Lines 643 to 644 in ea02e84

	parameterpattern = re.compile(
	beforethisafter % ('', 'parameter', 'parameter', r'\s*\(.*'), re.I), 'parameter'

The resulting (?P<after>\s*\(.*?) is a greedy match that would extend to the very last non-white-space character, including comments. Though I didn't find the corresponding code, but I guess in later processing of parameter, it must have assumed the last non-white-space character is ), so stripping it off would results in the real content of parameter statement without (), which is not the case when comment is present. (The same logic should also make sense for dimension)

This can be fixed by explicitly match out the (optional) comment, i.e., change the corresponding regexp to: \s*(?P<before>(?=\s*(\b(parameter)\b)))\s*(?P<this>(\b(parameter)\b))\s*(?P<after>\s*\(.*?)\s*(?P<comment>!.*)?\s*\Z. (NOTE that this quick fix could be problematic if the code happens to contain !, such as parameter (a="!"))

original:

fixed:

[charris]

@HaoZeke Ping.

[HaoZeke]

Excellent, I will get to this by the weekend for sure. @RibomBalt it would be pragmatic to fix the regex (and thanks for digging into it) but off the top of my head comments should have been stripped in readfortrancode (~Line 358 of crackfortran.py) and likely where the bug originates.