How to get path to `.pyx` file instead of `.so` file?

[user202729]

I'm installing sage in editable mode with meson.

Currently __file__ returns the path to the .so file

sage: from sage.rings import integer
sage: integer.__file__
'/home/user202729/sage/build/cp311/src/sage/rings/integer.cpython-311-x86_64-linux-gnu.so'

so is inspect module functions

sage: import inspect
sage: inspect.getfile(integer)
'/home/user202729/sage/build/cp311/src/sage/rings/integer.cpython-311-x86_64-linux-gnu.so'

How do I get path to the .pyx file? The best I can come up with is something like this

sage: def find_mesonpy_finder():
....:	  """
....:	  Look for the finder object whose class name is 'MesonpyMetaFinder'
....:	  in sys.meta_path.
....:	  """
....:	  for finder in sys.meta_path:
....:		  cls_name = type(finder).__name__
....:		  if 'MesonpyMetaFinder' in cls_name:
....:			  return finder
....:	  return None
....: 
....: def reverse_lookup(node, path_tuple=()):
....:	  """
....:	  Recursively traverse the nested dictionary 'node' and return a dict mapping:
....:		 file_path (str) -> lookup path (tuple of keys)
....:	  Leaves in the node are assumed to be file paths.
....:	  """
....:	  mapping = {}
....:	  if isinstance(node, dict):
....:		  for key, value in node.items():
....:			  new_path = path_tuple + (key,)
....:			  # if the value is itself a dict then process recursively
....:			  if isinstance(value, dict):
....:				  mapping.update(reverse_lookup(value, new_path))
....:			  else:
....:				  mapping[value] = new_path
....:	  return mapping
....: 
....: finder = find_mesonpy_finder()
....: node = finder._rebuild()
....: rev_map = reverse_lookup(node)
....: 
sage: rev_map[integer.__file__]
('sage', 'rings', 'integer.cpython-311-x86_64-linux-gnu.so')
sage: node["sage"]["rings"]["integer.pyx"]
'/home/user202729/sage/src/sage/rings/integer.pyx'

but it accesses private method _rebuild.

[eli-schwartz]

Currently __file__ returns the path to the .so file

It is supposed to, since that's set up by cpython. Also, that will be the only file available anyways, when installing via a wheel or a linux distro package or something.

I'm not sure I understand the goal. If you want source code specifically, you should probably prefer inspect.getsourcefile instead of inspect.getfile, though. It provides a clear indicator of when no source code is available, only binary: it will return None.

[user202729]

But…

sage: import inspect
sage: from sage.rings import integer
sage: inspect.getsourcefile(integer)
[prints nothing, it's None, even though /home/user202729/sage/src/sage/rings/integer.pyx exists on my machine]

The goal is to implement sage_getfile function in a way that works in meson editable install. Currently SageMath implements a hack as follows

    for suffix in import_machinery.EXTENSION_SUFFIXES:
        if sourcefile.endswith(suffix):
            return sourcefile.removesuffix(suffix)+os.path.extsep+'pyx'
    return sourcefile

but it does not work with out-of-tree build (i.e. the build directory is not the same as the source directory).

You can assume a environment variable SAGE_SRC exists that provides the path to the source containing .pyx files.

See also sagemath/sage#39369

[eli-schwartz]

[prints nothing, it's None, even though /home/user202729/sage/src/sage/rings/integer.pyx exists on my machine]

Yes, that is the point, getsourcefile will either return an accurate answer or if it cannot provide an accurate answer it will return None.

The fact that it prints None can be used as a flag that you "need to find a better heuristic".

but it does not work with out-of-tree build (i.e. the build directory is not the same as the source directory).

It also doesn't work with non-editable installs. Tools like gdb deal with this by utilizing embedded GCC / clang metadata that includes compiler-provided file and line info, and supports those sources being remapped into e.g. /usr/src/debug so that if you install a Linux distro package of a C/C++ program, you can also install gdb-compatible source code. This requires having a universal specification for how that should all behave. CPython doesn't have a standard for this.

Maybe you can hack it by providing a '-DSOURCEFILE=' + meson.current_source_dir() / 'integer.pyx' argument when compiling, use that to set a module attribute, and then having sage_getfile(integer) check if the module provides a integer.__sage_file__ property. But this would only work if the user keeps the build directory around, and not when compiling on one machine and installing on another (distros) because no tooling would know how to collect that information and bundle up the sources to distribute in "debugsrc" packages.

At least it would work in all situations that are currently supported, without relying on internal implementation details of an editable install finder. It would also work if you do a non-editable install using e.g. a git clone of sage with pip install . that you keep persistently around.