Prototype: Runtime inspection and Pyi generation

[CLOVIS-AI]

The goal of this PR is to add a new inspect feature to PyO3.
When this feature is enabled, the PyO3 macros extract information about the Python data structures.
The generated information can be accessed at runtime.

This feature can be used to implement Rust functions that behave the same as dict(...), etc.
In particular, this feature enables the automatic generation of Interface files (.pyi) which can be used for type checking (via MyPy, PyCharm…) or documentation (after small modifications, they can be fed to Sphinx or other tools).

Current status:

• collect data for modules:
  • name
  • classes
  • top-level fields
  • submodules?
• collect data for classes:
  • name
  • base class
  • direct fields
  • fields in impl blocks
• collect data for fields:
  • name
  • kind (method, static method, …)
  • parameters
    • name
    • kind (regular, positional-only, vararg, keyword arg)
    • has a default value
    • type
  • return type
• convert types from Rust to Python
  • for PyO3 builtins (PyDict -> dict, PyAny -> Any…) (no generics information available)
  • for Rust stdlib builtins (Vec<Foo> -> List[Foo], u16 -> int…) (generic information is available)
• merge information from the #[pyclass] and the #[pymethods] blocks at runtime into a single inspect() method
• entry in CHANGELOG.md
• docs to all new functions and / or detail in the guide
• tests for all new or changed functions
• create the inspect feature and gate all new behaviors behind it
• interface generation
  • generate a valid syntax for a .pyi file for each module
  • example in the documentation to write a build script / unit test that generates it
• replace the &'static lifetimes with normal lifetimes, so objects can be constructed at runtime (useful for editing the .pyi?)

Future possible enhancements:

• Include the documentation in the extracted information, so the interface files can be used for HTML documentation via Sphinx or other
• Find a way to make this compatible with multiple-pymethods (the current implementation only works for a single #[pymethods] block)
• Is there a way to have Maturin transparently handle the PyI generation?

[CLOVIS-AI]

Current state of this prototype:

• First and foremost, I've been figuring out what needs to be done as I go from one thing to the next, so the commit history / code is not greatly organized.
• I'm still learning how to write proc-macros, so there's a high likelihood that I'm breaking other things along the way. (and a lot of unit tests are broken)
• This prototype seems to show that the objective is possible.

Current generation:

#[pyclass]
struct Complicated {
    #[pyo3(get, set)]
    value: usize,
}

#[allow(unused_variables)]
#[pymethods]
impl Complicated {
    #[new]
    fn new(foo: PyObject, parent: PyObject) -> Self {
        unreachable!("This is just a stub")
    }

    #[pyo3(name = "staticmeth")]
    #[staticmethod]
    fn static_method(input: PyObject) -> Complicated {
        unreachable!("This is just a stub")
    }

    #[pyo3(name = "classmeth")]
    #[classmethod]
    fn class_method(cls: &PyType, input: ClassMethodInput) -> Complicated {
        unreachable!("This is just a stub")
    }
}

#[derive(FromPyObject)]
enum ClassMethodInput {
    // Complicated(PyCell<Complicated>),
    String(String),
    Int(usize),
}

generates:

class Complicated:
    @property
    def value(self) -> int: ...

    @value.setter
    def value(self, value: int) -> None: ...

    def __new__(cls, /, foo: PyAny, parent: PyAny) -> None: ...

    @staticmethod
    def staticmeth(input: PyAny) -> Any: ...

    @classmethod
    def classmeth(cls, /, input: Any) -> Any: ...

[CLOVIS-AI]

Hi @davidhewitt!

I opened #2454 to discuss the various ideas of this prototype. I think it's is good enough to show what the steps needed are, but it's far from perfect so I'd like your feedback on what the good/bad parts, so I can refactor out the good parts into independent PRs so we can finally stabilize this :)