#[pymodule] mod some_module { ... } v3 #3815

Tpt · 2024-02-09T15:14:22Z

Based on #2367 and #3294

Allows to export classes, native classes, functions and submodules and provide an init function

See test/test_module.rs for an example

Future work:

update examples, README and guide
investigate having #[pyclass] and #[pyfunction] directly in the #[pymodule]
properly register submodules in sys.modules

codspeed-hq · 2024-02-09T15:32:27Z

CodSpeed Performance Report

Merging #3815 will not alter performance

_{Comparing Tpt:mod-module (391b8f9) with main (0f92b67)}

Summary

✅ 67 untouched benchmarks

pyo3-macros-backend/src/module.rs

Tpt · 2024-02-09T16:30:44Z

src/impl_/pymodule.rs

+
+impl<T: PyTypeInfo> PyAddToModule for T {
+    fn add_to_module(module: &PyModule) -> PyResult<()> {
+        module.add(Self::NAME, Self::type_object(module.py()))


add_class uses T::lazy_type_object().get_or_try_init(py)? instead of T::type_object(py). Is there a reason to prefer one or the other?

T::lazy_type_object().get_or_try_init(py)? will fail more gracefully instead of panic if for some reason constructing the #[pyclass] fails, so it'd be nice to keep that property if we can.

Done. I have added PyTypeInfo::try_type_object_bound. Not sure if it's the best name.

davidhewitt · 2024-02-10T14:09:46Z

👍 thank you; I will do my best to review fully tonight or tomorrow.

Tpt · 2024-02-10T14:38:36Z

👍 thank you; I will do my best to review fully tonight or tomorrow.

Thank you! The day when I have a look a PyO3 things is Friday so if you ask me any significant revision it's going to wait for next Friday. So, please, don't do it this week-end if you have other things to do.

davidhewitt · 2024-02-13T23:27:53Z

👍 based on that, and the volume of reviews I've been keeping up with, I'm now aiming to read this tomorrow! 😂

davidhewitt

Thank you for picking this up! Overall I think this looks good, and while there's bits we will definitely want like #[pyclass] and #[pyfunction] direct support, this is already a great first step.

I think the thing which I'd like discussed further is the new #[pyo3] use ... and #[pymodule_init] attributes?

For #[pyo3] use, I'm so used to seeing #[pyo3(...)] with options that #[pyo3] without them just looks a bit funny to me now, but maybe it is the right choice. #[py] use? #[py(use)]? #[pyo3(use)]? #[pyimport]?

For #[pymodule_init], similar bike shedding. #[pyo3(module_init)]? Could maybe even just be fn __module_init__ and we pick up the magic dunder name?

All ideas welcome!

pytests/src/lib.rs

davidhewitt · 2024-02-12T21:20:44Z

src/impl_/pymodule.rs

+
+impl<T: PyTypeInfo> PyAddToModule for T {
+    fn add_to_module(module: &PyModule) -> PyResult<()> {
+        module.add(Self::NAME, Self::type_object(module.py()))


T::lazy_type_object().get_or_try_init(py)? will fail more gracefully instead of panic if for some reason constructing the #[pyclass] fails, so it'd be nice to keep that property if we can.

pyo3-macros/src/lib.rs

pytests/src/lib.rs

pyo3-macros-backend/src/module.rs

pytests/src/lib.rs

Tpt · 2024-02-16T14:58:41Z

Thank you for the review!

For #[pyo3] use, I'm so used to seeing #[pyo3(...)] with options that #[pyo3] without them just looks a bit funny to me now, but maybe it is the right choice. #[py] use? #[py(use)]? #[pyo3(use)]? #[pyimport]?

I agre that #[pyo3] without options is strange. My impression is that it's only use to set modifiers inside other macros. What about #[pyexport]? "exporting" is more or less what the macro is doing.

For #[pymodule_init], similar bike shedding. #[pyo3(module_init)]? Could maybe even just be fn __module_init__ and we pick up the magic dunder name?

#[pyo3(module_init)] is weird imho, the other usages of #[pyo3] seems to me mostly for options, not completely changing behaviors. __module_init__ is a bit strange because this is not related to a Python magic function. What about keeping #[pymodule_init]?

~~Note: I still got two things to fix:~~

~~CI fails on old Rust version because of function pointer casts in const context~~

~~Use T::lazy_type_object().get_or_try_init(py)? instead of T::type_object(py)~~ Edit 19/02/2023: done

davidhewitt · 2024-02-19T10:01:24Z

👍thanks! I will do my best to give this a second review tonight.

davidhewitt

Thanks, overall this looks great and I'm keen to use this myself!

I think the TypeInfo changes can / should probably be reverted, and let's make a final decision on a better name for the "use" attribute than #[pyo3].

Regarding releasing this, I'm trying to decide if it's better to release this in 0.21 or 0.22. There's still a couple of features missing from this like supporting the function / class inside the module and it'd be nice to have those as well as make this a headline feature in its own right rather than a secondary note behind the huge Bound API changes.

I don't want this to get into a state with horrible conflicts, nor block the rest of your progress. I wonder, can we do something a bit unorthodox like merge it anyway but hide it behind a secret environment variable?

Or we just include it in 0.21 but call it experimental and aim for complete and documented support in 0.22. Maybe that's fine.

davidhewitt · 2024-02-20T23:30:22Z

src/type_object.rs

+    /// Returns the safe abstraction over the type object or an error if initialization fails
+    #[inline]
+    fn try_type_object_bound(py: Python<'_>) -> PyResult<Bound<'_, PyType>> {
+        Ok(Self::type_object_bound(py))
+    }


While I think there's potentially value in having a fallible way to get the type object on this trait, I think that's probably a design question worth having in its own PR.

Since the previous week I think LazyTypeObject now uses the Bound API, so I think you should be able to use that again without needing this here.

While I think there's potentially value in having a fallible way to get the type object on this trait, I think that's probably a design question worth having in its own PR.

Agreed

Since the previous week I think LazyTypeObject now uses the Bound API, so I think you should be able to use that again without needing this here.

I have rebased my MR and I am using it in dc81d76

About why this new function, my reasoning is: In the add_to_module function implemented on types I need to find a way to call LazyTypeObject::get_or_try_init on pyclass derivation to properly propagate errors while calling the usual PyTypeInfo::type_object_bound on other types.
I see two ways to archive this:

make #[pyclass] generate a static add_to_module function like #[pymodule] and #[pyfunction] and implement the PyAddToModule trait on all types that are not created via #[pyclass] (custom exceptions...). To archive this easily I need a trait that is implemented by everyone but types created by #[pyclass]. I have not found such trait and was a bit reluctant to implement it.

The approach in dc81d76 you have reviewed and that I have reverted.

If the new MR state without dc81d76 is fine with you we can maybe merge it and figure out a way to call LazyTypeObject::get_or_try_init in a follow up.

Yep understood, let's work out this in a follow-up. Even some builtin types like the datetime types can fail to import, e.g. #3818, so there is definitely room for PyO3's existing API to improve.

tests/ui/invalid_pymodule_in_root_module.rs

pytests/src/lib.rs

adamreichold · 2024-02-21T08:01:40Z

Or we just include it in 0.21 but call it experimental and aim for complete and documented support in 0.22. Maybe that's fine.

I'd say experimental-inspect as a feature name provides some precedence here, e.g. experimental-declarative-modules?

davidhewitt · 2024-02-21T09:24:47Z

I'd say experimental-inspect as a feature name provides some precedence here, e.g. experimental-declarative-modules?

Seems reasonable to me, I just added a commit for testing feature groups in #3834 which will extend to this trivially.

Tpt · 2024-02-21T09:36:36Z

Thank you!

I think the TypeInfo changes can / should probably be reverted, and let's make a final decision on a better name for the "use" attribute than #[pyo3].

Done. Code in dc81d76

Or we just include it in 0.21 but call it experimental and aim for complete and documented support in 0.22. Maybe that's fine.
I'd say experimental-inspect as a feature name provides some precedence here, e.g. experimental-declarative-modules?
Seems reasonable to me, I just added a commit for testing feature groups in #3834 which will extend to this trivially.

Sounds great to me too! I have implemented it!

Based on PyO3#2367 and PyO3#3294 Allows to export classes, native classes, functions and submodules and provide an init function See test/test_module.rs for an example Future work: - update examples, README and guide - investigate having #[pyclass] and #[pyfunction] directly in the #[pymodule] Co-authored-by: David Hewitt <mail@davidhewitt.dev> Co-authored-by: Georg Brandl <georg@python.org>

…ture

davidhewitt · 2024-02-23T09:47:18Z

Thanks for rebasing! I'll catch up with all the discussion and give a final review here a bit later, hopefully with intent to merge!

davidhewitt

Thanks, I am happy with this implementation and very excited to see this feature making progress. I've wanted this literally for years, see #694 - nearly the first issue I opened on PyO3 back from when I first started contributing 😂

Are you happy to make an issue with all the next steps agreed from this PR? That way we can be sure to remember everything. If we describe the design relatively clearly we might find other contributors interested in taking on pieces of the puzzle too.

Here's a few thoughts I've had just now in addition to the other comments scattered across this PR:

documentation (e.g. in the guide) - even for 0.21 it would be nice to mention this experimental feature (but it wouldn't be a blocker)
let's consider deprecating #[pyfn] once this is stable
it would be nice to make #[pymodule_init] accept -> () or -> Result<()> as the return type, to remove the #[clippy::unnecessary_wraps] bit that sometimes comes up
it would be nice to explore the relationship between this and "two-phase initialization e.g. Implement PEP 489 - Multi-phase extension module initialization #2245. In particular I think we should look at #[pymodule_data] as a way to create a per-module data mechanism, but that can come much later.

Tpt · 2024-02-26T08:57:50Z

Thank you!

Are you happy to make an issue with all the next steps agreed from this PR? That way we can be sure to remember everything. If we describe the design relatively clearly we might find other contributors interested in taking on pieces of the puzzle too.

Done: #3900 Please add anything I missed.

Tpt force-pushed the mod-module branch 2 times, most recently from 846e128 to 6adfe20 Compare February 9, 2024 15:29

Tpt force-pushed the mod-module branch from 6adfe20 to bd7ad6b Compare February 9, 2024 16:25

Tpt commented Feb 9, 2024

View reviewed changes

pyo3-macros-backend/src/module.rs Show resolved Hide resolved

Tpt commented Feb 9, 2024

View reviewed changes

Tpt force-pushed the mod-module branch from bd7ad6b to 7b36a5a Compare February 9, 2024 16:48

davidhewitt mentioned this pull request Feb 14, 2024

add #[pymodule] mod some_module { ... }, v2 #3294

Closed

davidhewitt reviewed Feb 15, 2024

View reviewed changes

Tpt force-pushed the mod-module branch 7 times, most recently from 72495f1 to f71feec Compare February 19, 2024 09:52

davidhewitt reviewed Feb 20, 2024

View reviewed changes

Tpt force-pushed the mod-module branch from f71feec to 649a18e Compare February 21, 2024 09:36

Tpt force-pushed the mod-module branch 3 times, most recently from 9c8eac1 to e75c50a Compare February 21, 2024 10:17

davidhewitt mentioned this pull request Feb 21, 2024

fix either feature conditional compilation, again #3834

Merged

Tpt and others added 6 commits February 23, 2024 10:12

tests: group exported imports

ed39b31

Consolidate pymodule macro code to avoid duplicates

f8d39d2

Makes pymodule_init take Bound<'_, PyModule>

e3d6f82

Renames #[pyo3] to #[pymodule_export]

cff28c5

Gates #[pymodule] mod behind the experimental-declarative-modules fea…

beb75da

…ture

Tpt force-pushed the mod-module branch from e75c50a to beb75da Compare February 23, 2024 09:15

Properly fails on functions inside of declarative modules

391b8f9

davidhewitt mentioned this pull request Feb 24, 2024

Update ModuleInitializer to handle Bound API #3894

Closed

davidhewitt approved these changes Feb 24, 2024

View reviewed changes

davidhewitt added this pull request to the merge queue Feb 24, 2024

Merged via the queue into PyO3:main with commit e0e3981 Feb 24, 2024
40 checks passed

Tpt deleted the mod-module branch February 24, 2024 17:17

Tpt mentioned this pull request Feb 26, 2024

Declarative modules next steps #3900

Open

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#[pymodule] mod some_module { ... } v3 #3815

#[pymodule] mod some_module { ... } v3 #3815

Tpt commented Feb 9, 2024 •

edited

codspeed-hq bot commented Feb 9, 2024 •

edited

Tpt Feb 9, 2024

davidhewitt Feb 12, 2024

Tpt Feb 19, 2024

davidhewitt commented Feb 10, 2024

Tpt commented Feb 10, 2024

davidhewitt commented Feb 13, 2024

davidhewitt left a comment

davidhewitt Feb 12, 2024

Tpt commented Feb 16, 2024 •

edited

davidhewitt commented Feb 19, 2024

davidhewitt left a comment

davidhewitt Feb 20, 2024

Tpt Feb 21, 2024

davidhewitt Feb 24, 2024

adamreichold commented Feb 21, 2024

davidhewitt commented Feb 21, 2024 •

edited

Tpt commented Feb 21, 2024

davidhewitt commented Feb 23, 2024

davidhewitt left a comment

Tpt commented Feb 26, 2024

#[pymodule] mod some_module { ... } v3 #3815

#[pymodule] mod some_module { ... } v3 #3815

Conversation

Tpt commented Feb 9, 2024 • edited

codspeed-hq bot commented Feb 9, 2024 • edited

Merging #3815 will not alter performance

Summary

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidhewitt commented Feb 10, 2024

Tpt commented Feb 10, 2024

davidhewitt commented Feb 13, 2024

davidhewitt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Tpt commented Feb 16, 2024 • edited

davidhewitt commented Feb 19, 2024

davidhewitt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamreichold commented Feb 21, 2024

davidhewitt commented Feb 21, 2024 • edited

Tpt commented Feb 21, 2024

davidhewitt commented Feb 23, 2024

davidhewitt left a comment

Choose a reason for hiding this comment

Tpt commented Feb 26, 2024

Tpt commented Feb 9, 2024 •

edited

codspeed-hq bot commented Feb 9, 2024 •

edited

Tpt commented Feb 16, 2024 •

edited

davidhewitt commented Feb 21, 2024 •

edited